Re: [Nmh-workers] I need to learn more about MIME
Norm wrote: I give up. I will never become facile enough with MIME concepts to meaningfully participate in discussions of nmh that heavily involve Mime, that is to say, these days, with most discussions about nmh. I don't agree, FWIW. Questions help the discussions, I think. And the discussions are useful for background on the implementation (which should be self-contained, but isn't). David ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Ken Hornstein k...@pobox.com writes: (two months ago) In order to participate more meaningfully in these discussions I need to learn more about MIME. I give up. I will never become facile enough with MIME concepts to meaningfully participate in discussions of nmh that heavily involve Mime, that is to say, these days, with most discussions about nmh. So be it. It's been a good ride. Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Ralph Corderoy ra...@inputplus.co.uk writes: Hi Norm, The problem was that, for reasons I do not remember, Chart.pj had Microsoft line endings and your blanks, above includes '\r', and probably other white space. Confession: It took me an hour to figure that out. Useful ways to look for odd bytes in a file ./foo. One approach is to delete what you think is valid and see what's left. tr -d '\012 -~' foo | od -c Or look at their frequencies. $ tr -d '\012 -~' foo | hexdump -ve '/1 %02x\n' | sort | uniq -c 2 0d 1 a3 1 c2 $ `cat -A' can also be helpful. If you're having trouble tracking down the precise location then ask grep for byte offsets. LC_ALL=C grep -Eboa '[^ -~]+' foo | cat -A Thank you very much for the tutorial. I have a request (a request, not a demand!). When a command has both a '-' option and a '--' option for the same feature, use the '--' version in examples. Thus: grep --extended-regexp --byte-offset --only-matching --text instead of grep -Eboa Of course given a choice of a '-' example or no example at all, I'll take the former. Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Hi Norm, tr -d '\012 -~' foo | od -c I have a request (a request, not a demand!). When a command has both a '-' option and a '--' option for the same feature, use the '--' version in examples. I take your point, but the problem is... I don't know the long versions and my fingers certainly know only the short ones. Indeed, faced with `grep --text' I have to scurry off and find out what on earth is meant. :-) I quite like Go's standard library's take on things, single - but multiple letters possible, e.g. -i -verbose -f, but they rule out combining so no -if, and I think I'd tire of all that ` -'. Given POSIX eschews long options altogether and the work needed for me to substitute, I'd rather stick as is. Oft used single-letter options often occur in groups anyway, e.g. hexdump's -ve, and I rearrange to make them look like words for easier recall, e.g. netstat's -tulepn and rsync's -PacivHAX. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
On Sep 14, 2014, at 8:18 AM, n...@dad.org wrote: I have a request (a request, not a demand!). When a command has both a '-' option and a '--' option for the same feature, use the '--' version in examples. Long options names are gnu extensions, and not present in non-gnu versions of the utilities. We use the POSIX command syntax everywhere. (If we don't, that's a bug.) --lyndon signature.asc Description: Message signed with OpenPGP using GPGMail ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Hi David, I suspect you're running afowl of the 'linespace' test. Yeah, I do that a lot because body:component= turns a blank line in replied text into one with a trailing space. Yes, that's really icky. Would be nice if mhl supported some kind of `strip' to prune those off afterwards. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Ken Hornstein k...@pobox.com writes: The above referenced file, Chart.pj, contains only ASCII characters, and has no line longer than 63 characters. Why did nmh use a Content-Transfer-Encoding of quoted-printable instead of text/plain? That decision is made in scan_content() in uip/mhbuildsbr.c. Well, let me rephrase that. For MIME _bodies_, that's where the decision is made. For headers that happens in sbr/encode_rfc2047.c. The exact decision is here: int wants_q_p = (containsnul || linelen || linespace || checksw); containsnul means the content contains at least one byte of '0x00'. linelen means 'longer than maxunencoded', which defaults to 78. linespace means blanks at the end of the line. checksw means you're generating an MD5 checksum (which honestly, we should get rid of that code). I suspect you're running afowl of the 'linespace' test. The problem was that, for reasons I do not remember, Chart.pj had Microsoft line endings and your blanks, above includes '\r', and probably other white space. Confession: It took me an hour to figure that out. In the unlikely event, that you care, norms-cool-editor, will, by default, never write a file with lines ending with white space, except that by default, if a file had any Microsoft line endings it will write it with all Microsoft line endings. There are interactive options to override both of those defaults. Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Hi Norm, The problem was that, for reasons I do not remember, Chart.pj had Microsoft line endings and your blanks, above includes '\r', and probably other white space. Confession: It took me an hour to figure that out. Useful ways to look for odd bytes in a file ./foo. One approach is to delete what you think is valid and see what's left. tr -d '\012 -~' foo | od -c Or look at their frequencies. $ tr -d '\012 -~' foo | hexdump -ve '/1 %02x\n' | sort | uniq -c 2 0d 1 a3 1 c2 $ `cat -A' can also be helpful. If you're having trouble tracking down the precise location then ask grep for byte offsets. LC_ALL=C grep -Eboa '[^ -~]+' foo | cat -A Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
The problem was that, for reasons I do not remember, Chart.pj had Microsoft line endings and your blanks, above includes '\r', and probably other white space. Confession: It took me an hour to figure that out. As an aside ... I'm not quite sure what happens if we have a file with CR LF line endings that is text content. It looks like ... it will count the last character as a space (since '\r' counts as a space with isspace()) and will be marked as needing quoted-printable encoding. And \r will be encoded by the quoted-printable encoder. That might end up on the receiving MUA as lines be delimited by CR CR LF. I am unclear if we can do better; it seems that in this situation there are no good answers. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Ken Hornstein k...@pobox.com writes: And of course feel free to ask questions if you're confused! I have scanned all of and have read and understood much of RFC 2045 and RFC 2046; though how much I will retain is problematic. The going was slow because of a character flaw that limits the amount of pain I can endure each day. Also, I took two days off to write a program to decode quoted-printable, as a form of occupational therapy (Yeah, I know, Perl's MIME::QuotedPrint does that.) I did not find the example in RFC 2049 particularly helpful. But I have composed and sent a variety of test messages to myself, and looked at how they came back to me. I have a slew of questions. I hope that they don't abuse your patience. Here is the first installment. RFC 2045 seems to require that mail lines have Microsoft line endings, but fetchmail delivers mail with UNIX line endings. Did they arrive here with Microsoft line endings? The remainder of the questions in this Email apply to nmh outgoing mail. Is it true that, that possibly apart from mhbuild, the nmh user does not have explicit control of the content type of a message? Based on my experimentation, nmh must have a fairly complex heuristic for determining content type. For example, I attached a file named Chart.pj, That is, a file whose suffix was not .java. But human intelligence reveals it to be a Java source file. But nmh knew it was a java file; nmh set: Content-Type: text/x-java; charset=us-ascii; name=Chart.pj Is there any documentation of that heuristic, maybe in source code comments? The above referenced file, Chart.pj, contains only ASCII characters, and has no line longer than 63 characters. Why did nmh use a Content-Transfer-Encoding of quoted-printable instead of text/plain? When I send a test message with non-ASCII characters sometimes nmh uses a Content-Transfer-Encoding of quoted-printable and sometimes base64. I can't predict which it will use. More generally, is there any documentation of how the Content-Transfer-Encoding is determined? maybe in source code comments? Is it correct that nmh does not support the partial subtype, described in RFC 2045 Section 5.2.2. Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
n...@dad.org writes: Is it true that, that possibly apart from mhbuild, the nmh user does not have explicit control of the content type of a message? Based on my experimentation, nmh must have a fairly complex heuristic for determining content type. For example, I attached a file named Chart.pj, That is, a file whose suffix was not .java. But human intelligence reveals it to be a Java source file. But nmh knew it was a java file; nmh set: Content-Type: text/x-java; charset=us-ascii; name=Chart.pj Is there any documentation of that heuristic, maybe in source code comments? My memory on this is really fuzzy, but I think that if you use the attach stuff it uses the file command to get the file type. Jon ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Jon Steinhart j...@fourwinds.com writes: n...@dad.org writes: Is it true that, that possibly apart from mhbuild, the nmh user does not have explicit control of the content type of a message? Based on my experimentation, nmh must have a fairly complex heuristic for determining content type. For example, I attached a file named Chart.pj, That is, a file whose suffix was not .java. But human intelligence reveals it to be a Java source file. But nmh knew it was a java file; nmh set: Content-Type: text/x-java; charset=us-ascii; name=Chart.pj Is there any documentation of that heuristic, maybe in source code comments? My memory on this is really fuzzy, but I think that if you use the attach stuff it uses the file command to get the file type. The file command, here, thinks that Java source files are Perl5 module source text. Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
I have scanned all of and have read and understood much of RFC 2045 and RFC 2046; though how much I will retain is problematic. The going was slow because of a character flaw that limits the amount of pain I can endure each day. Also, I took two days off to write a program to decode quoted-printable, as a form of occupational therapy (Yeah, I know, Perl's MIME::QuotedPrint does that.) Wow, that's some dedication! Congrats! RFC 2045 seems to require that mail lines have Microsoft line endings, but fetchmail delivers mail with UNIX line endings. Did they arrive here with Microsoft line endings? So ... this is kind of an issue that gets glossed over. It's subtle as to what's going on here. As you and everyone else know text files on Unix have lines that simply end in a LF. All of the email RFCs (including non-MIME RFCs like RFC 822 and earlier) specify that lines end with CR LF. The convention has always been that when you exchange email with other systems you make sure that lines end with CR LF but you convert to the local line convention for local storage. Mostly nmh doesn't have to deal with this; email is treated as traditional Unix text files with just LF line endings, and mail in a spool file already has a LF line ending; whatever put it there stripped out the CRs for us. But there are some important exceptions: - Retrieving email via POP and sending email via SMTP; both of these protocols specify CR LF line endings, so nmh has to strip/append CRs where appropriate. - Dealing with text conent encoded with base64; that specifically has to have CR LF for a line separator. We used to get that wrong, but that's been fixed for 1.6. Is it true that, that possibly apart from mhbuild, the nmh user does not have explicit control of the content type of a message? Well, that's a pretty big exception; with mhbuild you can literally create any MIME message you can imagine. Based on my experimentation, nmh must have a fairly complex heuristic for determining content type. For example, I attached a file named Chart.pj, That is, a file whose suffix was not .java. But human intelligence reveals it to be a Java source file. But nmh knew it was a java file; nmh set: Content-Type: text/x-java; charset=us-ascii; name=Chart.pj Is there any documentation of that heuristic, maybe in source code comments? We kind of skip over the details here. The specifics are that we call a function called mime_type(), in sbr/mime_type.c. If you have a 'file' command that supports the --mime-type option, it will use that. If that doesn't work or doesn't exist, we use entries in the user's or system nmh profile (mhn.defaults). So if you have: mhshow-suffix-text/x-java: .pj That would cause that to happen. The above referenced file, Chart.pj, contains only ASCII characters, and has no line longer than 63 characters. Why did nmh use a Content-Transfer-Encoding of quoted-printable instead of text/plain? That decision is made in scan_content() in uip/mhbuildsbr.c. Well, let me rephrase that. For MIME _bodies_, that's where the decision is made. For headers that happens in sbr/encode_rfc2047.c. The exact decision is here: int wants_q_p = (containsnul || linelen || linespace || checksw); containsnul means the content contains at least one byte of '0x00'. linelen means 'longer than maxunencoded', which defaults to 78. linespace means blanks at the end of the line. checksw means you're generating an MD5 checksum (which honestly, we should get rid of that code). I suspect you're running afowl of the 'linespace' test. When I send a test message with non-ASCII characters sometimes nmh uses a Content-Transfer-Encoding of quoted-printable and sometimes base64. I can't predict which it will use. For MIME text parts, nmh will never pick base64 (by my reading of the code). For 8-bit headers, nmh will pick the shorter encoding (this is mentioned on the mhbuild man page). If you can find a counter-example I would love to see it! Is it correct that nmh does not support the partial subtype, described in RFC 2045 Section 5.2.2. Actually, that is false; nmh (and MH) do support the partial subtype! Although I doubt it's been tested lately. There's a whole section about message/partial in the mhstore man page. The -split option to send(1) is likewise untested. Before you ask, I'm not sure how big the split size is. And I have a sneaking suspicion that few MUAs understand message/partial at this point. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Ken wrote: [Norm:] Is it true that, that possibly apart from mhbuild, the nmh user does not have explicit control of the content type of a message? Well, that's a pretty big exception; with mhbuild you can literally create any MIME message you can imagine. attach users can change the content type (and anything else) by editing the message after running mime, which runs mhbuild (buildmimeproc, really) at the What now? prompt. So it's not necessary to play with mhbuild directives. Is there any documentation of that heuristic, maybe in source code comments? In addition to what Ken said, it is documented in the send(1) man page. I suspect you're running afowl of the 'linespace' test. Yeah, I do that a lot because body:component= turns a blank line in replied text into one with a trailing space. David ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Hi Ken, http://www.nongnu.org/nmh/rfc.html I don't know who started that list, but I've been adding stuff to that based on what I use during nmh development. I hadn't needed to use RFC 2049, so that's why I didn't add it. But it probably makes sense to add it at some point. I find the list useful; I don't need them often enough to recall the numbers. It would be nice to see ones that are relevant to nmh there even if nmh doesn't implement them (yet). Your recent mention of RFC 6530 would be a contender. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
I don't know who started that list, but I've been adding stuff to that based on what I use during nmh development. I hadn't needed to use RFC 2049, so that's why I didn't add it. But it probably makes sense to add it at some point. I find the list useful; I don't need them often enough to recall the numbers. It would be nice to see ones that are relevant to nmh there even if nmh doesn't implement them (yet). Your recent mention of RFC 6530 would be a contender. Sure, next time I edit that list (it's a bit of a pain; I have to use CVS, ewww!) I'll add both of those. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Ken Hornstein k...@pobox.com writes: We have a set of links here: = http://www.nongnu.org/nmh/rfc.html - There is a pretty good example of a more complicated message in RFC 2049, Appendix A. Shouln't http://www.nongnu.org/nmh/rfc.html include RFC 2049, http://tools.ietf.org/html/rfc2049 ? Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
http://www.nongnu.org/nmh/rfc.html - There is a pretty good example of a more complicated message in RFC 2049, Appendix A. Shouln't http://www.nongnu.org/nmh/rfc.html include RFC 2049, http://tools.ietf.org/html/rfc2049 ? I don't know who started that list, but I've been adding stuff to that based on what I use during nmh development. I hadn't needed to use RFC 2049, so that's why I didn't add it. But it probably makes sense to add it at some point. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Hi Norm, Can anybody suggest a course of study so that I can learn enough about MIME to able to participate, meaningfully, in most of these discussions? The link to Jerry's book given earlier is good. There's also http://www.hunnysoft.com/mime/mime-guide.html as a high-level overview. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
In order to participate more meaningfully in these discussions I need to learn more about MIME. Studying and becoming familiar with the relevant RFCs is much more than I want to or am able to do. I tried starting with http://en.wikipedia.org/wiki/MIME, but the tree of links overwhelmed me. Can anybody suggest a course of study so that I can learn enough about MIME to able to participate, meaningfully, in most of these discussions? We have a set of links here: http://www.nongnu.org/nmh/rfc.html RFC 2045 is sort of a broad overview of the format of a MIME message. That explains the extra headers you need for a MIME message. It's going to be pretty dry reading, though. As a companion with that, look at RFC 2046 to cover the common media types (like multipart). I see the Wikipedia entry is a bit confusing, as it covers a lot of MIME types that are not used by email (HTTP also uses MIME types). My advice is: - Take a look at this message, and compare the headers here to the descriptions in RFC 2045. This should be a pretty simple message, so it should be easy enough to undertand. I know, I know ... you're reluctant to look at the RFCs. At least give RFC 2045 a try; looking at an example message might make it more concrete. I wouldn't try reading the whole thing from start to finish; jump around and read the text for specific headers. - Take a look at a multipart message, and see the description in RFC 2046 how multipart messages are formatted. - There is a pretty good example of a more complicated message in RFC 2049, Appendix A. The basic framework is actually not very complicated; it's just a lot of stuff you need to know is spread out over multiple documents. At this stage you can ignore much of the extra stuff until you master the basics. And of course feel free to ask questions if you're confused! --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
On Wed, Aug 20, 2014 at 09:40:44AM -0700, n...@dad.org wrote: In order to participate more meaningfully in these discussions I need to learn more about MIME. Studying and becoming familiar with the relevant RFCs is much more than I want to or am able to do. I tried starting with http://en.wikipedia.org/wiki/MIME, but the tree of links overwhelmed me. Can anybody suggest a course of study so that I can learn enough about MIME to able to participate, meaningfully, in most of these discussions? There's an O'Reilly book called _Programming Internet Email_ http://shop.oreilly.com/product/9781565924796.do that has a whole chapter on MIME. That might be a good tutorial. The book is from 1999, but I suspect it's still mostly correct. -- Peter Davis The Tech Curmudgeon www.techcurmudgeon.com ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] I need to learn more about MIME
Norm wrote: In order to participate more meaningfully in these discussions I need to learn more about MIME. Jerry's book has a no-nonsense intro to MIME: http://oreilly.com/openbook/mh/tocs/intmime.htm David ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
