Re: [Nmh-workers] nmh internals: full MIME integration

2014-08-03 Thread David Levine
Ralph wrote: Whilst looking at pick's source, I found MHPDEBUG; I don't think it's documented but could be useful for those learning pick? Perhaps it should be -debug instead? Done. MHPDEBUG is now deprecated. David ___ Nmh-workers mailing list

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-31 Thread Ralph Corderoy
Hi Earl, Would be much nicer if it travelled internally as UTF-8 and then popped into the US-ASCII draft as Subject: Re: =3D?utf-8?B?wqE=3D?=3DHola world! and into the UTF-8 draft as Re: =C2=A1Hola world!. But this behavior does not require the data to be stored in UTF-8. No,

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-30 Thread Ken Hornstein
Mostly, but there's more duff emails coming along from the base64-everything brigade, even though the RFCs say to use it minimally IIRC. Well, technically that's not what they say, assuming we're talking about headers (RFC 2047). §5 says: Use of 'encoded-word's to represent strings of purely

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-30 Thread Ralph Corderoy
Hi Ken, I suppose in theory we should replace that with specific tests rather than using isspace() Might be easier to tie it in with the RFCs if it used macro names borrowed from it. we don't handle the case where a locale says 0x00-0x7F does not match ASCII (are there such locales?). Are

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-30 Thread Earl Hood
On Wed, Jul 30, 2014 at 5:25 AM, Ralph Corderoy wrote: Would be much nicer if it travelled internally as UTF-8 and then popped into the US-ASCII draft as Subject: Re: =?utf-8?B?wqE=?=Hola world! and into the UTF-8 draft as Re: ¡Hola world!. But this behavior does not require the data

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-30 Thread Ken Hornstein
Caution is required if nmh decides to normalize header data when it stores a message. Right now, I am in the camp of leaving things as nmh gets it for the following reasons: Just FYI, don't know how others feel but my plans were to always deal with this on as part of display/parsing, NOT message

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-29 Thread Ken Hornstein
The reason why I like the idea of using utf-8 internally is that it encompasses anything we can (reasonably) expect to see. It makes us lossless internally. If the display side of the environment can't handle things, well, too bad. We can't reprogram people's terminals or character sets for

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-29 Thread Ralph Corderoy
Hi Ken, Well, I guess we could make it work both ways. Right now it's not really decoded before it hits the format engine. We could keep with that logic. Or if you wanted to convert it to ASCII ... well, I don't see a better option than converting it and substituting an appropriate

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-29 Thread Lyndon Nerenberg
On Jul 29, 2014, at 4:26 AM, Ken Hornstein k...@pobox.com wrote: You had mentioned this earlier, and I took a look at it then. I did not see anything in Bio(3) that handled character transliteration. Bio doesn't. upasfs invoked tcs(1) to do the translations. signature.asc Description:

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-29 Thread Ken Hornstein
You had mentioned this earlier, and I took a look at it then. I did not see anything in Bio(3) that handled character transliteration. Bio doesn't. upasfs invoked tcs(1) to do the translations. Alright. So ... I'm trying to understand the gain of using Bio(3). I'm not seeing it yet, but

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-29 Thread Lyndon Nerenberg
On Jul 29, 2014, at 7:41 PM, Ken Hornstein k...@pobox.com wrote: So here's the thing ... right now we (mostly) don't have to think about processing UTF-8. We get bytes in from decoding and squirt them out. There's no processing; we leave that up to the terminal to handle it. We're

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-28 Thread Lyndon Nerenberg
On Jul 28, 2014, at 7:29 PM, Ken Hornstein k...@pobox.com wrote: It's not even that ... consider the existing APIs. If we consider iconv(3) to be a required dependency (something I'm seriously considering) then it's not so bad. But ... I guess I don't see the gain converting everything

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-27 Thread Robert Elz
Date:Sat, 26 Jul 2014 22:07:48 -0700 From:Lyndon Nerenberg lyn...@orthanc.ca Message-ID: 84da1252-321f-49d5-baa5-0ace76e14...@orthanc.ca | The ABNF has never allowed spaces between the header tag (for want of | a word) and the : (going back to RFC 822). No,

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-27 Thread David Levine
Ken wrote: We could make MIME part iterators (or simply call the selector module I outlined in an earlier email) and when the iterator reached a MIME part that hadn't been parsed yet it would then parse the rest of the message. I like the iterator approach. Just to clarify something: I

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-27 Thread Ken Hornstein
Just to clarify something: I expect that callers will have to access all pieces of the Content struct through accessors. If not, they should in order to support lazy evaluation and separation of concerns. It'll also make debugging, etc., a heckuva lot easier. I am wondering if we should take

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-27 Thread David Levine
Ken wrote: I am wondering if we should take this opportunity to simply make the Content structure opaque. It would prevent a lot of API abuse that exists today. I'm fine with that and willing to help. But it will be a lot of work. David ___

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread Ralph Corderoy
Hi Ken, Right now a call to the MIME parsing routines end up slurping in the whole message, but that's not desirable for a lot of programs (scan, pick). It seems like parsing all of the messages headers is generally worthwhile; that (usually) fits within a single stdio buffer, so doing

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread David Levine
Ralph wrote: Whilst looking at pick's source, I found MHPDEBUG; I don't think it's documented but could be useful for those learning pick? Perhaps it should be -debug instead? It's documented, very briefly, in the mh-profile(5) man page along with a couple of other debug environment

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread Ken Hornstein
If we're having lazy evaluation of MIME parts, which is good, can it also cover the headers? `pick --list-id f...@bar.com' isn't concerned with decoding Subject and all those Received headers. It may not sound like much, but we have folders with tens of thousands of emails. get_header() could

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread Ralph Corderoy
Hi Ken, If we're having lazy evaluation of MIME parts, which is good, can it also cover the headers? `pick --list-id f...@bar.com' isn't concerned with decoding Subject and all those Received headers. It may not sound like much, but we have folders with tens of thousands of emails.

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread Lyndon Nerenberg
On Jul 26, 2014, at 12:07 PM, Ralph Corderoy ra...@inputplus.co.uk wrote: pick may have changed a bit since the above version, but I still shouldn't have to care how much whitespace continuation lines are indented. Shouldn't pick be matching against a logical view of a single line, with

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread Ken Hornstein
pick may have changed a bit since the above version, but I still shouldn't have to care how much whitespace continuation lines are indented. Shouldn't pick be matching against a logical view of a single line, with `CRLF WS*' becoming a single space? Yeah, I agree with you there. I would

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread Ken Hornstein
I think in the general sense of matching a generic header, the semantics of header folding are clear. So did I, until I tried writing the spec. Easier said than done :-( I just reread §2.2.3 of RFC 5322; it seems pretty clear to me. So if you could give further detail as to what you mean,

Re: [Nmh-workers] nmh internals: full MIME integration

2014-07-26 Thread Lyndon Nerenberg
On Jul 26, 2014, at 10:02 PM, Ken Hornstein k...@pobox.com wrote: (As an aside, I see that pick does use ^foo[ \t]*: to match on a header, but my reading of RFC 5322 is that spaces are not allowed between the header name and the colon ... but I guess the old syntax did?) The ABNF has never

[Nmh-workers] nmh internals: full MIME integration

2014-07-25 Thread Ken Hornstein
Warning: this email will be discussing technical details of nmh internals and if you haven't paid that much attention to those sorts of things it will likely go over your head. The changes discussed here wouldn't directly affect nmh behaavior (although it would make a lot of things a lot easier