[Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Joel Uckelman
I've noticed recently that I'm getting some mojibake in messages from a few sources. Both examples I have handy have a quoted-printable UTF-8 encoded text/html part, and one also has a quoted-printable UTF-8 encoded text/plain part. The one which is HTML only happens also to be in German, and

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ken Hornstein
Just out of curiousity ... did you produce this message with nmh? I ask because it didn't actually have proper MIME encoding. This can happen when you use nmh but don't run mhbuild on the message, for example (my plans are to make this automatically happen for 1.6). I've noticed recently that

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ken Hornstein
The munged character in your fist example looks like it's supposed to be c3 bc c3, but instead is 83 c2 bc, if I did that right. It takes more than one step to get from here to there, such as losing bits and wrong endian? Actually, I think Joel was trying to say für, which has the middle letter

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ralph Corderoy
Hi Joel, The one which is HTML only happens also to be in German, and what's getting munged are the umlauted vowels nmh doesn't interpret HTML, so what command is being run to do that and with what options, e.g. lynx. `ps xfww' in another terminal whilst the corruption is being viewed might

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Joel Uckelman
Thus spake Ken Hornstein: Just out of curiousity ... did you produce this message with nmh? I ask because it didn't actually have proper MIME encoding. This can happen when you use nmh but don't run mhbuild on the message, for example (my plans are to make this automatically happen for 1.6).

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ralph Corderoy
Hi Joel, I think I found something related to the cause: I have this line in my mhn.defaults: mhshow-show-text/html: %p/usr/bin/lynx -force_html -dump '%f' | less I have a similar single line. mhshow-show-text/html: lynx -dump -width `tput cols` '%F' | expand | sed 's/ *$//'

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ken Hornstein
It's just as you said---I didn't run mhbuild. Having that run automatically wouldn't be a bad idea. Is there a way to have that run automatically in 1.5? Yeah, if you put: automimeproc: 1 in your mh_profile, that will do it. That has some side effects; any line beginning with a '#' will be

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Joel Uckelman
Thus spake Ralph Corderoy: Hi Joel, I think I found something related to the cause: I have this line in my mhn.defaults: mhshow-show-text/html: %p/usr/bin/lynx -force_html -dump '%f' | less I have a similar single line. mhshow-show-text/html: lynx -dump -width `tput cols`

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ken Hornstein
Does one of the MIME RFCs require an encoding declaration when the charset is already given in the MIME header? If so, then virtually every email in my inbox which has a text/html part is wrong: It's sort of a corner case, but RFC 2854 says that the charset MIME parameter indicates the character

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ralph Corderoy
Hi Joel, I find it gives the behaviour you describe if the HTML file contains a charset declaration that's incorrect for the content of the HTML, e.g. ISO-8859-1 when mhstore shows the glyph is UTF-8 encoded. Lynx obeys the charset in the file. Might be worth searching for `charset' in

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Joel Uckelman
Thus spake Ralph Corderoy: Perhaps see if you can get `lynx -dump' as specified in your ~/.mh_profile to give correct output on a HTML file you create, feeding the output to cat one time and less another. If I add -assume_charset=UTF-8 as an option for lynx, both the HTML written by mhstore

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ken Hornstein
The problem with that is that the HTML being sent to lynx might not be in a UTF-8 compatible encoding. Right, that's what I was talking about in my message. That doesn't currently exist. %a puts all of the MIME parameters out there, but that's inconvenient to deal with. That should be easy

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Joel Uckelman
Thus spake Ken Hornstein: The problem with that is that the HTML being sent to lynx might not be in a UTF-8 compatible encoding. Right, that's what I was talking about in my message. That doesn't currently exist. %a puts all of the MIME parameters out there, but that's inconvenient to

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ken Hornstein
I didn't know about %a. It turns out that's sufficient, if kludgy: mhshow-show-text/html: %p/usr/bin/lynx -assume_charset=`echo %a | sed 's/.*charset=\([^]\+\).*/\1/'` -force_html -dump '%f' | less The only problem with that is if charset doesn't exist as a parameter, you don't get the default

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Ken Hornstein
Challenge accepted: mhshow-show-text/html: %p/usr/bin/lynx `echo %a | sed -n 's/.*charset=\([^]\+\).*/-assume_charset=\1/p'` -force_html -dump '%f' | less Well played :-) Although ... will that end up with a zero-length parameter to lynx? Or will that be eaten by the shell? I think you

Re: [Nmh-workers] mojibake in UTF-8 encoded quoted-printable messages

2013-10-24 Thread Joel Uckelman
Thus spake Ken Hornstein: Challenge accepted: mhshow-show-text/html: %p/usr/bin/lynx `echo %a | sed -n 's/.*charset=\([^]\+\).*/-assume_charset=\1/p'` -force_html -dump '%f' | less Well played :-) Although ... will that end up with a zero-length parameter to lynx? Or will that be

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Joel Uckelman
Thus spake Ken Hornstein: So, I just found out about this: http://offlineimap.org It seems like it's close to what people are interested in. The big wrinkle is that right now the local store is Maildir; it occurs to me that it should be straightforward to add nmh folder support to

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Ken Hornstein
It seems to me there are two IMAP-related things people have wanted: 1) Have nmh commands act on IMAP-stored messages. 2) Expose an nmh folder via IMAP. #1 would let you use nmh against arbitrary IMAP accounts. #2 would let you access your existing nmh storage over IMAP. I think both would

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Lyndon Nerenberg
On Oct 24, 2013, at 11:35 AM, Joel Uckelman uckel...@nomic.net wrote: I looked this over a bit and wasn't able to satisfy myself as to what OfflineIMAP would do. For many years now my primary email engine has been IMAP. The driving force behind this is that I need to access my mail folders

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Ken Hornstein
But that describes a simple 1:1 mapping case. Nothing there says the proof extends to the 1:n mapping case (i.e. multiple clients). Fair enough ... I'm just trying to imagine exactly a) what conflicting clients would be doing, exactly, to conflict, and b) what SHOULD happen when a conflict

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread chad
On 24 Oct 2013, at 18:17, Ken Hornstein k...@pobox.com wrote: But that describes a simple 1:1 mapping case. Nothing there says the proof extends to the 1:n mapping case (i.e. multiple clients). Fair enough ... I'm just trying to imagine exactly a) what conflicting clients would be doing,

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Lyndon Nerenberg
On Oct 24, 2013, at 6:17 PM, Ken Hornstein k...@pobox.com wrote: But that describes a simple 1:1 mapping case. Nothing there says the proof extends to the 1:n mapping case (i.e. multiple clients). Fair enough ... I'm just trying to imagine exactly a) what conflicting clients would be

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Jerrad Pierce
Wouldn't a FUSE IMAP layer largely solve the problem of conflicts by working on the live data store? Perhaps a customization of something like: http://imapfs.sourceforge.net/ http://www.sr71.net/projects/gmailfs/ ___ Nmh-workers mailing list

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Lyndon Nerenberg
On Oct 24, 2013, at 7:22 PM, Jerrad Pierce belg4...@pthbb.org wrote: Wouldn't a FUSE IMAP layer largely solve the problem of conflicts by working on the live data store? Perhaps a customization of something like: No. nmh works in many places FUSE will never enjoin.

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Ken Hornstein
It's mostly about refile and delete, in the IMAP-MH case. Client A deletes message 1. Client B moves it to folder foo. Who wins? Especially when B syncs after A, thus message 1 is no longer in place on the server. (These are *very* simple examples of what you have to deal with ...) It looks

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Lyndon Nerenberg
On Oct 24, 2013, at 7:24 PM, Ken Hornstein k...@pobox.com wrote: Of course, the question really should be: what SHOULD happen in that case? Like I said, it gets complicated. The internet calendaring folks are still having ulcers over this. ___

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Ken Hornstein
Wouldn't a FUSE IMAP layer largely solve the problem of conflicts by working on the live data store? Perhaps a customization of something like: Well, I note this from the FUSE IMAP web page: Note: My project is old and has a lot of unresolved issues and design problems. Please don't try to

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Jerrad Pierce
have to manage the mapping between MH message numbers and IMAP messages, which involves a synchronization process. Also, it just seems like to Yeah, if I were doing it I'd probably not support all of MH's numbering and sequence goodness... for simplicity/sanity's sake. Just allow basic sortm to

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Ken Hornstein
have to manage the mapping between MH message numbers and IMAP messages, which involves a synchronization process. Also, it just seems like to Yeah, if I were doing it I'd probably not support all of MH's numbering and sequence goodness... for simplicity/sanity's sake. Just allow basic sortm to

Re: [Nmh-workers] IMAP, again

2013-10-24 Thread Ken Hornstein
For anyone who's looking to implement something new, I think that git's internal architecture might be a good starting place, written in C. In slightly related news, I looked at notmuch recently, and it made me miss MH. I've heard that before, but I don't see how looking at git helps anything