Re: [Nmh-workers] Fixing decoding of quoted messages in replies
On Sat, 06 Nov 2010 11:08:52 +0700, Robert Elz said: Date:Fri, 05 Nov 2010 21:27:33 + From:Peter Maydell pmayd...@chiark.greenend.org.uk Message-ID: e1petor-0002hx...@mnementh.archaic.org.uk | [Aside: 'columns' is | usually the same as 'characters' except that some Chinese/Japanese | characters are two columns wide on screen. Other languages have characters that use no horizontal space on the screen at all (they go above, or below, the previous (or potentially, I suppose, next, though I have no personal experience with that) character). Of course, once you start moving the code in this direction, you should also start to think about r to l scripts instead of assuming l to r is correct for everyone. Another thing to consider is the increasing difficulty in finding really good monospaced fonts, especially non-ascii ones, and xterm-ish things that handle them well (documenting this stuff would probably be useful, if anybody is in the mood). For the record, exmh hasn't fully addressed these issues either, and that's *with* the added advantage of having Tk doing all the canvas management - it sort of blindly assumes that Tk/Tcl supports it and it can just pretend the world is left-to-right, top-to-bottom. There's really no sane way to deal with bidirectional and vertical scripts unless you throw away the xterm paradigm and go to a fully functional 2-d canvas. pgpGxd0RjDRaF.pgp Description: PGP signature ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
Peter Maydell wrote: markus schnalke wrote: during the next weeks I intend to fix the annoying problem of still encoded message text in quoted replies. I completely agree. I'll try to make some time to take another look at this some time this week. (I'm at the Ubuntu Developer Summit so I'm in a free software kind of mood anyway :-)) Having thought about this a little more, here are six proposed changes to nmh which I think will together make dealing with MIME rather nicer. I have attempted to stick to what are the changes we want to make to UI, config and to individual nmh binaries? -- we should agree on these before we start getting into the technical details of how to change things. So: (1) things which currently claim to mean columns or characters but actually count bytes (eg scan -width, the %4(from) syntax in format fields) should count columns. [Aside: 'columns' is usually the same as 'characters' except that some Chinese/Japanese characters are two columns wide on screen. Since the affected switches/syntax are generally for making lined up displays on screen we should care about the display width of the text.] (2) the default setup should always MIME-decode headers. Either: (a) we need to add decode() calls to all the default format templates where they aren't already present (b) have a general config switch for default to raw vs default to decode, set it to 'decode' and remove any existing decode() calls from format templates (3) scan's body component escape should give you decoded text, by which I mean: * select the first text/plain bit of multipart MIME * decode quoted-printable and base-64 * convert to the character encoding of the current locale [we can probably steal a lot of code from mhshow for this] We can add a 'raw' option to this for the rare case where you really did want the undecoded text. (4) mhl's body component should decode as per (3). This is what will allow 'repl' to put you in an editor with the decoded text rather than the raw MIME. (5) (Contrary to what I first suggested) I think that the conversion of outbound messages to a suitable charset and adding of MIME headers should be done in send. We may be able to use the existing -mime option to control this, or we could play safe and add a new switch. Rationale: (a) I want MIME-ing to be automatic but I don't want to auto invoke mhbuild for all users because it does a lot of interpretation of the body text which would be very user-unfriendly to have happen without you explicitly asking for it. (b) send / post is the code which does things like looking up addresses in the aliases file, so we can't mime encode headers until it's done. And it already is the code which wraps headers to sensible lengths, ensures mandatory headers are present and generally makes the mail be a valid format, so it's not out of place to implement it there. So send should: * check for non-ASCII characters * when present: pick an encoding (first usable one from a user-configurable list like us-ascii,iso-8859-1,utf-8) * encode body and headers as per MIME, adding MIME headers to match [We need to take care not to trip mhbuild's earlier work up, but in theory mhbuild should already have got rid of all the non-ascii characters, so if send just passes through anything that's already a valid MIME message then mhbuild and any preexisting script-based hacks by users should be unaffected.] (6) Reading MIME email currently works but can feel a bit clunky. Since the changes in (4) basically mean that plain old 'show' can cope with at least some MIME messages, I think we could let the user choose to use show for everything. We would probably need some new mhl component file syntax to say display a summary of the MIME parts that weren't shown (ie attachments). So do people think I've got the changes to the user visible bits of interface/config right? -- PMM ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
[2010-11-05 21:27] Peter Maydell pmayd...@chiark.greenend.org.uk So send should: * check for non-ASCII characters * when present: pick an encoding (first usable one from a user-configurable list like us-ascii,iso-8859-1,utf-8) * encode body and headers as per MIME, adding MIME headers to match Just one quick thought: What about signed and encrypted messages? Where and how would this be done and would it interfere? meillo ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
On Fri, Nov 5, 2010 at 4:48 PM, markus schnalke mei...@marmaro.de wrote: So send should: * check for non-ASCII characters * when present: pick an encoding (first usable one from a user-configurable list like us-ascii,iso-8859-1,utf-8) * encode body and headers as per MIME, adding MIME headers to match Just one quick thought: What about signed and encrypted messages? Where and how would this be done and would it interfere? During send. I.e. Have an option for send to support signing/encrypted. It is only at the send stage that you know a message will not be further modified by the user. With the current motivation by some to do work on nmh, supporting SMIME/PGP should be considered. If some of the mail parsing and MIME handling is improved, adding such support may become easier. --ewh ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
Date:Fri, 05 Nov 2010 21:27:33 + From:Peter Maydell pmayd...@chiark.greenend.org.uk Message-ID: e1petor-0002hx...@mnementh.archaic.org.uk | [Aside: 'columns' is | usually the same as 'characters' except that some Chinese/Japanese | characters are two columns wide on screen. Other languages have characters that use no horizontal space on the screen at all (they go above, or below, the previous (or potentially, I suppose, next, though I have no personal experience with that) character). Of course, once you start moving the code in this direction, you should also start to think about r to l scripts instead of assuming l to r is correct for everyone. kre ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
[2010-10-26 19:18] Peter Maydell pmayd...@chiark.greenend.org.uk markus schnalke wrote: during the next weeks I intend to fix the annoying problem of still encoded message text in quoted replies. That would certainly be a cool thing to have working. I'm still becoming familiar with the code base, in order to find out how everything works and how to do it in a fairly good way. You might be interested in this: http://www.mail-archive.com/nmh-workers@nongnu.org/msg01508.html The referenced mail helped a lot, thanks. Basically there are a number of intertwined problems. My major concern would be that in fixing de-encoding when composing replies we don't accidentally paint ourselves into a corner for the UI/config settings for getting the larger issues right. Right. meillo ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
[2010-10-31 19:22] Ralph Corderoy ra...@inputplus.co.uk I was thinking of a scan(1) replacement that would colour-code the results, if I wanted. You can do this by piping the output through a highlighter program, or even by using a customized format file for scan. To begin with, I'd be happy with configuration being in Python. I hope you don't find supporters for that. ;-) And then an mhshow(1) that assumes a UTF-8 terminal and converts headers and body to that for one seamless less(1) It should surely not assume anything but find out (MM_CHARSET) what is wanted. Also not less(1) but moreproc. But on the seamless output, I agree. instance without a Press return to show content... prompt. -nopause fix that, or? meillo ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
[2010-10-26 21:03] Oliver Kiddle okid...@yahoo.co.uk markus schnalke wrote: (1) Showing messages with different charsets requires a line like mhshow-charset-iso-8859-1: iconv -f iso-8859-1 | %s in the profile if your terminal is on UTF-8. It might be worth to include such processing directly into nmh in order to make it more usable out-of-the-box. Iconv is already available (if compiled in). That would be good but you lose some flexibility. In some cases, I use the //TRANSLIT option on iconv. Also, I actually treat iso-8859-1 as windows-1252 because mails from Windows systems are sometimes incorrectly identified as iso-8859-1. mhshow-charset- entries should have precedence, of course. Then everything would be like now but text in foreign charsets gets converted to the native charset if nothing else is said. meillo ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
And then an mhshow(1) that assumes a UTF-8 terminal and converts headers and body to that for one seamless less(1) I personally want mhshow to go away. It should all be integrated into show. ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
AFAIK this problem is still unsolved. I only found a hack in the FAQ: Subject: 05.16 How can I convert quoted-printable to 8bit in quoted text in replies? From: Jarle F. Greipsland jarle at idt.unit.no Date: 22 Aug 1995 10:42:07 +0200 wow. below you'll find a less convoluted hack. i seem to be using it a LOT lately. perhaps it should be folded into the faq. steve -- #!/bin/bash repl -filter /dev/null -nodraftfolder -nowhatnowproc $* (cat ~/Mail/draft; echo -e \n Original Message ; \ mhshow cur -type text -nopause -nomoreproc \ | egrep -v '^Return-Path|^Delivery-Date|^part[ ]+text/plain' \ | egrep -v '^X-|^MIME' \ | sed 's/^/ /' \ ) ~/Mail/drafts/999 comp -use -draftfolder drafts -draftmessage 999 rm -f ~/Mail/draft rm -f ~/Mail/drafts/999 ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
Hi meillo, Using nmh though is currently pretty tough. ;-) Agreed, in this modern world. I persist with it but that's because it's what I know. I occasionally have flights of fancy about gradually replacing some of its commands with Python ones. Am I the only one? The C code base of nmh has accumulated lots of cruft over the years and contains code and tests that's no longer required on any modern Unix. But there's not much of a test suite to aid trying to clean it up. I was thinking of a scan(1) replacement that would colour-code the results, if I wanted. To begin with, I'd be happy with configuration being in Python. And then an mhshow(1) that assumes a UTF-8 terminal and converts headers and body to that for one seamless less(1) instance without a Press return to show content... prompt. Unfortunately, the spirit is willing but the tuits are lacking. :) Good to see some activity on nmh regardless. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
markus schnalke wrote: during the next weeks I intend to fix the annoying problem of still encoded message text in quoted replies. That would certainly be a cool thing to have working. Has anyone of you already worked on this? Has nayone information that can help me? You might be interested in this: http://www.mail-archive.com/nmh-workers@nongnu.org/msg01508.html which I wrote back in 2008(!) about nmh's MIME handling issues. Basically there are a number of intertwined problems. My major concern would be that in fixing de-encoding when composing replies we don't accidentally paint ourselves into a corner for the UI/config settings for getting the larger issues right. For users in non-English speaking countries such issues matter much. Using nmh though is currently pretty tough. ;-) I completely agree. I'll try to make some time to take another look at this some time this week. (I'm at the Ubuntu Developer Summit so I'm in a free software kind of mood anyway :-)) -- PMM ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Fixing decoding of quoted messages in replies
Thus spake markus schnalke: Hoi nmh workers, during the next weeks I intend to fix the annoying problem of still encoded message text in quoted replies. The problem appears when running repl -format on a message with Content-Transfer-Encoding like quoted-printable or base64. Let me be the first to offer you encouragement in fixing this. It's bothered me for quite some time now. -- J. ___ Nmh-workers mailing list Nmh-workers@nongnu.org http://lists.nongnu.org/mailman/listinfo/nmh-workers
