Re: [nmh-workers] Formatting HTML to Text: netrik.
In message , Conrad Hughes wrote: > - replfilter depends on 'par', which users may not have installed, but U... I don't have that either. What is it and where do I get it? >Finally, I'd almost be inclined to have nmh-without-replfilter display a >message about replfilter, for example maybe in whatnowproc, so after >grinding your teeth about the undecoded base64 you at least see a >message suggesting a remedy for this after exiting the editor. I >realise though that accurate detection of circumstances where it would >be helpful to display such a message might not easy, but it would save a >certain amount of repetition. Seconded. Some folks... me included... need to be very explicitly knocked upside the head in order to make sure that we get the message. Or maybe replfilter should just become part of the (shipped) default configuration. Lord knows that well over than 50% of all emails I've received over the past several years contain either base64 or HTML or both, so that would seem to make some sense. Regards, rfg P.S. Speaking as a relic of a now long bygone era (which I am), I really do wish that all of this base64 and HTNMLized email stuff would get off my lawn. (Yes, I'm an old geezer.) It all annoys me very much, because I know damn well that all of this stupid HTML stuff... which makes all mails about a factor of ten bigger... isn't actually carrying any useful additional information that could not have been represented and expressed just as well with good old plain text. But I already lost this battle at least a decade or two ago. Sigh. Oh well. We live and we adapt. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
In message <20190627232032.737b9159...@pb-smtp1.pobox.com>, Ken Hornstein wrote: >>I had just sort-of jury-rigged my own very local and very idiosyncratic >>mechanism for dealing with the problem/issue some years ago, and I have >>been just trying to struggle along with it for all this time because >>I just haven't had the time to find a proper fix... which I've always >>assumed is burred in the documentation someplace. But if there really >>is no good solution in the case of nmh, then I guess that eventually... >>and proabbly sooner rather than later... I'm going to have to switch >>mail clients at long last, although I sure will miss some of the nicer >>nmh features. > >Geez, I thought we handled that pretty well. Apologies if I offended, unintentionally. I should have gone further to clarify that I'm quite completely sure that all of the issues I've encountered are due either to my own ignorance, or to the fact that I had jury-rigged my personal nmh install, long long ago, approximately in the bronze age, and I've just never had the time to re-do any of this stuff properly, or even to -learne- the specifics of what I have been doing wrong, let alone the "new" and proper way of doing things. (I';ve been using nmh and its predecessor nmh since about 1982, so that will give you some idea of the fact that I'm set in my ways, more than a little bit.) >For base64 emails, we should handle that fine for display, full stop. Perhaps, but for display, since so much of the email I've received for low these many years now has been either partially or totally HTMLized crap, *and* because I have some rather specialized needs, long ago I invented and implemenmted my own personal quirky little "solution", which involved running HTMLized MIME parts thrugh the linx browser to render them as plain text. I freely admit that it was most probably a *really* bad idea for me to have strayed so far from the beaten parth, but I do a lot of anti-spam work, and since the dawn of time this case caused me to insist on always seeing the FULL HEADERS of each and every email message I read. (And I'm sure this was part of what motivated me to go down this route, although I have paid for it in daily agony ever since.) So anyway, here's what I have now, and you're all invited to tell me what obvious garbage it is, and how I should be doing things The Right Way... For starters, I have in my ~/.mh_profile the following lines: showproc: mtext showmimeproc: mtext On my system "mtext" is a trivial Perl script I wrote that simply figures out the full pathname of the "current" NMH message and then passes that to a different small Perl script I wrote (called "textualize") that does most of the heavy lifting. Here are those two short Perl scripts: https://pastebin.com/raw/XgqbekaZ https://pastebin.com/raw/EMSgdXaD Together, these two work well enough to show me what I want to see when I do "show", but the whole mess breaks down badly (and is very neraly entirely useless) when I try to "repl" any message. >If you are building from source, _if_ you have one of the common text-based >HTML browsers available (I use w3m, but we also support lynx and elinks), >then show(1) does the right thing, and pretty reasonably I would say. OK, well, but as noted above, I've screwed things up, badly, so what should I *actually* have in my .mh_profile file for the definitions of showproc and showmimeproc? what are the current (proper) defaults for those? And do the defaults have options to show me FULL headers for each message? If not, how do I make that happen? >If you are installing from an OS package ... well, what happens there varies. >At least for MacOS X, it will use w3m (this all is configured at install >time). But if you've put your own entry in for mhshow-show-text/html, then >you wouldn't see that. We've done that since 1.6 (released in 2014). Here is my current, much fiddled ~/.mh_profile. Tell me -everything- I should fix and I'll fix it all: https://pastebin.com/raw/PEMJ06VH (Yes, this .mh_profile has been reused and recycled and fiddled and ignored ad infinitum from literally EONS ago. In you see anything in there that hasn't even been supported for 10+ years, don't be surprised.) >As for repl(1), we've shipped with replyfilter in $(DOCDIR)/contrib >since 1.5 (released in 2012), and I use it on every message. I won't >say it's perfect, but 95% of the time it does the right thing for me, >and handles HTML and base64 email just fine. I've just upgraded everything, so I'm on a fresh new FreeBSD 12.0 system with a fresh and shiny new nmh-1.7.1 pre-built package installed, so I do believe that I have everything needed in order to get rolling. Checking now, i see that I *do* have the following: /usr/local/share/doc/nmh/contrib/replyfilter >Unfortunately without a >complete rewrite of mhl it was hard to put it in there transparently, so >it does require extra configuration (see the comments at the top of
Re: [nmh-workers] Formatting HTML to Text: netrik.
> - replfilter depends on 'par', which users may not have installed, but
>the failure mode in its absence is suboptimal: the quoted text is
>absent, and the error shown on the command line is something like
>
> Pipe reader process exited with 72057594037927935
>
>.. could well be enough just to list the required support tools in
>the instructions at the start of replfilter.
Fair enough; it's down a bit farther, but putting a bit at the top saying
you might want to look at that (and the HTML converter) would be helpful.
You could also use "fmt", but I've found par tends to work better.
> - Reading the manpage I see that .mh_profile comments are supposed to
>start '#:', but lines starting '#' give every practical appearance
>of working as comments too — I've wrongly used that for years
>— except that if you have two of them in a row, nmh commands fail
>with "no blank lines are permitted in .mh_profile". The error at
>least seems confused. Is the colon in '#:' just for easy parsing?
>Would it break things for # to introduce a comment?
W yes.
You may notice that on closer inspection that mh-profile actually looks
a lot like a message header, in that no blank lines are permitted and
it consists of a header field name ('name:') and header field text.
This is not a coincidence. The same routine used for parsing message
headers is also used to parse the profile (and context files, and
message sequence files ... sigh).
So we'd have to either introduce some special-case code during profile
parsing, or change the email parser code (ugh). Both of these are hard;
the function used during message parsing (m_getfld()) really takes over
the input stream and does a fair amount of caching for efficiency, so
you can't easily look at the input stream and say, 'Oh, this starts with
a #, skip this line'. And for changing m_getfld() ... well, take a look
at it sometime and tell me if YOU want to mess with it. Welcome to
how the sausage is made :-/
What you're doing when you put in '#:' is creating a special profile
entry called '#' which is not used for anything, but it looks like a
header field just enough that m_getfld() is happy. I forget who
pointed this out on the mailing list many years ago, but the easiest
solution was just to document the current behavior.
I have been playing around with a flex-based email header parser; if
that gets working it would be very easy to create a slight modification
that handled '#' based comments for things like the profile. That
would be part of my "full MIME parsing" work and wouldn't be done for
a while.
>Finally, I'd almost be inclined to have nmh-without-replfilter display a
>message about replfilter, for example maybe in whatnowproc, so after
>grinding your teeth about the undecoded base64 you at least see a
>message suggesting a remedy for this after exiting the editor. I
>realise though that accurate detection of circumstances where it would
>be helpful to display such a message might not easy, but it would save a
>certain amount of repetition.
I am sympathetic to that idea ... the problem is that requires making
mhl be smarter about what is and isn't a MIME body. Right now mhl knows
nothing of MIME, it just sees the message body as one big text blob.
Making it do MIME is a lot of work. In a perfect "full MIME parsing"
world it would just DTRT and replyfilter could vanish. There are sadly
no wonderful solutions.
--Ken
--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
Ken> As for repl(1), we've shipped with replyfilter in $(DOCDIR)/contrib Ken> since 1.5 (released in 2012), and I use it on every message. Just to say thanks for your patience on this: has been bugging me for a while, and finally just sitting down and unpacking replfilter made a huge difference. Two minor issues cropped up for me while setting it up though; these may already have been addressed (I use Debian, so am on nmh 1.6-16), but just in case: - replfilter depends on 'par', which users may not have installed, but the failure mode in its absence is suboptimal: the quoted text is absent, and the error shown on the command line is something like Pipe reader process exited with 72057594037927935 .. could well be enough just to list the required support tools in the instructions at the start of replfilter. - Reading the manpage I see that .mh_profile comments are supposed to start '#:', but lines starting '#' give every practical appearance of working as comments too — I've wrongly used that for years — except that if you have two of them in a row, nmh commands fail with "no blank lines are permitted in .mh_profile". The error at least seems confused. Is the colon in '#:' just for easy parsing? Would it break things for # to introduce a comment? Finally, I'd almost be inclined to have nmh-without-replfilter display a message about replfilter, for example maybe in whatnowproc, so after grinding your teeth about the undecoded base64 you at least see a message suggesting a remedy for this after exiting the editor. I realise though that accurate detection of circumstances where it would be helpful to display such a message might not easy, but it would save a certain amount of repetition. Conrad -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
>I had just sort-of jury-rigged my own very local and very idiosyncratic >mechanism for dealing with the problem/issue some years ago, and I have >been just trying to struggle along with it for all this time because >I just haven't had the time to find a proper fix... which I've always >assumed is burred in the documentation someplace. But if there really >is no good solution in the case of nmh, then I guess that eventually... >and proabbly sooner rather than later... I'm going to have to switch >mail clients at long last, although I sure will miss some of the nicer >nmh features. Geez, I thought we handled that pretty well. For base64 emails, we should handle that fine for display, full stop. If you are building from source, _if_ you have one of the common text-based HTML browsers available (I use w3m, but we also support lynx and elinks), then show(1) does the right thing, and pretty reasonably I would say. If you are installing from an OS package ... well, what happens there varies. At least for MacOS X, it will use w3m (this all is configured at install time). But if you've put your own entry in for mhshow-show-text/html, then you wouldn't see that. We've done that since 1.6 (released in 2014). As for repl(1), we've shipped with replyfilter in $(DOCDIR)/contrib since 1.5 (released in 2012), and I use it on every message. I won't say it's perfect, but 95% of the time it does the right thing for me, and handles HTML and base64 email just fine. Unfortunately without a complete rewrite of mhl it was hard to put it in there transparently, so it does require extra configuration (see the comments at the top of replyfilter). We talked about this a LOT on the mailing list during it's development, and here's the snippet from the NEWS file: - Preliminary support for improved MIME handling when replying to messages! Yes, a long requested feature has a solution. A perl script called replyfilter is available; it is designed to act as a mhl external filter to process MIME messages in a more logical way. It is available in $(srcdir)/docs/contrib/replyfilter or is typically installed as $(prefix)/share/doc/nmh/contrib/replyfilter. See the comments at the top of replyfilter for usage information; it will likely require some adjustment for your site. replyfilter requires the MIME-Tools and MailTools perl modules. So, I hope it doesn't seem like I'm crapping on you, because you're not the first person who is unaware of replyfilter (you can search the mailing list archives for people who have asked this question before, and I can write this email in my sleep now). I thought we did a pretty good job of putting this information out there, but clearly in the 7 years since 1.5 has come out everyone hasn't gotten the word yet. So my question to YOU is ... what could we have done better to let you know? (Yes, in a perfect world 'repl' would just do the right thing automatically without any extra configuration, but that's a heavy lift). --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
Ronald F. Guilmette wrote: > Quite simply, all I would wish for would be something that would -properly- > convert -both- HTMLized emails -and- "Content-Transfer-Encoding: base64" > emails (like this one I'm responding to) into good old fashioned ascii, > at least for purposes of the "show" and "repl" commands. I have my > jury-rigged solution working adequately well for the base64 encoding > still, but only for the "show" command, which means that I have to do > some manual cutting-and-pasting when/if I want to reply to a base64 > encoded email. :-( I would also like that for the cases where I want to use show. I use mh-e, and I mostly have things configured right: 1) use text/plain if it exists. 2) format text/html is no text/plain (3) but often text/plain is bullshit-pseudo-HTML and you need to avoid it. The additional problem is that reply yanks text from formatted text/html rather than text/plain. The good HTML formatters in mh-e (Emacs) are slow, and the fast ones do a poor job. I would rather show did things and mh-e used that, but too many moving parts. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
In message <20190627202344.7389f...@enterprise.home.antispaceman.com>, spaceman wrote: >All these in combination you end with a reasonable reply to HTML emails. >The downside is that you don't get to keep the original email unless you >make a copy of it and it's fairly hacky. Thanks for all the tips, but this is a non-starter for me. I need to preserve originals. I would think there should be some way of doing that *and* getting nicely TEXTified emails, no? -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
Hi,
Ronald F. Guilmette wrote:
> Quite simply, all I would wish for would be something that would -properly-
> convert -both- HTMLized emails -and- "Content-Transfer-Encoding: base64"
> emails (like this one I'm responding to) into good old fashioned ascii,
> at least for purposes of the "show" and "repl" commands. I have my
> jury-rigged solution working adequately well for the base64 encoding
> still, but only for the "show" command, which means that I have to do
> some manual cutting-and-pasting when/if I want to reply to a base64
> encoded email. :-(
mhfixmsg does this pretty well I think, you just have to use it in
your procmail rules. It will convert the messages to plain ascii (from
base64) as well as convert html to text and add it as plain/text section
in the mime. I use the following:
mhfixmsg-format-text/html: charset="%{charset}"; /usr/bin/w3m -I ${charset} -T
text/html -dump
This works in combination with replyfilter (a perl script distributed
with nmh or used to be) which nicely puts only the plain text section in
the reply for you (I am not sure if nmh would do this, I don't see why
not).
All these in combination you end with a reasonable reply to HTML emails.
The downside is that you don't get to keep the original email unless you
make a copy of it and it's fairly hacky.
Regards,
spaceman
--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
In message <20190627125114.70a3921...@orac.inputplus.co.uk>, Ralph Corderoy wrote: >Revisiting once again the issue of nice text from horrible HTML emails... And here, all this time, I thought that it was just me! I assumed that I had just failed to read the documentation well enough or long enough to work out some appropriate was of causing nmh to properly render the (now prevalent) HTMLized and/or base64 encoded emails of the modern era. I had just sort-of jury-rigged my own very local and very idiosyncratic mechanism for dealing with the problem/issue some years ago, and I have been just trying to struggle along with it for all this time because I just haven't had the time to find a proper fix... which I've always assumed is burred in the documentation someplace. But if there really is no good solution in the case of nmh, then I guess that eventually... and proabbly sooner rather than later... I'm going to have to switch mail clients at long last, although I sure will miss some of the nicer nmh features. Quite simply, all I would wish for would be something that would -properly- convert -both- HTMLized emails -and- "Content-Transfer-Encoding: base64" emails (like this one I'm responding to) into good old fashioned ascii, at least for purposes of the "show" and "repl" commands. I have my jury-rigged solution working adequately well for the base64 encoding still, but only for the "show" command, which means that I have to do some manual cutting-and-pasting when/if I want to reply to a base64 encoded email. :-( Regards, rfg -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] success using the OAUTH2 with gmail.
>> Um, is it? I just looked at my copy, and it's there! And according to >> the revision history that was added in 2016, and nmh 1.7 was released >> in 2017 (and I didn't update the NEWS file for 1.7.1; dammit). > >Sorry, I should be more precise. >The values which are valid for "authservice" are not, as far as I can, >mentioned. Reading that with fresh eyes yeah, I see what you mean. We kind of skate on the details with regards to that, don't we? It's meant to be generic, in theory, but we don't really say anywhere "Hey, we have included the right magic for gmail so you just need to say -authservice gmail". We do point to the mhlogin(1) man page, but we don't go into great detail there either. Thanks for this feedback! It's always good to get someone new looking at this, because since I was involved with the integration of this code (not as much as David and Eric, though) I was familiar with the basic ideas so I didn't see what was missing. Back in: https://lists.gnu.org/archive/html/nmh-workers/2019-05/msg0.html One of my bullet points for a possible 1.8 release was: - Write a man page explaining about the network security for inc/post and friends and how it works in nmh, since it's kind of confusing. And this sounds like a perfect candidate for this. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
Ken Hornstein wrote in <20190627175008.639987b...@pb-smtp21.pobox.com>: |>Thinking about it, the "ext" in SSL_set_tlsext_host_name |>could appear strange in five years from now. | |As opposed to the REST of the OpenSSL API? :-) ..seen that way.. But the problem is real: #?0|kent:$ grep -ri deprecated /usr/include/openssl/|wc -l 85 And this lists prominent things like and as new as TLSv1_2_XY(). --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] success using the OAUTH2 with gmail.
Ken Hornstein wrote: >> -authservice is missing from the inc.1 man page. > Um, is it? I just looked at my copy, and it's there! And according to > the revision history that was added in 2016, and nmh 1.7 was released > in 2017 (and I didn't update the NEWS file for 1.7.1; dammit). Sorry, I should be more precise. The values which are valid for "authservice" are not, as far as I can, mentioned. > I believe that -authservice gmail does that already (it should use our > compiled-in default values for gmail, including our registered client > identifier). okay, I suggest an example in inc. > If you mean --gmail does everything like -initialtls, -host, -port, -sasl > -saslmech xoauth2 and -authservice gmail, w actually, I have > some thoughts on that. > Let's say in a hypothetical future we support IMAP. That means that nearly > every command would take a whole pile of arguments like -initialtls, > -host, -port, -sasl, and more. Obviously changing your profile for every > nmh command would be awful. So there should be some way of handling that. > What I had thought maybe was tying profile entries to mailboxes, so if > you did "scan my-imap-server:foo" it could possibly look in your profile > and find: Interesting idea. > You get the idea. But thinking about this more makes me think that we > should extend this a bit so it's not tied to folders, but a generic > connection profile defaults and we could provide ones that work with > Gmail. I don't have it all jelled in my head how this would look and > you'd need to do something to ADD to an existing connection profile so > you could supply your own username, for example. But it seems like it > should be doable. But I guess my idea is that you should be able to > do something like > inc -conn gmail -user myu...@gmail.com > and the right stuff should happen. Make sense? Yeah, that's what I'd want to happen. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] success using the OAUTH2 with gmail.
>-authservice is missing from the inc.1 man page. Um, is it? I just looked at my copy, and it's there! And according to the revision history that was added in 2016, and nmh 1.7 was released in 2017 (and I didn't update the NEWS file for 1.7.1; dammit). >I think that maybe we should have --gmail which sets all the right >parameters. I believe that -authservice gmail does that already (it should use our compiled-in default values for gmail, including our registered client identifier). If you mean --gmail does everything like -initialtls, -host, -port, -sasl -saslmech xoauth2 and -authservice gmail, w actually, I have some thoughts on that. Let's say in a hypothetical future we support IMAP. That means that nearly every command would take a whole pile of arguments like -initialtls, -host, -port, -sasl, and more. Obviously changing your profile for every nmh command would be awful. So there should be some way of handling that. What I had thought maybe was tying profile entries to mailboxes, so if you did "scan my-imap-server:foo" it could possibly look in your profile and find: my-imap-server: -host my.server.com -port imap -tls -sasl -saslmech GSSAPI -user me You get the idea. But thinking about this more makes me think that we should extend this a bit so it's not tied to folders, but a generic connection profile defaults and we could provide ones that work with Gmail. I don't have it all jelled in my head how this would look and you'd need to do something to ADD to an existing connection profile so you could supply your own username, for example. But it seems like it should be doable. But I guess my idea is that you should be able to do something like inc -conn gmail -user myu...@gmail.com and the right stuff should happen. Make sense? --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
>Thinking about it, the "ext" in SSL_set_tlsext_host_name >could appear strange in five years from now. As opposed to the REST of the OpenSSL API? :-) --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
Ken Hornstein wrote in <20190627171410.ea24e7b...@pb-smtp21.pobox.com>: |>I use that protected via |> |> #ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME | |I did see that ... but I also was worried that since OpenSSL makes no |guarantees that this define will stick around in the future, depending |on that may come back to bite me. I'd rather simply just put it in |unconditionally and force everyone to be using 1.0.0 or newer. Fair enough. Though i am afraid that regarding OpenSSL bit rot will have to be expected; the _CTRL_ series looked the most stable to me. Thinking about it, the "ext" in SSL_set_tlsext_host_name could appear strange in five years from now. Btw. i was lazy and simply call this function, even if SSLv3 was still around by then (more than today): OpenSSL and derivates do not perform any checks, it is just that the hostname set will be used for SNI if possible, and not otherwise. Unlikely this has changed. (Despite that noone uses SSLv3 no more.) --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] fetchmail and SNI (and pop.gmail.com)
Ken Hornstein wrote: > It looks like Debian buster is the earliest version of Debian which has > nmh 1.7.1. And it looks like that will be officially released next week. > If you upgraded, would that be enough for you to switch away from > fetchmail? :-) We support XOAUTH2! I won't upgrade, I just installed from source. I needed libcurl-dev. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] success using the OAUTH2 with gmail.
I've made sure I am running 1.7 (from git), and I went through mhlogin correctly. I pasted the result in, and then I did this with a second account. The token seems to be stored into Mail/oauth-gmail. %inc -initialtls -host pop.gmail.com -user mcharl...@gmail.com -port pop3s -snoop -sasl -saslmech xoauth2 -authservice gmail -authservice is missing from the inc.1 man page. I think that maybe we should have --gmail which sets all the right parameters. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] fetchmail and SNI (and pop.gmail.com)
>> And geez Mike, we talked about this a lot! Wasn't a secret! > >I read the man page. I wonder if my man pages are coming from debian, while >my binaries are manually installed. It looks like Debian buster is the earliest version of Debian which has nmh 1.7.1. And it looks like that will be officially released next week. If you upgraded, would that be enough for you to switch away from fetchmail? :-) We support XOAUTH2! --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
>I use that protected via > > #ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME I did see that ... but I also was worried that since OpenSSL makes no guarantees that this define will stick around in the future, depending on that may come back to bite me. I'd rather simply just put it in unconditionally and force everyone to be using 1.0.0 or newer. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
Ken Hornstein wrote: > When researching the issue Michael Richardson brought up today, it make > me realize we really should be calling SSL_set_tlsext_host_name() so we > send the TLS extension "server name indicator". Which is easy, it's > literally one line of code. But that makes me ask a larger question: we > have some autoconf goo to support older libraries (pre OpenSSL 1.0.2) > that didn't support the function X509_VERIFY_PARAM_set1_host(), and I > lack the energy to research if SSL_set_tlsext_host_name() exists in > pre-1.0.2 OpenSSL. I think at this point we should consider OpenSSL > 1.0.2 the minimum supported version of OpenSSL for nmh. This would > guarantee we are doing TLS 1.2 everywhere and clean up some #ifdefs. > Objections? I concur. If you have <1.0.2, then you probably don't have useful TLS, and should build without it. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] fetchmail and SNI (and pop.gmail.com)
Ralph Corderoy wrote: >> I have used: >> >> fetchmail --verbose --sslcertpath="/etc/ssl/certs" --sslcertck >> --proto POP3 --mda "rcvstore -sequence gmail +inbox" >> --logfile /var/tmp/gmail.log pop.gmail.com >> >> to get my gmail downloaded for some time now. > Has your OpenSSL been upgraded recently? Yes-ish, I'm usually running something from git. >> It seems that fetchmail doesn't enable SNI for it's TLS connection > Try adding `--sslproto TLS1' to fetchmail's arguments. That worked perfectly. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
>That would make RHEL6 users, at least, sad: > >$ rpm -q openssl >openssl-1.0.1e-57.el6.x86_64 >openssl-1.0.1e-57.el6.i686 I feel your pain since we use a lot of CentOS 6 at work, but you don't have much longer to use it, right? I think support for it only goes until next year, unless you pay for extended lifecycle support. Maybe we can come out with a newer release of nmh before then, but it's not like it's tomorrow. But you motivated me enough to look ... I see that 1.0.1 DOES actually include the necessary function (SSL_set_tlsext_host_name()). It looks like that was added for 1.0.0. >I am not exactly confident that replacing that with 1.0.2 or later would >be feasible --- didn't they break ABI to some extent in that revision? Ummm 'maybe'. There is no ABI compatibility guarantee, that is for sure. It looks like what bit us was that going from 1.0.2 to 1.1.0 a library function (SSL_library_init) was turned into a macro. But there is nothing stopping you from installing a newer OpenSSL into /usr/local and linking nmh against that; it wouldn't conflict with anything installed. I feel that since SSL_set_tlsext_host_name() has been around for approximately forever I'm fine with just adding it and assuming that everyone is at 1.0.0 or newer (but I just know someone will show up still using 0.9.8). But it does beg a larger question ... should we still force a minimum version of 1.0.2? The reason I ask is our current code has an #ifdef for the function X509_VERIFY_PARAM_set1_host() which controls the verification of the name of the server certificate against the passed-in hostname, which is pretty important; without that no hostname verification of the server certificate is done. I don't know if we think this is important enough that we require nmh have this functionality or not (you can always turn it off with a command line switch). --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] fetchmail and SNI (and pop.gmail.com)
Ken Hornstein wrote: > And geez Mike, we talked about this a lot! Wasn't a secret! I read the man page. I wonder if my man pages are coming from debian, while my binaries are manually installed. SNI === Server Name Indicator, which lets a server know which name a client meant to connect to, and therefore, which certificate to respond to. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
Ken Hornstein wrote in <20190627150420.4ff107a...@pb-smtp21.pobox.com>: |Everyone, | |When researching the issue Michael Richardson brought up today, it make |me realize we really should be calling SSL_set_tlsext_host_name() so we |send the TLS extension "server name indicator". Which is easy, it's |literally one line of code. But that makes me ask a larger question: we |have some autoconf goo to support older libraries (pre OpenSSL 1.0.2) |that didn't support the function X509_VERIFY_PARAM_set1_host(), and I |lack the energy to research if SSL_set_tlsext_host_name() exists in |pre-1.0.2 OpenSSL. I think at this point we should consider OpenSSL |1.0.2 the minimum supported version of OpenSSL for nmh. This would |guarantee we are doing TLS 1.2 everywhere and clean up some #ifdefs. |Objections? I use that protected via #ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME which seems to work everywhere i tried. --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Formatting HTML to Text: netrik.
Ralph Corderoy wrote in <20190627125114.70a3921...@orac.inputplus.co.uk>: |Revisiting once again the issue of nice text from horrible HTML emails, |I found another textual web browser, like lynx(1), etc., but |http://netrik.sourceforge.net/ uses colour in its --dump output. |It doesn't list the URL destinations though, e.g. the noisy `[42]' that |links prefixes to the anchor's text. I also haven't looked into what |remote accesses it may make. You could try to take out the very simple and extremely primitive HTML filter code of the MUA i maintain and embed it into nmh. It only uses standard POSIX C interfaces, so it should not be too hard. I do not use anything else for the few HTML things i get, i have never seen it fail. (I have test mails around which count as terror of a pathological maniac. The only one where i can see some garbage is actually from the german computer magazine c't, and it looks like [-- #1.2 1007/75296 text/html, quoted-printable, utf-8 --] 96 Silex: Neue Malware legt schlecht gesicherte Geräte im Internet of Things still -- Die Malware Silex kapert mit Default-Credentials IoT-Geräte, um sie lahmzulegen. Ihr 14-jähriger Entwickler handelt offenbar aus Spaß. [%p(2,2)]» https://www.heise.de/security/meldung/Silex-Neue-Malware-legt-schlecht-gesicherte-Geraete-im-Internet-of-Things-still- 4455677.html[%hr(=)] Wherever "96", "%p(2,2)" and "%hr(=)" come from and whatever they are, i never looked into this. I only ever see this from these guys. But WWF, Naturschutzbund and Conversation work for years, as well as some german shops, and Change.org and a lot of other mails in the HTML test box do, too. Guaranteed no loading of external data, anyway. --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
Ken Hornstein writes: > I think at this point we should consider OpenSSL > 1.0.2 the minimum supported version of OpenSSL for nmh. This would > guarantee we are doing TLS 1.2 everywhere and clean up some #ifdefs. > Objections? That would make RHEL6 users, at least, sad: $ rpm -q openssl openssl-1.0.1e-57.el6.x86_64 openssl-1.0.1e-57.el6.i686 I am not exactly confident that replacing that with 1.0.2 or later would be feasible --- didn't they break ABI to some extent in that revision? regards, tom lane -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Making OpenSSL 1.0.2 minimum version
>I think at this point we should consider OpenSSL >1.0.2 the minimum supported version of OpenSSL for nmh. This would >guarantee we are doing TLS 1.2 everywhere and clean up some #ifdefs. >Objections? One additional thing ... writing the code to check the version of OpenSSL is a mild to moderate pain. It seems like most people are slowly switching to pkg-config for things like this. What do people think of using pkg-config for this? Openssl 1.0.2 distributes a pkg-config file so it sure seems like every instance of it would include it (that would make pkg-config a build-time dependency if you wanted openssl support, though). --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] fetchmail and SNI (and pop.gmail.com)
>> It seems that fetchmail doesn't enable SNI for it's TLS connection > >Try adding `--sslproto TLS1' to fetchmail's arguments. I guess the core issue is that for Google servers when using TLS 1.2 SNI isn't required, but for TLS 1.3 it is; well, let me rephrase that. If you negotiate TLS 1.3 you get the bogus certificate if you don't send a SNI. But it seems like the 'right' solution is we should be sending a SNI to avoid this problem? --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] burst behavior
ralph wrote: > Hi Paul, > > > An obvious fix (for me) is to pre-process the digest, and > > hyphen-escape all lines which follow the "END OF DIGEST" line and > > which begin with a '-', by adding an extra '- ' at the start of line. > > If I do that, then burst will do the right thing, and ignore all of > > the trailer text. > > Isn't it simpler to > > sed '/^- *END OF [^ ]* DIGEST/q' > > if you're preprocessing? If your sed has -i then you could run it on > the digest first to do the change in situ. Yes, it certainly would be easier. (Too many trees -- where's the forest??) In fact, a few minutes before posting I thought to myself, "I'll bet Ralph replies with a one-liner for this." :-) paul =-- paul fox, p...@foxharp.boston.ma.us (arlington, ma, where it's 71.2 degrees) -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] burst behavior
>If I were to become extra-ambitious, and create a patch which added a >"-endofdigest" switch which did something similar, would there be >interest? Yes ... but I would personally prefer if it was more generic. Like if the -endofdigest switch took a regular expression, because as far as I can tell there is no standard when it comes to digest format nowadays (sigh). --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] fetchmail and SNI (and pop.gmail.com)
Hi Michael, > I have used: > >fetchmail --verbose --sslcertpath="/etc/ssl/certs" --sslcertck >--proto POP3 --mda "rcvstore -sequence gmail +inbox" >--logfile /var/tmp/gmail.log pop.gmail.com > > to get my gmail downloaded for some time now. Has your OpenSSL been upgraded recently? > It seems that fetchmail doesn't enable SNI for it's TLS connection Try adding `--sslproto TLS1' to fetchmail's arguments. -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] burst behavior
Hi Paul, > An obvious fix (for me) is to pre-process the digest, and > hyphen-escape all lines which follow the "END OF DIGEST" line and > which begin with a '-', by adding an extra '- ' at the start of line. > If I do that, then burst will do the right thing, and ignore all of > the trailer text. Isn't it simpler to sed '/^- *END OF [^ ]* DIGEST/q' if you're preprocessing? If your sed has -i then you could run it on the digest first to do the change in situ. -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] burst behavior
I'm on a list which, for historical reasons, is available only in digest form. (Sigh.) Like many digests, it includes hyphen-rich separator lines in the trailing text of the digest, causing burst to create a flurry of malformed messages when it parses according to RFC 934. Hoping for the digest creators to start honoring 934 (or convert to mime!) at this point is clearly just wishful thinking. An obvious fix (for me) is to pre-process the digest, and hyphen-escape all lines which follow the "END OF DIGEST" line and which begin with a '-', by adding an extra '- ' at the start of line. If I do that, then burst will do the right thing, and ignore all of the trailer text. Is there a better/easier way to handle this? If I were to become extra-ambitious, and create a patch which added a "-endofdigest" switch which did something similar, would there be interest? I think the new code would be something like: If the first non-whitespace text after a hyphen separator line is a case-insensitive match on "END OF [^ ]* DIGEST.*", then start ignoring hyphen separator lines. paul =-- paul fox, p...@foxharp.boston.ma.us (arlington, ma, where it's 69.4 degrees) -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] Making OpenSSL 1.0.2 minimum version
Everyone, When researching the issue Michael Richardson brought up today, it make me realize we really should be calling SSL_set_tlsext_host_name() so we send the TLS extension "server name indicator". Which is easy, it's literally one line of code. But that makes me ask a larger question: we have some autoconf goo to support older libraries (pre OpenSSL 1.0.2) that didn't support the function X509_VERIFY_PARAM_set1_host(), and I lack the energy to research if SSL_set_tlsext_host_name() exists in pre-1.0.2 OpenSSL. I think at this point we should consider OpenSSL 1.0.2 the minimum supported version of OpenSSL for nmh. This would guarantee we are doing TLS 1.2 everywhere and clean up some #ifdefs. Objections? --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] fetchmail and SNI (and pop.gmail.com)
>I don't think that inc has any TLS support. You are incorrect! Supported as of 1.7 when the unified security framework was implemented. From the NEWS file: - Complete unification of network security support. All network protocols (currently, POP and SMTP) have been refactored to use a common set of security routines. This means all protocols support all SASL mechanisms (via the Cyrus-SASL library) and TLS. TLS support has been strengthened to perform certificate name validation and to require TLS 1.1 as a minimum protocol. Also, all protocols can make use of the OAuth2/XOAUTH SASL mechanism, which is supported by Gmail. The last may be interesting to you. I had not heard of SNI before, but a quick test suggests to me that we work fine with pop.gmail.com (we don't error out, at least). The Interwebs suggest I should use a special API call to make that work and I definitely didn't do that, but it seems to be ok? And geez Mike, we talked about this a lot! Wasn't a secret! --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] fetchmail and SNI (and pop.gmail.com)
I have used: fetchmail --verbose --sslcertpath="/etc/ssl/certs" --sslcertck --proto POP3 --mda "rcvstore -sequence gmail +inbox" --logfile /var/tmp/gmail.log pop.gmail.com to get my gmail downloaded for some time now. It seems that fetchmail doesn't enable SNI for it's TLS connection, and I don't see any new versions of fetchmail in years. It looks like pop.gmail.com wants SNI: fetchmail: Trying to connect to 2607:f8b0:4001:c16::6c/995...connected. fetchmail: Server certificate: fetchmail: Unknown Organization fetchmail: Issuer CommonName: invalid2.invalid fetchmail: Subject CommonName: invalid2.invalid fetchmail: Server CommonName mismatch: invalid2.invalid != pop.gmail.com fetchmail: pop.gmail.com key fingerprint: 90:4A:C8:D5:44:5A:D0:6A:8A:10:FF:CD:8B:11:BE:16 fetchmail: Server certificate verification error: self signed certificate fetchmail: Missing trust anchor certificate: /OU=No SNI provided; please fix your client./CN=invalid2.invalid [nice hack to send a message back to the user Google...] I don't think that inc has any TLS support. (kerberos support, yes) Maybe there are other ways to skin this cat? -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] Formatting HTML to Text: netrik.
Hi, Revisiting once again the issue of nice text from horrible HTML emails, I found another textual web browser, like lynx(1), etc., but http://netrik.sourceforge.net/ uses colour in its --dump output. It doesn't list the URL destinations though, e.g. the noisy `[42]' that links prefixes to the anchor's text. I also haven't looked into what remote accesses it may make. -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
