Re: whatnow: problems with edit--draft left
Thus said Ralph Corderoy on Sun, 09 Jun 2024 22:16:25 +0100: > What am I doing wrong? I don't see 'success' in That's interesting, I'm not sure why the source wouldn't have it. You can see the rendered man page here: http://man.openbsd.org/ed However, I'm now confused why the man page as rendered doesn't match the sources as you have discovered. It seems that it us using the Ex macro: .Sh EXIT STATUS .Ex -std ed I'm no man page expert but it seems that .Ex is a macro that loads a standard string that is used commonly between commands where 0 is success and >0 is error. And it takes the name of the utility as an argument. Indeed it is found in mdoc_validate.c in the definition of post_ex(): http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/mandoc/mdoc_validate.c?rev=1.306&content-type=text/x-cvsweb-markup roff_word_alloc(mdoc, n->line, n->pos, "on success, and\\~>0 if an error occurs."); Andy
Re: whatnow: problems with edit--draft left
Thus said Ralph Corderoy on Thu, 06 Jun 2024 11:15:43 +0100:
> My guess is you're imperfect in your ed-ing. :-)
Or perhaps the ed utility is poorly implemented. Man page for OpenBSD's ed:
EXIT STATUS
The ed utility exits 0 on success, and >0 if an error occurs.
Does this statement mean that there was an error during the sequence of
ed commands or does it mean that there was an error in saving?
At any rate, observe:
$ ed /tmp/foo
/tmp/foo: No such file or directory
a
foo
.
wq
4
$ echo $?
0
$ ed /tmp/foo
4
/bar
?
$a
bar
.
wq
8
$ echo $?
0
So, in my case, ed exited 0. Is this a bug in OpenBSD's ed which
claims to be "compliant with the IEEE Std 1003.1-2008 ("POSIX.1")
specification", or a bug in the man page?
Andy
Re: Review code changes for handling huge lines?
Thus said David Levine on Sun, 07 Apr 2024 12:09:05 -0400: > Just to note that we would need a test suite addition that shows what > the change fixes. Yes, that's on the todo list. I'm not very familiar with the test harness but it should definitely be done if we're going to include it. > Also, there are these gcc warnings with these settings (from > build_nmh -d), based on what Fedora has used: Ok, I'll have to look into the warnings. I didn't see any when I built it on OpenBSD, but then I used OpenBSD's ports infrastructure to build it, not "build_nmh -d" Andy
Re: Review code changes for handling huge lines?
Thus said "Andy Bradford" on 04 Apr 2024 16:45:29 -0600: > Any interest in renewing a discussion about this? Also, I have found at least one contact mechanism that I'm pursuing but I'm not very hopeful that it will result in any kind of resolution---there is a "Send Feedback" option, however, I've utilized it in the past and received no confirmation. The bug still exists because the messages are still being sent with huge lines. There is no rush to get this code in, but I just wanted to bring this up again just in case anyone else has an opinion (assuming we have new subscribers since I last mentioned it). Sadly, I agree with Ralph's comments here: https://lists.nongnu.org/archive/html/nmh-workers/2022-11/msg00014.html The best option is to have the sender comply with RFC 5322, Section 2.1.1 which clearly restricts line lengths. We'll see where my efforts to contact end up. Thanks, Andy
Re: Review code changes for handling huge lines?
Thus said "Andy Bradford" on 04 Apr 2024 16:45:29 -0600: > A day later I also provided a patch against master: > > https://lists.nongnu.org/archive/html/nmh-workers/2022-11/msg7.html And that link is actually not the latest---I found some bugs along the way and addressed them in a separate patch. Here is the actual latest: https://lists.nongnu.org/archive/html/nmh-workers/2022-11/msg00028.html Andy
Review code changes for handling huge lines?
Hello, Back in August of 2022, I identified a "problem" with inc not handling long lines (to the tune of megabytes of characters all on a single line which is clearly contrary to all standards and RFCs): https://lists.nongnu.org/archive/html/nmh-workers/2019-09/msg7.html I have tried contacting the sender, which is a big organization in which finding an appropriate contact has been difficult, if not impossible so far. A couple months later in November of 2022, I produced a patch to address the issue and submitted it here: https://lists.nongnu.org/archive/html/nmh-workers/2022-11/msg0.html A day later I also provided a patch against master: https://lists.nongnu.org/archive/html/nmh-workers/2022-11/msg7.html There was some interest but I believe it got lost in the nmh 1.7.1 shuffle. After patching up my OS to the latest, I found that it was now running nmh 1.8, but was missing the code and so I tried applying that patch locally against my system and have been running it since then (January of this year 2024) without issues. Any interest in renewing a discussion about this? Thanks, Andy
Re: Trivia: space before '(others)' from 'folder' command
Thus said "Andy Bradford" on 20 Jan 2024 08:48:36 -0700: > I think it would be nice if it's aligned properly like other nmh > commands try to accomplish. Time to backpedal my comment a bit as it's not about proper alignment per se; I was distracted by the folders output which aligns multiple folder names. I suppose in this case, the argument is that because there is no current message, the presence of the extra spaces in the context of "folder" (without the 's') is distracting, or unnecessary because "folder" only deals with one folder at a time. Not only that, it's unlikely that one folder command needs to be aligned with a previous folder command, so a condition on all could make sense. Andy
Re: Trivia: space before '(others)' from 'folder' command
Thus said Ralph Corderoy on Sat, 20 Jan 2024 16:00:46 +:
> folders +OpenSourceML | awk '{l=length($1)}l>m{m=l;p=$0}END{print p}'
I see, I have one folder name that is fairly long and as a result, this
double-space shifted things just enough to make it wider than 80:
https://git.savannah.nongnu.org/cgit/nmh.git/tree/uip/folder.c?h=1.8-release#n588
But, as you say, it does appear to be just coincidence. If I run folders
against the longer folder name, clearly all of the subfolders in there
cause the lines of output to exceed the 80 character wide display and
wrap in the terminal.
Thanks for the nifty awk command to sort things out.
Andy
Re: Trivia: space before '(others)' from 'folder' command
Thus said Ken Hornstein on Fri, 19 Jan 2024 22:33:54 -0500: > Yeah, that is exactly why it happens. Honestly it doesn't personally > bother me, but what do others think? While I've never noticed it in the context of "folder -pack" (because I don't use it) I think it would be nice if it's aligned properly like other nmh commands try to accomplish. Not sure if this is related, but I have a similar experience with the folders command. For example, I have a folder OpenSourceML where I stuff various subfolders of things I'm subscribed to and on my 80 character wide terminal it seems to be having trouble figuring out the width. It seems to be generating a line that is exactly 81 characters long and so in my terminal it wraps the output exactly on the final period (.) leaving a much longer (visually) list of output than otherwise is necesary that looks like: $ folders +OpenSourceML FOLDER# MESSAGES RANGE; CUR(OTHERS) OpenSourceML+ hasno messages ;(others) . OpenSourceML/Bind has 338 messages (1- 338); cur= 338; (others) . OpenSourceML/OpenBSD hasno messages ;(others) . OpenSourceML/fossil has 16425 messages (1-16425);(others) . OpenSourceML/nongnu hasno messages ;(others) . Is this perhaps an off-by-one error? Aside from that everything seems to be impressively aligned just fine. Andy
Re: Where is my editor?
Thus said Ken Hornstein on Fri, 19 Jan 2024 16:45:18 -0500: > Like many threads involving ancient greybeards, it kind of devolved > into a discussion about the "true" vi and how vim wasn't vi enough It wouldn't be a good devolution if it didn't involve vi vs emacs or another editor! > And the consensus of everyone was prompter was fine as a default. lso, > Andy, you later said in that thread: > > Sounds like I might have to add something to my profile now after > this change is made to avoid prompter. Now that you quote it, I definitely remember saying it. And as I most recently hinted, I still don't have EDITOR/VISUAL set; I've now set the editor in the mh profile. > But to be fair, we did have a reasonable discussion about this change > (that it seems we all forgot about). Yes, indeed we did. Thanks for reminding me. It's amazing that it only took 6 years for me to finally get impacted by the results of the discussion! Watch out debian, if you think you have stable releases that live on forever, watch out for nmh as a contender. Andy
Re: Where is my editor?
Thus said Robert Elz on Fri, 19 Jan 2024 15:10:26 +0700: > Yes, I have had the "editor" line in my profile since MH days, (I have > been a user since almost forever) so what nmh did as default I never > knew, which is why I avoided saying that... >From looking at the past sources it used to use a C macro defined in h/nmh.h: #define DEFAULT_EDITOR "vi" I'm curious now how many installations of nmh do not have vi available... that being said, I do know Linux based OS developers make a lot of strange decisions about what is in the "base" OS, so it's entirely possible that vi may not exist in some flavors of Linux. How many non-POSIX systems is nmh running on? > I hope it is back working like you expected it now. It is now, thanks. I was able to figure out that EOT was what ended my experience with prompter and changed the default editor in .mh_profile and then I could just use "edit vi" at the "What now?" prompt. That's how I composed my original email starting this thread. It wasn't completely foreign (being a long-time Unix user I assumed it was just reading stdin and EOT would end it), just unexpected. I suppose I could set EDITOR but I've never had to on OpenBSD. Andy
Re: Where is my editor?
Thus said "Andy Bradford" on 18 Jan 2024 23:05:09 -0700: > Thus said Robert Elz on Fri, 19 Jan 2024 12:30:08 +0700: > > > That's "prompter" - has always been mh's default. Ahh, I didn't read the fine print... you said "mh's default", but I misunderstood it for "nmh's default". So it's a change to align with mh default behavior, and of course since I've never used mh I wouldn't know. Andy
Re: Where is my editor?
Thus said Robert Elz on Fri, 19 Jan 2024 12:30:08 +0700: > That's "prompter" - has always been mh's default. Not always: https://git.savannah.nongnu.org/cgit/nmh.git/log/sbr/geteditor.c?h=1.8-release Looks like it was changed in 1.8 (if I read that correctly). I wasn't aware of "prompter" before, thanks for the education. Andy
Re: Where is my editor?
Thus said Robert Elz on Fri, 19 Jan 2024 12:30:08 +0700: > That's "prompter" - has always been mh's default. Well, I'm confused. If it was always the default, why am I only now seeing it after upgrading to 1.8? I haven't changed anything in my .mh_profile in years: $ ls -l ~/.mh_profile -rw-r--r-- 1 amb amb 788 Aug 16 2019 /home/amb/.mh_profile I'll update ~/.mh_profile with your suggestion for editor, but I'm confused how it ever worked with my editor. I've been using nmh for years now and never had any dealings with "prompter". Thanks, Andy
Where is my editor?
Hello, After upgrading to 1.8 running commands like comp, repl, etc, no longer bring up my editor (vi) with the data already filled in. For example, to type this email I typed comp, and then it entered some kind of interactive dialog asking me for the To header, the Subject, and other headers, then it printed and then let me start typing a message. None of this is what it used to be. I used to type comp and immediately vi would open with all of the headers and other details from my components in there. Similarly, I used to type repl (followed by my favorite arguments) and it would populate my editor with a bunch of details that I could edit, but now I have some strange interactive thing. What can I do to get the original behavior back? Thanks, Andy
Re: Setting send(1)'s draftfolder breaks mhmail.
Thus said Ralph Corderoy on Thu, 18 Jan 2024 15:22:28 +: > The suggestion was to duplicate the setting by adding "send: > -draftfolder drafts". I think this breaks mhmail(1). Except, according to the man page for mhmail: PROFILE COMPONENTS mhmail does not consult the user's .mh_profile Perhaps it breaks send? Andy
Re: Strange problem replying to message
Thus said aalin...@riseup.net on Mon, 15 Jan 2024 08:57:51 -0500: > Now when I look at it instead of base64 it says 8bit. Am I confused or > is that what mhfixmsg did, changed the message from base64 to 8bit? According to the manpage for mhfixmsg, it's supposed to make a backup copy of the original message before making alterations. Hopefully that worked, and hopefully the backup hasn't yet been overwritten, and then you can inspect the backup to see what the original looked like. If you are unaware, a backup is the same message number prefixed with a comma (,) in the folder where it is found. So for example, if your message number was 1234, you should have a ,1234 in the folder that you can look at. Andy
Re: message-Id has localhost
Thus said Ken Hornstein on Sun, 31 Dec 2023 15:00:38 -0500: > >Can we just use "localname" from mts.conf? > > We COULD, it would just be wrong for some people. I definitely don't want nmh generating Message-ID unless I ask it to do so as my MTA is already correctly configured. The MTA that I use, qmail, does not use the hostname, it uses one of 2 configuration files... /var/qmail/control/me by default unless /var/qmail/control/idhost exists in which case it uses that. However, that only allows me to customize the right-side of the Message-ID. The left-side is predetermined and uses basically what the RFC recommends (a timestamp and a PID). Andy
Re: message-Id has localhost
Thus said Ken Hornstein on Sun, 31 Dec 2023 11:56:38 -0500: > I am personally skeptical that people actually configure this. I also find it hard to beleive that someone wants the MUA to have a specific Message-ID for their email, but there is at least one software that I'm aware of that does act upon the contents of it: http://smarden.org/qconfirm/qconfirm-check-mid.1.html It seems reasonable to believe that I may want control over the Message-ID for the purpose of making this work, however, it's definitely not required that the MUA be responsible for this. > My personal feeling is that the people who (a) care about generating a > local Message-ID, and (b) actually care WHAT appears right of the '@' > either need to configure their system appropriately or write code to > change nmh behavior. I think that's reasonable. I don't see anything necessarily wrong with nmh being able to generate a Message-ID so I'm sure patches would be considered if someone offered. Andy
Re: mhbuild and long header fields
Thus said David Levine on Sun, 17 Sep 2023 21:51:04 -0400: > How about this? > > mhbuild now folds header field bodies to avoid lines with more > then 78 bytes. > > This is required by RFC 5322 2.1.1. Line Length Limits. > Minor correction; replace "then" with "than". Also, RFC 5322 2.1.1 does not "require" that the lines be no more than 78 bytes, only that it's highly recommended that they not exceed that. The actual line limit is 998 bytes. Thanks, Andy
Re: (Not-so) hypothetical question: What to do about NULs?
Thus said Ken Hornstein on Wed, 22 Feb 2023 20:59:31 -0500: > I had an inkling popular MTAs would DTRT. Well, qmail's hardly "popular" these days, but Professor Bernstein had a penchant to make string handling robust to avoid exploits, so he got NUL handling as a benefit. I run minority MTA with minority MUA (at least for as long as the email cartel continues to permit legitimate email). Andy
Re: (Not-so) hypothetical question: What to do about NULs?
Thus said Ken Hornstein on Tue, 21 Feb 2023 21:29:16 -0500: > So you're told "I am sending this many bytes exactly", and you don't > have to deal with "lines", so the implementations I've seen tend to > call read() (or the equivalent) until they get the correct number of > bytes, and because you're not dealing with "lines" you don't treat > them as C strings. While POP's LIST does actually include the size of the message in bytes, that's prior to any CRLF mangling that happens so it cannot be used reliably as a method for determining when to stop reading. Unfortunate. I notice however, that some components of my email infrastructure pass NULs through without problems and some do not. qmail successfully queued a message with a NUL in both the header and the body, but other parts (e.g. recipient validation tools) did not fare as well, and of course we knew that inc would truncate (and it did because the lines with NUL were truncated). I suspect that qmail worked for the most part because of stralloc: http://cr.yp.to/lib/stralloc.html Andy
Re: (Not-so) hypothetical question: What to do about NULs?
Thus said Ken Hornstein on Tue, 21 Feb 2023 07:17:19 -0500: > I'm sitting down to write or modify nmh code. Right now we have a lot > of code that assumes NUL-terminated C strings are safe to represent > email everywhere. My question is: is that a valid assumption? I don't think nmh should produce anything that contains NUL bytes, but whether or not it should accept such is a different question (as you mention the 16 million byte line of text in an email message that I keep getting from a certain sender that cannot be bothered to follow the RFC which clearly states that base64 MIME data should be 78 characters and clearly not longer than 998). When I was poking around in the POP code I didn't notice any special handling of NUL bytes. It's possible that this would result in truncation. If that's what we do now, I suspect it's alright to continue to do so; at least until we find legitimate emails in the wild that do not conform (again think 16M character lines). nmh's POP code has been silently truncating long lines (e.g. those greater than 1023 bytes) for years and crashing on lines that were longer than 32,767 bytes). I only recently discovered this while trying to figure out what to do with a 16M character line. I went back through old emails and sure enough, I had a lot of truncation. I never noticed because most of them were in long lines of HTML that I don't ever bother reading. So I guess what I'm saying is, I think it's alright to continue to treat messages as C-strings (until it isn't). Andy
Re: (Not-so) hypothetical question: What to do about NULs?
Thus said Ken Hornstein on Mon, 20 Feb 2023 21:11:48 -0500: > Facinating! I am curious: who/what sent this to you! Do you remember > the MIME type? 0.11 % (percent) of my messages have Content-Transfer-Encoding of binary at the beginning of the line somewhere in the message. Here are the headers from one that dates all the way back to 2001 (this message does not appear to have any actual "binary" content in it). ---BEGIN Content-Type: multipart/mixed; boundary="--=_154292612-6290-0" Content-Transfer-Encoding: binary MIME-Version: 1.0 X-Mailer: MIME-tools 5.41 (Entity 5.404) From: "Jato Boa" Date: Xxx, 00 Xxx 2001 16:09:27 +0800 This is a multi-part message in MIME format... =_154292612-6290-0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline [ascii data] =_154292612-6290-0 Content-Type: image/jpg; name="OosI Fric Ghesuf kurfIzKi chruzGi Awt.jpg" Content-Disposition: attachment; filename="OosI Fric Ghesuf kurfIzKi chruzGi Awt.jpg" Content-Transfer-Encoding: base64 [base64 data] ---END-- Also, I have quite a few from the Bugtraq mailing list that have a C-T-E of binary. The headers indicate binary, but the rest of the body doesn't seem to imply it (doesn't need it probably), but then there are some like this: https://seclists.org/bugtraq/2004/Aug/223 Here are relevant headers and the binary values were replaced with : ---BEGIN Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: binary MIME-Version: 1.0 X-Mailer: MIME-tools 5.411 (Entity 5.404) From: "Jrme" ATHIAS To: bugt...@securityfocus.com Subject: First vulnerabilities in the SP2 - XP ?... X-Spam-BMF-Status: No, hits=0.00 required=0.90 http://www.heise.de/security/artikel/50051 Regards, Jrme ATHIAS ---END-- Today, I think this message would instead be quoted-printable or some other encoding. Here's another example from a well known online seller of goods that used messagelabs to send out customer order statuses: ---BEGIN Content-Transfer-Encoding: binary Content-Type: multipart/related; boundary="_--=_79242061420" MIME-Version: 1.0 X-Mailer: MIME::Lite 3.01 (F2.72; A1.60; B2.21; Q2.21) Date: Xxx, 00 Xxx 16:36:04 UT From: [online store redacted] This is a multi-part message in MIME format. --_--=_79242061420 Content-Transfer-Encoding: binary Content-Type: multipart/alternative; boundary="_--=_79242061421" MIME-Version: 1.0 X-Mailer: MIME::Lite 3.01 (F2.72; A1.60; B2.21; Q2.21) Date: Xxx, 00 Xxx 16:36:04 UT This is a multi-part message in MIME format. --_--=_79242061421 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain MIME-Version: 1.0 X-Mailer: MIME::Lite 3.01 (F2.72; A1.60; B2.21; Q2.21) Date: Xxx, 00 Xxx 16:36:04 UT [quoted printable data] ---END-- Here's a more recent email from another online provider of services with replaced where binary value was found: ---BEGIN Content-Type: text/html Content-Disposition: inline Content-Transfer-Encoding: binary MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) ... Copyright 2021 ... ---END-- Are these bugs in email client implementations? I've looked at a handful of the messages that I have which have a header of C-T-E binary and the body of the message is almost always some other C-T-E (mostly quoted-printable) or non-binary. But sometimes it seems justified. Maybe they just throw the C-T-E on there "just in case" as a sloppy way of getting by? > I guess what I was hoping for was a consensus on what we SHOULD do > when we encounter a NUL byte, because I haven't heard that yet! Like > what should the code do, precisely? I'm not sure. Does any one have any example of having received a NUL byte in an email? I'm having a hard time convincing grep to look for one. Andy
Re: nmh 1.8?
Thus said Ralph Corderoy on Mon, 02 Jan 2023 13:21:16 +: > > > Has anyone had a chance to review my proposed changes to inc to be > > > able to handle long lines from POP sources? > > Is the latest in Git? I doubt it. I sent an updated patch but never heard about it making into Git. It would be nice if I could find some time to add some test cases for it, but I didn't have as much time during the holidays as I thought I would: https://lists.nongnu.org/archive/html/nmh-workers/2022-11/msg00028.html As I said in that posting, I've been running a similar patch against 1.7.1 on my system and haven't encountered any problems using it. Andy
Re: nmh 1.8?
Thus said David Levine on Sat, 31 Dec 2022 22:00:04 +0100: > Do you or anyone else have anything else you'd like to put in before > starting the 1.8 release cycle? Has anyone had a chance to review my proposed changes to inc to be able to handle long lines from POP sources? While it's not common (most big email providers like Hotmail, Gmail, etc, all conform to RFCs), there are occasional emails (mostly from online web stores with shoddy software) that do send out non-conforming emails. Thanks, Andy
Re: inc and non-compliant long lines redux
Thus said "Andy Bradford" on 22 Nov 2022 09:38:08 -0700:
> I'm running a similar patch on my nmh 1.7.1 installation and will
> report any problems that I find in "production".
So far I haven't found any problems and I've been running it daily. I
did successfully receive one such of the original problematic emails
that I reported and it works just fine. Here is a breakdown of the
longest lines in the email:
$ cd ~/Mail/inbox
$ cat 3462 | awk '{ print NR, length }' | sort -k 2n | tail
5 71
9 74
19 75
28 75
11 89
38 91
88 126
65 170
103 225
115 4467120
As can be seen, line 115 is one massively long line. Oddly enough, I had
trouble forwarding it on my SMTP server because apparently I hadn't
allocated enough memory for it to forward such messages; after I
increased the amount of memory it was able to traverse the queue so I
could POP it.
I've also attempted to contact the producers of such emails, however,
again I haven't even received an acknowledgement of the bug.
Andy
Re: inc and non-compliant long lines redux
Thus said David Levine on Sun, 27 Nov 2022 19:14:09 -0800: > Would it be much trouble for you to add tests for the test suite? I can look into it when I get some spare time. I'm not yet very familiar with the test code, but I did take a brief look and noticed that there is a fake pop server implementation that can be utilized. Thanks, Andy
Re: inc and non-compliant long lines redux
Thus said Ken Hornstein on Mon, 21 Nov 2022 21:41:34 -0500:
> netsec_read() is supposed to RETURN the number of bytes read, just
> like read() does (that's why it's return value is prototyped ssize_t).
> But it just returns OK on success.
Attached is the previous patch remade against master. It also includes a
bug fix against the previous that I noticed while testing; a simple
off-by-one error when checking for \r at the end of a line. I again
tested it with my 6 sample messages that gave angst to inc earlier and
they work fine now (that's how I noticed the off-by-one error).
I'm running a similar patch on my nmh 1.7.1 installation and will report
any problems that I find in "production".
Suggestions?
Andy
diff --git a/h/netsec.h b/h/netsec.h
index 6c69c82c..42370fa7 100644
--- a/h/netsec.h
+++ b/h/netsec.h
@@ -166,8 +166,8 @@ char *netsec_readline(netsec_context *ns_context, size_t *length,
* Returns number of bytes read, or -1 on error.
*/
-ssize_t netsec_read(netsec_context *ns_context, void *buffer, size_t size,
- char **errstr);
+ssize_t netsec_read(netsec_context *ns_context, unsigned char *buffer,
+ size_t size, char **errstr);
/*
* Write data to the network; if encryption is being performed, we will
diff --git a/sbr/netsec.c b/sbr/netsec.c
index bf0ff9cf..c5f478e2 100644
--- a/sbr/netsec.c
+++ b/sbr/netsec.c
@@ -382,9 +382,10 @@ netsec_set_timeout(netsec_context *nsc, int timeout)
*/
ssize_t
-netsec_read(netsec_context *nsc, void *buffer, size_t size, char **errstr)
+netsec_read(netsec_context *nsc, unsigned char *buffer, size_t size,
+ char **errstr)
{
-int retlen;
+ssize_t retlen;
/*
* If our buffer is empty, then we should fill it now
@@ -415,7 +416,7 @@ netsec_read(netsec_context *nsc, void *buffer, size_t size, char **errstr)
nsc->ns_inbuflen -= size;
}
-return OK;
+return retlen;
}
/*
diff --git a/uip/inc.c b/uip/inc.c
index dd0a646d..ddbc13cb 100644
--- a/uip/inc.c
+++ b/uip/inc.c
@@ -900,7 +900,7 @@ pop_action(void *closure, char *s)
int n;
pc = closure;
-n = fprintf(pc->mailout, "%s\n", s);
+n = fprintf(pc->mailout, "%s", s);
if (n < 0)
return NOTOK;
pc->written += n; /* Count linefeed too. */
diff --git a/uip/popsbr.c b/uip/popsbr.c
index 8a064ff5..d96dde22 100644
--- a/uip/popsbr.c
+++ b/uip/popsbr.c
@@ -565,8 +565,13 @@ static int
traverse (int (*action)(void *, char *), void *closure, const char *fmt, ...)
{
int result, snoopstate;
+size_t len, unused, inoffset, slen, count;
va_list ap;
char buffer[sizeof(response)];
+size_t buflen;
+char *errstr, *ptr, *sptr, *tptr;
+bool checkline = true;
+bool needdata = true;
va_start(ap, fmt);
result = vcommand (fmt, ap);
@@ -579,21 +584,94 @@ traverse (int (*action)(void *, char *), void *closure, const char *fmt, ...)
if ((snoopstate = netsec_get_snoop(nsc)))
netsec_set_snoop(nsc, 0);
+unused = sizeof(buffer);
+inoffset = 0;
+buflen = 0;
for (;;) {
-result = multiline();
-if (result == OK) {
-result = (*action)(closure, response);
-if (result == OK)
-continue;
-} else if (result == DONE) {
-strncpy(response, buffer, sizeof(response));
-result = OK;
-}
-break;
+ if (needdata) {
+ len = netsec_read(nsc, buffer + inoffset, unused, &errstr);
+ if (len < 0) {
+ strncpy(response, errstr, sizeof(response));
+ response[sizeof(response) - 1] = '\0';
+ free(errstr);
+ netsec_set_snoop(nsc, snoopstate);
+ return NOTOK;
+ }
+ tptr = buffer;
+ buflen += len;
+ needdata = false;
+ }
+ if (checkline) {
+ if (buflen >= LEN(TRM) + 2 &&
+ strncmp(TRM "\r\n", tptr, LEN(TRM) + 2) == 0)
+ return OK;
+ if (buflen >= LEN(TRM TRM) &&
+ strncmp(TRM TRM, tptr, LEN(TRM TRM)) == 0) {
+ tptr += LEN(TRM);
+ buflen -= LEN(TRM);
+ }
+ }
+ ptr = tptr;
+ checkline = false;
+ count = 0;
+ while (count < buflen) {
+ count++;
+ if (*ptr++ == '\n') {
+ sptr = tptr;
+ if (count > 1 && *(ptr - 2) == '\r') {
+ *--ptr = '\0';
+ checkline = true;
+ }
+ *--ptr = '\n'; ptr++;
+ slen = ptr - tptr;
+ strncpy(response, sptr, slen);
+ response[slen] = '\0';
+ buflen -= count;
+ tptr += count;
+ (*action) (closure, response);
+ break;
+ }
+ }
+ if (checkline) {
+ if (buflen > 0) {
+ if (buflen < LEN(TRM) + 2 || buflen < LEN(TRM TRM)) {
+ needdata = true;
+ inoffset = buflen;
+ unused = sizeof(buffer) - buflen;
+ memmove(buffer, tptr, buflen);
+ }
+ } else {
+ needdata = true;
+ inoffset = 0;
+ unused = sizeof(buffer);
+ }
+ continue;
+ }
+ if (count == buflen) {
+ sptr = tptr;
+ if (buflen > 0 && tptr[buflen - 1] == '\r')
+ ptr--;
+ slen = ptr - tptr;
+ strncpy(response, sptr, slen);
+ response[slen] = '\0';
+ buflen -= slen;
+ (*action) (cl
Re: inc and non-compliant long lines redux
Thus said Ken Hornstein on Mon, 21 Nov 2022 21:41:34 -0500: > netsec_read() is supposed to RETURN the number of bytes read, just > like read() does (that's why it's return value is prototyped ssize_t). Aha, that does make more sense. I thought it was odd that it just returned OK, but didn't think to actually check the documentation in netsec.h (except to add another variable to the declaration); otherwise I would have noticed. Let me see if I can rework it to just return the number of bytes rather than taking another parameter (and the "buffer" type change). Thanks, Andy
Re: inc and non-compliant long lines redux
Thus said Ken Hornstein on Wed, 16 Nov 2022 09:19:11 -0500:
> I think the correct solution is when nmh is doing a RETR inside of
> POP we should use netsec_read() and perform the necessary CR-LF
> translations on the returned buffer. The only complicated thing I see
> is dealing with the case where a CR is at the end of one buffer and
> the LF is at the start of the next buffer; I think that is
> straightforward.
I have a working prototype that I'm testing that uses netsec_read(). I
had to expand the function declaration for netsec_read() because it
didn't return the number of bytes that the caller received. Not sure if
that was intentional and I'm just misunderstanding netsec_read().
I pretty much just stuffed all the logic into traverse() for now just to
simplify the flow, though, I imagine it's still more complicated than I
should like. I tested it with 5 emails that I have received that were
known to cause problems, and have been receiving other "normal" emails
since I put it into production.
The patch is currently against 1.7.1 since that is what I run on my
system, so I'll rework it for master when I get more free time. I've
attached the patch so others can review for correctness, though, as I
said, it's for 1.7.1 so it won't apply cleanly to master yet.
Thanks,
Andy
Index: h/netsec.h
==
--- h/netsec.h
+++ h/netsec.h
@@ -164,12 +164,12 @@
* errstr - Error size
*
* Returns number of bytes read, or -1 on error.
*/
-ssize_t netsec_read(netsec_context *ns_context, void *buffer, size_t size,
- char **errstr);
+ssize_t netsec_read(netsec_context *ns_context, char *buffer, size_t size,
+ size_t *retlen, char **errstr);
/*
* Write data to the network; if encryption is being performed, we will
* do it. Data may be buffered; use netsec_flush() to flush any outstanding
* data to the network.
Index: sbr/netsec.c
==
--- sbr/netsec.c
+++ sbr/netsec.c
@@ -372,14 +372,13 @@
* Read data from the network. Basically, return anything in our buffer,
* otherwise fill from the network.
*/
ssize_t
-netsec_read(netsec_context *nsc, void *buffer, size_t size, char **errstr)
+netsec_read(netsec_context *nsc, char *buffer, size_t size,
+ size_t *retlen, char **errstr)
{
-int retlen;
-
/*
* If our buffer is empty, then we should fill it now
*/
if (nsc->ns_inbuflen == 0) {
@@ -390,15 +389,15 @@
/*
* netsec_fillread only returns if the buffer is full, so we can
* assume here that this has something in it.
*/
-retlen = min(size, nsc->ns_inbuflen);
+*retlen = min(size, nsc->ns_inbuflen);
-memcpy(buffer, nsc->ns_inptr, retlen);
+memcpy(buffer, nsc->ns_inptr, *retlen);
-if (retlen == (int) nsc->ns_inbuflen) {
+if (*retlen == nsc->ns_inbuflen) {
/*
* We've emptied our buffer, so reset everything.
*/
nsc->ns_inptr = nsc->ns_inbuffer;
nsc->ns_inbuflen = 0;
Index: uip/inc.c
==
--- uip/inc.c
+++ uip/inc.c
@@ -934,9 +934,9 @@
}
static int
pop_action (char *s)
{
-fprintf (pf, "%s\n", s);
-stop += strlen (s) + 1;
+fprintf (pf, "%s", s);
+stop += strlen (s);
return 0; /* Is return value used? This was missing before 1999-07-15. */
}
Index: uip/popsbr.c
==
--- uip/popsbr.c
+++ uip/popsbr.c
@@ -489,12 +489,17 @@
static int
traverse (int (*action)(char *), const char *fmt, ...)
{
int result, snoopstate;
+size_t len, unused, inoffset, slen, count;
va_list ap;
char buffer[sizeof(response)];
+size_t buflen;
+char *errstr, *ptr, *sptr, *tptr;
+boolean checkline = true;
+boolean needdata = true;
va_start(ap, fmt);
result = vcommand (fmt, ap);
va_end(ap);
@@ -503,25 +508,94 @@
strncpy (buffer, response, sizeof(buffer));
if ((snoopstate = netsec_get_snoop(nsc)))
netsec_set_snoop(nsc, 0);
-for (;;)
- switch (multiline ()) {
- case NOTOK:
+unused = sizeof(buffer);
+inoffset = 0;
+buflen = 0;
+for (;;) {
+ if (needdata) {
+ result = netsec_read(nsc, buffer + inoffset, unused, &len, &errstr);
+ if (result != OK) {
+ strncpy(response, errstr, sizeof(response));
+ response[sizeof(response) - 1] = '\0';
+ free(errstr);
netsec_set_snoop(nsc, snoopstate);
return NOTOK;
-
- case DONE:
- strncpy (response, buffer, sizeof(response));
- netsec_set_snoop(nsc, snoopstate);
+ }
+ tptr = buffer;
+ buflen += len;
+ needdata = false;
+ }
+ if (checkline) {
+ if (buflen >= LEN(TRM) + 2 &&
+ strncmp(TRM "\r\n", tptr, LEN(TRM) + 2) == 0)
return OK;
-
- case OK:
+ if (buflen >= LEN(TRM TRM) &&
+ strncmp(TRM TRM, tptr, LEN(TRM TRM)) == 0) {
+ t
Re: inc and non-compliant long lines redux
Thus said Ken Hornstein on Wed, 16 Nov 2022 09:19:11 -0500: > The only complicated thing I see is dealing with the case where a CR > is at the end of one buffer and the LF is at the start of the next > buffer; I think that is straightforward. Precisely. Though in my current patch proposal, the problem isn't with CR/LF but with the two .. signal terminator coming from POP. The two dots must be getting split and the first dot is at the end of a buffer and the second is at the beginning of the next buffer. > Unfortunately my life is still difficult right now, otherwise I'd work > on fixing that. I may take a stab at it if I get some time. I'll submit patches and solicit feedback if I make any more progress. Thanks for looking. Andy
Re: inc and non-compliant long lines redux
Thus said Ken Hornstein on Tue, 15 Nov 2022 15:44:59 -0500: > My reading of the RFC is that the 512 limit is _just_ for responses to > commands. For the response to something like RETR it punts it off to > RFC 822, which it seems like people are ignoring anyway. This is also how I interpreted it. I left the command processing alone. The changes I made only kick in after the RETR command has been successfully started. Also, I recognize Ralph's statement that this is not so much "people ignoring" the RFCs, but really just one "sender". In my 23+ years of using nmh for email this is the only "sender" I have ever encountered that generates such messages. So it does seem a bit drastic to take on the added risk of "fixing" nmh. :-) > I used netsec_readline() because it handles the case of stripping line > endings so we could easily convert CR-LF to Unix style LF-end messages. > But really we could simply use the netsec_read() and just convert every > CR-LF pair to a simple LF. I originally looked at netsec_read() as an alternative, but thought it would be easier (maybe not simpler) to use netsec_readline() since it already had logic for dealing with CR/LF. And the more I looked at using netsec_read(), the more it seemed like I would be reinventing netsec_readline(). Perhaps this direction was misguided and I should have stuck with netsec_read(). Instead, what pop_retr() (and more specifically traverse()) could do is after successfully starting the RETR command, it could just switch to using netsec_read() and handling any CR/LF encountered until it sees the CRLF.CRLF indicator. There wouldn't even be any need for inc to distinguish between "header" and "body" in this mode, just read() data until we get CRLF.CRLF, then return so DELE and QUIT can be sent. For now, I'm dealing with another synchronization problem with my original patch (as reported in a different email). Basically the CRLF.CRLF is getting flagged early in these lines: ng> " and render this text in the browser= . Optionally the output file can be stored and rendered in the reposit= The POP server dutifully sends the lines as: ng> " and render this text in the browser= .. Optionally the output file can be stored and rendered in the reposit= However, the way the patch reads chunks of data must be causing the .. to be split into two parts, and so it sees the first one and thinks it has arrived at the end of the RETR command. The message is written to disk excluding the dot and everything that follows. Anyway, I've reverted my changes locally for now until I can figure out how to make sure that the .. are not split while reading chunks. I imagine that this same challenge will be had even when using netsec_read() instead. Andy
Re: inc and non-compliant long lines redux
Thus said Ralph Corderoy on Tue, 15 Nov 2022 17:27:47 +: > It's one peer of one user. No one else suffers. (Sorry, Andy.) > Seems a shame to complicate the code. Well, actually, it's a system that is used to notify more than just me, and I cannot say that I'm the only one impacted, however, I'm pretty sure I'm the only one who knows there is a problem at this point. I agree, however, that modifying nmh code, and more specifically the changes that I made in netsec_readline() are a shame. I'm not proud of the changes by any measure. > Andy, if it were the last, what kind of size would be needed, i.e. how > long is the longest line from this brain-damaged peer, including CR > LF? The longest line that I have observed was 11,370,773 bytes, however, it could be any variable length based upon the size of the attachment being included in the email. > - Upping the truncation limit to avoid truncating Andy's case. What specifically is meant by this? Do you mean allocate more memory? There currently isn't really a "truncation limit" except for the bug that I identified where lines longer than 1024 characters get truncated. The real problem is with messages that have huge lines, inc fails with an error: inc: Unable to find a line terminator after 32768 bytes When this happens, it leaves all the emails previously POP'ed in the session on the server, so the next time I run inc, I get duplicates. Personally, I would be fine if inc decided to simply insert a CR/LF every 998 bytes when it encounters long lines, and in fact, this is what I originally accomplished, but then decided it was only a little more effort to make inc simply continue reading more data until it finally got the CR/LF. I'm still trying to contact the organization responsible for sending the messages, but so far haven't reached anyone with enough technical knowledge to understand the problem. Maybe I'll get lucky and find that one shining star programmer that can fix their broken form to email notification tool. Andy
Re: inc and non-compliant long lines redux
Thus said "Andy Bradford" on 09 Nov 2022 21:18:55 -0700: > I've been running it for a couple of days and it seems to work as I > haven't discovered any corrupted message or lost anything; it even > preserves the huge lines that shouldn't be present. Well, I've run into my first bug with this particular patch. I'm trying to track it down. Somehow these lines are causing truncation of the rest of the message and leaving the POP connection in a potentially unknown state: ng> " and render this text in the browser= . Optionally the output file can be stored and rendered in the reposit= I suspect that for some reason the buffer is aligning just well enough that for some reason the has_prefix() being called in multiline() is flagging it as a the terminator and cutting off the communication early even though there is more data to be read. Things don't end well. I haven't been receiving emails properly since that message arrived in my POP mail store, so I may not be getting everything. Guess I'm eating my own dog food for now. I'll watch the mailing list archives for responses while I dig into the code some more to see why it might be bailing out early. Andy
Re: inc and non-compliant long lines redux
Thus said Ken Hornstein on Mon, 14 Nov 2022 21:27:23 -0500: > Officially such a message violates RFC 5322 (line length limit of 998 > plus CR LF), which means it violates the POP3 RFC as well. But I > suspect we just have to deal. Sigh. Absolutely, it's one of the most blatant violations I've seen. I'm trying to contact the sender/generator of such messages, but these days, as you rightly pointed out three years ago, it's hard to find someone willing to listen. > netsec_readline() can handle a maximum line length of 32k, which I > chose based on it being way larger than what email protocols say is > the limit of a "line". My only question is: should there be a > hardcoded limit in terms of line length, or should we have the limit > be whatever you can malloc()? Yes, as I looked at the code, I realized that netsec_readline() was plenty generous in allowing the newline to appear anywhere in the first 32k bytes. In practice, this never really happens, but in the event that it exceeded the 1024, it actually would truncate the line; again a very rare event in my opinion. The patch as I proposed it doesn't require more memory to be allocated as it simply scans until it fines a CR/LF, and as far as I know there is no limit. I have not yet tested what happens when there is no CR/LF, however I'm not sure such a message could ever exist in an SMTP/POP environment. Andy
Re: inc and non-compliant long lines redux
Thus said Ralph Corderoy on Sun, 13 Nov 2022 21:29:23 +: > Is this only seen with inc(1) reading from a particular source? I believe the only problem that I found was inc and reading from a POP source; sorry for not being more clear. Specifically in popsbr.c and pop_getline(), it would call netsec_readline() which uses a buffer of 65535 for holding data, and then pop_getline() would stuff it into a buffer that was BUFSIZ (1024 on my system), thus truncating whatever was returned to 1024. For most cases, this actually worked out fine because netsec_readline() would always find a newline before 1024 bytes and return a pointer with a length of < 1024. But in the case of really long lines (probably pretty rare in SMTP and POP), it would truncate them. Andy
Misplaced header include in some files for sbr/globals.h?
Hello, While compiling on OpenBSD I encountered some failures due the placement of this line: #include "sbr/globals.h" For some reason it's nested in an #ifdef block, and I think it should actually be outside of the block. For example: https://git.savannah.nongnu.org/cgit/nmh.git/commit/uip/rcvtty.c?id=472d87ae1864d7486a8d2d241eb2d82481a27c16 Why shouldn't the include be above with all the rest of the includes? As it is, I compilation fails due to "undeclared identifier" errors such as: uip/rcvtty.c:96:31: error: use of undeclared identifier 'invo_name' arguments = getarguments (invo_name, argc, argv, 1); Is there a reason why the include is nested in the #ifdef? Thanks, Andy
Re: inc and non-compliant long lines redux
Thus said David Levine on Thu, 10 Nov 2022 05:51:12 -0800:
> Andy, are you able to create patches against HEAD?
Attached is a new patch that applies cleanly against master (17734c9c).
I had trouble compiling on OpenBSD, but I eventually got it working
after a bit of hacking, however, I did not include those bits in my
patch since they are not specific to the long line problem (I'll bring
up in a different thread).
The new patch compiles and I tested inc using 3 malformed messages and
it worked fine. I did not test imaptest.c nor the changes for smtp.c,
but the changes there were merely to preserve the original behavior of
the functions modified (function signature changed).
Thanks,
Andy
diff --git a/h/netsec.h b/h/netsec.h
index 6c69c82c..73e9585b 100644
--- a/h/netsec.h
+++ b/h/netsec.h
@@ -139,19 +139,20 @@ void netsec_set_timeout(netsec_context *ns_context, int timeout);
* Read a "line" from the network. This reads one CR/LF terminated line.
* Returns a pointer to a NUL-terminated string. This memory is valid
* until the next call to any read function. Will return an error if
- * the line does not terminate with a CR/LF.
+ * the line does not terminate with a CR/LF unless strict is false.
*
* Arguments:
*
* ns_context - Network security context
* length - Returned length of string
* errstr - Error string
+ * strict - Strict CR/LF checking
*
* Returns pointer to string, or NULL on error.
*/
char *netsec_readline(netsec_context *ns_context, size_t *length,
- char **errstr);
+ char **errstr, bool strict);
/*
* Read bytes from the network.
diff --git a/mts/smtp/smtp.c b/mts/smtp/smtp.c
index bf338b12..20722bfd 100644
--- a/mts/smtp/smtp.c
+++ b/mts/smtp/smtp.c
@@ -817,7 +817,7 @@ again: ;
rc = sizeof(sm_reply.text) - 1;
for (more = false; (buffer = netsec_readline(nsc, &buflen,
- &errstr)) != NULL ; ) {
+ &errstr, true)) != NULL ; ) {
if (doingEHLO
&& has_prefix(buffer, "250")
@@ -1035,7 +1035,7 @@ sm_sasl_callback(enum sasl_message_type mtype, unsigned const char *indata,
netsec_set_snoop_callback(nsc, netsec_b64_snoop_decoder, &snoopoffset);
snoopoffset = 4;
- line = netsec_readline(nsc, &len, errstr);
+ line = netsec_readline(nsc, &len, errstr, true);
netsec_set_snoop_callback(nsc, NULL, NULL);
if (line == NULL)
@@ -1093,7 +1093,7 @@ sm_sasl_callback(enum sasl_message_type mtype, unsigned const char *indata,
/*
* Finish the protocol; we're looking for a 235 message.
*/
- line = netsec_readline(nsc, &len, errstr);
+ line = netsec_readline(nsc, &len, errstr, true);
if (line == NULL)
return NOTOK;
diff --git a/sbr/netsec.c b/sbr/netsec.c
index bf0ff9cf..12cae575 100644
--- a/sbr/netsec.c
+++ b/sbr/netsec.c
@@ -50,6 +50,8 @@ static SSL_CTX *sslctx = NULL; /* SSL Context */
/* I'm going to hardcode this for now; maybe make it adjustable later? */
#define NETSEC_BUFSIZE 65536
+bool addnl;
+
/*
* Our context structure, which holds all of the relevant information
* about a connection.
@@ -434,7 +436,7 @@ netsec_read(netsec_context *nsc, void *buffer, size_t size, char **errstr)
*/
char *
-netsec_readline(netsec_context *nsc, size_t *len, char **errstr)
+netsec_readline(netsec_context *nsc, size_t *len, char **errstr, bool strict)
{
unsigned char *ptr = nsc->ns_inptr;
size_t count = 0, offset;
@@ -443,10 +445,12 @@ retry:
/*
* Search through our existing buffer for a LF
*/
+addnl = false;
-while (count < nsc->ns_inbuflen) {
+while (count < nsc->ns_inbuflen && count < BUFSIZ - 2) {
count++;
if (*ptr++ == '\n') {
+ addnl = true;
char *sptr = (char *) nsc->ns_inptr;
if (count > 1 && *(ptr - 2) == '\r')
ptr--;
@@ -478,14 +482,30 @@ retry:
}
/*
- * Hm, we didn't find a \n. If we've already searched half of the input
+ * Hm, we didn't find a \n. Return half of a BUFSIZ buffer it not strictly
+ * searching for \n.
+ * Otherwise, if we've already searched half of the input
* buffer, return an error.
*/
-if (count >= nsc->ns_inbufsize / 2) {
- netsec_err(errstr, "Unable to find a line terminator after %zu bytes",
- count);
- return NULL;
+if (strict == true) {
+ if (count >= nsc->ns_inbufsize / 2) {
+ netsec_err(errstr,
+ "Unable to find a line terminator after %zu bytes",
+ count);
+ return NULL;
+ }
+} else {
+ if (count) {
+ count = min(BUFSIZ / 2, count);
+ ptr = nsc->ns_inptr + count;
+ char *sptr = (char *) nsc->ns_inptr;
+ if (len)
+ *len = ptr - nsc->ns_inptr;
+ nsc->ns_inptr += count;
+ nsc->ns_inbuflen -= count;
+ return sptr;
+ }
}
/*
diff --git a/uip/imaptest.c b/uip/imaptest.c
index 9e19aac6..938c2506 100644
--- a/uip/imaptest.c
+++ b/uip/imaptest.c
@@ -291,7 +291,7 @@ main (int argc, char **argv)
die("%s", errstr);
}
-if ((cp = netsec_readline(nsc
Re: inc and non-compliant long lines redux
Thus said Michael Richardson on Thu, 10 Nov 2022 13:08:02 +: > Not sure where TRUE/FALSE is supposed to be declared, but that didn't > compile. Not sure where they went... they used to be defined in mh.h: #ifndef FALSE #define FALSE false #endif #ifndef TRUE #define TRUE true #endif Thanks for pointing out that they're missing. I guess a lot more has changed between 1.7.1 and master than I had thought. I'm just about done with reworking the patch for master. Thanks, Andy
Re: inc and non-compliant long lines redux
Thus said David Levine on Thu, 10 Nov 2022 05:51:12 -0800: > Andy, are you able to create patches against HEAD? Yes, I'll see about getting it applied to HEAD. Thanks, Andy
inc and non-compliant long lines redux
Hello again,
It's been a few years since I first reported a problem with inc's
handling of long lines (e.g. those that are clearly longer than RFC
mandated lengths). Oddly enough, I have only ever seen it coming from
one particular sender (and the problem still persists to this day).
Every other email sender from whom I receive email has never presented a
problem.
For historical context:
https://lists.nongnu.org/archive/html/nmh-workers/2019-09/msg7.html
At any rate, I've been mulling over the source for the past few days and
decided to take a stab at addressing the issue. Along the way I found 2
other bugs. Not only does inc fail with long lines that are greater than
32768 bytes, it also truncates lines that are less than that value and
longer than 1024 bytes (not sure if this behavior is intentional, but it
seems like an oversight to me). The other minor bug is that it appends
an extra newline to the body after scanning the terminating dot (.) at
the end of the transaction, which I decided was fairly innocuous so I
didn't spend time on it.
Ken made the suggestion that inc should just be able to handle longer
lines. Turns out that was quite a challenge not being familiar with the
code and it's possible that my modifications are more complicated than
necessary, but attached is an attempt to make it work. As it turns out,
inc's ability to POP email had no concept of "header" and "body", but
merely "command" and "lines of content", so I sought to make it grok
when the header ends and the body starts. After it finds the end of
headers (a line with 0 bytes of data because CRNL are stripped from the
line) it then stops requiring CRNL and reads in chunks of data instead,
while still looking for CRNL for conversion. (I thought it should be
strict with CRNL parsing during headers, but perhaps that's a flaw in
this particular change I've made). When I got that working I then found
that most of the message was still missing, and that's when I discovered
that inc discarded any characters after 1024 bytes on a line.
I've been running it for a couple of days and it seems to work as I
haven't discovered any corrupted message or lost anything; it even
preserves the huge lines that shouldn't be present.
Can this be done better another way? Is it sufficient?
The patch is against 1.7.1 since that is what is installed on my system.
Thanks,
Andy
Index: h/netsec.h
==
--- h/netsec.h
+++ h/netsec.h
@@ -149,11 +149,11 @@
*
* Returns pointer to string, or NULL on error.
*/
char *netsec_readline(netsec_context *ns_context, size_t *length,
- char **errstr);
+ char **errstr, int strict);
/*
* Read bytes from the network.
*
* Arguments:
Index: mts/smtp/smtp.c
==
--- mts/smtp/smtp.c
+++ mts/smtp/smtp.c
@@ -783,11 +783,11 @@
sm_reply.text[0] = 0;
rp = sm_reply.text;
rc = sizeof(sm_reply.text) - 1;
for (more = FALSE; (buffer = netsec_readline(nsc, &buflen,
- &errstr)) != NULL ; ) {
+ &errstr, TRUE)) != NULL ; ) {
if (doingEHLO
&& has_prefix(buffer, "250")
&& (buffer[3] == '-' || doingEHLO == 2)
&& buffer[4]) {
@@ -993,11 +993,11 @@
* by base64 response data.
*/
netsec_set_snoop_callback(nsc, netsec_b64_snoop_decoder, &snoopoffset);
snoopoffset = 4;
- line = netsec_readline(nsc, &len, errstr);
+ line = netsec_readline(nsc, &len, errstr, TRUE);
netsec_set_snoop_callback(nsc, NULL, NULL);
if (line == NULL)
return NOTOK;
@@ -1049,11 +1049,11 @@
case NETSEC_SASL_FINISH:
/*
* Finish the protocol; we're looking for a 235 message.
*/
- line = netsec_readline(nsc, &len, errstr);
+ line = netsec_readline(nsc, &len, errstr, TRUE);
if (line == NULL)
return NOTOK;
if (!has_prefix(line, "235 ")) {
if (len > 4)
Index: sbr/netsec.c
==
--- sbr/netsec.c
+++ sbr/netsec.c
@@ -44,10 +44,12 @@
#endif /* TLS_SUPPORT */
/* I'm going to hardcode this for now; maybe make it adjustable later? */
#define NETSEC_BUFSIZE 65536
+int addnl;
+
/*
* Our context structure, which holds all of the relevant information
* about a connection.
*/
@@ -424,23 +426,25 @@
* - If your data may contain embedded NULs, this won't work. You should
* be using netsec_read() in that case.
*/
char *
-netsec_readline(netsec_context *nsc, size_t *len, char **errstr)
+netsec_readline(netsec_context *nsc, size_t *len, char **errstr, int strict)
{
unsigned char *ptr = nsc->ns_inptr;
size_t count = 0, offset;
retry:
/*
* Search through our existing buffer for a LF
*/
+addnl = FALSE;
-while (count < nsc->ns_inbuflen) {
+while (count < nsc->ns_inbuflen && count < BUFSIZ - 2) {
count++;
Re: new release?
Thus said David Levine on Tue, 19 Apr 2022 19:18:49 -0700: > In the meantime, if those who build from the repo could pull and > report the results of "make check", that will help reveal areas that > we need to address. OpenBSD 6.9 amd64: $ make check cc -DHAVE_CONFIG_H -I.-DNMHBINDIR='"/usr/local/bin"' -DNMHLIBEXECDIR='"/usr/local/libexec/nmh"' -DNMHETCDIR='"/usr/local/etc/nmh"' -DNMHDOCDIR='"/usr/local/share/doc/nmh"' -DMAILSPOOL='"/var/mail"' -DSENDMAILPATH='"/usr/sbin/sendmail"' -g -O2 -Wall -Wextra -MT sbr/sbr_libmh_a-fmt_rfc2047.o -MD -MP -MF sbr/.deps/sbr_libmh_a-fmt_rfc2047.Tpo -c -o sbr/sbr_libmh_a-fmt_rfc2047.o `test -f 'sbr/fmt_rfc2047.c' || echo './'`sbr/fmt_rfc2047.c sbr/fmt_rfc2047.c:391:12: error: use of undeclared identifier 'initialdstlen' return initialdstlen - dstlen; ^ sbr/fmt_rfc2047.c:399:12: error: use of undeclared identifier 'initialdstlen' return initialdstlen - dstlen - 1; ^ sbr/fmt_rfc2047.c:72:66: warning: unused parameter 'dest_charset' [-Wunused-parameter] decode_rfc2047 (char *str, char *dst, size_t dstlen, const char *dest_charset) ^ 1 warning and 2 errors generated. *** Error 1 in /tmp/nmh (Makefile:3435 'sbr/sbr_libmh_a-fmt_rfc2047.o') Andy
Re: crufty mhn.default.sh stuff
Thus said Paul Fox on Sat, 18 Dec 2021 16:20:05 -0500: > And others that Doug didn't introduce, but which are somewhat long in > the tooth: > xv > soffice I still prefer to use xv for image display. Also, while Star Office is long gone as a name, soffice still works as a compatible name for Libre Office (and in fact, seems to be the official name with a symbolic link for libreoffice, at least on OpenBSD): $ which xv /usr/local/bin/xv $ which soffice /usr/local/bin/soffice $ ls -l /usr/local/bin/libreoffice /usr/local/bin/soffice lrwxr-xr-x 1 root wheel7 Sep 4 10:51 /usr/local/bin/libreoffice@ -> soffice -rwxr-xr-x 1 root bin959 Apr 19 2021 /usr/local/bin/soffice* Just my $0.02 Andy
Re: mts.conf not being read?
Thus said David Levine on Tue, 17 Aug 2021 18:51:38 -0700: > Does your profile have an -mts sendmail/pipe switch for send? send > -help would show it. I do have this: postproc: /usr/local/libexec/nmh/spost But nothing for send. Also, I notice in analyzing the trace that the process chain looks something like: comp, vi, mhbuild (which reads mts.conf), spost, post, and finally sendmail. And apparently, spost is actually just a shell script that calls post with -mts sendmail/pipe. :-) It's possible that at one point I ran into trouble using mts.conf (maybe I didn't realize that SMTP had changed to SUBMISSION) and somehow figured out that I could use spost instead. Andy
Re: mts.conf not being read?
Thus said Bakul Shah on Tue, 17 Aug 2021 18:38:21 -0700: > Try ktrace -di comp Wow, how did I miss that in the man page! Now after adding -i I see that it does in fact read mts.conf. Now back to my original question. How is it possible that when mts.conf is configured as: mts: smtp servers: localhost That it didn't somehow use the "new default port of 587" and result in failures in my environment which has no daemon on port 587. Furthermore, it didn't even use port 25 because otherwise I would have evidence in my MTA logs showing that the messages were submitted via SMTP and not via pipe. Thanks, Andy
mts.conf not being read?
Hello, While participating in the recent thread, "Delivering mail through nmh via SMTP...", I decided to check my mts.conf and discovered that mts wasn't actually configured as "sendmail/pipe" as I had thought (probably got wiped away when I upgraded); so I started wondering why the change hadn't impacted me at all. In my mts.conf I had the following: mts: smtp servers: localhost Yet, as I said in the previous thread, I have nothing listening on port 587: $ telnet localhost 587 Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused Trying ::1... telnet: connect to address ::1: Connection refused Yet, whenever I send email, it does in fact get injected into my MTA's queue via a pipe! How is that possible and why isn't it attempting to connect to port 587? So I decided to use ktrace (similar to strace on Linux) to figure out if it was even reading mts.conf, and it does not even open the file. I ran: ktrace -d comp Composed a message and then at the "What now?" pormpt I typed send and let it rip and the email arrived as expected, but MTA logs indicate it was submitted via pipe sendmail interface. Given this, I'm curious how it could even be working since it isn't reading my mts.conf? Thanks, Andy
Re: Delivering mail through nmh via SMTP...
Thus said Ken Hornstein on Tue, 17 Aug 2021 21:01:18 -0400: > 587 is the port that is supposed to be used by MUAs to submit mail. > nmh is a MUA; it seems perfectly logical to me that the default that > is used by nmh is the submission port. Yeah, I can see your point though I still think it was an abrupt change [though in light of the dwindling number of nmh/exmh users perhaps it's perfectly fine for the community]. Fortunately I haven't been impacted by this (I would know because I don't run anything on port 587) so I'm really arguing more out of principle of least surprise than anything. I prefer to use "sendmail/pipe" for the mts setting which then allows me to easily use scripts to filter emails as they are submitted. Sure, that could also be done with an SMTP server, but it's much easier with a pipe. Andy
Re: Delivering mail through nmh via SMTP...
Thus said "Andy Bradford" on 17 Aug 2021 18:50:01 -0600: > Thus said David Levine on Tue, 17 Aug 2021 05:28:29 -0700: > > > Starting with nmh 1.7, nmh defaults to port 587 instead of 25 for > > SMTP submission. > > Personally I think this was a mistake. SMTP is port 25, SUBMISSION is > port 587. If something says SMTP, what comes to mind is port 25, not > 587. Let me clarify what I meant since I was looking at mts.conf when I read your comment... (obviously you did mention submission). Here is what is in mts.conf (much of which is mirrored in mh-tailor(5): # The delivery method to use, which must be one of the following: # smtp: nmh opens a socket connection to the appropriate port #on the servers listed below and speaks SMTP to the #first one that responds. This is the default. # sendmail/smtp: nmh pipes messages directly to the sendmail program, #speaking SMTP. Can be abbreviated to "sendmail". # sendmail/pipe: nmh pipes messages directly to the sendmail program, #using the -t option so that addresses are retrieved #from the message. I think a new value for mts should have been invented called "submission" that would be used in place of smtp. Also, given that is it not possible to change the smtp/submission port in mts.conf, this further bolsters the argument that it should probably not have changed to 587 by default and a new mts method invented. Andy
Re: Delivering mail through nmh via SMTP...
Thus said David Levine on Tue, 17 Aug 2021 05:28:29 -0700: > Starting with nmh 1.7, nmh defaults to port 587 instead of 25 for SMTP > submission. Personally I think this was a mistake. SMTP is port 25, SUBMISSION is port 587. If something says SMTP, what comes to mind is port 25, not 587. $ grep smtp /etc/services; grep submission /etc/services smtp25/tcp mail smtps 465/tcp # SSL-wrapped SMTP submission 587/tcp msa # mail message submission submission 587/udp msa # mail message submission Andy
Re: inc not sending password to POP3 server
Ken asked: > What version of nmh were you running before? Oddly enough, it would seem that I was also running nmh 1.7.1 prior to upgrading the OS. I see that I also had a local modification that maybe I forgot to communicate to anyone... Index: sbr/ruserpass.c --- sbr/ruserpass.c.orig +++ sbr/ruserpass.c @@ -170,7 +170,7 @@ ruserpass(const char *host, char **aname, char **apass mypass = *aname; } -*aname = mh_xstrdup(mypass); +*apass = mh_xstrdup(mypass); } } Andy
Re: inc not sending password to POP3 server
[Apologies that this will break threading, but I cannot yet receive email] Ken asked: > Is your password in a .netrc file? Something different? Do you have a > "credentials" entry in your .mh_profile? I don't have a .netrc file at all: $ ls -ld ~/.netrc ls: /home/amb/.netrc: No such file or directory Neither is there "credentials" in .mh_profile: $ grep -i credentials ~/.mh_profile; echo $? 1 inc prompts for the password (as can be seen in the inc output in the original email). I've started looking at the nmh sources to see if I can figure out what's going on. If someone spots it sooner than I, please send response to the mailing list. Andy
inc not sending password to POP3 server
Hello, I recently upgraded to OpenBSD 6.9 and consequently also got nmh 1.7.1 and now find that inc no longer sends the password to my POP3 server: $ inc +MyIncTmp -snoop -nochangecur -user username -host localhost Trying to connect to "localhost" ... Connecting to 127.0.0.1:110... <= +OK <83350.1625494797@localhost> => USER username <= +OK Password (localhost:username): => PASS (null) <= -ERR authorization failed => QUIT inc: -ERR authorization failed I also used ktrace to try to see what inc was doing and see that it reads the password: 65472 inc CALL read(5,0x75ee3000,0x1) 65472 inc GIO fd 5 read 9 bytes " " 65472 inc RET read 9 I also found this interesting behavior: 65472 inc CALL sendsyslog(0xcf7bdc67,43,0<>) 65472 inc GIO fd -1 wrote 43 bytes "<10>inc: vfprintf %s NULL in "PASS %s"" 65472 inc RET sendsyslog 0 I wonder why inc decided to send that particular information to syslog instead of stderr (where I would have actually seen it)? And of course, as snoop has already revealed, it sends (null) for the password: 65472 inc CALL write(4,0x4a3c2000,0xd) 65472 inc GIO fd 4 wrote 13 bytes "PASS (null)\r " 65472 inc RET write 13/0xd Any idea why doesn't send the real password? I will probably have to read any responses on the mailing list archives since I'm not yet able to receive email. Andy
inc not sending password to POP3 server
Hello, I recently upgraded to OpenBSD 6.9 and consequently also got nmh 1.7.1 and now find that inc no longer sends the password to my POP3 server: $ inc +MyIncTmp -snoop -nochangecur -user username -host localhost Trying to connect to "localhost" ... Connecting to 127.0.0.1:110... <= +OK <83350.1625494797@localhost> => USER username <= +OK Password (localhost:username): => PASS (null) <= -ERR authorization failed => QUIT inc: -ERR authorization failed I also used ktrace to try to see what inc was doing and see that it reads the password: 65472 inc CALL read(5,0x75ee3000,0x1) 65472 inc GIO fd 5 read 9 bytes " " 65472 inc RET read 9 I also found this interesting behavior: 65472 inc CALL sendsyslog(0xcf7bdc67,43,0<>) 65472 inc GIO fd -1 wrote 43 bytes "<10>inc: vfprintf %s NULL in "PASS %s"" 65472 inc RET sendsyslog 0 I wonder why inc decided to send that particular information to syslog instead of stderr (where I would have actually seen it)? And of course, as snoop has already revealed, it sends (null) for the password: 65472 inc CALL write(4,0x4a3c2000,0xd) 65472 inc GIO fd 4 wrote 13 bytes "PASS (null)\r " 65472 inc RET write 13/0xd Any idea why doesn't send the real password? I will probably have to read any responses on the mailing list archives since I'm not yet able to receive email. Andy
Re: Is nmh suitable for managing multiple email accounts?
Thus said Michael Richardson on Sun, 07 Mar 2021 14:04:18 -0500: > But that doesn't work anymore unless you control the DNS for all zones > in question, and can set up SPF correctly. (Mostly, that actually does > apply to me) As a counterpoint, I have never setup SPF and generally have no problems. I do, however, control the DNS. As a counter counterpoint, however, not many people run their own email anymore. :-) Andy
Re: Synchronization Problem
Thus said Ken Hornstein on Mon, 31 Aug 2020 13:25:17 -0400: > Right, I mean ... that's probably sufficient. But there aren't any > universal solutions. If you want to wrap every nmh command in a global > lock, that's also an option. The only command I run the risk of running concurrently is inc, so this is what I have: $ cat /usr/local/bin/inc #!/bin/sh REALINC=`dirname "$0"`/inc.real INCLOCK=`mhpath +`/.inc.lock exec setlock "$INCLOCK" "$REALINC" "$@" Andy -- I believe in only one thing: liberty; but I do not believe in liberty enough to want to force it upon anyone. --- H. L. Mencken TAI64 timestamp: 40005f4d5609
Re: Forwarding html email
Thus said aalin...@riseup.net on Fri, 10 Jan 2020 16:15:37 -0500: > This gets me what I want and if that's the way it's done, so be it. My > only complaint is that I don't see the message I'm forwarding. You can always re-edit after you have run the mime command with "edit" if I'm not mistaken. Thanks, Andy -- TAI64 timestamp: 40005e1a762a
Re: Extracting Links from HTML Message
Thus said "Martin McCormick" on Mon, 06 Jan 2020 07:03:14 -0600: > https://www.radioid.net/verify/token=SHTUI469E1GVO35SDTGK180BAHKLOP9FPF > R9GWQD This URL looks like it has an embedded newline in it (maybe the result of quoted-printable content being poorly handled by the senders text/plain conversion. What if you just try putting it all in one URL: https://www.radioid.net/verify/token=SHTUI469E1GVO35SDTGK180BAHKLOP9FPFR9GWQD Andy -- TAI64 timestamp: 40005e139e47
Re: format and output all received: lines in an e-mail message
Thus said Ralph Corderoy on Sat, 23 Nov 2019 12:03:42 +: > Andy is correct. Mailman has a `nodupes' flag for every subscription > and it's set for Andy's subscription. Not any longer. :-) I didn't realize I had control over this. Andy -- TAI64 timestamp: 40005dd9a302
Re: format and output all received: lines in an e-mail message
Thus said Steffen Nurpmeso on Thu, 21 Nov 2019 23:14:29 +0100: > I have heared someone revived qmail and wants to include some patches > for builtin TLS etc. That sounded very much interesting, especially if > its mailing-list manager would be maintained again! There has been some momentum around creating a newer version that has a forward path. I imagine there are still some diehards out there like me who just continue to use the original software (mine is minimally patched for example) because it "just works." I still use unpatched ezmlm for my own personal uses for the same reasons, though I have added an extra binary; but the rest of the additional things that were added to it I haven't needed. Curiously, the email you sent me appears to have never arrived via the MLM that nmh-workers uses---I suspect it has one of those fancy features that thinks an email that was sent to one address shouldn't be also sent to another address (i.e. if the address of the To/Cc recipient is also a member of the MLM exclude one of them). Personally, I prefer to distinguish between direct replies and replies via MLM so I don't mind the extra because my filters funnel things appropriately. Andy -- TAI64 timestamp: 40005dd80178
Re: format and output all received: lines in an e-mail message
Thus said Greg Minshall on Wed, 20 Nov 2019 10:41:34 +0530: > then, i'd like to use something like fmttest(1) to print out all the > "Received:" lines in an e-mail message. ideally, each "Received:" line > would come out on a separate line; less ideally, but i'm sure very > practical, a very long line would come out, with some odd ascii code > separating the individual lines. I usually use 822field (from mess822 [1]) for this kind of thing which takes all received lines and reformats them one per line (odd ascii code separating them is a newline). For example, your message looks like: $ 822field received < `mhpath cur` | tail -6 from eggs.gnu.org ([2001:470:142:3::10]:33280) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iXIOr-0005CP-UD for nmh-workers@nongnu.org; Wed, 20 Nov 2019 00:19:34 -0500 from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iXIOq-00089z-NS for nmh-workers@nongnu.org; Wed, 20 Nov 2019 00:19:33 -0500 from hiwela.pair.com ([209.68.5.201]:21038) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iXIOq-00088d-K6 for nmh-workers@nongnu.org; Wed, 20 Nov 2019 00:19:32 -0500 from hiwela.pair.com (localhost [127.0.0.1]) by hiwela.pair.com (Postfix) with ESMTP id 001419805E3 for ; Wed, 20 Nov 2019 00:11:43 -0500 (EST) from minshall-entroware-apollo.cliq.com (unknown [59.95.74.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by hiwela.pair.com (Postfix) with ESMTPSA id 89AEF8F084C for ; Wed, 20 Nov 2019 00:11:43 -0500 (EST) from apollo2.minshall.org (localhost [IPv6:::1]) by minshall-entroware-apollo.cliq.com (Postfix) with ESMTP id E305D6089A for ; Wed, 20 Nov 2019 10:41:34 +0530 (IST) Andy [1] https://cr.yp.to/mess822.html -- TAI64 timestamp: 40005dd69ee2
Re: [nmh-workers] FSF is changing Mailman list settings unless you opt out (fwd)
Thus said Ken Hornstein on Thu, 26 Sep 2019 12:36:41 -0400: > You can read the details in the message for the complete technical > reasons why this is happening; the other option is to do what is > called "Munge From" and I personally think this is 100x worse (I am on > a Yahoo mailing list where this is done and I hate it). I won't mind if the Subject stops being mangled (I use the list-post header for filtering). I also won't mind if the body stops being mangled with additional MIME parts; the same information is already available in the list-(un)?subscribe headers) and I'm of the opinion that list control mechanisms belong in headers. Besides, how often does someone accidentally get subscribed to a mailing list anymore? I don't think however, that "Munge From" sounds like a good idea. Thanks, Andy -- TAI64 timestamp: 40005d8d7d75 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] inc: Unable to find a line terminator after 32768 bytes
Thus said Ken Hornstein on Tue, 10 Sep 2019 09:15:00 -0400: > So how big WAS this message, actually? I'm trying to understand the > scope of the problem. The entire size of the message on disk (including additional trace headers added by my MTA) is 11,374,046 while the size of the offending line is 11,370,773. That means that the rest of the message headers and text/plain part of the message occupy 3,273 bytes. > Really, I think that the best course of action would be that inc > always tries to write something out (unless it encounters something > like an I/O error) and exits cleanly. Actually I failed to report that inc *did* write out something. It wrote out until the MIME content started, so it got up to the headers of the MIME part and then while trying to scan the next line issued that error---the resulting file was truncated. I found the problem later when I was reading messages with EXMH which showed the attachment, but when I saved the attachment it was a 0 byte file. EXMH must be lenient when it comes to missing MIME part markers (maybe it just assumes end-of-file is good enough). Thanks, Andy -- TAI64 timestamp: 40005d77a9cf -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] inc: Unable to find a line terminator after 32768 bytes
Thus said Ken Hornstein on Mon, 09 Sep 2019 17:04:05 -0400: > In a perfect world I think we SHOULD parse those messages (up to the > limits of virtual memory), but right now we don't. That's actually how I figured this problem out. I found that my POP3 daemon kept crashing and when I investigated it, I found that it was because it didn't have sufficient memory to respond to inc's RETR command. After I increased the amount of memory that the POP3 daemon was allowed to allocate, the RETR command succeeded, but then I ended up with an inc that refused to incorporate emails. Whether or not we think making inc handle nonconforming lines is worth tackling, it might be a good idea to make inc handle the failure a little better. What happened instead was that inc exited after having partially RETR'ieved the message, without having told the POP3 server to DELE the ones it had already successfully pulled down. So each time I ran inc, it would pull down the messages, die on the same bogus message, and repeat; so that I ended up with a few duplicates. I think issuing a warning and leaving a bad message on the server would be better than aborting the entire POP3 session and causing a repeat. > Based on my personal experience ... you may not be able to find anyone > who really cares about fixing that (I have run into some people who > care about fixing broken email, most of the time I get ignored or > blown off). Just to warn you. Yeah, I just wanted to double-check my facts before I sent off an email asking them if they are aware of their misbehaving mail system. I'll see how they react (if they even get the message---it's difficult to find functioning postmaster@ addresses these days). Thanks, Andy -- TAI64 timestamp: 40005d7706d8 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] inc: Unable to find a line terminator after 32768 bytes
Hello, This is the first time I've ever seen such an error from inc. In looking at the message that is causing the problem, apparently it's a MIME message that has a base64 encoded MIME body that is all on one line that even sed has a hard time parsing: $ time (cat bigmessage | sed -ne '62p' | wc) 1 1 11370773 1m08.62s real 0m15.01s user 0m23.09s system This just seems ridiculous. I'm tempted to modify my SMTP server to reject such messages, but I'm curious to know if this is simply a bug in the sending software or if something has changed? Unfortunately, I cannot share the entire message, but I'm not sure I need to. It should be easily reproduced by simply base64 encoding a large file and putting it all on one line. Here are the redacted MIME headers: --=_Part_8167195_1805258438.1568043535371 Content-Type: application/octet-stream; name="attachment.eml" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.eml" Is this something I should report to the sender as a clear violation of RFC5322, which as far as I can tell, restricts line lengths to 998 characters, or is there something special about MIME that supersedes the limit and which means inc needs fixing? Thanks, Andy -- TAI64 timestamp: 40005d76af72 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Stupid 'pick' question...
Thus said "Valdis Kl?tnieks" on Sat, 08 Jun 2019 21:26:46 -0400: > In a world of Microsoft Office attachments, is having -search go > through the body by default as well still a good idea? Maybe having a > separate -searchbody would be better? Hard to say what it *should* be. In my environment, the majority of messages that I use -search with do have searchable content in the body of the message, so I've never really been concerned about it. Andy -- TAI64 timestamp: 40005d011078 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Flagging messages
Thus said aalin...@riseup.net on Sat, 25 May 2019 10:37:17 -0400: > Scenario: you receive an email from Geico informing you your car > insurance payment is due in 4 weeks. In mutt or Apple Mail you can > flag that message so you're reminded of it each time you log in. I'm not sure what "log in" means in your environment. If you're using X you could do something like this: Identify a message for which you want a reminder Use "mark -seq remind" to add the message to the remind sequence or use "refile -link +Reminders" to link the message into a folder Then in your .xsession you could place: REMINDERS=`scan +folder -width 132 remind` test "$REMINDERS" && xmessage -default okay -center -timeout 60 "$REMINDERS" Or if using a folder: REMINDERS=`scan +Reminders -width 132` test "$REMINDERS" && xmessage -default okay -center -timeout 60 "$REMINDERS" Then when done you remove the message from either the sequence or the folder. Andy -- TAI64 timestamp: 40005cec1f2e -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Thank You
Thus said Ken Hornstein on Tue, 30 Apr 2019 13:23:37 -0400: > Thank you for that pointer; sadly, I went through the whole process of > getting it started, it DID pop up the right window to prompt for MFA, > but the domain I use is configured to require administrator approval > to use another email application. I think there may also be an option to use a personal token (which may have to be enabled). I cannot search for it at the moment, but I do recall seeing some discussion in the Davmail forums about using a token instead of MFA. Thanks, Andy -- TAI64 timestamp: 40005cc8a461 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Thank You
Thus said berg...@merctech.com on Thu, 25 Apr 2019 20:08:02 -0400: > $WORK migrated my email to Office365 this week. I've had a similar experience and was able to reintegrate nmh by using Davmail to provide a POP interface to Office365---if they don't provide one for you already. Use search engines to discover more about Davmail. Thanks, Andy -- TAI64 timestamp: 40005cc67ec3 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] closefds() _before_ fork?
Thus said Ken Hornstein on Tue, 23 Apr 2019 19:35:11 -0400: > >setlock /tmp/mit.lock inc +MyIncTmp ... > > Well, my reading of the setlock man page (assuming this is the FreeBSD > one) is that it is not depending on a _descriptor_ to be passed down > (I don't even know how that would work), but in fact it is opening a > particular _file_ and locking it. Sorry, I should have referenced which setlock I intended. I didn't realize there was more than one setlock. I specifically had in mind setlock from daemontools (see http://cr.yp.to/daemontools.html for more information), though it is not the only command that implements this interface. Basically, it opens a file and attempts to obtain an exclusive lock the file descriptor for that file. If/when it successfully obtains the exclusive lock it then uses exec() to execute a new process, and that new process inherits the open file descriptor. As long as that process leaves the file descriptor open, it continues to hold the lock on said file which prevents any similarly invoked processes from running concurrently. The lock goes away either when the file descriptor is intentionally closed, or the process exits or crashes. > My change makes it so we no longer close all descriptors when creating > a child process (in the majority of cases we didn't). The particular > change I have started implementing is that any descriptors _created by > nmh_ will be marked as close-on-exec, so they will no longer be available > to child processes (well, technically, child processes that have called > exec(), but you know what I mean). Yes, I see what you mean---also, I didn't realize that the approach in nmh was so varied, but then, I haven't looked at the sources in-depth. This change sounds less drastic than I intially thought. I've run into problems before (most notably with Bash) where after setlock does exec() the new process will close inherited open file descriptors which basically breaks this particular way of resource control (sometimes referred to as Bernstein chaining). Thanks for the clarification; it does sound like it will be an improvement. Andy -- TAI64 timestamp: 40005cbfdd37 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] closefds() _before_ fork?
Thus said Paul Vixie on Tue, 23 Apr 2019 05:59:05 -0700: > to that end, i propose that we treat any open descriptor N>2 at the > time of an exec() to be a bug, which is to be fixed by setting > O_CLOEXEC at time of creation. What impact will such a decision have on tools like setlock which open a filedescriptor, obtain an exclusive lock, and then exec a process? For example, I may use something like the following in a script to prevent multiple processes from incorporating email: setlock /tmp/mit.lock inc +MyIncTmp ... Thanks, Andy -- TAI64 timestamp: 40005cbf7b26 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] The State of exmh.
Thus said Ralph Corderoy on Thu, 08 Nov 2018 10:36:12 +: > A workaround is to subscribe the list to a public archiver so Google > can see everything from the switch-over date onwards, as long as the > list-info page makes this clear as otherwise new posters still think > it's semi-private. If you didn't see my previous email... It was already done long ago by MARC; these archives go back to 1998: https://marc.info/?l=exmh-users&r=1&w=2 https://marc.info/?l=exmh-workers&r=1&w=2 I don't know if they are 100% complete, but I don't know why they wouldn't be. Andy -- TAI64 timestamp: 40005be4538c -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] The State of exmh.
Thus said Ken Hornstein on Wed, 07 Nov 2018 15:52:32 -0500: > >I considered poking around > >https://www.redhat.com/mailman/listinfo/exmh-users for signs of life, > >but those archives are subscriber only. > > You know, this has always driven me nuts. Why is it set like this? Can > that be opened up? I prefer these anyway: https://marc.info/?l=exmh-users&r=1&w=2 https://marc.info/?l=exmh-workers&r=1&w=2 Andy -- TAI64 timestamp: 40005be451bc -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] post 1.71 ug: "long line"/single newline paragraphs
Thus said Ken Hornstein on Thu, 24 May 2018 19:28:25 -0400: > So, I don't think this is necessarily a nmh problem, like I said. From > a practical standpoint, you're probably one of the few people who > actually USES the text/plain content instead of the text/html content. Count me as another one of those few people who actually uses the text/plain content. :-) For the most part, I don't read emails that are HTML only unless I absolutely have to and if there's a text/plain I manage with that unless it's terrible. Andy -- TAI64 timestamp: 40005b082e22 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Unnecessary dependency on vi???
Thus said Ralph Corderoy on Tue, 20 Mar 2018 12:56:09 -: > For evermore, programs that only offer one means of invoking an editor > have had to checking first $VISUAL, falling back to $EDITOR. :-) You mean like the following chunk of code: :-) http://www.fossil-scm.org/index.html/artifact?udc=1&ln=1180-1186+1196-1204&name=8d7c320c6bbe086b Andy -- TAI64 timestamp: 40005ab16cc3 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Unnecessary dependency on vi???
Thus said Ken Hornstein on Sun, 18 Mar 2018 19:49:03 -0400: > If you're a long-time MH user, I admit that I am surprised you never > set anything in your profile; it seems like the default was prompter > for a long time (although, like I said earlier, that depends on your > specific site configuration; I've been using (N)MH since 2,000 and I think this is the first time I've run into prompter; though I must admit that I used Exmh a couple years before I began investigating the command line tools, but as far as I can remember, I've never had to interface with prompter. As you say, perhaps this was due to specific configuration of which I was unaware, but I've usually just used whatever the OS vendor shipped as defaults. Andy -- TAI64 timestamp: 40005aaf0524 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Unnecessary dependency on vi???
Thus said Ken Hornstein on Sat, 17 Mar 2018 20:25:05 -0400: > Yeah, I tried it quickly and it seems simple enough. And people who > have editor in their profile or use EDITOR/VISUAL won't notice a > change. Under what conditions will this change? I have neither EDITOR/VISUAL nor profile settings for editor, but maybe that won't matter because my usage patterns will never invoke prompter? Right now, when I run comp from the command line, I get a vi editor with with components in it. Is this where prompter comes in? Sounds like I might have to add something to my profile now after this change is made to avoid prompter. Andy -- TAI64 timestamp: 40005aae8212 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Unnecessary dependency on vi???
Thus said Paul Vixie on Sat, 17 Mar 2018 08:55:41 -0700: > i have not run comp without first setting VISUAL for at least two > decades, but when i used to do this, it would print a message like > "type your message below, and then hit control-D" and then read from > standard input. when did that change to requiring an external editor? I generally don't set VISUAL or EDITOR unless I absolutely have to (e.g. on Ubuntu which defaults to nano) as I generally expect vi to be the default. $ set | grep -E '(EDITOR|VISUAL)' $ echo $? 1 Andy -- TAI64 timestamp: 40005aad45d8 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Unnecessary dependency on vi???
Thus said Paul Fox on Wed, 14 Mar 2018 14:59:57 -0400: > The big exception that I remember was his implementation of infinite > undo using '.', which broke a corner case of the redo command, but is > so easy to use. Oddly enough, that is one exception that I praise and the one difference between nvi and vim that I cannot live without. I use infinite undo via '.' all the time, but not only that, I'm impressed by how I can ``reverse'' the direction of the undo and '.' repeats that decision. Andy -- TAI64 timestamp: 40005aac2904 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Unnecessary dependency on vi???
Thus said Paul Fox on Tue, 13 Mar 2018 09:50:24 -0400: > well, part of me wants to take offense at that, since it's not like > vim is completely compatible with the "real" vi. nvi is much closer, > in that regard, and should really be the rewrite that gets to use the > /usr/bin/vi name. As a long-time nvi user, I'm always annoyed that vim doesn't work more like vi. :-) Andy -- TAI64 timestamp: 40005aa96d14 -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] What OS/Architecture Do You Run nmh On?
Thus said Ken Hornstein on Fri, 16 Feb 2018 10:44:06 -0500: > Right, I was wondering if Dovecot was cool with other Maildir clients > modifying its store; the answer is "yes". I was also wondering if we > do a "refile +/path/to/dovecot/folder 42", should we put the message > in "new" or "cur"? Since it seems (in the singular instance of > Dovecot) that "new" is for messages that are flagged as \Recent, and I > don't think refiled messages should count as \Recent, then that > suggests that the message should go into cur. I suppose that depends on the meaning of the word ``deliver'' and also whether or not the refile command is moving message 42 from the same filesystem, or crossing filesystem boundaries. In the former case, it's an atomic operation, but in the latter case, to retain the ``crash proof'' nature of Maildir, it would seem that one should ``deliver'' the message first into Maildir/tmp, followed by an atomic move into either Maildir/new or Maildir/cur. Andy -- TAI64 timestamp: 40005a8753f1 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] What OS/Architecture Do You Run nmh On?
Thus said Ken Hornstein on Thu, 15 Feb 2018 22:35:04 -0500: > Well, you can read the many messages on this topic from someone who > will remain nameless, but who's name rhymes with "Maul Pixie". In the scenario that he who shall remain nameless presented, I can definitely see the need for it. In this case, nmh is running on the same server where the mail is stored, and it would be nice to have both command-line access with nmh and remote access with IMAP. If one wants to access a Maildir from multiple locations (e.g. IMAP server, mutt, nmh, whatever), then it would seem worthwhile to discuss Maildir support for nmh. > (and while our mailstore has a number of advantages, it also has a > number of disadvantages). Is there a documented pros/cons discussion of MH mailstore vs Maildir. They do have similar structures in that a Maildir stores individual messages in individual files. I've often wondered what I might gain from switching to Maildir (it's benefits are clear with respect to locking and crash prevention). Thanks, Andy -- TAI64 timestamp: 40005a875219 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] What OS/Architecture Do You Run nmh On?
Thus said Ken Hornstein on Wed, 14 Feb 2018 14:22:56 -0500: > If we scan a Maildir folder, should we be in charge of moving stuff > from new into cur? Maybe? Maybe not? You tell me! Let's look at what qmail-pop3d does when a client ``scans'' the list of messages: $ find new -type f| wc -l 1 $ find cur -type f| wc -l 1 So, I have one message in Maildir/new and one in Maildir/cur, and if I login to my qmail-pop3d, run LIST, nothing happens to the file on disk, there remains one in Maildir/cur and one in Maildir/new. +OK LIST +OK 1 559 2 563 . However, as soon as I send QUIT to my qmail-pop3d daemon, it moves that message from Maildir/new to Maildir/cur: $ find cur -type f| wc -l 2 $ find new -type f| wc -l 0 Does that mean that MH scan should do the same? Andy -- TAI64 timestamp: 40005a850851 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] What OS/Architecture Do You Run nmh On?
Thus said Ken Hornstein on Tue, 13 Feb 2018 10:09:07 -0500: > But Bernstein ignored MH because he was not trying to invent a MAILBOX > format, he was trying to invent a mailDROP ... really, I went back and > looked. Yes, I know people now use Maildir as a mailbox, and I think > that's weird, but it wasn't his intention. As a longtime (and current) qmail user, I'm interested in understanding this argument more in detail; e.g. where did you go back and look to support the claim that he was trying to invent a maildrop? >From http://cr.yp.to/qmail.html: Reliable: qmail's straight-paper-path philosophy guarantees that a message, once accepted into the system, will never be lost. qmail also optionally supports maildir, a new, super-reliable user mailbox format. Maildirs, unlike mbox files and mh folders, won't be corrupted if the system crashes during delivery. Even better, not only can a user safely read his mail over NFS, but any number of NFS clients can deliver mail to him at the same time. That certainly seems to me that he intended for it to replace a MAILBOX format (unless perhaps I misunderstand what is meant by it). Also, theres: http://cr.yp.to/proto/maildir.html Which again would seem to indicate that it was intended to be a mailbox and not just a maildrop; here again, it's entirely possible that I misunderstand the difference to which you alluded above, but I would be interested in what you found that suggests it was intended solely as a maildrop format. Thanks, Andy -- TAI64 timestamp: 40005a85016a -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] What OS/Architecture Do You Run nmh On?
Thus said Ken Hornstein on Tue, 13 Feb 2018 11:30:38 -0500: > >We've talked a lot about the subtle change to move MH to Maildir, and > >we never quite work out all the wrinkles, and I'd sure like to that. > > I hear people say this, and I have to wonder ... what's the goal here? Again, as one of the few remaining qmail users, I've never really seen the need for MH to move to Maildir. What exactly would I gain by that? I use inc to pull mail via POP from a qmail server, and then inc places it into my filesystem using MH storage routines (I suppose it's slocal really). -- TAI64 timestamp: 40005a850298 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Where are we at for 1.7.1?
Thus said "Todd C. Miller" on Wed, 31 Jan 2018 14:11:09 -0700: > The OpenBSD C compiler does not look in /usr/local/include or > /usr/local/lib by default but the nmh port's Makefile does. I should have cracked open the Makefile and I would have seen the obvious CONFIGURE_ENV options being passed in. > CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure Yes, now it builds and the test passes, and now I see that test/common.sh is actually ``built'' on demand from test/common.sh.in with the rest of the build process so that it defaults ICONV_ENABLED differently based on how it was configured. That answers my previous question about how ICONV_ENABLED could ever be altered (it happens at ``build'' time). $ make TESTS=test/mhshow/test-charset check make test/fakehttp test/fakepop test/fakesmtp test/getcanon test/getcwidth test/getfullname test/runpty test/common.sh `test/fakehttp' is up to date. `test/fakepop' is up to date. `test/fakesmtp' is up to date. `test/getcanon' is up to date. `test/getcwidth' is up to date. `test/getfullname' is up to date. `test/runpty' is up to date. `test/common.sh' is up to date. make check-TESTS PASS: test/mhshow/test-charset = 1 test passed = Thanks, Andy -- TAI64 timestamp: 40005a73d21d -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Where are we at for 1.7.1?
Thus said "Andy Bradford" on 31 Jan 2018 13:26:53 -0700: > I believe the actual nmh port does enable iconv support, so this would > not come out there. It only was detected because I didn't bother to do > things in the ``standard'' way. Apparently I did already have libiconv installed, I just assumed I didn't have it because it was failing. As indicated in my previous email, apparently configure is not detecting it. Also, what's interesting is that apparently while my locally built nmh doesn't have iconv, apparently the one built from ports does: $ ldd uip/mhshow uip/mhshow: StartEnd Type Open Ref GrpRef Name 19619000 3962f000 exe 10 0 uip/mhshow 0f939000 2f951000 rlib 01 0 /usr/lib/libtermcap.so.14.0 094de000 29507000 rlib 01 0 /usr/lib/libc.so.89.3 0726d000 0726d000 rtld 01 0 /usr/libexec/ld.so $ ldd /usr/local/bin/mhshow /usr/local/bin/mhshow: StartEnd Type Open Ref GrpRef Name 19629000 3963c000 exe 10 0 /usr/local/bin/mhshow 02c5b000 22c73000 rlib 01 0 /usr/lib/libtermcap.so.14.0 0d635000 2d715000 rlib 01 0 /usr/local/lib/libiconv.so.6.0 0a9c1000 2a9ea000 rlib 01 0 /usr/lib/libc.so.89.3 05527000 05527000 rtld 01 0 /usr/libexec/ld.so So it's interesting that even though I have libiconv installed, when I build as indicated, I don't get iconv support linked in, but somehow the port did (perhaps due to the LIB_DEPENDS) as mentioned previously. Which means that when building from ports, nmh will have iconv, but when building stand-alone, it will not (given that it couldn't find iconv.h). Of course, all of this does not eliminate the problem of the test not working correctly when ICONV_ENABLED=0. Thanks, Andy -- TAI64 timestamp: 40005a722bd7 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Where are we at for 1.7.1?
Thus said Ken Hornstein on Wed, 31 Jan 2018 09:22:20 -0500: > Yeah ... I am kind of surprised OpenBSD doesn't ship with iconv, but I > have to believe it's available via a package. Like I said, it's part > of POSIX so I think requiring it is perfectly reasonable. Maybe he > does have it and our autoconf tests are wrong; if that's the case we > should spend our energy on fixing that. While it doesn't ship with it directly, it is a package, and I do actually have it installed: $ pkg_info | grep iconv libiconv-1.14p3 character set conversion library Here's what I found in config.log: configure:6048: checking iconv.h usability configure:6048: cc -c -g -O2 -Wall -Wextra conftest.c >&5 conftest.c:72:19: error: iconv.h: No such file or directory configure:6048: $? = 1 Here's where the header is: $ pkg_info -L libiconv | grep iconv.h /usr/local/include/iconv.h Andy -- TAI64 timestamp: 40005a7227fa -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Where are we at for 1.7.1?
Thus said Ralph Corderoy on Wed, 31 Jan 2018 12:40:28 +: > > This begs a larger question ... should we just specify iconv as a > > requirement for the next release? > > You're thinking those without it, e.g. Andy, could easily get iconv > with a package install? You're right, I did not go through any extra effort beyond: ./configure make make checks I believe the actual nmh port does enable iconv support, so this would not come out there. It only was detected because I didn't bother to do things in the ``standard'' way. Andy -- TAI64 timestamp: 40005a7226b2 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Where are we at for 1.7.1?
Thus said Ralph Corderoy on Tue, 30 Jan 2018 17:01:58 +: > That's this early part. Does it attempt the printf? If so, > $ICONV_ENABLED isn't 0, and iconv(1) quite reasonably disliked > converting from `?UTF-8'. No, it does not attempt the printf because ICONV_ENABLED=0 and I'm not even sure what is supposed to alter ICONV_ENABLED (I ran with sh -x and only ever see it accessed once). I don't see it anywhere in the test case in question, nor do I see it being set in test/common.sh. > > $ cat $MHTMPDIR/inbox/$msgnum > > Subject: invalid parameter value charset > > MIME-Version: 1.0 > > Content-Type: text/plain; charset*=invalid''%0Dus-ascii > > It expects an error because the charset is invalid. The parameter's > value is in the `invalid' character set, with no specified language, > and it is a carriage return followed by `us-ascii'. So, even though $actual and $expected are identical, the output is incorrect in both? That doesn't seem correct; rather, it would seem to me that if ICONV_ENABLED=0 that the output is actually correct output, and that inspecting iconv_elides_question_marks is the wrong thing to do (at least in this environment). Accordingly, I just tried the following and now the test passes: $ diff -u test/mhshow/test-charset.orig test/mhshow/test-charset --- test/mhshow/test-charset.orig Thu Jan 18 18:23:34 2018 +++ test/mhshow/test-charsetTue Jan 30 13:40:48 2018 @@ -144,7 +144,7 @@ EOF run_prog mhshow $msgnum > $actual 2>&1 -if [ $iconv_elides_question_marks -eq 1 ]; then +if [ $ICONV_ENABLED -eq 0 -o $iconv_elides_question_marks -eq 1 ]; then check "$expected" "$actual" else grep "Can't convert .*us-ascii to UTF-8" "$actual" >/dev/null || failed=1 However, I'm not certain if that's the right thing to do here, but ti does pass: $ make TESTS=test/mhshow/test-charset check make test/fakehttp test/fakepop test/fakesmtp test/getcanon test/getcwidth test/getfullname test/runpty test/common.sh `test/fakehttp' is up to date. `test/fakepop' is up to date. `test/fakesmtp' is up to date. `test/getcanon' is up to date. `test/getcwidth' is up to date. `test/getfullname' is up to date. `test/runpty' is up to date. `test/common.sh' is up to date. make check-TESTS PASS: test/mhshow/test-charset = 1 test passed = Thanks, Andy -- TAI64 timestamp: 40005a70d9d6 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Where are we at for 1.7.1?
Thus said "Andy Bradford" on 29 Jan 2018 21:16:14 -0700: > I'll try to figure out why it's failing and report back, ... So, here's where I'm at; I see the script makes it to here: if [ $iconv_elides_question_marks -eq 1 ]; then check "$expected" "$actual" else grep "Can't convert .*us-ascii to UTF-8" "$actual" >/dev/null || failed=1 fi It enters the else because iconv_elides_question_marks was previously set to 0 and never altered thereafter. So, why does it expect an error in $actual under these conditions? Here is what I have in $actual and $expected: $ diff $actual $expected $ echo $? 0 $ cat $actual [ Message inbox:12 ] Subject: invalid parameter value charset MIME-Version: 1.0 [ part - text/plain - 0B ] So, why is this test failing? It seems to think that there should be some kind of error in the file that it expects mhshow to produce, but it's not going to. Here is the given input for mhshow (and output that mhshow produces): $ cat $MHTMPDIR/inbox/$msgnum Subject: invalid parameter value charset MIME-Version: 1.0 Content-Type: text/plain; charset*=invalid''%0Dus-ascii $ mhshow $msgnum [ Message inbox:12 ] Subject: invalid parameter value charset MIME-Version: 1.0 [ part - text/plain - 0B ] Here are some more relevant bits that were setup prior to running this test; LC_ variables were removed from the environment and a local appended to $MH (the .mh_profile): $ env | grep LC $ echo $? 1 $ tail -1 $MH locale: en_US.UTF-8 So why does it think it needs to change the test under these conditions from a simple diff to a grep? Thanks, Andy -- TAI64 timestamp: 40005a6ffceb -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Where are we at for 1.7.1?
Thus said Ken Hornstein on Mon, 29 Jan 2018 21:12:43 -0500: > So I went and looked again at people who tried out 1.7.1-RC2; it seems > there were some benign warnings (which we didn't clean up, AFAIK) and > Andy Bradford reported a failure on the test suite that we never > resolved. Is that correct? Should we try to run that to ground? I'll try to figure out why it's failing and report back, but it's probably not a show-stopper, more likely a test that didn't account for some environmental difference. Andy -- TAI64 timestamp: 40005a6ff1b3 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Announcing the second release candidate of nmh 1.7.1
Thus said David Levine on Tue, 23 Jan 2018 07:45:29 -0500: > I usually cheat and just run the test directly: > > test/ali/test-ali Ok, I'll give that a try. Interestingly: $ make check TESTS=test/mhshow/test-charset make test/fakehttp test/fakepop test/fakesmtp test/getcanon test/getcwidth test/getfullname test/runpty test/common.sh `test/fakehttp' is up to date. `test/fakepop' is up to date. `test/fakesmtp' is up to date. `test/getcanon' is up to date. `test/getcwidth' is up to date. `test/getfullname' is up to date. `test/runpty' is up to date. `test/common.sh' is up to date. make check-TESTS ... first named test failure: replacement character in parameter value FAIL: test/mhshow/test-charset === 1 of 1 test failed Please report to nmh-workers@nongnu.org === *** Error 1 in . (Makefile:4853 'check-TESTS': @failed=0; all=0; xfail=0; xpass=0; skip=0; srcdir=.; export srcdir; list=' test/mhshow/tes...) *** Error 1 in /tmp/nmh-1.7.1-RC2 (Makefile:5020 'check-am') Let's see what failed: $ ls test/testdir/ 19455.actual19455.expected inst/ $ diff test/testdir/19455.*; echo $? 0 $ md5 test/testdir/19455.* MD5 (test/testdir/19455.actual) = cc15d943cbe66365f49268516f99e00c MD5 (test/testdir/19455.expected) = cc15d943cbe66365f49268516f99e00c $ cat test/testdir/19455.expected [ Message inbox:12 ] Subject: invalid parameter value charset MIME-Version: 1.0 [ part - text/plain - 0B ] So why is this test failing? Maybe I misunderstand the significance of these files and how they're used. Time to do some more digging... $ test/mhshow/test-charset first named test failure: replacement character in parameter value $ echo $? 1 And with -x: + [ 0 -eq 1 ] + grep Can't convert .*us-ascii to UTF-8 /tmp/nmh-1.7.1-RC2/test/testdir/38552.actual + > /dev/null + failed=1 So it's entering this: if [ $iconv_elides_question_marks -eq 1 ]; then check "$expected" "$actual" else grep "Can't convert .*us-ascii to UTF-8" "$actual" >/dev/null || failed=1 fi Is it not honoring the locale: setting in .mh_profile? And I'm out of time, so I'll have to look into it more tomorrow. Thanks, Andy -- TAI64 timestamp: 40005a6820a8 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Announcing the second release candidate of nmh 1.7.1
Thus said Ralph Corderoy on Tue, 23 Jan 2018 11:12:17 +: > The vital lines of interest are in the lead up to that. Should be > something starting with a context diff. There was no lead up to it, that's why I didn't include it---and also the reason why I was trying out that MH_TEST_NOCLEANUP variable; here is what is output to my terminal: PASS: test/mhparam/test-mhparam PASS: test/mhpath/test-mhpath PASS: test/mhshow/test-binary first named test failure: replacement character in parameter value FAIL: test/mhshow/test-charset PASS: test/mhshow/test-cte-binary PASS: test/mhshow/test-msg-buffer-boundaries PASS: test/mhshow/test-qp I'm going to experiment some more to see where the output is going. Andy -- TAI64 timestamp: 40005a67f827 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Announcing the second release candidate of nmh 1.7.1
Thus said Ralph Corderoy on Tue, 23 Jan 2018 11:03:46 +:
> What's the `cc' line from make's output that compile's post.c? And
> what version of a C compiler is it? Can't get the warning here with
> gcc 7.2.1 20171224 and `-std=c99 -pedantic -pedantic-errors -Wall
> -Wextra -Wformat=2 -Wno-format-nonliteral'.
depbase=`echo uip/post.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`; cc
-DHAVE_CONFIG_H -I. -g -O2 -Wall -Wextra -MT uip/post.o -MD -MP -MF
$depbase.Tpo -c -o uip/post.o uip/post.c && mv -f $depbase.Tpo $depbase.Po
uip/post.c: In function 'putadr':
uip/post.c:1239: warning: 'shlen' may be used uninitialized in this function
$ cc -v
Reading specs from /usr/lib/gcc-lib/i386-unknown-openbsd6.1/4.2.1/specs
Target: i386-unknown-openbsd6.1
Configured with: OpenBSD/i386 system compiler
Thread model: posix
gcc version 4.2.1 20070719
> Does swapping the strlen and true line above shut it up?
I'm not sure why it would, but I tried:
if (savehdr) {
saveappend = 1;
shlen = strlen(savehdr);
}
And it still issued the same warning.
Also, I tried compiling the same code on OpenBSD 6.2 which apparently
now uses clang, and I didn't get any warning:
depbase=`echo uip/post.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`; cc
-DHAVE_CONFIG_H -I. -g -O2 -Wall -Wextra -MT uip/post.o -MD -MP -MF
$depbase.Tpo -c -o uip/post.o uip/post.c && mv -f $depbase.Tpo $depbase.Po
cc -g -O2 -Wall -Wextra -Qunused-arguments -o uip/post uip/post.o
uip/aliasbr.o mts/libmts.a sbr/libmh.a -lssl -lcrypto 2>&1 | sed -e 's/:
w\(arning: s.*() is .* misused, please use\)/: W\1/'
$ cc -v
OpenBSD clang version 4.0.0 (tags/RELEASE_400/final) (based on LLVM 4.0.0)
Target: i386-unknown-openbsd6.2
Thread model: posix
InstalledDir: /usr/bin
Interesting.
Andy
--
TAI64 timestamp: 40005a67f057
--
Nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Announcing the second release candidate of nmh 1.7.1
Thus said Ken Hornstein on Mon, 22 Jan 2018 14:20:32 -0500: > This is a patch release for 1.7, and fixes some output problems with > the format engine, issues with rcvdist(1) passing switches to post(8), > and a number of problems discovered with the test suite. Speaking of the test suite, I ran it (on OpenBSD 6.1) and encountered a failure I don't recall seeing previously: first named test failure: replacement character in parameter value FAIL: test/mhshow/test-charset ... === 2 of 104 tests failed (8 tests were not run) Please report to nmh-workers@nongnu.org === I'm not sure why it failed, and while reading through test/common.sh, I saw the following: If you're reading this you can set MH_TEST_NOCLEANUP to prevent the test suite from cleaning up the results of a test run, if you need to do manual debugging on a test. So I then ran: env MH_TEST_NOCLEANUP=1 make check And got a whopping 34 failures... not sure if that's expected or not. 2 of the test cases asked me interactively for Disposition of a message (I opted for replace because I didn't know one way or the other). === 34 of 104 tests failed (8 tests were not run) Please report to nmh-workers@nongnu.org === So, then I decided to run it a third time, again without the environment variable and got 4 failures (2 more than before without the variable) among which: inc: field name "ThisHeaderFieldNameIsWyTooLong" exceeds 997 bytes ??Format error (message 12) in component 5 inc: m_getfld() called with bogus state of -2 FAIL: test/folder/test-packf And inc: field name "ThisHeaderFieldNameIsWyTooLong" exceeds 997 bytes ??Format error (message 24) in component 5 inc: m_getfld() called with bogus state of -2' FAIL: test/inc/test-inc-scanout Maybe the environment variable to not clean up corrupted the test environment and I need some way to clean it up? I can rerun ``make check'' multiple times and get a consistent 2 failures, but if I use that environment variable it then hits 34 failures, and then without again 4. Also, how do I check just a single test? Thanks, Andy -- TAI64 timestamp: 40005a66dde6 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] Announcing the second release candidate of nmh 1.7.1
Thus said Ken Hornstein on Mon, 22 Jan 2018 14:20:32 -0500: > If you encounter any problems or issues with this release candidate, > please don't hesitate to contact nmh-workers@nongnu.org. This may be a known issue (or not an issue at all): uip/post.c: In function 'putadr': uip/post.c:1239: warning: 'shlen' may be used uninitialized in this function >From a cursory look, it could be uninitialized if the savehdr parameter is NULL and I don't see any code that guards savehdr from not being NULL. I also cannot easily say whether any of the calls to putadr will ever send in NULL. Should shlen have a default value, or should putadr not allow the caller to send in NULL? uip/fmttest.c: In function 'main': uip/fmttest.c:123: warning: 'outputsize' may be used uninitialized in this function Looks like this one will require even more investigation to discover whether the warning is just a red herring or not. Andy -- TAI64 timestamp: 40005a66d8b1 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] why does mhfixmsg dislike long text lines?
Thus said David Levine on Mon, 22 Jan 2018 20:43:50 -0500: > > The comment in mhfixmsg which I quoted at the beginning of > > this thread seems to be saying that sometimes message components > > described as text/* are really binary files, and that the > > 998-character limit is used in mhfixmsg (only) as a heuristic to > > identify this situation. > > I wouldn't call it a heuristic. It's definitive, according to RFC > 2045. One of the reasons why RFC 2045 has this definition can be found in RFC 5321 (previously 2821, previously 821) where a Text Line is defined: 4.5.3.1.6. Text Line The maximum total length of a text line including the is 1000 octets (not counting the leading dot duplicated for transparency). This number may be increased by the use of SMTP Service Extensions. So, regardless of the on-disk format or how a message might meet RFC 5322, if it wants to be sent via SMTP, it will have to be encoded in some fashion (enter MIME, uuencode, etc...). Andy -- TAI64 timestamp: 40005a66d3a7 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] proposed patch for shell metacharacter failure in nmh-1.7
Thus said Ken Hornstein on Sat, 20 Jan 2018 13:59:20 -0500: > Forgive me if this is a dumb question, but ... why do you care what > the timezone is in your Date: headers? Personally, I've always found the timezone in the Date header quite useful to convey a sense of time. If all timezones in Date headers are UTC, then it becomes difficult for interlocutors to to predict whether or not they're likely to receive a response to an email. For example, the Date header in your email indicates to me that you are somewhere in EST and if I were respond to this email at 22:00 MST time, I wouldn't likely expect an answer until the next morning. Indeed, his script does include this comment: # it's annoying to view Date: headers marked in a different time zone; # that annoyance wasn't important in a world where invalid time zones were # infrequent, but EVERY SINGLE MESSAGE from Concordia's new Exchange # servers is stamped in UTC :-( I too have been extremely annoyed by the fact that Outlook/Exchange emails are all in UTC as it destroys the abovementioned utility. Had your Date header been in UTC, I could not make any such inference, however, given the current state of communicating with Exchange users, would converting the Date header to my local time be of any use to me? Yes, because at least now I can gauge it relative to my local time. Fortunately, my email client (exmh) does this for me without actually altering the message itself. Your Date header in (exmh) shows up as: Date: Sat, 20 Jan 2018 13:59:20 -0500 (11:59 MST) While on disk it remains: Date: Sat, 20 Jan 2018 13:59:20 -0500 Andy -- TAI64 timestamp: 40005a63b250 -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] 1.7's `make clean' is Overzealous.
Thus said Ken Hornstein on Tue, 05 Dec 2017 10:43:36 -0500: > But it sounds like with the way things are currently set > up, for 1.7 the configuration files will actually end up in > /usr/local/lib/nmh/nmh, which I think most of us would agree kind of > sucks. You're right, I didn't try building nmh-1.7 via OpenBSD ports. I just updated the Makefile in ports to build nmh-1.7 and indeed it does now have the double nmh directory: $ find /usr/ports/pobj/nmh-1.7/fake-i386 -type d -name nmh /usr/ports/pobj/nmh-1.7/fake-i386/usr/local/lib/nmh /usr/ports/pobj/nmh-1.7/fake-i386/usr/local/lib/nmh/nmh /usr/ports/pobj/nmh-1.7/fake-i386/usr/local/libexec/nmh /usr/ports/pobj/nmh-1.7/fake-i386/usr/local/share/doc/nmh Has something changed in nmh-1.7 that would make sysconfdir be handled differently, whereas in nmh-1.6 it worked fine? Thanks, Andy -- TAI64 timestamp: 40005a2701da -- Nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
