Re: [nmh-workers] logging outgoing messages
On Jul 9, 2019, at 5:56 PM, Ronald F. Guilmette wrote: > > In message <20190710004749.89c1b163...@pb-smtp1.pobox.com>, > Ken Hornstein wrote: > >> If I could make sendmail/pipe punch the user in the face every time a >> message was sent using it... > > Please don't. I'm using it. > > It is MUCH faster than trying to feed the message to Postfix > (aka Sendmail) via SMTP/587 because in the case of just piping the > message, Postfix doesn't make me wait until it has done the > DNS lookups it thinks it needs to do in order to process > the message. I've just added send: -push to ~/.mh_profile. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Ronald F. Guilmette wrote: >> If I could make sendmail/pipe punch the user in the face every time a >> message was sent using it... > Please don't. I'm using it. > It is MUCH faster than trying to feed the message to Postfix > (aka Sendmail) via SMTP/587 because in the case of just piping the > message, Postfix doesn't make me wait until it has done the > DNS lookups it thinks it needs to do in order to process > the message. Then you are missing an entry in /etc/hosts for 127.0.0.1 and ::1, or you are missing your hostname in that file. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
In message <20190710004749.89c1b163...@pb-smtp1.pobox.com>, Ken Hornstein wrote: >If I could make sendmail/pipe punch the user in the face every time a >message was sent using it... Please don't. I'm using it. It is MUCH faster than trying to feed the message to Postfix (aka Sendmail) via SMTP/587 because in the case of just piping the message, Postfix doesn't make me wait until it has done the DNS lookups it thinks it needs to do in order to process the message. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Date:Tue, 09 Jul 2019 19:39:08 -0400 From:Ken Hornstein Message-ID: <20190709233912.db2aa73...@pb-smtp20.pobox.com> | - We don't, in general, want to have any more #ifdefs in the code unless | they are completely unavoidable I agree with that, and even when ifdef's are added, they should be positive, not double negative, so #ifndef NOSYSLOG is just perferse, where #ifdef USE_SYSLOG would work just as well (it does mean the name needs to be explicitly defined to get the new code, but that's the way it should be, rather than requiring a new name to be defined to retain the previous behaviour). | - It is not clear to me that you can state with certainly that the | 250 response code will contain the queue identifier No, you can't, but these days it almost always does. | As a practical matter I've never had to give anyone the queue | identifier of a message Then you have been lucky, are are relatively new to e-mail tracking. The need is less common today, than it once was, since more and more e-mail is direct from sender's MTA to recipient's - but back when more mail relaying was done (when there was more than just "the internet") the queue-id along with the transfer timestamp was the info that you could send to a relay postmaster and actually expect them to look and see what happened to the message (because that generally makes it trivial for them to do ... send the message-id instead, or worse, just addresses, and it was typically a lot more work, which resulted in requests for tracking sometimes simply being ignored). | (because it's not normally logged on the client; Any rational (MTA) client does. MUA's typically don't, but those should not really ever be talking to anything but their local MTA. What is different now than what used to be true, is what people regard as their local MTA, which in the past would normally have been either on the same host, or at absolute worst, in the same organisation as the sender (and usually same department of the organisation when it is big enough to have such things). (Organisation can mean household for personal e-mail). Way too many people today are relying upon "free" giant e-mail providers to do everything for them. | really, most people are happy with recipients and a time, and | they are really happy if you have a message-id). time yes, recipients, no, that's close to useless (though it can work if the recipient receives very little e-mail and the site postmaster is in a helpful mood at the time a request is made). The message-id is better, but the queue-id (along with time) is perfect. (With a message-id (with or without the time) the first thing that normally needs doing to track a message is to convert that into the relevant local queue-id - avoiding the need for that step makes it just that much more likely that a message can, and will, be traced.) | But there's no way we could | accept this patch as-is, because it depends on the specific format of | your ISP's response message. Yes, that's no good - the only rational solution is to log all of the 250 response (after the 250 itself). It is not only the queue-id in it which can be helpful, sometimes the wording of the response can give more info to the site which issued it (when you ask them to see what happened) - it can differ for mail which was queued there, immediately forwarded before being ack'd, or which was delivered locally. | It looks like your ISP's format is "250 id=QUEUE-ID". So that's 3 | servers and 3 different formats. I used to have my mailer say something like 250 OK Receipt: queue-id to make it clear that that was the info that I wanted sent to me if you wanted me to track your e-mail. Not sure if I still do that or not, it has been a while since munnari did any significant e-mail relaying (it used to connect 5 or 6 different e-mail environments). | I think this should be a lot more generic. So ... an alternate proposal. Personally, I'd just suggest keeping the local MTA, having post deliver to that, and let it do the logging - it will also do queueing in case your local ISP link is down (like you want to send e-mail while in an airplane...). That is, I wouldn't bother with this at all - there is an alternative (and better) solution already available. kre -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>Could we log the entire result, and let the post hook take care of the >various queue formats? That was what I suggested. Clearly nmh shouldn't be in the business of figuring out what (if any) the queue identifier is based on the SMTP DATA response message. >> I am neutral about this being made to work with sendmail/pipe; it would >> actually be a lot of work to do that. We could just accept that it is >> one more thing that doesn't work with sendmail/pipe. > >slowly, that interface is dying. I have mixed feelings about that. If I could make sendmail/pipe punch the user in the face every time a message was sent using it, I would. Sadly, that does not seem possible given the current constraints of POSIX. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
Ken Hornstein wrote: > - It is not clear to me that you can state with certainly that the > 250 response code will contain the queue identifier (that is, in > fact, not a concept that appears anywhere that I can find in the SMTP > RFCs). As a practical matter I've never had to give anyone the queue > identifier of a message (because it's not normally logged on the I have both asked for it and provided it when debugging bizarre situations. Could we log the entire result, and let the post hook take care of the various queue formats? > I am neutral about this being made to work with sendmail/pipe; it would > actually be a lot of work to do that. We could just accept that it is > one more thing that doesn't work with sendmail/pipe. slowly, that interface is dying. I have mixed feelings about that. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
On Tue, 09 Jul 2019 17:43:06 -0400, Steven Winikoff said:
> sm_reply.length = rp - sm_reply.text;
> sm_reply.text[sm_reply.length] = 0;
> +#ifndef NOSYSLOG
> +if (strncmp(sm_reply.text, "OK id=", 6) == 0)
> +{
This is highly dependent on the remote MTA.
Google, for instance, returns:
250 2.0.0 OK 1562718444 r17sm180161qtf.26 - gsmtp
The 250 is required. 2.0.0 (or similar) if the remote server does extended
return codes. The rest is up for grabs.
pgp9KL4F2M5Mn.pgp
Description: PGP signature
--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] logging outgoing messages
>Is there any interest in adding an improved version of this to the code >base? So ... maybe? But, some thoughts. - We don't, in general, want to have any more #ifdefs in the code unless they are completely unavoidable (e.g., operating system differences or optional third-party libraries like OpenSSL). So this would require some run-time configuration. - It is not clear to me that you can state with certainly that the 250 response code will contain the queue identifier (that is, in fact, not a concept that appears anywhere that I can find in the SMTP RFCs). As a practical matter I've never had to give anyone the queue identifier of a message (because it's not normally logged on the client; really, most people are happy with recipients and a time, and they are really happy if you have a message-id). But ... fine. I can imagine cases where it would be helpful. But there's no way we could accept this patch as-is, because it depends on the specific format of your ISP's response message. I tested out two email providers that I use and their formats differ, specifically: 250 2.0.0 Ok: queued as QUEUE-ID 250 2.0.0 QUEUE-ID Message accepted for delivery It looks like your ISP's format is "250 id=QUEUE-ID". So that's 3 servers and 3 different formats. I think this should be a lot more generic. So ... an alternate proposal. Right now we have the hooks interface (see doc/nmh/README-HOOKS). It should be relatively straightforward to create a "post hook" that could be invoked when email is submitted to a mail server. One of the arguments to the post hook could be the response message from the SMTP server. Another one of the arguments to the post hook could be the message draft so you could interrogate it with scan(1) to extract out anything useful you might want like the message-id. You could then use your favorite shell tools to process the SMTP server response and then send it to syslog with logger(1). I realize that would be a lot more work than the code you submitted, and we'd have to think about the arguments to the post hook, but I don't see any huge blockers other than WRITING the code. I am neutral about this being made to work with sendmail/pipe; it would actually be a lot of work to do that. We could just accept that it is one more thing that doesn't work with sendmail/pipe. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
[nmh-workers] logging outgoing messages
I recently modified my configuration for nmh-1.7.1 to connect directly to
my ISP's sendmail, rather than going through sendmail on my desktop Linux
system.
This works perfectly, but as a side effect I lost all logging of outgoing
messages. This isn't the end of the world, but it's a pain because there
are times when outgoing messages may need to be traced, and in cases like
that it's important to be able to quote the ISP's assigned QID.
...so I hacked the appended patch together, and it seems to work, but I'm
sure there must be a better way to do this. Problems with my code include
(but are almost certainly not limited to) the following:
- It doesn't check to see whether it's connecting to a local sendmail
(if so, each message will be logged twice).
- It logs successful delivery, but not delivery failures (no matter
the reason).
- It seems to break two of the tests in make check:
# make check
[...]
PASS: test/post/test-messageid
send: message not delivered to anyone
FAIL: test/post/test-mts
send: message not delivered to anyone
FAIL: test/post/test-post-aliases
PASS: test/post/test-post-basic
[...]
===
2 of 112 tests failed
Please report to nmh-workers@nongnu.org
===
Makefile:4763: recipe for target 'check-TESTS' failed
make[1]: *** [check-TESTS] Error 1
make[1]: Leaving directory '/big/local/pkg/nmh/nmh-1.7.1'
Makefile:5019: recipe for target 'check-am' failed
Is there any interest in adding an improved version of this to the code
base?
- Steven
8<- cut here >8
--- h/mts.h.original2017-11-22 09:37:56.0 -0500
+++ h/mts.h 2019-07-09 16:51:34.260314328 -0400
@@ -57,3 +57,15 @@
* Global MailDelivery File
*/
extern char *maildelivery;
+
+#ifndef NOSYSLOG
+# define SYSLOG_FIELD_SIZE 1024
+# define SYSLOG_FIELD_LAST 1023
+
+ char syslog_from [SYSLOG_FIELD_SIZE];
+ char syslog_to[SYSLOG_FIELD_SIZE];
+ char syslog_msgid [SYSLOG_FIELD_SIZE];
+ char syslog_server[SYSLOG_FIELD_SIZE];
+ char syslog_port [SYSLOG_FIELD_SIZE];
+ char syslog_qid [SYSLOG_FIELD_SIZE];
+#endif
--- mts/smtp/smtp.c.original2018-03-06 14:05:55.0 -0500
+++ mts/smtp/smtp.c 2019-07-09 17:17:34.382593987 -0400
@@ -13,6 +13,9 @@
#include
#include
+#ifndef NOSYSLOG
+ #include
+#endif
/*
* This module implements an interface to SendMail very similar
@@ -74,6 +77,24 @@
int debug, int sasl, const char *saslmech, const char *user,
const char *oauth_svc, int tls)
{
+#ifndef NOSYSLOG
+/** ensure fields are properly initialized to empty strings: **/
+
+syslog_from [0] = '\0';
+syslog_to [0] = '\0';
+syslog_msgid[0] = '\0';
+syslog_qid [0] = '\0';
+
+
+/** ...except server and port, which can take their real values: **/
+
+(void)strncpy(syslog_server, server, SYSLOG_FIELD_SIZE);
+syslog_server[SYSLOG_FIELD_LAST] = '\0';
+
+(void)strncpy(syslog_port, port, SYSLOG_FIELD_SIZE);
+syslog_port[SYSLOG_FIELD_LAST] = '\0';
+#endif
+
if (sm_mts == MTS_SMTP)
return smtp_init (client, server, port, watch, verbose,
debug, sasl, saslmech, user, oauth_svc, tls);
@@ -454,6 +475,13 @@
}
}
+#ifndef NOSYSLOG
+/** record from field for syslog: **/
+
+(void)strncpy(syslog_from, from, SYSLOG_FIELD_SIZE);
+syslog_from[SYSLOG_FIELD_LAST] = '\0';
+#endif
+
switch (smtalk (SM_MAIL, "MAIL FROM:<%s>%s", from, mail_parameters)) {
case 250:
sm_addrs = 0;
@@ -473,6 +501,37 @@
int
sm_wadr (char *mbox, char *host, char *path)
{
+#ifndef NOSYSLOG
+{
+ /** record recipient address for syslog: **/
+
+ char address[SYSLOG_FIELD_SIZE];
+ int length, used, remaining;
+
+ if (host && *host)
+ (void)snprintf(address, SYSLOG_FIELD_SIZE, "<%s%s@%s>",
+ FENDNULL(path), mbox, host);
+ else
+ (void)snprintf(address, SYSLOG_FIELD_SIZE, "<%s%s>",
+ FENDNULL(path), mbox);
+
+ length= strlen(address);
+ used = strlen(syslog_to);
+ remaining = (SYSLOG_FIELD_SIZE - used) - 1;
+
+ if ((used == 0) && (length < remaining))
+ {
+ (void)strcat(syslog_to, address);
+ }
+ else if ((length + 1) < remaining)
+ {
+ (void)strcat(syslog_to, " ");
+ (void)strcat(syslog_to, address);
+ }
+
+ syslog_to[SYSLOG_FIELD_LAST] = '\0';
+}
+#endif
switch (smtalk (SM_RCPT, host && *host ? "RCPT TO:<%s%s@%s>"
: "RCPT TO:<%s%s>",
FENDNULL(path), mbox, host)) {
@@ -717,6 +776,19 @@
Re: [nmh-workers] success using the OAUTH2 with gmail.
>> Let's say in a hypothetical future we support IMAP. That means that >> nearly every command would take a whole pile of arguments like >> -initialtls, -host, -port, -sasl, and more. Obviously changing your >> profile for every nmh command would be awful. So there should be some >> way of handling that. What I had thought maybe was tying profile >> entries to mailboxes, so if you did "scan my-imap-server:foo" it could >> possibly look in your profile and find: >> >> my-imap-server: -host my.server.com -port imap -tls -sasl >> -saslmech GSSAPI -user me > >That seems very specific and introduces a new colon operator that >restricts what's available for other features later. Well, it was just a strawman like I said, I'm not really sure about it yet. It seems a bit clumsy that you have to add that folder-specific config file to bind an IMAP folder, but maybe extending the folder(1) command would work? More thought is needed. >How about allowing an mh-profile(5) in a folder's directory with its >content having higher priority than the ancestor folders' .mh_profile >and the general ~/.mh_profile. This could be used for more general >things, e.g. the template used for replies to emails in that folder, or >the preferred format for scanning it. In that vein, when playing around with DavMail I found that it cached a lot of message headers with IMAP but not the message bodies, so a theoretical scan template which fetched the first part of the body of the message has lousy performance, so that's a case where you would want a folder-specific scan template. Something else to think about. >> You get the idea. But thinking about this more makes me think that we >> should extend this a bit so it's not tied to folders, but a generic >> connection profile defaults and we could provide ones that work with >> Gmail. I don't have it all jelled in my head how this would look and >> you'd need to do something to ADD to an existing connection profile so >> you could supply your own username, for example. But it seems like it >> should be doable. But I guess my idea is that you should be able to >> do something like >> >> inc -conn gmail -user myu...@gmail.com >> >> and the right stuff should happen. Make sense? > >If `foo: -bar xyzzy...' is in an .mh_profile then often, depending on >the value's complexity, it can be interpolated with «`mhparam foo`» in >the shell. What if a similar capability existed on the value side of an >.mh_profile's `key: value'. Except it could cope with the interpolated >value having quotes. > >This would allow collections of options to be defined and then >referenced with a shorthand. Either back quotes copied from sh(1), or >`-use foo' so it can work easily at the shell too. Well ... I see where you are going. But ... It occurs to me that one problem with our configuration, especially for newer users, is that it's all spread out between different tools. For example, your SMTP servers settings are in send(1) (and if you want to be complete, probably whom(1) as well), POP server settings belong in inc(1) and msgchk(1), and with a theoretical IMAP server it would be a lot of tools. This is confusing. When users are given information about the settings for an email provider, they are usually given them all at once. "Here's the SMTP server, here's the POP server, here's the IMAP server", etc etc. So if we had some facility where we could say "Here's where you put the block of network settings for GMail", for example, and then we could have people just refer to the "gmail" settings, that would be a huge win. Making those things easily be extractable via mhparam would also be a huge win. Again, I'm not sure what all of this would look like right now. This is just idle musings. --Ken -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [nmh-workers] Ordinal Indicators.
Hi Valdis, > > Is -42nd handled? > > I admit being totally mystified as to what situations require proper > handling of negative ordinals Well, from here the one after next is the 2nd, and the one before last is the -2nd. Regardless, the code and documentation should match, and it seems easiest to fix the code rather than explain the exceptions in the man page. IIRC that was the impetus behind McIlroy adding the BUGS section; his peers found it simpler to fix than document. -- Cheers, Ralph. -- nmh-workers https://lists.nongnu.org/mailman/listinfo/nmh-workers
