RE: How to terminate smtpd filters?
I should have included more information. OS is FreeBSD 13.2. smtpd is 7.3.0
which is the latest in the pkg collection.
Originally, the only change I made to the filter, was add
#!/usr/local/bin/python3.9 and made afilter.py executable.
All other filters terminate correctly when smtpd exits.
filter afilter proc-exec "/data/afilter.py"
listen on bridge20 port smtp tls pki mail.tekrealm.net auth-optional
filter { afilter, trusted, check_dyndns, check_rdns,
check_fcrdns, senderscore, rspamd }
listen on bridge20 port smtps smtps pki mail.tekrealm.net auth
mask-src filter { afilter, rspamd }
action "outbound" relay helo mail.tekrealm.net filter "afilter"
Ultimately, I was able to get it to exit correctly after the confirmation that
it should exit when stdin closes.
I will submit a patch to the github repo after I beat on this for a few days or
so.
Thank you for your assistance.
-Andrew
-Original Message-
From: Tassilo Philipp
Sent: Tuesday, April 2, 2024 1:22 AM
To: gil...@poolp.org
Cc: and...@tekrealm.net; misc@opensmtpd.org
Subject: Re: How to terminate smtpd filters?
I agree with Gilles, your filter should react on stdin closing, but not sure
how your filter is set up.
Also, just a guess... are you running smtpd on Linux?
Linux doesn't kill children when the parent process dies, maybe that's related?
(To make it do that prctl(2) would need to be used w/
PR_SET_PDEATHSIG.)
On Tue, Apr 02, 2024 at 06:53:39AM +, gil...@poolp.org wrote:
> April 2, 2024 4:47 AM, and...@tekrealm.net wrote:
>
>> What signals a termination for smtpd filters?
>>
>> I'm using the code at
>> https://github.com/Beutlin/howto-opensmtpd-filters-and-reports,
>> Which works great, except for when smtpd gets shutdown. The script
>> continues to run and consumes up to 100% cpu time, while keeping the
>> smtpd parent? process running.
>>
>> I've tried adding a SIGTERM handler to the code which didn't work, as
>> well as I saw mentioned that the filter should exit on EOF, so I
>> tried wrapping
>> parser.dispatch() in
>> a try/except EOFError and using sys.exit. That didn't work either.
>>
>> I've read smtpd-filters, and looked at various other filters and I am
>> not understanding what tells the filter to shutdown.
>>
>
> The filter is connected to smtpd through its stdin, so it can
> terminate when there's an EOF on stdin.
>
> This is the proper way to do it and how all filters I wrote work but
> maybe a bug has crawled in the handling of filter termination and it
> went unnoticed, I don´t think I ever terminated smtpd in years besides system
> restarts.
>
> What system are you running on ?
>
[PATCH] DSNs to follow RFCs more closely
Hi,
Find below the first stab at a final patch making generated bounce
mails follow more closely the RFCs 3461, 3464 and 6522. The patch
includes the patch from my previous mail in this thread.
The patch is based on opensmtp 7.5.0rc1 (w/ the additional patch in
the rc1 announcement thread on this ml).
It does, in short:
- uses the correct content type(s) for DSNs, as specified in RFC3464 and
RFC6522
- adds xtext_decode() utility func (following the style/interface of the
base64_decode() utility func)
- extends the data structures of the bounce message and envelope, as
necesary, to write the required fields outlined by RFC3461 and RFC3464
into the machine readable sub mime part, namely, per-message field:
* Original-Envelope-Id: (required if ENVID= was used, see RFC3461)
and per recipiend fields:
* Original-Recipient: (required if ORCPT= was used, see RFC3461)
* Arrival-Date: (optional, but added it while at it...)
* Diagnostic-Code: (optional, but added it while at it...)
- adds a check whether the SMTP ENVID= param is valid xtext (RFC3461)
- minor: removes a space in already set Final-Recipient: field
(following more closely the example layout from RFC3464)
One thing I wasn't sure about what to do is *where* to handle the xtext
decoding of the ENVID= and ORCPT= SMTP params. Right now the SMTP
handling part just considers them "opaque text", as they are just handed
down the relays, unmodified. When copying those into a DSN, they must be
decoded, though, and the decoded result is considered being printable
US-ASCII. From RFC3461 4.2 about ORCPT:
`Due to limitations in the Delivery Status Notification format, the
value of the original recipient address prior to encoding as "xtext"
MUST consist entirely of printable (graphic and white space)
characters from the US-ASCII repertoire.`
A functional identical wording is used in 4.4 for the ENVID= param.
The patch below keeps the SMTP behaviour to just treat those params as
opaque strings, but checks if the decoded part is US-ASCII when
generating a bounce - and if it's not, doesn't write it to the
respective DSN field(s).
I think the SMTP handling might be the better place to check for that,
and refuse such SMTP params outright, maybe?
Thanks
--- ./usr.sbin/smtpd/bounce.c
+++ ./usr.sbin/smtpd/bounce.c
@@ -57,6 +57,9 @@
char*report;
uint8_t esc_class;
uint8_t esc_code;
+ char*errorline;
+ /* decoded xtext SMTP ORCPT param, len always <= encoded */
+ char orcpt[DSN_ORCPT_LEN+1];
};
struct bounce_message {
@@ -68,6 +71,9 @@
char*to;
time_t timeout;
TAILQ_HEAD(, bounce_envelope)envelopes;
+ time_t creation;
+ /* decoded xtext SMTP ENVID param, len always <= encoded */
+ char envid[DSN_ENVID_LEN+1];
};
struct bounce_session {
@@ -120,6 +126,18 @@
}
}
+static int
+count_printable_ascii_chars(const char *s)
+{
+ const char *p = s;
+ for (; *p; ++p) {
+ if (!isprint((unsigned char)*p))
+ break;
+ }
+ return p - s;
+}
+
+
void
bounce_add(uint64_t evpid)
{
@@ -127,6 +145,7 @@
struct envelope evp;
struct bounce_messagekey, *msg;
struct bounce_envelope *be;
+ int n;
bounce_init();
@@ -164,6 +183,14 @@
msg->msgid = key.msgid;
msg->bounce = key.bounce;
+ msg->creation = evp.creation;
+ n = xtext_decode(evp.dsn_envid, msg->envid, sizeof(msg->envid));
+ if (n == -1 || n != count_printable_ascii_chars(msg->envid)) {
+ msg->envid[0] = '\0';
+ log_debug("debug: bounce: ill-formed ENVID param (decoded
xtext not "
+ "printable ascii, not setting DSN Original-Envelope-Id:
header");
+ }
+
TAILQ_INIT(>envelopes);
msg->smtpname = xstrdup(evp.smtpname);
@@ -187,11 +214,18 @@
be = xmalloc(sizeof *be);
be->id = evpid;
be->report = xstrdup(buf);
+ be->errorline = xstrdup(evp.errorline);
(void)strlcpy(be->dest.user, evp.dest.user, sizeof(be->dest.user));
(void)strlcpy(be->dest.domain, evp.dest.domain,
sizeof(be->dest.domain));
be->esc_class = evp.esc_class;
be->esc_code = evp.esc_code;
+ n = xtext_decode(evp.dsn_orcpt, be->orcpt, sizeof(be->orcpt));
+ if (n == -1 || n != count_printable_ascii_chars(be->orcpt)) {
+ be->orcpt[0] = '\0';
+ log_debug("debug: bounce: ill-formed ORCPT param (decoded xtext not
"
+ "printable ascii, not setting DSN
Re: [PATCH] DSNs to follow more closely RFCs
April 2, 2024 6:00 PM, "Tassilo Philipp" wrote: > Hi, > > Find attached the first stab at a final patch making generated bounce mails > follow more closely the > RFCs 3461, 3464 and 6522. The attached file includes the patch from my > previous mail in this > thread. > > The patch is based on opensmtp 7.5.0rc1 (w/ the additional patch in the rc1 > announcement thread on > this ml). > > It does, in short: > > - uses the correct content type(s) for DSNs, as specified in RFC3464 and > RFC6522 > > - adds xtext_decode() utility func (following the style/interface of the > base64_decode() utility func) > > - extends the data structures of the bounce message and envelope, as > necesary, to write the required fields outlined by RFC3461 and RFC3464 > into the machine readable sub mime part, namely, per-message field: > > * Original-Envelope-Id: (required if ENVID= was used, see RFC3461) > > and per recipiend fields: > > * Original-Recipient: (required if ORCPT= was used, see RFC3461) > * Arrival-Date: (optional, but added it while at it...) > * Diagnostic-Code: (optional, but added it while at it...) > > - adds a check whether the SMTP ENVID= param is valid xtext (RFC3461) > > - minor: removes a space in already set Final-Recipient: field > (following more closely the example layout from RFC3464) > > One thing I wasn't sure about what to do is *where* to handle the xtext > decoding of the ENVID= and > ORCPT= SMTP params. Right now the SMTP handling part just considers them > "opaque text", as they are > just handed down the relays, unmodified. When copying those into a DSN, they > must be decoded, > though, and the decoded result is considered being printable US-ASCII. >From > RFC3461 4.2 about > ORCPT: > > `Due to limitations in the Delivery Status Notification format, the > value of the original recipient address prior to encoding as "xtext" > MUST consist entirely of printable (graphic and white space) > characters from the US-ASCII repertoire.` > > A functional identical wording is used in 4.4 for the ENVID= param. > > The attached patch keeps the SMTP behaviour to just treat those params as > opaque strings, but > checks if the decoded part is US-ASCII when generating a bounce - and if it's > not, doesn't write it > to the respective DSN field(s). > > I think the SMTP handling might be the better place to check for that, and > refuse such SMTP params > outright, maybe? > > Thanks > Can you please inline the diff to your mail ?
[PATCH] DSNs to follow more closely RFCs
Hi, Find attached the first stab at a final patch making generated bounce mails follow more closely the RFCs 3461, 3464 and 6522. The attached file includes the patch from my previous mail in this thread. The patch is based on opensmtp 7.5.0rc1 (w/ the additional patch in the rc1 announcement thread on this ml). It does, in short: - uses the correct content type(s) for DSNs, as specified in RFC3464 and RFC6522 - adds xtext_decode() utility func (following the style/interface of the base64_decode() utility func) - extends the data structures of the bounce message and envelope, as necesary, to write the required fields outlined by RFC3461 and RFC3464 into the machine readable sub mime part, namely, per-message field: * Original-Envelope-Id: (required if ENVID= was used, see RFC3461) and per recipiend fields: * Original-Recipient: (required if ORCPT= was used, see RFC3461) * Arrival-Date: (optional, but added it while at it...) * Diagnostic-Code: (optional, but added it while at it...) - adds a check whether the SMTP ENVID= param is valid xtext (RFC3461) - minor: removes a space in already set Final-Recipient: field (following more closely the example layout from RFC3464) One thing I wasn't sure about what to do is *where* to handle the xtext decoding of the ENVID= and ORCPT= SMTP params. Right now the SMTP handling part just considers them "opaque text", as they are just handed down the relays, unmodified. When copying those into a DSN, they must be decoded, though, and the decoded result is considered being printable US-ASCII. From RFC3461 4.2 about ORCPT: `Due to limitations in the Delivery Status Notification format, the value of the original recipient address prior to encoding as "xtext" MUST consist entirely of printable (graphic and white space) characters from the US-ASCII repertoire.` A functional identical wording is used in 4.4 for the ENVID= param. The attached patch keeps the SMTP behaviour to just treat those params as opaque strings, but checks if the decoded part is US-ASCII when generating a bounce - and if it's not, doesn't write it to the respective DSN field(s). I think the SMTP handling might be the better place to check for that, and refuse such SMTP params outright, maybe? Thanks On Tue, Mar 19, 2024 at 04:22:13PM +0100, Tassilo Philipp wrote: Alright, find attached a first patch, fixing up some content-type headers, as outlined by RFC3464 and RFC6522 - in detail: The patch's first hunk follows RFC3464, which specifies that a DSN should have a top-level type of "multipart/report" with a parameter "report-type=delivery-status"; the actual format is described in RFC6522, and is specified having two or three sub mime parts, a human readable message, a machine parsable part with content-type "message/delivery-status" and an optional 3rd part of either "text/rfc822-headers" or "message/rfc822". The latter is part of the second hunk of the patch, b/c "text/rfc822-headers" was used in any case, even if the full message was returned in the DSN. Also, the end of the second hunk removes a check, which I think is unintuitive to begin with, but unsure. Basically, it only returned the headers for NOTIFY=SUCCESS DSNs when RET=HDRS was also set (explicitly or implicitly). Here's the section from RFC3461: When the value of the RET parameter is FULL, the full message SHOULD be returned for any DSN which conveys notification of delivery failure. (However, if the length of the message is greater than some implementation-specified length, the MTA MAY return only the headers even if the RET parameter specified FULL.) If a DSN contains no notifications of delivery failure, the MTA SHOULD return only the headers. The original code does what the last line implies, but only when RET=HDRS is set, meaning it ignores it for anything but NOTIFY=SUCCESS, and returns always the full message. I'm not sure what would be best to do here, all in all the RFC says 'SHOULD'... either way, the check there should reflect the same logic used in the if...else... block of the second hunk, which is what the patch shoots for. I chose to keep it simple and simply do what RET= asks for. So, if the original logic is kept, the second hunk's if...else... block would need an additional "s->msg->bounce.type == B_DELIVERED" check. I tested the patch with different combinations of the RET= and NOTIFY= parameters, and so far it seems to work fine. PS: I'm working on related other patches, at the moment, which I'll mail when they are done, namely adding a "Original-Envelope-ID" header in the DSN when the ENVID= param was present, as well as "Original-Recipient" for any ORCPT= param. Those are specified in RFC3461 section 6.3., but they are a bit more involved as the params need to be decoded and recoded, and I just haven't found the time, yet. Thanks! On Wed, Mar 13, 2024 at 12:04:24PM +0100, Tassilo
Re: How to terminate smtpd filters?
I agree with Gilles, your filter should react on stdin closing, but not sure how your filter is set up. Also, just a guess... are you running smtpd on Linux? Linux doesn't kill children when the parent process dies, maybe that's related? (To make it do that prctl(2) would need to be used w/ PR_SET_PDEATHSIG.) On Tue, Apr 02, 2024 at 06:53:39AM +, gil...@poolp.org wrote: April 2, 2024 4:47 AM, and...@tekrealm.net wrote: What signals a termination for smtpd filters? I'm using the code at https://github.com/Beutlin/howto-opensmtpd-filters-and-reports, Which works great, except for when smtpd gets shutdown. The script continues to run and consumes up to 100% cpu time, while keeping the smtpd parent? process running. I've tried adding a SIGTERM handler to the code which didn't work, as well as I saw mentioned that the filter should exit on EOF, so I tried wrapping parser.dispatch() in a try/except EOFError and using sys.exit. That didn't work either. I've read smtpd-filters, and looked at various other filters and I am not understanding what tells the filter to shutdown. The filter is connected to smtpd through its stdin, so it can terminate when there's an EOF on stdin. This is the proper way to do it and how all filters I wrote work but maybe a bug has crawled in the handling of filter termination and it went unnoticed, I don´t think I ever terminated smtpd in years besides system restarts. What system are you running on ?
Re: How to terminate smtpd filters?
April 2, 2024 4:47 AM, and...@tekrealm.net wrote: > What signals a termination for smtpd filters? > > I'm using the code at > https://github.com/Beutlin/howto-opensmtpd-filters-and-reports, > Which works great, except for when smtpd gets shutdown. The script continues > to run and > consumes up to 100% cpu time, while keeping the smtpd parent? process > running. > > I've tried adding a SIGTERM handler to the code which didn't work, as well > as I saw > mentioned that the filter should exit on EOF, so I tried wrapping > parser.dispatch() in > a try/except EOFError and using sys.exit. That didn't work either. > > I've read smtpd-filters, and looked at various other filters and I am not > understanding > what tells the filter to shutdown. > The filter is connected to smtpd through its stdin, so it can terminate when there's an EOF on stdin. This is the proper way to do it and how all filters I wrote work but maybe a bug has crawled in the handling of filter termination and it went unnoticed, I don´t think I ever terminated smtpd in years besides system restarts. What system are you running on ?
