[pfx] Re: Milter own Postfix-prepended Received
Matus UHLAR - fantomas via Postfix-users: > >> > Bill Cole via Postfix-users escribi? el 11/12/2023 a las 15:31: > >> >> On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) > >> >> Carlos Velasco via Postfix-users > >> >> is rumored to have said: > >> >> [...] > >> >>> And doing the same work in 2 different places can be called software > >> >>> efficiency? > >> >> No, but the "fix" here would be a divergence from how Milter has > >> >> worked > >> >> since it was created and semi-documented by Sendmail Inc. It is de > >> >> facto > >> >> controlled by the current developers of Sendmail, but I don't believe > >> >> anyone is working to make Milter better, at least not in ways that > >> >> would > >> >> break compatibility. > > >> On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) > >> Carlos Velasco via Postfix-users > >> is rumored to have said: > >> > No one is talking here about breaking any compatibility, re-read the > >> > messages. > > >Bill Cole via Postfix-users: > >> What did I miss? Are you not asking for Postfix to support providing > >> milters with a header that none of them expect and which no other Milter > >> implementation supports? > > On 11.12.23 10:31, Wietse Venema via Postfix-users wrote: > >He asked to make this configurable. I declined because the human > >cost (of having two incompatible ways to convey the connection info) > >would in my opinion exceed the gain from saving a few machine cycles. I was referring to a request to make the Postfix-generated Received: header available to Milters, so that they would not have to generate a fake Received: header from information that the Milter receives with the smfi_connect() callback. If the OP made requests other than fixing the inability to replace or remove his prepended Received-SPF header, then I could not discern that in the chaotic discussion. > if application called from milter was able to distinguish between > headers added locally (thus trusted) and headers received from the > network (untrusted), it could effectively use the locally added > headers. See "I was referring..." above. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
> Bill Cole via Postfix-users escribi? el 11/12/2023 a las 15:31: >> On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) >> Carlos Velasco via Postfix-users >> is rumored to have said: >> [...] >>> And doing the same work in 2 different places can be called software >>> efficiency? >> No, but the "fix" here would be a divergence from how Milter has >> worked >> since it was created and semi-documented by Sendmail Inc. It is de >> facto >> controlled by the current developers of Sendmail, but I don't believe >> anyone is working to make Milter better, at least not in ways that >> would >> break compatibility. On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) Carlos Velasco via Postfix-users is rumored to have said: > No one is talking here about breaking any compatibility, re-read the > messages. Bill Cole via Postfix-users: What did I miss? Are you not asking for Postfix to support providing milters with a header that none of them expect and which no other Milter implementation supports? On 11.12.23 10:31, Wietse Venema via Postfix-users wrote: He asked to make this configurable. I declined because the human cost (of having two incompatible ways to convey the connection info) would in my opinion exceed the gain from saving a few machine cycles. if application called from milter was able to distinguish between headers added locally (thus trusted) and headers received from the network (untrusted), it could effectively use the locally added headers. SpamAssassin trusts all headers before locally added Received:, however spamass-milter and amavisd-milter add it as first header, thus all other headers added by local milters (spf,dkim,arc,dmarc...) are not trusted. Unless the protocol supports providing this information, ability to see Received: header at proper place would increase SA effectiveness, which is I believe what OP asks for. And I would be happy as well. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. REALITY.SYS corrupted. Press any key to reboot Universe. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: And doing the same work in 2 different places can be called software efficiency? Bill Cole via Postfix-users escribió el 11/12/2023 a las 15:31: since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. On 11.12.23 15:37, Carlos Velasco via Postfix-users wrote: No one is talking here about breaking any compatibility, re-read the messages. _You_ have complained why Received: is not seen by milter, here: https://marc.info/?l=postfix-users=170223488205099=2 The answer has been given and documented: this is how milter works from the beginning. However, I agree that this makes work hard for SpamAssassin, because this way SA does not know, which adders have been added by local milters/policy servers and thus can be trusted - SA only trusts headers before locally added Received: ... headers added by spf,dkim,dmarc milters would be very useful for SA processing. So, it would be great if postfix could optionally add (or, better, not remove) locally added Received: header, although milters would need to implement this feature first. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. "The box said 'Requires Windows 95 or better', so I bought a Macintosh". ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Bill Cole via Postfix-users: > On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) > Carlos Velasco via Postfix-users > is rumored to have said: > > > Bill Cole via Postfix-users escribi? el 11/12/2023 a las 15:31: > >> On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) > >> Carlos Velasco via Postfix-users > >> is rumored to have said: > >> [...] > >>> And doing the same work in 2 different places can be called software > >>> efficiency? > >> No, but the "fix" here would be a divergence from how Milter has > >> worked > >> since it was created and semi-documented by Sendmail Inc. It is de > >> facto > >> controlled by the current developers of Sendmail, but I don't believe > >> anyone is working to make Milter better, at least not in ways that > >> would > >> break compatibility. > > No one is talking here about breaking any compatibility, re-read the > > messages. > > What did I miss? Are you not asking for Postfix to support providing > milters with a header that none of them expect and which no other Milter > implementation supports? He asked to make this configurable. I declined because the human cost (of having two incompatible ways to convey the connection info) would in my opinion exceed the gain from saving a few machine cycles. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) Carlos Velasco via Postfix-users is rumored to have said: Bill Cole via Postfix-users escribió el 11/12/2023 a las 15:31: On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: [...] And doing the same work in 2 different places can be called software efficiency? No, but the "fix" here would be a divergence from how Milter has worked since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. No one is talking here about breaking any compatibility, re-read the messages. What did I miss? Are you not asking for Postfix to support providing milters with a header that none of them expect and which no other Milter implementation supports? That seems to me like a compatibility break. Or you could call it a unique extension to a protocol which lacks a defined extension mechanism or even a formal standardization. Postfix would be doing something with a shared (if not exactly open...) de facto standard that is effectively controlled by the Sendmail project. What could go wrong? Beyond that, I don't think that your justifications for such an addition have made any sense. Milters can ask for any available macros needed explicitly, avoiding the need to parse a Received header in the milter itself. The only reason SpamAssassin requires a synthetic header is that it has no interface for passing the known untainted values of the parameters it wants to use. SA knows nothing about the milter API or "Sendmail Macros" or which MTA+milter layer you are using, so if it needs that it can only look in a header built in the milter e.g, as logged by the MD instance here (which logs excessively...): Dec 11 09:31:59 shiny mimedefang.pl[13203]: 4Spkhs2mXtzXcP2f: Macros are mail_mailer mail_addr daemon_port client_port mail_host auth_authen auth_type j cipher _ client_connections daemon_name tls_version cipher_bits From those macros, MD constructed this Received header for handoff to SpamAssassin: Received: from skinnyclam.scconsult.com (skinnyclam.scconsult.com [192.168.254.125]) by shiny.scconsult.com (envelope-sender ) (MIMEDefang) with ESMTPSA id 4Spkhs2mXtzXcP2f for ; Mon, 11 Dec 2023 09:31:59 -0500 SpamAssassin can reliably and usefully parse out of that header the fact that the message arrived with authentication over an encrypted session. If one had a particular need for some other defined macro one could request it in the milter and stuff it into a header as appropriate. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Bill Cole via Postfix-users escribió el 11/12/2023 a las 15:31: On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: [...] And doing the same work in 2 different places can be called software efficiency? No, but the "fix" here would be a divergence from how Milter has worked since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. No one is talking here about breaking any compatibility, re-read the messages. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: [...] And doing the same work in 2 different places can be called software efficiency? No, but the "fix" here would be a divergence from how Milter has worked since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-10 at 16:16:27 UTC-0500 (Sun, 10 Dec 2023 22:16:27 +0100) Carlos Velasco via Postfix-users is rumored to have said: Wietse Venema via Postfix-users escribió el 10/12/2023 a las 21:53: 2. the "own Received" header not being passed to milter. I understand it works as expected, but could be great to be able to decide this with a configuration parameter. That is because every Milter in the real world gets the client info from the smfi_connect() callback function and from Milter macros, instead of parsing Received: headers. That statement is absolutely false. Quite a bold choice there... Many milters, like amavisd, use other filters for mail content inspection, like SpamAssassin. Spamassassin uses the "Received" headers for some tasks, like this: https://cwiki.apache.org/confluence/display/spamassassin/TrustPath# As a contributor & PMC member for both SpamAssassin (definitely not a milter) and MIMEDefang (definitely a milter, which uses SpamAssassin) I assure you that the Received header which notionally reflects the current transaction DOES NOT exist when the milter gets the message data, and for SA to do its "trust path" audit of Received headers, the MIMEDefang milter MUST construct one for SA to use. Amavis is slightly different because it can be used as a SMTP proxy (between 2 Postfix smtpd processes) or a milter. When used as a proxy, it sees the whole message with the locally-added Received headers. When used as a milter, it works with what Postfix provides it (the relevant macros) to construct a Received header for SA. All of this is documented accurately. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Wietse: > I asked for a copy of the (headers of the) resulting message that > Postfix delivers. > - Does it have a Received-SPF header? > - Does it have two? Carlos Velasco: > 1. Deleting the header in the milter or doing nothing in the milter > has the same result: final email has only 1 Received-SPF header. > > Received-SPF: Pass (mailfrom) identity=mailfrom; >client-ip=2620:137:e000::1:20; helo=out1.vger.email; >envelope-from=linux-kernel-ow...@vger.kernel.org; receiver= > Received: from out1.vger.email (out1.vger.email >[IPv6:2620:137:e000::1:20]) <-- *this is the own Received generated by > postfix* Thank you for showing concrete information. It seems that the order of these headers is the problem. The Postfix Milter implementation expects that the header order looks like: Received: from out1.vger.email ... (Postfix-generated) Received-SPF: (and other Postfix-generated headers) ...headers from remote SMTP client... ...date/message-id/etc. headers that Postfix might add to an original submission.. Unfortunatelty the PREPEND action will put the Received-SPF header BEFORE the Postfix-generated Received: header. And since the Postfix Milter implementation hides the first header, a Milter cannot delete that Received-SPF: header. I'll do some tests to confirm that this is the problem. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Wietse Venema via Postfix-users escribió el 10/12/2023 a las 22:54:
Carlos Velasco via Postfix-users:
Wietse Venema via Postfix-users escribi? el 10/12/2023 a las 21:53:
Carlos Velasco via Postfix-users:
Wietse Venema via Postfix-users escribi? el 10/12/2023 a las 19:44:
Carlos Velasco via Postfix-users:
*** And there is the milter, is custom made ***
You need to reduce complexity.
- If you remove the Milter, is the header still duplicated?
No.
Duplication only happens when, in milter, I delete that header and readd it
again.
Obviously, you are not deleting that header!
How do you know that the header is readlly deleted? Can you show a
message that the header is removed when you aren't adding that
header with the Milter?
Obviously, I am deleting that header, like all others!
header received in the header callback:
milter[548374]: header: Received-SPF: Pass (mailfrom) identity=mailfrom;
client-ip=2620:137:e000::1:20; helo=out1.vger.email;
envelope-from=linux-kernel-ow...@vger.kernel.org; receiver=
Header removed in the eom callback, NO error reported:
postfix/cleanup[563179]: reply: SMFIR_CHGHEADER data 18 bytes
postfix/cleanup[563179]: cleanup_del_header: 1 "Received-SPF"
postfix/cleanup[563179]: cleanup_find_header_start: index 1 name "Received-SPF"
postfix/cleanup[563179]: cleanup_find_header_start: index 1 name Received-SPF
type 78 offset -1
I asked for a copy of the (headers of the) resulting message that
Postfix delivers.
- Does it have a Received-SPF header?
- Does it have two?
1. Deleting the header in the milter or doing nothing in the milter has the
same result: final email has only 1 Received-SPF header.
2. Deleting the header in the milter and re-adding it result: final email gets
the 2 Received-SPF headers, but in different positions.
In 1:
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2620:137:e000::1:20;
helo=out1.vger.email; envelope-from=linux-kernel-ow...@vger.kernel.org;
receiver=
Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) <--
*this is the own Received generated by postfix*
by server.domain.com (ESMTP Server) with ESMTP id A754FC0087
for ; Sun, 10 Dec 2023 23:03:11 +0100 (CET)
*** Rest of headers seen in milter ***
In 2:
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2620:137:e000::1:20;
helo=out1.vger.email; envelope-from=linux-kernel-ow...@vger.kernel.org;
receiver=
Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) <--
*this is the own Received generated by postfix*
by server.domain.com (ESMTP Server) with ESMTP id A754FC0087
for ; Sun, 10 Dec 2023 23:03:11 +0100 (CET)
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2620:137:e000::1:20;
helo=out1.vger.email; envelope-from=linux-kernel-ow...@vger.kernel.org;
receiver= <-- *repeated*
*** Rest of headers seen in milter ***
So, my understanding is that the policyd generated header is somehow out of the
scope in the milter, although it shows up as a normal header.
I have found this in some milter documentation, so I assume there is not error
control in the protocol:
"Note that the status returned indicates only whether or not the filter's message
was successfully sent to the MTA, not whether or not the MTA performed the requested
operation"
Test. Deleting a non-existant header produces the same logs and no errors:
postfix/cleanup[569527]: reply: SMFIR_CHGHEADER data 15 bytes
postfix/cleanup[569527]: cleanup_del_header: 1 "qwrrewwre"
postfix/cleanup[569527]: cleanup_find_header_start: index 1 name "qwrrewwre"
postfix/cleanup[569527]: cleanup_find_header_start: index 1 name qwrrewwre type
78 offset -1
BTW, The Postfix SMFIR_CHGHEADER implementation will delete a header
when the caller specifies an empty header value. That must be for some
compatibility reason.
I'm doing this correct. I can delete other headers without problem, *including the own
Received header*, using chgheader("Received", 1, NULL). And this one really
should be really out of scope.
Regards,
Carlos Velasco
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Carlos Velasco via Postfix-users: > > Wietse Venema via Postfix-users escribi? el 10/12/2023 a las 21:53: > > Carlos Velasco via Postfix-users: > >> Wietse Venema via Postfix-users escribi? el 10/12/2023 a las 19:44: > >>> Carlos Velasco via Postfix-users: > *** And there is the milter, is custom made *** > >>> You need to reduce complexity. > >>> > >>> - If you remove the Milter, is the header still duplicated? > >> No. > >> Duplication only happens when, in milter, I delete that header and readd > >> it again. > > Obviously, you are not deleting that header! > > > > How do you know that the header is readlly deleted? Can you show a > > message that the header is removed when you aren't adding that > > header with the Milter? > Obviously, I am deleting that header, like all others! > > header received in the header callback: > milter[548374]: header: Received-SPF: Pass (mailfrom) identity=mailfrom; > client-ip=2620:137:e000::1:20; helo=out1.vger.email; > envelope-from=linux-kernel-ow...@vger.kernel.org; receiver= > > Header removed in the eom callback, NO error reported: > postfix/cleanup[563179]: reply: SMFIR_CHGHEADER data 18 bytes > postfix/cleanup[563179]: cleanup_del_header: 1 "Received-SPF" > postfix/cleanup[563179]: cleanup_find_header_start: index 1 name > "Received-SPF" > postfix/cleanup[563179]: cleanup_find_header_start: index 1 name Received-SPF > type 78 offset -1 I asked for a copy of the (headers of the) resulting message that Postfix delivers. - Does it have a Received-SPF header? - Does it have two? BTW, The Postfix SMFIR_CHGHEADER implementation will delete a header when the caller specifies an empty header value. That must be for some compatibility reason. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Jaroslaw Rafa via Postfix-users escribió el 10/12/2023 a las 22:32: Dnia 10.12.2023 o godz. 22:16:27 Carlos Velasco via Postfix-users pisze: That is because every Milter in the real world gets the client info >from the smfi_connect() callback function and from Milter macros, instead of parsing Received: headers. That statement is absolutely false. Many milters, like amavisd, use other filters for mail content inspection, like SpamAssassin. Spamassassin uses the "Received" headers for some tasks, like this: That's the reason why milters that are specifically meant to call spamassassin, like spamass-milter, are regenerating this header on their own (using information from smfi_connect()) and pass it to spamassassin. And doing the same work in 2 different places can be called software efficiency? Regards, Carlos Velasco___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Dnia 10.12.2023 o godz. 22:16:27 Carlos Velasco via Postfix-users pisze: > > >That is because every Milter in the real world gets the client info > >from the smfi_connect() callback function and from Milter macros, > >instead of parsing Received: headers. > That statement is absolutely false. > Many milters, like amavisd, use other filters for mail content inspection, > like SpamAssassin. Spamassassin uses the "Received" headers for some > tasks, like this: That's the reason why milters that are specifically meant to call spamassassin, like spamass-milter, are regenerating this header on their own (using information from smfi_connect()) and pass it to spamassassin. -- Regards, Jaroslaw Rafa r...@rafa.eu.org -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub." ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Wietse Venema via Postfix-users escribió el 10/12/2023 a las 21:53: Carlos Velasco via Postfix-users: Wietse Venema via Postfix-users escribi? el 10/12/2023 a las 19:44: Carlos Velasco via Postfix-users: *** And there is the milter, is custom made *** You need to reduce complexity. - If you remove the Milter, is the header still duplicated? No. Duplication only happens when, in milter, I delete that header and readd it again. Obviously, you are not deleting that header! How do you know that the header is readlly deleted? Can you show a message that the header is removed when you aren't adding that header with the Milter? Obviously, I am deleting that header, like all others! header received in the header callback: milter[548374]: header: Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=linux-kernel-ow...@vger.kernel.org; receiver= Header removed in the eom callback, NO error reported: postfix/cleanup[563179]: reply: SMFIR_CHGHEADER data 18 bytes postfix/cleanup[563179]: cleanup_del_header: 1 "Received-SPF" postfix/cleanup[563179]: cleanup_find_header_start: index 1 name "Received-SPF" postfix/cleanup[563179]: cleanup_find_header_start: index 1 name Received-SPF type 78 offset -1 2. the "own Received" header not being passed to milter. I understand it works as expected, but could be great to be able to decide this with a configuration parameter. That is because every Milter in the real world gets the client info from the smfi_connect() callback function and from Milter macros, instead of parsing Received: headers. That statement is absolutely false. Many milters, like amavisd, use other filters for mail content inspection, like SpamAssassin. Spamassassin uses the "Received" headers for some tasks, like this: https://cwiki.apache.org/confluence/display/spamassassin/TrustPath# Regards, Carlos Velasco ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Carlos Velasco via Postfix-users: > Wietse Venema via Postfix-users escribi? el 10/12/2023 a las 19:44: > > Carlos Velasco via Postfix-users: > >> *** And there is the milter, is custom made *** > > You need to reduce complexity. > > > > - If you remove the Milter, is the header still duplicated? > No. > Duplication only happens when, in milter, I delete that header and readd it > again. Obviously, you are not deleting that header! How do you know that the header is readlly deleted? Can you show a message that the header is removed when you aren't adding that header with the Milter? > 2. the "own Received" header not being passed to milter. I understand > it works as expected, but could be great to be able to decide this > with a configuration parameter. That is because every Milter in the real world gets the client info from the smfi_connect() callback function and from Milter macros, instead of parsing Received: headers. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Wietse Venema via Postfix-users escribió el 10/12/2023 a las 19:44:
Carlos Velasco via Postfix-users:
*** And there is the milter, is custom made ***
You need to reduce complexity.
- If you remove the Milter, is the header still duplicated?
No.
Duplication only happens when, in milter, I delete that header and readd it
again.
- If you keep the milter and rmeove the polocy lookup, is eom other
header duplicated?
Mmm no. I don't quite understand what you mean by this test, but no.
The only one that generates that header is the policy daemon, if it is not
executed, this header does not exist later in the milter. And the milters
doesn't add anyone itself.
Please note that I'm really referring to 3 *different* problems/issues:
1. the duplicate header thing (not really important although weird).
2. the "own Received" header not being passed to milter. I understand it works
as expected, but could be great to be able to decide this with a configuration parameter.
3. the "own Received" header, despite not appearing in the milter (header callback), *can
be deleted*, in eom callback using chgheader("Received",1)
I have encountered these issues because, due to the filter (milter) that I use,
all the message headers are deleted, and then re-added.
Regards,
Carlos Velasco
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Carlos Velasco via Postfix-users: > *** And there is the milter, is custom made *** You need to reduce complexity. - If you remove the Milter, is the header still duplicated? - If you keep the milter and rmeove the polocy lookup, is eom other header duplicated? Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Wietse Venema via Postfix-users escribió el 10/12/2023 a las 18:49:
Carlos Velasco via Postfix-users:
That means you are somehow calling the policy server twice.
You didn't mention what version of the SPF policy server you are
using. Recent versions (under the project name SPF Engine, since
it's not just a policy server anymore) also provide a milter front
end if you would rather do all the processing in milters. It uses
the same SPF processing code, so it's merely a choice of how you
would prefer to integrate with Postfix.
No, it's called only 1 time, verified with debugs. The duplicated
header issue is located in postfix when mixing policyd and milter,
something weird happens.
Sorry, when the Received-SPF header is generated by a policy service,
and that header is added twice, then you are calling that service
twice. Postfix does not duplicate headers.
If you disagree, then I need to see the proof in your debug logging.
If you have been making changes to Postfix source code, then you
are out of support.
Is only called 1 time. You are ignoring the test and troubleshooting I have
done.
To know if it is called more than 1 time is pretty easy, just putting
debugLevel = 4 in policyd-spf.conf and look at the syslog.
Dec 10 18:59:19 mail:info postfix/smtpd: postfix/smtpd[537116]: connect from
out1.vger.email[2620:137:e000::1:20]:33610
*** Here it goes ***
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Starting
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"request=smtpd_access_policy"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"protocol_state=RCPT"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"protocol_name=ESMTP"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"client_address=2620:137:e000::1:20"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"client_name=out1.vger.email"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"client_port=33610"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"reverse_client_name=out1.vger.email"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"server_address=2a01:4f8:202:6022::2"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"server_port=25"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"helo_name=out1.vger.email"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"sender=linux-kernel-ow...@vger.kernel.org"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"recipient=mtgh...@newipnet.com"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"recipient_count=0"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"queue_id="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"instance=8321c.6575fc78.234b4.0"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"size=3915"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"etrn_domain="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line: "stress="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"sasl_method="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"sasl_username="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"sasl_sender="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"ccert_subject="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"ccert_issuer="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"ccert_fingerprint="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"ccert_pubkey_fingerprint="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"encryption_protocol="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"encryption_cipher="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"encryption_keysize=0"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"policy_context="
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"compatibility_level=3.6"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line:
"mail_version=3.8.3"
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Read line: ""
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Found the end of
entry
Dec 10 18:59:20 mail:info policyd-spf: policyd-spf[537122]: Config:
{'debugLevel': 4, 'HELO_reject': 'Fail', 'Mail_From_reject': 'Fail',
'PermError_reject': 'False', 'TempError_Defer': 'False', 'skip_addresses':
'127.0.0.0/8,:::127.0.0.0/104,::1', 'TestOnly': 1,
'SPF_Enhanced_Status_Codes': 'Yes', 'Header_Type': 'SPF', 'Hide_Receiver':
'Yes', 'Authserv_Id': 'hermes', 'Lookup_Time': 20, 'Whitelist_Lookup_Time': 10,
[pfx] Re: Milter own Postfix-prepended Received
Carlos Velasco via Postfix-users: > > That means you are somehow calling the policy server twice. > > > > You didn't mention what version of the SPF policy server you are > > using. Recent versions (under the project name SPF Engine, since > > it's not just a policy server anymore) also provide a milter front > > end if you would rather do all the processing in milters. It uses > > the same SPF processing code, so it's merely a choice of how you > > would prefer to integrate with Postfix. > > No, it's called only 1 time, verified with debugs. The duplicated > header issue is located in postfix when mixing policyd and milter, > something weird happens. Sorry, when the Received-SPF header is generated by a policy service, and that header is added twice, then you are calling that service twice. Postfix does not duplicate headers. If you disagree, then I need to see the proof in your debug logging. If you have been making changes to Postfix source code, then you are out of support. > Is milter handled by cleanup? The smtpd process handles only the SMTP commands. It delegates header/body activity to the cleanup daemon (it forwards the Milter socket file descriptor along with smtpd_milters configuration). Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Scott Kitterman via Postfix-users escribió el 10/12/2023 a las 18:09: On December 10, 2023 3:54:13 PM UTC, Carlos Velasco via Postfix-users wrote: Wietse Venema via Postfix-users escribió el 10/12/2023 a las 15:53: Carlos Velasco via Postfix-users: 2. Duplicated SMTP Access Policy Delegation This issue is related to headers too. I'm using pypolicyd-spf as policy daemon to check SPF. It is working fine. The execution happens before milter. This is ok. Prepended "Received-SPF" header by pypolicyd-spf show up in the milter. This is ok. But in the final email, the one received by the IMAP server, "Received-SPF" header is duplicated, one in the location shown at the milter process (heder callback), but another new one, just above the "own Postfix-prepended Received: header". This is weird. If the milter adds Received-SPF twice, then you are processing a message twice with that Milter. Output from: postconf -n postconf -Mf Is requuired fo further support. I have collected the postconfs (see below), but I I've done some tests and I think I know what's happening. Please note that "Received-SPF" is not coming from milter, it comes from policy daemon (inside smtpd_recipient_restrictions: check_policy_service unix:private/policyd-spf). So, policyd-spf is executed first, along all smtpd_recipient_restrictions. After, in the milter, the first header to show up in header callback is this "Received-SPF". Now comes the "weird thing"... If I don't touch headers in the milter, all is fine, "Received-SPF" is not duplicated, but note that, in the final email, this header is located *above* own Received (the one missing in milter, that is, my point 1). Test 1: If I delete this "Received-SPF" header in the milter (with chgheader with index 1 and undef), the header still show up in the final email, again, above own Received. Test 2: Delete this "Received-SPF" header in the milter (chgheader) and recreate again (with addheader). Then duplicated "Received-SPF" headers show up in the final email, one above own Received and another one in the placer inserted by the milter. So, I think there is a "ghost" Received-SPF header coming from policyd that doesn't works well with the milter. Info: the reason to delete/add headers is because the filter can, sometimes, change the whole mail, so the headers are first deleted and then recreated, this is how I hit this weird issue. As a workaround, I can just ignore/delete the "Received-SPF" header in the milter, as it is *always* there in the final mail, above own Received. That means you are somehow calling the policy server twice. You didn't mention what version of the SPF policy server you are using. Recent versions (under the project name SPF Engine, since it's not just a policy server anymore) also provide a milter front end if you would rather do all the processing in milters. It uses the same SPF processing code, so it's merely a choice of how you would prefer to integrate with Postfix. No, it's called only 1 time, verified with debugs. The duplicated header issue is located in postfix when mixing policyd and milter, something weird happens. I will think about passing it to milter protocol, but really the main problem in my email is point 1. I'm trying to understand the postfix source code. I see the own Received is created in function common_pre_message_handling, but I don't see how milter is handled at all, and I don't see any milter call in common_post_message_handling. Is milter handled by cleanup? Regards, Carlos Velasco___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On December 10, 2023 3:54:13 PM UTC, Carlos Velasco via Postfix-users wrote: > >Wietse Venema via Postfix-users escribió el 10/12/2023 a las 15:53: >> Carlos Velasco via Postfix-users: >>> 2. Duplicated SMTP Access Policy Delegation >>> This issue is related to headers too. >>> I'm using pypolicyd-spf as policy daemon to check SPF. It is working fine. >>> The execution happens before milter. This is ok. >>> Prepended "Received-SPF" header by pypolicyd-spf show up in the milter. >>> This is ok. >>> But in the final email, the one received by the IMAP server, >>> "Received-SPF" header is duplicated, one in the location shown at >>> the milter process (heder callback), but another new one, just >>> above the "own Postfix-prepended Received: header". This is weird. >> If the milter adds Received-SPF twice, then you are processing a message >> twice with that Milter. >> >> Output from: >> >> postconf -n >> postconf -Mf >> >> Is requuired fo further support. > >I have collected the postconfs (see below), but I I've done some tests and I >think I know what's happening. >Please note that "Received-SPF" is not coming from milter, it comes from >policy daemon (inside smtpd_recipient_restrictions: check_policy_service >unix:private/policyd-spf). > >So, policyd-spf is executed first, along all smtpd_recipient_restrictions. >After, in the milter, the first header to show up in header callback is this >"Received-SPF". >Now comes the "weird thing"... If I don't touch headers in the milter, all is >fine, "Received-SPF" is not duplicated, but note that, in the final email, >this header is located *above* own Received (the one missing in milter, that >is, my point 1). > >Test 1: If I delete this "Received-SPF" header in the milter (with chgheader >with index 1 and undef), the header still show up in the final email, again, >above own Received. >Test 2: Delete this "Received-SPF" header in the milter (chgheader) and >recreate again (with addheader). Then duplicated "Received-SPF" headers show >up in the final email, one above own Received and another one in the placer >inserted by the milter. > >So, I think there is a "ghost" Received-SPF header coming from policyd that >doesn't works well with the milter. > >Info: the reason to delete/add headers is because the filter can, sometimes, >change the whole mail, so the headers are first deleted and then recreated, >this is how I hit this weird issue. > >As a workaround, I can just ignore/delete the "Received-SPF" header in the >milter, as it is *always* there in the final mail, above own Received. > That means you are somehow calling the policy server twice. You didn't mention what version of the SPF policy server you are using. Recent versions (under the project name SPF Engine, since it's not just a policy server anymore) also provide a milter front end if you would rather do all the processing in milters. It uses the same SPF processing code, so it's merely a choice of how you would prefer to integrate with Postfix. Scott K ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Wietse Venema via Postfix-users escribió el 10/12/2023 a las 15:53: Carlos Velasco via Postfix-users: 2. Duplicated SMTP Access Policy Delegation This issue is related to headers too. I'm using pypolicyd-spf as policy daemon to check SPF. It is working fine. The execution happens before milter. This is ok. Prepended "Received-SPF" header by pypolicyd-spf show up in the milter. This is ok. But in the final email, the one received by the IMAP server, "Received-SPF" header is duplicated, one in the location shown at the milter process (heder callback), but another new one, just above the "own Postfix-prepended Received: header". This is weird. If the milter adds Received-SPF twice, then you are processing a message twice with that Milter. Output from: postconf -n postconf -Mf Is requuired fo further support. I have collected the postconfs (see below), but I I've done some tests and I think I know what's happening. Please note that "Received-SPF" is not coming from milter, it comes from policy daemon (inside smtpd_recipient_restrictions: check_policy_service unix:private/policyd-spf). So, policyd-spf is executed first, along all smtpd_recipient_restrictions. After, in the milter, the first header to show up in header callback is this "Received-SPF". Now comes the "weird thing"... If I don't touch headers in the milter, all is fine, "Received-SPF" is not duplicated, but note that, in the final email, this header is located *above* own Received (the one missing in milter, that is, my point 1). Test 1: If I delete this "Received-SPF" header in the milter (with chgheader with index 1 and undef), the header still show up in the final email, again, above own Received. Test 2: Delete this "Received-SPF" header in the milter (chgheader) and recreate again (with addheader). Then duplicated "Received-SPF" headers show up in the final email, one above own Received and another one in the placer inserted by the milter. So, I think there is a "ghost" Received-SPF header coming from policyd that doesn't works well with the milter. Info: the reason to delete/add headers is because the filter can, sometimes, change the whole mail, so the headers are first deleted and then recreated, this is how I hit this weird issue. As a workaround, I can just ignore/delete the "Received-SPF" header in the milter, as it is *always* there in the final mail, above own Received. postconf -n: https://pastebin.com/y3auzWHy postconf -Mf https://pastebin.com/a2t3jdgm Regards, Carlos Velasco ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Carlos Velasco via Postfix-users: > 2. Duplicated SMTP Access Policy Delegation > This issue is related to headers too. > I'm using pypolicyd-spf as policy daemon to check SPF. It is working fine. > The execution happens before milter. This is ok. > Prepended "Received-SPF" header by pypolicyd-spf show up in the milter. This > is ok. > But in the final email, the one received by the IMAP server, > "Received-SPF" header is duplicated, one in the location shown at > the milter process (heder callback), but another new one, just > above the "own Postfix-prepended Received: header". This is weird. If the milter adds Received-SPF twice, then you are processing a message twice with that Milter. Output from: postconf -n postconf -Mf Is requuired fo further support. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Hi, Things have gotten stranger for case 1. If milter "chgheader" is used, the Index 1 is the hidden (prepended) header of postfix. So, you must start in index = 2 to delete the first header received in header callback. Regards, Carlos Velasco Carlos Velasco via Postfix-users escribió el 10/12/2023 a las 14:26: Hello, I'm migrating a postfix server from "Before-Queue Content Filter" to "before-queue Milter". Postfix version is 3.8.3 The process is going well but I have found 2 weird things: 1. Postfix hides its own Postfix-prepended Received: header, for compatibility with Sendmail. This was unexpected for me but later I found it documented in postfix doc. But is there any configuration option to disable this (this is, to make the Postfix-prepended Received available in milter)? The reason for this is that content filters, like SpamAssassin, use the Received headers to make some tests, specially the last one (the one hidden), where SpamAssassin finds out about the last hop connection. 2. Duplicated SMTP Access Policy Delegation This issue is related to headers too. I'm using pypolicyd-spf as policy daemon to check SPF. It is working fine. The execution happens before milter. This is ok. Prepended "Received-SPF" header by pypolicyd-spf show up in the milter. This is ok. But in the final email, the one received by the IMAP server, "Received-SPF" header is duplicated, one in the location shown at the milter process (heder callback), but another new one, just above the "own Postfix-prepended Received: header". This is weird. I was trying to look into postfix code to see if I could make some custom make changes for this, but I'm totally lost, any help would be appreciated. Regards, Carlos Velasco ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
