[pfx] Re: Postfix Milter, the gift that keeps on giving (was: PATCH: using Milter to change a PREPENDed header)
Carlos Velasco via Postfix-users: > I think you are absolutely correct in your analysis. > I've been looking over the code and, although there is a lot I > still don't understand, your yesterday patch seems more a (good) > workaround to the real problem. > I also located the cleanup_header_done_callback where I can make > an (ugly) change to expose the header, but, IMHO, a flag should > be the way to go. Wrong place. Header hiding is controlled with the auto_hdrs list. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Postfix Milter, the gift that keeps on giving (was: PATCH: using Milter to change a PREPENDed header)
Wietse Venema via Postfix-users escribió el 11/12/2023 a las 22:30: Wietse Venema: Patch below. Carlos Velasco: Tested patch against 3.8.3, now it works as expected. Thank you. No duplicated "Received-SPF" and *removing "Received" in position 1 is now not the own generated, is really the first one Received seen in the milter.* Your second comment (text between "*" and "*") led me to a better problem analysis. - The real problem was not with the Postfix order of headers. - The real problem also was not with the Postfix code that exposes message headers with remote smfi_header() calls in the Milter; that code was already fixed to handle prepended headers on 20141018. - The real problem was that the Postfix code that handles header update/delete requests from a Milter was not fixed on 20141018. Thus, the Postfix code that handles header update/delete requests was still naively skipping the first header, making calls to delete the prepended Received-SPF: header ineffective, and mis-directing calls to delete the first Milter-visible Received: header, instead deleting the invisible Postfix-generated Received: header (wtf). To maintain protocol compatibility, the Postfix code that handles header update/delete requests will need to skip the Postfix-generated Received: header, instead of the first message header. Once that mess is handled with, we can consider adding a flag to expose Postfix-generated Received: headers to Milters. I think you are absolutely correct in your analysis. I've been looking over the code and, although there is a lot I still don't understand, your yesterday patch seems more a (good) workaround to the real problem. I also located the cleanup_header_done_callback where I can make an (ugly) change to expose the header, but, IMHO, a flag should be the way to go. Regards, Carlos Velasco ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Postfix Milter, the gift that keeps on giving (was: PATCH: using Milter to change a PREPENDed header)
Wietse Venema: > Patch below. Carlos Velasco: > Tested patch against 3.8.3, now it works as expected. Thank you. > No duplicated "Received-SPF" and *removing "Received" in position > 1 is now not the own generated, is really the first one Received > seen in the milter.* Your second comment (text between "*" and "*") led me to a better problem analysis. - The real problem was not with the Postfix order of headers. - The real problem also was not with the Postfix code that exposes message headers with remote smfi_header() calls in the Milter; that code was already fixed to handle prepended headers on 20141018. - The real problem was that the Postfix code that handles header update/delete requests from a Milter was not fixed on 20141018. Thus, the Postfix code that handles header update/delete requests was still naively skipping the first header, making calls to delete the prepended Received-SPF: header ineffective, and mis-directing calls to delete the first Milter-visible Received: header, instead deleting the invisible Postfix-generated Received: header (wtf). To maintain protocol compatibility, the Postfix code that handles header update/delete requests will need to skip the Postfix-generated Received: header, instead of the first message header. Once that mess is handled with, we can consider adding a flag to expose Postfix-generated Received: headers to Milters. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Matus UHLAR - fantomas via Postfix-users: > >> > Bill Cole via Postfix-users escribi? el 11/12/2023 a las 15:31: > >> >> On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) > >> >> Carlos Velasco via Postfix-users > >> >> is rumored to have said: > >> >> [...] > >> >>> And doing the same work in 2 different places can be called software > >> >>> efficiency? > >> >> No, but the "fix" here would be a divergence from how Milter has > >> >> worked > >> >> since it was created and semi-documented by Sendmail Inc. It is de > >> >> facto > >> >> controlled by the current developers of Sendmail, but I don't believe > >> >> anyone is working to make Milter better, at least not in ways that > >> >> would > >> >> break compatibility. > > >> On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) > >> Carlos Velasco via Postfix-users > >> is rumored to have said: > >> > No one is talking here about breaking any compatibility, re-read the > >> > messages. > > >Bill Cole via Postfix-users: > >> What did I miss? Are you not asking for Postfix to support providing > >> milters with a header that none of them expect and which no other Milter > >> implementation supports? > > On 11.12.23 10:31, Wietse Venema via Postfix-users wrote: > >He asked to make this configurable. I declined because the human > >cost (of having two incompatible ways to convey the connection info) > >would in my opinion exceed the gain from saving a few machine cycles. I was referring to a request to make the Postfix-generated Received: header available to Milters, so that they would not have to generate a fake Received: header from information that the Milter receives with the smfi_connect() callback. If the OP made requests other than fixing the inability to replace or remove his prepended Received-SPF header, then I could not discern that in the chaotic discussion. > if application called from milter was able to distinguish between > headers added locally (thus trusted) and headers received from the > network (untrusted), it could effectively use the locally added > headers. See "I was referring..." above. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
> Bill Cole via Postfix-users escribi? el 11/12/2023 a las 15:31: >> On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) >> Carlos Velasco via Postfix-users >> is rumored to have said: >> [...] >>> And doing the same work in 2 different places can be called software >>> efficiency? >> No, but the "fix" here would be a divergence from how Milter has >> worked >> since it was created and semi-documented by Sendmail Inc. It is de >> facto >> controlled by the current developers of Sendmail, but I don't believe >> anyone is working to make Milter better, at least not in ways that >> would >> break compatibility. On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) Carlos Velasco via Postfix-users is rumored to have said: > No one is talking here about breaking any compatibility, re-read the > messages. Bill Cole via Postfix-users: What did I miss? Are you not asking for Postfix to support providing milters with a header that none of them expect and which no other Milter implementation supports? On 11.12.23 10:31, Wietse Venema via Postfix-users wrote: He asked to make this configurable. I declined because the human cost (of having two incompatible ways to convey the connection info) would in my opinion exceed the gain from saving a few machine cycles. if application called from milter was able to distinguish between headers added locally (thus trusted) and headers received from the network (untrusted), it could effectively use the locally added headers. SpamAssassin trusts all headers before locally added Received:, however spamass-milter and amavisd-milter add it as first header, thus all other headers added by local milters (spf,dkim,arc,dmarc...) are not trusted. Unless the protocol supports providing this information, ability to see Received: header at proper place would increase SA effectiveness, which is I believe what OP asks for. And I would be happy as well. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. REALITY.SYS corrupted. Press any key to reboot Universe. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: And doing the same work in 2 different places can be called software efficiency? Bill Cole via Postfix-users escribió el 11/12/2023 a las 15:31: since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. On 11.12.23 15:37, Carlos Velasco via Postfix-users wrote: No one is talking here about breaking any compatibility, re-read the messages. _You_ have complained why Received: is not seen by milter, here: https://marc.info/?l=postfix-users&m=170223488205099&w=2 The answer has been given and documented: this is how milter works from the beginning. However, I agree that this makes work hard for SpamAssassin, because this way SA does not know, which adders have been added by local milters/policy servers and thus can be trusted - SA only trusts headers before locally added Received: ... headers added by spf,dkim,dmarc milters would be very useful for SA processing. So, it would be great if postfix could optionally add (or, better, not remove) locally added Received: header, although milters would need to implement this feature first. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. "The box said 'Requires Windows 95 or better', so I bought a Macintosh". ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Bill Cole via Postfix-users: > On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) > Carlos Velasco via Postfix-users > is rumored to have said: > > > Bill Cole via Postfix-users escribi? el 11/12/2023 a las 15:31: > >> On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) > >> Carlos Velasco via Postfix-users > >> is rumored to have said: > >> [...] > >>> And doing the same work in 2 different places can be called software > >>> efficiency? > >> No, but the "fix" here would be a divergence from how Milter has > >> worked > >> since it was created and semi-documented by Sendmail Inc. It is de > >> facto > >> controlled by the current developers of Sendmail, but I don't believe > >> anyone is working to make Milter better, at least not in ways that > >> would > >> break compatibility. > > No one is talking here about breaking any compatibility, re-read the > > messages. > > What did I miss? Are you not asking for Postfix to support providing > milters with a header that none of them expect and which no other Milter > implementation supports? He asked to make this configurable. I declined because the human cost (of having two incompatible ways to convey the connection info) would in my opinion exceed the gain from saving a few machine cycles. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-11 at 09:37:39 UTC-0500 (Mon, 11 Dec 2023 15:37:39 +0100) Carlos Velasco via Postfix-users is rumored to have said: Bill Cole via Postfix-users escribió el 11/12/2023 a las 15:31: On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: [...] And doing the same work in 2 different places can be called software efficiency? No, but the "fix" here would be a divergence from how Milter has worked since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. No one is talking here about breaking any compatibility, re-read the messages. What did I miss? Are you not asking for Postfix to support providing milters with a header that none of them expect and which no other Milter implementation supports? That seems to me like a compatibility break. Or you could call it a unique extension to a protocol which lacks a defined extension mechanism or even a formal standardization. Postfix would be doing something with a shared (if not exactly open...) de facto standard that is effectively controlled by the Sendmail project. What could go wrong? Beyond that, I don't think that your justifications for such an addition have made any sense. Milters can ask for any available macros needed explicitly, avoiding the need to parse a Received header in the milter itself. The only reason SpamAssassin requires a synthetic header is that it has no interface for passing the known untainted values of the parameters it wants to use. SA knows nothing about the milter API or "Sendmail Macros" or which MTA+milter layer you are using, so if it needs that it can only look in a header built in the milter e.g, as logged by the MD instance here (which logs excessively...): Dec 11 09:31:59 shiny mimedefang.pl[13203]: 4Spkhs2mXtzXcP2f: Macros are mail_mailer mail_addr daemon_port client_port mail_host auth_authen auth_type j cipher _ client_connections daemon_name tls_version cipher_bits From those macros, MD constructed this Received header for handoff to SpamAssassin: Received: from skinnyclam.scconsult.com (skinnyclam.scconsult.com [192.168.254.125]) by shiny.scconsult.com (envelope-sender ) (MIMEDefang) with ESMTPSA id 4Spkhs2mXtzXcP2f for ; Mon, 11 Dec 2023 09:31:59 -0500 SpamAssassin can reliably and usefully parse out of that header the fact that the message arrived with authentication over an encrypted session. If one had a particular need for some other defined macro one could request it in the milter and stuff it into a header as appropriate. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: PATCH: using Milter to change a PREPENDed header
Carlos Velasco via Postfix-users: > > Wietse Venema via Postfix-users escribi? el 11/12/2023 a las 1:11: > > Patch below. > Tested patch against 3.8.3, now it works as expected. Thank you. > No duplicated "Received-SPF" and removing "Received" in position 1 is now not > the own generated, is really the first one Received seen in the milter. IOW it works as expected. > I still think that passing the own generated Received to the milter > should be optionally parametrizable, with default configuration > to no. Doing the same thing in 2 different programs doesn't make > sense, and also, the milter lacks important information that postfix > have, like authenticated connection, protocol, ciphers, ... All that info is available in the smfi_connect() callback. It may seem wasteful that some programs use that info to re-create a fake Received: header, and then parse that fake header to extracxt the same information again, but that is nothing compared to the cost of having two ways to inspect a message using the Milter protocol. Humans are much more expsive than computers. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
Bill Cole via Postfix-users escribió el 11/12/2023 a las 15:31: On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: [...] And doing the same work in 2 different places can be called software efficiency? No, but the "fix" here would be a divergence from how Milter has worked since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. No one is talking here about breaking any compatibility, re-read the messages. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-10 at 16:37:16 UTC-0500 (Sun, 10 Dec 2023 22:37:16 +0100) Carlos Velasco via Postfix-users is rumored to have said: [...] And doing the same work in 2 different places can be called software efficiency? No, but the "fix" here would be a divergence from how Milter has worked since it was created and semi-documented by Sendmail Inc. It is de facto controlled by the current developers of Sendmail, but I don't believe anyone is working to make Milter better, at least not in ways that would break compatibility. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Milter own Postfix-prepended Received
On 2023-12-10 at 16:16:27 UTC-0500 (Sun, 10 Dec 2023 22:16:27 +0100) Carlos Velasco via Postfix-users is rumored to have said: Wietse Venema via Postfix-users escribió el 10/12/2023 a las 21:53: 2. the "own Received" header not being passed to milter. I understand it works as expected, but could be great to be able to decide this with a configuration parameter. That is because every Milter in the real world gets the client info from the smfi_connect() callback function and from Milter macros, instead of parsing Received: headers. That statement is absolutely false. Quite a bold choice there... Many milters, like amavisd, use other filters for mail content inspection, like SpamAssassin. Spamassassin uses the "Received" headers for some tasks, like this: https://cwiki.apache.org/confluence/display/spamassassin/TrustPath# As a contributor & PMC member for both SpamAssassin (definitely not a milter) and MIMEDefang (definitely a milter, which uses SpamAssassin) I assure you that the Received header which notionally reflects the current transaction DOES NOT exist when the milter gets the message data, and for SA to do its "trust path" audit of Received headers, the MIMEDefang milter MUST construct one for SA to use. Amavis is slightly different because it can be used as a SMTP proxy (between 2 Postfix smtpd processes) or a milter. When used as a proxy, it sees the whole message with the locally-added Received headers. When used as a milter, it works with what Postfix provides it (the relevant macros) to construct a Received header for SA. All of this is documented accurately. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: PATCH: using Milter to change a PREPENDed header
Wietse Venema via Postfix-users escribió el 11/12/2023 a las 1:11: Patch below. Tested patch against 3.8.3, now it works as expected. Thank you. No duplicated "Received-SPF" and removing "Received" in position 1 is now not the own generated, is really the first one Received seen in the milter. I still think that passing the own generated Received to the milter should be optionally parametrizable, with default configuration to no. Doing the same thing in 2 different programs doesn't make sense, and also, the milter lacks important information that postfix have, like authenticated connection, protocol, ciphers, ... Regards, Carlos Velasco ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
