[pfx] Re: Feature request
Ralf Hildebrandt via Postfix-users: > Hi! > > I wonder if this is possible: > > If a PCRE/regexp style map is triggering, it can be quite hard to > find out WHICH pattern actually caused the action. > > So maybe postmap (when invoked with "-b", "-h" or "-q key") could emit > which regular expression (or which line it was in) actually matched. > > Yes, I could give all my regular expressions patterns a unique RHS or > find the regular expressions by divide-et-impera, but I'm being lazy. After a rule is compiled, Postfix no longer has access to source information, especially when information is stored externally such as indexed files and LDAP or SQL databases. Source information could be maintained with the en-memory tables but that would add some bloat, and maybe disappoitment when some tables have source info some don't. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature request
On 20/03/2024 13:17, Viktor Dukhovni via Postfix-users wrote:
> On Wed, Mar 20, 2024 at 01:42:16PM +0100, Ralf Hildebrandt via Postfix-users
> wrote:
>> Hi!
>>
>> I wonder if this is possible:
>>
>> If a PCRE/regexp style map is triggering, it can be quite hard to
>> find out WHICH pattern actually caused the action.
>>
>> So maybe postmap (when invoked with "-b", "-h" or "-q key") could emit
>> which regular expression (or which line it was in) actually matched.
>>
>> Yes, I could give all my regular expressions patterns a unique RHS or
>> find the regular expressions by divide-et-impera, but I'm being lazy.
> With bash <(command) inline file syntax, make the RHS unique on the fly:
>
> $ keystr=...
> $ remap=/etc/postfix/...
> $ postmap -q "$keystr" pcre:<(perl -pe 's/$/ LINE $./ unless
> m{^(if|endif|#)?\s}' "$remap")
>
> Better yet, don't be lazy, include a fingerprint string in your RHS
> reject rule values.
Postscreen doesn't have the option of unique RHS fingerprints; nonetheless, it
would useful to see which (of several)
ACLs was rejecting an incoming connection.
Allen C
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature request
On Wed, Mar 20, 2024 at 09:17:58AM -0400, Viktor Dukhovni via Postfix-users
wrote:
> With bash <(command) inline file syntax, make the RHS unique on the fly:
>
> $ keystr=...
> $ remap=/etc/postfix/...
> $ postmap -q "$keystr" pcre:<(perl -pe 's/$/ LINE $./ unless
> m{^(if|endif|#)?\s}' "$remap")
>
> Better yet, don't be lazy, include a fingerprint string in your RHS
> reject rule values.
Note that the recipe isn't correct for multi-line extended regular
expressions. It appends the tag to the first rather than last line.
But you're probably not using these. If you are, you'd need a fancier
Perl (or Python, ...) script to read folded logical lines from the input
file, and append at the end of a logical line.
--
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature request
On Wed, Mar 20, 2024 at 01:42:16PM +0100, Ralf Hildebrandt via Postfix-users
wrote:
> Hi!
>
> I wonder if this is possible:
>
> If a PCRE/regexp style map is triggering, it can be quite hard to
> find out WHICH pattern actually caused the action.
>
> So maybe postmap (when invoked with "-b", "-h" or "-q key") could emit
> which regular expression (or which line it was in) actually matched.
>
> Yes, I could give all my regular expressions patterns a unique RHS or
> find the regular expressions by divide-et-impera, but I'm being lazy.
With bash <(command) inline file syntax, make the RHS unique on the fly:
$ keystr=...
$ remap=/etc/postfix/...
$ postmap -q "$keystr" pcre:<(perl -pe 's/$/ LINE $./ unless
m{^(if|endif|#)?\s}' "$remap")
Better yet, don't be lazy, include a fingerprint string in your RHS
reject rule values.
--
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Matthias Schneider via Postfix-users: > Hi Jaroslaw, > > In this context, it's not about the ability to recognize the > message, as unique IDs and postfix long queue IDs can handle that > effectively within the 200-character limit. The primary concern > is having the capability to log full header values. As Gerald > illustrated with the example of logging Subject as INFO using > header_check, this feature is currently restricted by the static > 200-character limit. > > While this particular use case might not be applicable in your > scenario, it's worth noting that others find it beneficial for > various reasons. Having the flexibility to configure this limit > would cater to a broader range of user requirements. Postfix can already be configured to provide 'full headers' and much more: use a Milter. There is an endless list of things that might be useful in some niche scenario that serves 1% of the population. I am drawing the line where it currently stands. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Hi Jaroslaw, In this context, it's not about the ability to recognize the message, as unique IDs and postfix long queue IDs can handle that effectively within the 200-character limit. The primary concern is having the capability to log full header values. As Gerald illustrated with the example of logging Subject as INFO using header_check, this feature is currently restricted by the static 200-character limit. While this particular use case might not be applicable in your scenario, it's worth noting that others find it beneficial for various reasons. Having the flexibility to configure this limit would cater to a broader range of user requirements. Your understanding is much appreciated. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: "Jaroslaw Rafa via Postfix-users" An: "postfix-users" Gesendet: Donnerstag, 25. Januar 2024 10:01:49 Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check Dnia 24.01.2024 o godz. 23:21:10 Gerald Galster via Postfix-users pisze: > > As the amount of email increases it can be difficult to distinguish mails > to or from a correspondent. In this case it would help a lot to display > the subject as well but that's not part of envelope data. Therefore it's > convenient to log it using a header_check like /^Subject: / -> INFO, which > is subject to the 200 bytes limit. As headers consist of 7-bit ASCII > only, UTF8-Characters like Umlauts or Emojis must be encoded (qp/base64), > so 200 bytes sometimes is not equal to 200 chars and might result in > truncated subject lines. Even if every character is encoded as two bytes, does ANYBODY who sends an email use a subject so long that first 100 characters are not enough to recognize the message? It seems a completely unreal scenario to me. If somebody used so long subject lines, with the LAST characters being the most meaningful, then the recipients themselves wouldn't be able to recognize the messages properly, because most email clients don't display as many characters in the subject when displaying a list of messages in the inbox. -- Regards, Jaroslaw Rafa r...@rafa.eu.org -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub." ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Dnia 24.01.2024 o godz. 23:21:10 Gerald Galster via Postfix-users pisze: > > As the amount of email increases it can be difficult to distinguish mails > to or from a correspondent. In this case it would help a lot to display > the subject as well but that's not part of envelope data. Therefore it's > convenient to log it using a header_check like /^Subject: / -> INFO, which > is subject to the 200 bytes limit. As headers consist of 7-bit ASCII > only, UTF8-Characters like Umlauts or Emojis must be encoded (qp/base64), > so 200 bytes sometimes is not equal to 200 chars and might result in > truncated subject lines. Even if every character is encoded as two bytes, does ANYBODY who sends an email use a subject so long that first 100 characters are not enough to recognize the message? It seems a completely unreal scenario to me. If somebody used so long subject lines, with the LAST characters being the most meaningful, then the recipients themselves wouldn't be able to recognize the messages properly, because most email clients don't display as many characters in the subject when displaying a list of messages in the inbox. -- Regards, Jaroslaw Rafa r...@rafa.eu.org -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub." ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Hi Gerald, Thank you for providing an insightful perspective on the matter, and you've hit the nail on the head. Indeed, the 200-byte limit not only impacts the display of subjects but also affects headers with security content, such as Authentication-Results. A Milter cannot get any information about the smtp delivery attempts, only the TLS infomation from the recieving smtpd process. To obtain a full picture, we end up having to join the in-session information from the Milter to the log, which is asynchronous and can delay final delivery status for several days. In light of this, a higher or configurable limit in the log would simplify the process, eliminating the need for complex join operations between two separate logs. Considering the various use cases and scenarios, maintaining the 200-char limit as the default while allowing users to configure it based on their specific needs sounds like a pragmatic solution. This approach would provide flexibility for users who wish to log extensive information, such as antispam symbols in headers. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: "Gerald Galster via Postfix-users" An: "postfix-users" Gesendet: Mittwoch, 24. Januar 2024 23:21:10 Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check > Viktor Dukhovni via Postfix-users : > > On Wed, Jan 24, 2024 at 08:27:53PM +0100, Matthias Schneider via > Postfix-users wrote: > >> Using a Milter is an option, but it often involves correlating >> information from both the milter process and the log for a >> comprehensive view. > > Everything of interest can be added as a message header. Just to add another view, I guess the op wants to solve a problem like this: Consider a mailflow utility that correlates postfix logs according to queue ids and stores them to a database. A webinterface allows users to retrieve delivery information for their email addresses themselves. Postfix logging is sufficient to see if a delivery succeeded (250 OK - often including some form of queue-id) or bounced and sometimes this functionality is very convenient for users, that normally do not have access to maillogs. As the amount of email increases it can be difficult to distinguish mails to or from a correspondent. In this case it would help a lot to display the subject as well but that's not part of envelope data. Therefore it's convenient to log it using a header_check like /^Subject: / -> INFO, which is subject to the 200 bytes limit. As headers consist of 7-bit ASCII only, UTF8-Characters like Umlauts or Emojis must be encoded (qp/base64), so 200 bytes sometimes is not equal to 200 chars and might result in truncated subject lines. Programming a milter just to display a complete subject line would be quite time-consuming. Therefore I think the op considers a higher or configurable limit helpful. Best regards, Gerald ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
> Viktor Dukhovni via Postfix-users : > > On Wed, Jan 24, 2024 at 08:27:53PM +0100, Matthias Schneider via > Postfix-users wrote: > >> Using a Milter is an option, but it often involves correlating >> information from both the milter process and the log for a >> comprehensive view. > > Everything of interest can be added as a message header. Just to add another view, I guess the op wants to solve a problem like this: Consider a mailflow utility that correlates postfix logs according to queue ids and stores them to a database. A webinterface allows users to retrieve delivery information for their email addresses themselves. Postfix logging is sufficient to see if a delivery succeeded (250 OK - often including some form of queue-id) or bounced and sometimes this functionality is very convenient for users, that normally do not have access to maillogs. As the amount of email increases it can be difficult to distinguish mails to or from a correspondent. In this case it would help a lot to display the subject as well but that's not part of envelope data. Therefore it's convenient to log it using a header_check like /^Subject: / -> INFO, which is subject to the 200 bytes limit. As headers consist of 7-bit ASCII only, UTF8-Characters like Umlauts or Emojis must be encoded (qp/base64), so 200 bytes sometimes is not equal to 200 chars and might result in truncated subject lines. Programming a milter just to display a complete subject line would be quite time-consuming. Therefore I think the op considers a higher or configurable limit helpful. Best regards, Gerald ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
On Wed, Jan 24, 2024 at 08:27:53PM +0100, Matthias Schneider via Postfix-users wrote: > Using a Milter is an option, but it often involves correlating > information from both the milter process and the log for a > comprehensive view. Everything of interest can be added as a message header. > For example, capturing TLS details from both the smtp client process > and smtpd pid, along with recipient information and specific Headers > is crucial. You can PREPEND the TLS details for inspection by a milter. Why is this "crucial"? Postfix SMTP server TLS logging is fairly terse. Other than that TLS was used, what is so crucial to correlate with message headers? > As far as I know, there are currently no other supported protocols for > event streaming. Therefore, relying on syslog (nearly real-time) or > reading logs from a file (with some delay) remains the most effective > solution for obtaining delivery information. A milter can stream its findings to some central collection point using any protocol of its choice, including syslog. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Hi Victor, Thank you for diving into this topic. Using a Milter is an option, but it often involves correlating information from both the milter process and the log for a comprehensive view. For example, capturing TLS details from both the smtp client process and smtpd pid, along with recipient information and specific Headers is crucial. The proposed configurable header key/value limit would simplyfy this process. As far as I know, there are currently no other supported protocols for event streaming. Therefore, relying on syslog (nearly real-time) or reading logs from a file (with some delay) remains the most effective solution for obtaining delivery information. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: Viktor Dukhovni via Postfix-users An: postfix-users@postfix.org Gesendet: Wed, 24 Jan 2024 16:38:05 +0100 (CET) Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check On Wed, Jan 24, 2024 at 03:10:03PM +0100, Matthias Schneider via Postfix-users wrote: > Initially, I experimented with a Milter for logging the required > headers, but I found that employing a larger %s printf value proved to > be a more efficient solution. However, I'd like to point out that the > default configuration for MaxMessageSize in rsyslog is often set to > 8k, with the potential to increase it to 64k when using TCP. > Therefore, the syslog aspect can handle larger messages effectively. Why is syslog the right medium for accurate message header recording? What sort of key/value pairs are you looking to extract from headers? When ~2 decades ago I implemented header logging (actually MIME-skeleton logging, that recorded also the header of all nested MIME parts), I recorded the MIME skeletons of each message (separated by blank lines) to a daily disk file. Syslog was not a good vehicle for faithful recording of message metadata. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Claus Assmann via Postfix-users: > On Wed, Jan 24, 2024, Wietse Venema via Postfix-users wrote: > > 1) You can log full headers with a Milter. You will run into the > > length limit of the syslog() client (historically, 2 kBytes) before > > the Milter protocol limit (64 kBytes) which is less than the Postfix > > Just FYI: That limit can be increased to 256K or even 1M if > _FFR_MDS_NEGOTIATE is used for libmilter (and the MTA). I should check my claims before posting. - The 64kB limit is for Postfix when sending a body chunk to a Milter. It does not apply to headers. - When Postfix sends a header label and value to a Milter, the combined length has to fit into an unsigned 32-bit integer (length includes null terminators). That limit is inherent to the types used. - Postfix puts a limit of INT_MAX/2 on the payload size of a response from a Milter. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
On Wed, Jan 24, 2024, Wietse Venema via Postfix-users wrote: > 1) You can log full headers with a Milter. You will run into the > length limit of the syslog() client (historically, 2 kBytes) before > the Milter protocol limit (64 kBytes) which is less than the Postfix Just FYI: That limit can be increased to 256K or even 1M if _FFR_MDS_NEGOTIATE is used for libmilter (and the MTA). ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
On Wed, Jan 24, 2024 at 03:10:03PM +0100, Matthias Schneider via Postfix-users wrote: > Initially, I experimented with a Milter for logging the required > headers, but I found that employing a larger %s printf value proved to > be a more efficient solution. However, I'd like to point out that the > default configuration for MaxMessageSize in rsyslog is often set to > 8k, with the potential to increase it to 64k when using TCP. > Therefore, the syslog aspect can handle larger messages effectively. Why is syslog the right medium for accurate message header recording? What sort of key/value pairs are you looking to extract from headers? When ~2 decades ago I implemented header logging (actually MIME-skeleton logging, that recorded also the header of all nested MIME parts), I recorded the MIME skeletons of each message (separated by blank lines) to a daily disk file. Syslog was not a good vehicle for faithful recording of message metadata. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Initially, I experimented with a Milter for logging the required headers, but I found that employing a larger %s printf value proved to be a more efficient solution. However, I'd like to point out that the default configuration for MaxMessageSize in rsyslog is often set to 8k, with the potential to increase it to 64k when using TCP. Therefore, the syslog aspect can handle larger messages effectively. The issue I've been grappling with is more centered around the 200-character limit in the %.200s format, rather than any limitations imposed by syslog. I already use long queue IDs for unique identification of Mail transactions, and that's totally fine, its just about to have the ability to log custom headers without a small size limit. The current limit is also missing in the header_checks documentation (i totally agree that there _must be_ a limit) I hope this provides a clearer perspective. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: "Wietse Venema via Postfix-users" An: "Postfix users" Gesendet: Mittwoch, 24. Januar 2024 14:37:34 Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check 1) You can log full headers with a Milter. You will run into the length limit of the syslog() client (historically, 2 kBytes) before the Milter protocol limit (64 kBytes) which is less than the Postfix header_size_limit (default: 102400). 2) You can uniqely identify all Postfix transactions with a long Postfix queue ID. There is no need to log the same large header upon entry and exit. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
1) You can log full headers with a Milter. You will run into the length limit of the syslog() client (historically, 2 kBytes) before the Milter protocol limit (64 kBytes) which is less than the Postfix header_size_limit (default: 102400). 2) You can uniqely identify all Postfix transactions with a long Postfix queue ID. There is no need to log the same large header upon entry and exit. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Hi Jaroslaw, Thank you for your suggestion. I've indeed explored the option of adjusting the limits in the code, and I've been maintaining a patched version for my specific needs for quite some time now. However, I've also encountered others who faced similar challenges and found the patched version useful. While this approach works, having this feature as a configurable option in the upstream codebase would significantly simplify the process for users with similar requirements. It would make Postfix more adaptable to various use cases without the need for manual adjustments and custom builds. I appreciate your input and encourage discussions that contribute to the improvement of open-source projects like Postfix. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: "Jaroslaw Rafa via Postfix-users" An: "postfix-users" Gesendet: Mittwoch, 24. Januar 2024 12:18:54 Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check Dnia 24.01.2024 o godz. 08:20:33 Matthias Schneider via Postfix-users pisze: > > Upon reviewing the code, it appears there are only one limit on > vstring_sprintf, three limits on msg_info in the code, whereas the rest of > the %.200s limits are present on msg_warn lines. My request stems from > the need to log more comprehensive information, such as > Authentication-Results headers or DKIM Signatures, which often exceed the > current 200-character limit. ( they header key 'Authentication-Results: ' > is using 24 chars already) If you found the limits in the code, just change them to a value that suits you and rebuild Postfix for yourself. It's one of the advantages of being open source, anyway. -- Regards, Jaroslaw Rafa r...@rafa.eu.org -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub." ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Dnia 24.01.2024 o godz. 08:20:33 Matthias Schneider via Postfix-users pisze: > > Upon reviewing the code, it appears there are only one limit on > vstring_sprintf, three limits on msg_info in the code, whereas the rest of > the %.200s limits are present on msg_warn lines. My request stems from > the need to log more comprehensive information, such as > Authentication-Results headers or DKIM Signatures, which often exceed the > current 200-character limit. ( they header key 'Authentication-Results: ' > is using 24 chars already) If you found the limits in the code, just change them to a value that suits you and rebuild Postfix for yourself. It's one of the advantages of being open source, anyway. -- Regards, Jaroslaw Rafa r...@rafa.eu.org -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub." ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Hi Wietse, I understand your perspective on format string limits throughout Postfix. I'd like to clarify that my concern is specific to the 200-byte limit imposed on the logging of header key/values in INFO header_check lines. Upon reviewing the code, it appears there are only one limit on vstring_sprintf, three limits on msg_info in the code, whereas the rest of the %.200s limits are present on msg_warn lines. My request stems from the need to log more comprehensive information, such as Authentication-Results headers or DKIM Signatures, which often exceed the current 200-character limit. ( they header key 'Authentication-Results: ' is using 24 chars already) While I acknowledge the effectiveness of 200 bytes for unique identification, I would like to highlight that there are additional use cases beyond unique identification. In scenarios involving compliance and audit requirements, logging detailed header information becomes crucial. The existing limit can pose challenges in capturing and analyzing essential data for these purposes. I appreciate your consideration of these use cases and their implications. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: "Wietse Venema via Postfix-users" An: "Postfix users" Gesendet: Dienstag, 23. Januar 2024 15:40:45 Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check First, there are format string limits all over Postfix. As a matter of principle I would not make a special case for headers. Second, the existing 200 byte limit should be plenty sufficient to uniqiely identify every past, present, and future email message in this universe and in several other ones. For example, 200 bytes is more than 3x the length of SHA256 in hexadecimal notation. SHA256 alone is sufficient to uniquely identify every particle in the universe(*) with a negligible collision rate. If you can't do unique identification(**) in 200 bytes, then that is the problem, not Postfix. Wietse (*) https://protection.retarus.com/v1?u=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FEddington_number=3ii9Gwp=7l05hCnQ6I4bNCXds6AWd6=7s1=vf77jFIToTkQ7cExLRgDVB9xCtMAahT4xHyJlNSUET0 (**) https://protection.retarus.com/v1?u=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FUniversally_unique_identifier=3ii9Gwp=9Emu5N2QwHcOrnvf7mVKa=7s1=Oi96LJpjp44xzkAa66XQyvQeyXIUZGtaUpgT9RwTxwc ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
First, there are format string limits all over Postfix. As a matter of principle I would not make a special case for headers. Second, the existing 200 byte limit should be plenty sufficient to uniqiely identify every past, present, and future email message in this universe and in several other ones. For example, 200 bytes is more than 3x the length of SHA256 in hexadecimal notation. SHA256 alone is sufficient to uniquely identify every particle in the universe(*) with a negligible collision rate. If you can't do unique identification(**) in 200 bytes, then that is the problem, not Postfix. Wietse (*) https://en.wikipedia.org/wiki/Eddington_number (**) https://en.wikipedia.org/wiki/Universally_unique_identifier ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Hi Wietse, I appreciate your suggestion to use a collate script. However, the collate script, too, relies on the logfile as a reliable source. In my case, I'm keen on receiving the state directly from syslog, bypassing the disk. I understand there may be some potential data loss with this approach, but having the ability to log and receive the complete header key/values is crucial for my use case. While Postfix already has a feature in the header_checks to log header information (info/warn/reject), it becomes unreliable due to the 200-character limit. This limit poses a challenge as it restricts the completeness of key/value pairs in the log. Having a configurable option for the header log size would greatly improve the reliability and usability of this feature, ensuring that users can confidently rely on the logged information for comprehensive header analysis. I kindly urge you to consider incorporating this enhancement into future releases, as it would undoubtedly enhance the Postfix experience for users with similar needs. Thank you for your time and consideration. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: "Wietse Venema via Postfix-users" An: "Postfix users" Gesendet: Montag, 22. Januar 2024 16:49:10 Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check Matthias Schneider via Postfix-users: > Thanks for getting back to me quickly. > > While I understand your concerns about logging reliability, it's > important to highlight that, for some users, the log serves as the > sole source of truth to confirm whether an email was relayed or > rejected in the SMTP process. In that case, better record keeping may be needed. For example, some organizations maintain an archive of all email processed by Postfix. This can be made as reliable as the rest of Postfix. > The challenge we face is the need to correlate Header Information > with the log status of the SMTP process. Even when using the Milter > protocol to obtain headers, I still can't obtain the final state > of the mail from the Milter and must resort to joining information > from the log file. But you don't need headers to correlate receiving and delivering activity. $postfixsource/auxiliary/collate has a tool that groups records by queue ID. Long Postfix queue IDs are used only once within the same mail queue, as long as your clock is monotonic. Wietse > Your insights are much appreciated. > > Best regards, > > Matthias Schneider > > - Urspr?ngliche Mail - > Von: "Wietse Venema via Postfix-users" > An: "Postfix users" > Gesendet: Montag, 22. Januar 2024 16:14:03 > Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in > INFO/WARN/REJECT Header_Check > > Sorry, Postfix logging must not be used as if it is a reliable > channel for message processing. Postfx goes through great effort > to guarantee that message loss won't happen unless a file system > is damaged or unless a message is forcibly deleted from the queue. > > There are no such guarantees for logging. If logging were as reliable > as Postix email deliveries, then logging would use more resources > than all of Postfix combined. In fact, that was happening on ancient > Linux implementations that by default insisted on calling fsync(). > > In my opinion, the 'logging length limit' is a red herring. For > reliable header processing use a reliable mechanism. > > Wietse > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org > ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Matthias Schneider via Postfix-users: > Thanks for getting back to me quickly. > > While I understand your concerns about logging reliability, it's > important to highlight that, for some users, the log serves as the > sole source of truth to confirm whether an email was relayed or > rejected in the SMTP process. In that case, better record keeping may be needed. For example, some organizations maintain an archive of all email processed by Postfix. This can be made as reliable as the rest of Postfix. > The challenge we face is the need to correlate Header Information > with the log status of the SMTP process. Even when using the Milter > protocol to obtain headers, I still can't obtain the final state > of the mail from the Milter and must resort to joining information > from the log file. But you don't need headers to correlate receiving and delivering activity. $postfixsource/auxiliary/collate has a tool that groups records by queue ID. Long Postfix queue IDs are used only once within the same mail queue, as long as your clock is monotonic. Wietse > Your insights are much appreciated. > > Best regards, > > Matthias Schneider > > - Urspr?ngliche Mail - > Von: "Wietse Venema via Postfix-users" > An: "Postfix users" > Gesendet: Montag, 22. Januar 2024 16:14:03 > Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in > INFO/WARN/REJECT Header_Check > > Sorry, Postfix logging must not be used as if it is a reliable > channel for message processing. Postfx goes through great effort > to guarantee that message loss won't happen unless a file system > is damaged or unless a message is forcibly deleted from the queue. > > There are no such guarantees for logging. If logging were as reliable > as Postix email deliveries, then logging would use more resources > than all of Postfix combined. In fact, that was happening on ancient > Linux implementations that by default insisted on calling fsync(). > > In my opinion, the 'logging length limit' is a red herring. For > reliable header processing use a reliable mechanism. > > Wietse > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org > ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Thanks for getting back to me quickly. While I understand your concerns about logging reliability, it's important to highlight that, for some users, the log serves as the sole source of truth to confirm whether an email was relayed or rejected in the SMTP process. The challenge we face is the need to correlate Header Information with the log status of the SMTP process. Even when using the Milter protocol to obtain headers, I still can't obtain the final state of the mail from the Milter and must resort to joining information from the log file. Your insights are much appreciated. Best regards, Matthias Schneider - Ursprüngliche Mail - Von: "Wietse Venema via Postfix-users" An: "Postfix users" Gesendet: Montag, 22. Januar 2024 16:14:03 Betreff: [pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check Sorry, Postfix logging must not be used as if it is a reliable channel for message processing. Postfx goes through great effort to guarantee that message loss won't happen unless a file system is damaged or unless a message is forcibly deleted from the queue. There are no such guarantees for logging. If logging were as reliable as Postix email deliveries, then logging would use more resources than all of Postfix combined. In fact, that was happening on ancient Linux implementations that by default insisted on calling fsync(). In my opinion, the 'logging length limit' is a red herring. For reliable header processing use a reliable mechanism. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Feature Request: Adjustable Header Log Size Limit in INFO/WARN/REJECT Header_Check
Sorry, Postfix logging must not be used as if it is a reliable channel for message processing. Postfx goes through great effort to guarantee that message loss won't happen unless a file system is damaged or unless a message is forcibly deleted from the queue. There are no such guarantees for logging. If logging were as reliable as Postix email deliveries, then logging would use more resources than all of Postfix combined. In fact, that was happening on ancient Linux implementations that by default insisted on calling fsync(). In my opinion, the 'logging length limit' is a red herring. For reliable header processing use a reliable mechanism. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
