Thanks very much for the detailed response. My original issue was why dkim signatures were failing on some emails from email lists when arriving at my Postfix based domain (postfix-3.4.10-1.fc30.x86_64 - I know it needs updating: and that may be the only reasonable answer). I have only seen this sort of issue from two sources. So I tried subscribing to one of them with a gmail address and found the folded header difference, which, if the lines were merged meant that dkim signature verification worked (using dkimpy). (tbh, I thought I’d done very well at this point as I’d spotted what had gone wrong for DKIM).
That’s why I formed a hypothesis that (my) Postfix had changed this long header - as you note this is < 998 stmp limit. I presume it’s not directly involved now, as smtp_line_length_limit is the default (998) So, I tried creating an email using python and sending that to an account in my domain - that had the same <cr><lf><space> inserted after the ‘,’ in the ‘List-Unsubscribe’ header. But if I use the python to send to an account in gmail, the behaviour is different from the externally sourced email as the 'List-Unsubscribe’, when observed with the gmail client in a browser does have an extra <space>, but no <cr><lf>. I clearly need to do several more experiments to work out what is going on. It’s non-trivial when you control so little of the whole chain. Tim > On 29 Apr 2024, at 17:48, Wietse Venema via Postfix-users > <postfix-users@postfix.org> wrote: > > Tim Coote via Postfix-users: >> Hullo >> >> I've recently stumbled across this issue and wondered if it's a/ >> common, b/ how it can be addressed. >> >> SMTP headers are often 'folded' as they flow through MTAs. The >> standard approach to folding and unfolding is covered in rfcs 5322 >> and is relied on in 6377 (DKIM). Message signing (DKIM) is >> increasingly used to avoid spam/phishing and relies on consistent >> header formats when the signature is generated and validated. >> >> It is known that of the two folding mechanisms, one, called >> 'simpled', just assumes that the messages are not changed in any >> way, which is not, in general, practical. > > You appear to be confusing: > > 1) Header folding, which involves folding whitespace as described > in RFC 5322. > > When Postfix is allowed to rewrite ADDRESS headers > (controlled with local_header_rewrite_clients) then it may re-fold > address headers but NOT OTHER headers (such as List-Unsubscribe). > > 2) DKIM header or body canonicalization, either simple or relaxed, > which is a way to process message content before computing or > verifying its DKIM signature as described in RFC 6376. > > 3) Lines that exceed 998 bytes (not including <CR><LF>) cannot be > sent in SMTP. The result of sending such text is UNDEFINED. > > But this is unlikely the cause of a problem with a List-Unsubscribe > header that you show in a later email message. That header was > only 756 bytes before it was broken up. > > When a line is too long, the Postfix SMTP client inserts > <CR><LF><SPACE> (controlled by smtp_line_length_limit). > > Before Postfix inserted <CR><LF><SPACE>, some MTA would insert > a line break after ~1024 without adding a space. This would > "terminate" the message header, destroy the MIME structure, and > users would see message attachments as raw base64 content that > had become part of the main message body. > > For DKIM signatures to work across multiple email systems, a message > has to be already in RFC compliant form before it is signed, and > has to remain RFC compliant after signing. Messages that rely on > UNDEFINED behavior may have DKIM signature failures. > >> Unfortunately, the other folding mechanism (relaxed) is based on >> replacing <whitespace> with <CRLF><whitespace> (thus turning one >> line into two), and unfolding consists of deleting <CRLF> where >> it is preceded by <whitespace>. The approach is unfortunate as >> some headers don't have any <whitespace>, so the unfolded header >> is different from the original (signed) version. So if one of these >> headers is included in the signature it will fail. > >> AFAICT, Postfix folds headers and, if there is not whitespaces >> typically puts <CRLF><space> after a comma in the data structure >> of the header, thus breaking DKIM on unfolding (as there is an >> extra <space> character not in the original header used to construct >> the signature. > > As mentioned above, Postfix may re-fold ADDRESS headers (see point > 1 above) but not other headers (such as List-Unsubscribe). Additionally, > Postfix will 'break' long lines that cannot be sent within the SMTP > specification with <line-break><space> (see point 3 above). > >> This raises a few questions: >> - is my analysis correct: postfix header folding can break DKIM as described > > See points 1), 2) and 3) above. > >> - is the problem common (not very, but it's often silent, so it >> could just be that it's not being found as it's not being looked >> for) > > As mentioned before, DKIM is defined only for messages that are > fully RFC compliant before and after the mesage is signed. A message > that relies on UNDEFINED behavior may have its signatures broken > by any MTA that handles that message. > >> - is there a pragmatic fix in Postfix configuration (e.g. to >> increase ths maximum size of a header line?) > > See 3) above. If Postfix does not break too-long lines, then you > are depending on UNDEFINED behavior and other remote MTAs may still > break the signature. You just won't be able to verify that so easily > like you did with gmail. But, Postfix isn't changing the List-Unsubscribe > header in your later example. > > To change the SMTP specification, you'd have to convince the IETF > community for SMTP to allow arbitrary line lengths. > >> - is it legitimate to fold without <whitespace> according to rfc 5322 > > Did you mean fold a line with <line-break> instead of <line-break> > <white-space>? Postfix does not do that. Other MTAs might depending > on how long the line is. See 3) above. > >> - if there's an issue of incompatibility between folding/unfolding >> of headers, how will this be addressed in the longer term > > Did you mean 1) folding, 2) canonicalization, or 3) breaking long > lines that violate the SMTP specification with <line-break><space>? > > The longer term fix is to send messages that meet RFC recommendations > (lines SHOULD be <=78) and requirements (lines MUST be <= 998) > before and after they are signed. A proper base64 decoder will > ignore line breaks and whitespace, so there should be no need to > send jumbo header or body lines at all. > > In later email I see that you have a long List-Unsubscribe header, > but its total length is well under the Postfix smtp_line_length_limit, > so if it is folded after 'simple' DKIM signature generation, then > it wasn't done by Postfix. > > List-Unsubscribe: > <https://02.emailinboundprocessing.eu/enc_user/list_unsubscrib > ...more... > 8Mm7C2YB%2BBnyJwmHc0h5It%2FqZWoq%2BtCrJAsj1H2QUy71o%3D&1=1>, > <mailto:4_pao95zpiovuhqqz6207bba02rle66w6hglw8hnndffz5zt16k27b93@unsubscribe-02 > ...more... > q0xlfviyos65msrngz1rg78iv8xmc6wgekztesn6wrr-DO_NOT_DELETE> > > Those two fragments are 459 and 297 bytes, respectively. After > deliting the line break and one space, the result is 756 bytes > which is well under the Postfix smtp_line_length_limit. > > Wietse > _______________________________________________ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org
_______________________________________________ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
