[pfx] Re: long header folding and DKIM fails

[John Levine via Postfix-users]

It appears that Steffen Nurpmeso via Postfix-users  said:
W> |I did not want to insult you!
> |In mind i had these canon..py snippets
> |
> |  def strip_trailing_whitespace(content):
> |return re.sub(b"[\t ]+\r\n", b"\r\n", content)
> |
> |
> |  def compress_whitespace(content):
> |return re.sub(b"[\t ]+", b" ", content)
> |
> |  ...
> |
> |(x[0].lower().rstrip(),
> |  compress_whitespace(unfold_header_value(x[1])).strip() + b"\r\n")
> |  for x in headers]
> |
> |These strip()s remove any whitespace, as opposed to [\t ] only.

If you are complaining that it removes bare \r and \n, those are invalid,
we've repeatedly told you why they're invalid and you're not doing yourself
any favors by continued digging.

R's,
JOhn
































































a
>a

>In general "all the interoperability problems" are only (as far as
>*i* flew above it) about such whitespace issues.  Most dramatical,
>still, by opendkim and rspamd (git repository source).
>
>--steffen
>|
>|Der Kragenbaer,The moon bear,
>|der holt sich munter   he cheerfully and one by one
>|einen nach dem anderen runter  wa.ks himself off
>|(By Robert Gernhardt)
>___
>Postfix-users mailing list -- postfix-users@postfix.org
>To unsubscribe send an email to postfix-users-le...@postfix.org


___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Steffen Nurpmeso via Postfix-users wrote in
 <20240429215451.hPgOZwzc@steffen%sdaoden.eu>:
 |Scott Kitterman via Postfix-users wrote in
 | <53d75fd8-e109-4712-ba9c-4ea07aa2b...@kitterman.com>:
 ||On April 29, 2024 9:27:20 PM UTC, Steffen Nurpmeso via Postfix-users \
 || wrote:
 ||>Tim Coote via Postfix-users wrote in
 ||> :
 | ...
 ||>|That’s why I formed a hypothesis that (my) Postfix had changed this \
 ||>|long header - as you note this is < 998 stmp limit. I presume it’s \
 ||>|not directly involved now, as smtp_line_length_limit is the default \
 ||>|(998)
 | ...
 ||>Note that in practice only exim and the DKIM library it uses (from
 ||>2009) produce really standard-conforming results, all other
 ||>implementations i have tested have some edge cases.  For example,
 ||>opendkim and rspamd (as of git) use (at least in pars, or always)
 ||>the C function isspace(3) to normalize whitespace, which is not
 ||>allowed according to the DKIM RFC.  They all do it wrong.  The
 ||>most minimum and in practice very likely non-failing (or not at
 ||>all) is dkimpy (which uses the python function trim(), but only as
 ||>the very last step).  My upcoming sign-only milter will only
 ||>support relaxed/relaxed, but is hard-conforming, too.
 ||
 ||If you have examples of cases where dkimpy is wrong, please file bugs \
 ||or email me off list.  It aims to support the RFC requirements with \
 ||only minimal tweaks for interoperability.
 |
 |I did not want to insult you!
 |In mind i had these canon..py snippets
 |
 |  def strip_trailing_whitespace(content):
 |return re.sub(b"[\t ]+\r\n", b"\r\n", content)
 |
 |
 |  def compress_whitespace(content):
 |return re.sub(b"[\t ]+", b" ", content)
 |
 |  ...
 |
 |(x[0].lower().rstrip(),
 |  compress_whitespace(unfold_header_value(x[1])).strip() + b"\r\n")
 |  for x in headers]
 |
 |These strip()s remove any whitespace, as opposed to [\t ] only.

In general "all the interoperability problems" are only (as far as
*i* flew above it) about such whitespace issues.  Most dramatical,
still, by opendkim and rspamd (git repository source).

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Wietse Venema via Postfix-users wrote in
 <4vstkr2gkhzj...@spike.porcupine.org>:
 |Steffen Nurpmeso via Postfix-users:
 |> Wietse Venema via Postfix-users wrote in
 |>  <4vsq5f6q3nzj...@spike.porcupine.org>:
 |>|Tim Coote via Postfix-users:
 |>  ..
 |>|> SMTP headers are often 'folded' as they flow through MTAs. The
 |>|> standard approach to folding and unfolding is covered in rfcs 5322
 |>  ...
 |>|3) Lines that exceed 998 bytes (not including ) cannot be
 |>|   sent in SMTP. The result of sending such text is UNDEFINED.
 |>  ...
 |>|When a line is too long, the Postfix SMTP client inserts
 |>| (controlled by smtp_line_length_limit).  
 |> 
 |> This is a deficit of the entire RFC *822 series that a superficial
 |> whitespace is necessary at that point.  It was simply not on the
 |> table, and i got not even an answer on that in private
 |> communication (but maybe because of my way of speaking things out,
 |> i mean, isn't that just a desaster: this *breaks* the protocol).
 |> (On the other hand the dinosaur nmh just got (partially) proper
 |> line folding in September last year:
 |> 
 |>   http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=542cb12b6d0646b7117\
 |>   72ee97c1e2aacf2bada86
 |> 
 |> and they will fail for the artificial case as i said in
 |> 
 |>   https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00010.html
 |>   [https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00011.htm\
 |>   l])
 |> 
 |>|Before Postfix inserted , some MTA would insert
 |>|a line break after ~1024 without adding a space. This would
 |>|"terminate" the message header, destroy the MIME structure, and
 |> 
 |> This is really what they do?  Then .. my "artificial whitespace"
 |> thought of a decade ago would be wrong.  I would have thought they
 |> possibly reformat with RFC 2047 (if they can), as i said in the
 |> msg00011 thing above.
 |
 |Yes this was a real problem.
 |
 |For example, an email message like this:
 |
 |label: value
 |label: value-that exceeds-smtp-limit
 |...zero or more label: value...
 |MIME-Version-1.0
 |Content-type: multipart/mixed, boundary="foobar"
 |END_OF_PRIMARY_HEADER
 |
 |This is a multi-part message in MIME format.
 |
 |--foobar
 |first body part 
 | 
 |--foobar
 | first body part
 |
 |--foobar--
 | 
 |Was received as:
 |
 |label: value
 |label: value
 |label: first-part-of-value-that-exceeds-smtp-limit
 |END_OF_PRIMARY_HEADER
 | 
 |first body line with remainder-of-value-that-exceeds-smtp-limit
 |
 |>From here on all remaining content including headers and MIME
 |boundaries was received as one large non-MIME message body and was
 |displayed to the user in its encoded form includeing MIME boundaries
 |encoded text (quoted-printable or base62 encoded).
 |
 |Why did that happen? Because some MTA's input reader did not know
 |if it was reading a header or body line.

Very interesting.  All i knew until know is that there were quite
some messages without separating empty line in between headers and
body, "we" (the MUA i maintain) do a "find :" dance thus, and in
the worst case add one.  (This MUA comes from the 70s and still
mostly thinks email==(single-message) MBOX.)

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Scott Kitterman via Postfix-users wrote in
 <53d75fd8-e109-4712-ba9c-4ea07aa2b...@kitterman.com>:
 |On April 29, 2024 9:27:20 PM UTC, Steffen Nurpmeso via Postfix-users \
 | wrote:
 |>Tim Coote via Postfix-users wrote in
 |> :
 ...
 |>|That’s why I formed a hypothesis that (my) Postfix had changed this \
 |>|long header - as you note this is < 998 stmp limit. I presume it’s \
 |>|not directly involved now, as smtp_line_length_limit is the default (998)
 ...
 |>Note that in practice only exim and the DKIM library it uses (from
 |>2009) produce really standard-conforming results, all other
 |>implementations i have tested have some edge cases.  For example,
 |>opendkim and rspamd (as of git) use (at least in pars, or always)
 |>the C function isspace(3) to normalize whitespace, which is not
 |>allowed according to the DKIM RFC.  They all do it wrong.  The
 |>most minimum and in practice very likely non-failing (or not at
 |>all) is dkimpy (which uses the python function trim(), but only as
 |>the very last step).  My upcoming sign-only milter will only
 |>support relaxed/relaxed, but is hard-conforming, too.
 |
 |If you have examples of cases where dkimpy is wrong, please file bugs \
 |or email me off list.  It aims to support the RFC requirements with \
 |only minimal tweaks for interoperability.

I did not want to insult you!
In mind i had these canon..py snippets

  def strip_trailing_whitespace(content):
return re.sub(b"[\t ]+\r\n", b"\r\n", content)


  def compress_whitespace(content):
return re.sub(b"[\t ]+", b" ", content)

  ...

(x[0].lower().rstrip(),
  compress_whitespace(unfold_header_value(x[1])).strip() + b"\r\n")
  for x in headers]

These strip()s remove any whitespace, as opposed to [\t ] only.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Scott Kitterman via Postfix-users]

On April 29, 2024 9:27:20 PM UTC, Steffen Nurpmeso via Postfix-users 
 wrote:
>Tim Coote via Postfix-users wrote in
> :
> |Thanks very much for the detailed response. My original issue was why \
> |dkim signatures were failing on some emails from email lists when arriving \
> |at my Postfix based domain (postfix-3.4.10-1.fc30.x86_64 - I know it \
> |needs updating: and that may be the only reasonable answer). I have \
> |only seen this sort of issue from two sources. So I tried subscribing \
> |to one of them with a gmail address and found the folded header differen\
> |ce, which, if the lines were merged meant that dkim signature verification \
> |worked (using dkimpy).  (tbh, I thought I’d done very well at this \
> |point as I’d spotted what had gone wrong for DKIM).
> |
> |That’s why I formed a hypothesis that (my) Postfix had changed this \
> |long header - as you note this is < 998 stmp limit. I presume it’s \
> |not directly involved now, as smtp_line_length_limit is the default (998)
> |
> |So, I tried creating an email using python and sending that to an account \
> |in my domain - that had the same  inserted after the \
> |‘,’ in the ‘List-Unsubscribe’ header. But if I use the python to send \
> |to an account in gmail, the behaviour is different from the externally \
> |sourced email as the 'List-Unsubscribe’, when observed with the gmail \
> |client in a browser does have an extra , but no .
> |
> |I clearly need to do several more experiments to work out what is going \
> |on.  It’s non-trivial when you control so little of the whole chain.
>
>Note that in practice only exim and the DKIM library it uses (from
>2009) produce really standard-conforming results, all other
>implementations i have tested have some edge cases.  For example,
>opendkim and rspamd (as of git) use (at least in pars, or always)
>the C function isspace(3) to normalize whitespace, which is not
>allowed according to the DKIM RFC.  They all do it wrong.  The
>most minimum and in practice very likely non-failing (or not at
>all) is dkimpy (which uses the python function trim(), but only as
>the very last step).  My upcoming sign-only milter will only
>support relaxed/relaxed, but is hard-conforming, too.


If you have examples of cases where dkimpy is wrong, please file bugs or email 
me off list.  It aims to support the RFC requirements with only minimal tweaks 
for interoperability.

Scott K
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Tim Coote via Postfix-users wrote in
 :
 |Thanks very much for the detailed response. My original issue was why \
 |dkim signatures were failing on some emails from email lists when arriving \
 |at my Postfix based domain (postfix-3.4.10-1.fc30.x86_64 - I know it \
 |needs updating: and that may be the only reasonable answer). I have \
 |only seen this sort of issue from two sources. So I tried subscribing \
 |to one of them with a gmail address and found the folded header differen\
 |ce, which, if the lines were merged meant that dkim signature verification \
 |worked (using dkimpy).  (tbh, I thought I’d done very well at this \
 |point as I’d spotted what had gone wrong for DKIM).
 |
 |That’s why I formed a hypothesis that (my) Postfix had changed this \
 |long header - as you note this is < 998 stmp limit. I presume it’s \
 |not directly involved now, as smtp_line_length_limit is the default (998)
 |
 |So, I tried creating an email using python and sending that to an account \
 |in my domain - that had the same  inserted after the \
 |‘,’ in the ‘List-Unsubscribe’ header. But if I use the python to send \
 |to an account in gmail, the behaviour is different from the externally \
 |sourced email as the 'List-Unsubscribe’, when observed with the gmail \
 |client in a browser does have an extra , but no .
 |
 |I clearly need to do several more experiments to work out what is going \
 |on.  It’s non-trivial when you control so little of the whole chain.

Note that in practice only exim and the DKIM library it uses (from
2009) produce really standard-conforming results, all other
implementations i have tested have some edge cases.  For example,
opendkim and rspamd (as of git) use (at least in pars, or always)
the C function isspace(3) to normalize whitespace, which is not
allowed according to the DKIM RFC.  They all do it wrong.  The
most minimum and in practice very likely non-failing (or not at
all) is dkimpy (which uses the python function trim(), but only as
the very last step).  My upcoming sign-only milter will only
support relaxed/relaxed, but is hard-conforming, too.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Tim Coote via Postfix-users]

Thanks very much for the detailed response. My original issue was why dkim 
signatures were failing on some emails from email lists when arriving at my 
Postfix based domain (postfix-3.4.10-1.fc30.x86_64 - I know it needs updating: 
and that may be the only reasonable answer). I have only seen this sort of 
issue from two sources. So I tried subscribing to one of them with a gmail 
address and found the folded header difference, which, if the lines were merged 
meant that dkim signature verification worked (using dkimpy).  (tbh, I thought 
I’d done very well at this point as I’d spotted what had gone wrong for DKIM).

That’s why I formed a hypothesis that (my) Postfix had changed this long header 
- as you note this is < 998 stmp limit. I presume it’s not directly involved 
now, as smtp_line_length_limit is the default (998)

So, I tried creating an email using python and sending that to an account in my 
domain - that had the same  inserted after the ‘,’ in the 
‘List-Unsubscribe’ header. But if I use the python to send to an account in 
gmail, the behaviour is different from the externally sourced email as the 
'List-Unsubscribe’, when observed with the gmail client in a browser does have 
an extra , but no .

I clearly need to do several more experiments to work out what is going on.  
It’s non-trivial when you control so little of the whole chain.

Tim

> On 29 Apr 2024, at 17:48, Wietse Venema via Postfix-users 
>  wrote:
> 
> Tim Coote via Postfix-users:
>> Hullo
>> 
>> I've recently stumbled across this issue and wondered if it's a/
>> common, b/ how it can be addressed.
>> 
>> SMTP headers are often 'folded' as they flow through MTAs. The
>> standard approach to folding and unfolding is covered in rfcs 5322
>> and is relied on in 6377 (DKIM). Message signing (DKIM) is
>> increasingly used to avoid spam/phishing and relies on consistent
>> header formats when the signature is generated and validated.
>> 
>> It is known that of the two folding mechanisms, one, called
>> 'simpled', just assumes that the messages are not changed in any
>> way, which is not, in general, practical.
> 
> You appear to be confusing:
> 
> 1) Header folding, which involves folding whitespace as described
>   in RFC 5322.
> 
>   When Postfix is allowed to rewrite ADDRESS headers
>   (controlled with local_header_rewrite_clients) then it may re-fold
>   address headers but NOT OTHER headers (such as List-Unsubscribe).
> 
> 2) DKIM header or body canonicalization, either simple or relaxed,
>   which is a way to process message content before computing or
>   verifying its DKIM signature as described in RFC 6376.
> 
> 3) Lines that exceed 998 bytes (not including ) cannot be
>   sent in SMTP. The result of sending such text is UNDEFINED.
> 
>But this is unlikely the cause of a problem with a List-Unsubscribe
>header that you show in a later email message. That header was
>only 756 bytes before it was broken up.
> 
>When a line is too long, the Postfix SMTP client inserts
> (controlled by smtp_line_length_limit).  
> 
>Before Postfix inserted , some MTA would insert
>a line break after ~1024 without adding a space. This would
>"terminate" the message header, destroy the MIME structure, and
>users would see message attachments as raw base64 content that
>had become part of the main message body.
> 
> For DKIM signatures to work across multiple email systems, a message
> has to be already in RFC compliant form before it is signed, and
> has to remain RFC compliant after signing. Messages that rely on
> UNDEFINED behavior may have DKIM signature failures.
> 
>> Unfortunately, the other folding mechanism (relaxed) is based on
>> replacing  with  (thus turning one
>> line into two), and unfolding consists of deleting  where
>> it is preceded by . The approach is unfortunate as
>> some headers don't have any , so the unfolded header
>> is different from the original (signed) version. So if one of these
>> headers is included in the signature it will fail.
> 
>> AFAICT, Postfix folds headers and, if there is not whitespaces
>> typically puts  after a comma in the data structure
>> of the header, thus breaking DKIM on unfolding (as there is an
>> extra  character not in the original header used to construct
>> the signature.
> 
> As mentioned above, Postfix may re-fold ADDRESS headers (see point
> 1 above) but not other headers (such as List-Unsubscribe). Additionally,
> Postfix will 'break' long lines that cannot be sent within the SMTP
> specification with  (see point 3 above).
> 
>> This raises a few questions:
>> - is my analysis correct: postfix header folding can break DKIM as described
> 
> See points 1), 2) and 3) above.
> 
>> - is the problem common (not very, but it's often silent, so it
>> could just be that it's not being found as it's not being looked
>> for)
> 
> As mentioned before, DKIM is defined only for messages that are
> fully RFC compliant before and after the mesage is 

[pfx] Re: long header folding and DKIM fails

[Wietse Venema via Postfix-users]

Steffen Nurpmeso via Postfix-users:
> Wietse Venema via Postfix-users wrote in
>  <4vsq5f6q3nzj...@spike.porcupine.org>:
>  |Tim Coote via Postfix-users:
>  ..
>  |> SMTP headers are often 'folded' as they flow through MTAs. The
>  |> standard approach to folding and unfolding is covered in rfcs 5322
>  ...
>  |3) Lines that exceed 998 bytes (not including ) cannot be
>  |   sent in SMTP. The result of sending such text is UNDEFINED.
>  ...
>  |When a line is too long, the Postfix SMTP client inserts
>  | (controlled by smtp_line_length_limit).  
> 
> This is a deficit of the entire RFC *822 series that a superficial
> whitespace is necessary at that point.  It was simply not on the
> table, and i got not even an answer on that in private
> communication (but maybe because of my way of speaking things out,
> i mean, isn't that just a desaster: this *breaks* the protocol).
> (On the other hand the dinosaur nmh just got (partially) proper
> line folding in September last year:
> 
>   
> http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=542cb12b6d0646b711772ee97c1e2aacf2bada86
> 
> and they will fail for the artificial case as i said in
> 
>   https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00010.html
>   [https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00011.html])
> 
>  |Before Postfix inserted , some MTA would insert
>  |a line break after ~1024 without adding a space. This would
>  |"terminate" the message header, destroy the MIME structure, and
> 
> This is really what they do?  Then .. my "artificial whitespace"
> thought of a decade ago would be wrong.  I would have thought they
> possibly reformat with RFC 2047 (if they can), as i said in the
> msg00011 thing above.

Yes this was a real problem.

For example, an email message like this:

label: value
label: value-that exceeds-smtp-limit
...zero or more label: value...
MIME-Version-1.0
Content-type: multipart/mixed, boundary="foobar"
END_OF_PRIMARY_HEADER

This is a multi-part message in MIME format.

--foobar
first body part 
 
--foobar
 first body part

--foobar--

Was received as:

label: value
label: value
label: first-part-of-value-that-exceeds-smtp-limit
END_OF_PRIMARY_HEADER
 
first body line with remainder-of-value-that-exceeds-smtp-limit

>From here on all remaining content including headers and MIME
boundaries was received as one large non-MIME message body and was
displayed to the user in its encoded form includeing MIME boundaries
encoded text (quoted-printable or base62 encoded).

Why did that happen? Because some MTA's input reader did not know
if it was reading a header or body line.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Wietse Venema via Postfix-users wrote in
 <4vsq5f6q3nzj...@spike.porcupine.org>:
 |Tim Coote via Postfix-users:
 ..
 |> SMTP headers are often 'folded' as they flow through MTAs. The
 |> standard approach to folding and unfolding is covered in rfcs 5322
 ...
 |3) Lines that exceed 998 bytes (not including ) cannot be
 |   sent in SMTP. The result of sending such text is UNDEFINED.
 ...
 |When a line is too long, the Postfix SMTP client inserts
 | (controlled by smtp_line_length_limit).  

This is a deficit of the entire RFC *822 series that a superficial
whitespace is necessary at that point.  It was simply not on the
table, and i got not even an answer on that in private
communication (but maybe because of my way of speaking things out,
i mean, isn't that just a desaster: this *breaks* the protocol).
(On the other hand the dinosaur nmh just got (partially) proper
line folding in September last year:

  
http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=542cb12b6d0646b711772ee97c1e2aacf2bada86

and they will fail for the artificial case as i said in

  https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00010.html
  [https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00011.html])

 |Before Postfix inserted , some MTA would insert
 |a line break after ~1024 without adding a space. This would
 |"terminate" the message header, destroy the MIME structure, and

This is really what they do?  Then .. my "artificial whitespace"
thought of a decade ago would be wrong.  I would have thought they
possibly reformat with RFC 2047 (if they can), as i said in the
msg00011 thing above.

  ...

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Wietse Venema via Postfix-users]

Tim Coote via Postfix-users:
> Hullo
> 
> I've recently stumbled across this issue and wondered if it's a/
> common, b/ how it can be addressed.
> 
> SMTP headers are often 'folded' as they flow through MTAs. The
> standard approach to folding and unfolding is covered in rfcs 5322
> and is relied on in 6377 (DKIM). Message signing (DKIM) is
> increasingly used to avoid spam/phishing and relies on consistent
> header formats when the signature is generated and validated.
> 
> It is known that of the two folding mechanisms, one, called
> 'simpled', just assumes that the messages are not changed in any
> way, which is not, in general, practical.

You appear to be confusing:

1) Header folding, which involves folding whitespace as described
   in RFC 5322.

   When Postfix is allowed to rewrite ADDRESS headers
   (controlled with local_header_rewrite_clients) then it may re-fold
   address headers but NOT OTHER headers (such as List-Unsubscribe).

2) DKIM header or body canonicalization, either simple or relaxed,
   which is a way to process message content before computing or
   verifying its DKIM signature as described in RFC 6376.

3) Lines that exceed 998 bytes (not including ) cannot be
   sent in SMTP. The result of sending such text is UNDEFINED.

But this is unlikely the cause of a problem with a List-Unsubscribe
header that you show in a later email message. That header was
only 756 bytes before it was broken up.

When a line is too long, the Postfix SMTP client inserts
 (controlled by smtp_line_length_limit).  

Before Postfix inserted , some MTA would insert
a line break after ~1024 without adding a space. This would
"terminate" the message header, destroy the MIME structure, and
users would see message attachments as raw base64 content that
had become part of the main message body.

For DKIM signatures to work across multiple email systems, a message
has to be already in RFC compliant form before it is signed, and
has to remain RFC compliant after signing. Messages that rely on
UNDEFINED behavior may have DKIM signature failures.

> Unfortunately, the other folding mechanism (relaxed) is based on
> replacing  with  (thus turning one
> line into two), and unfolding consists of deleting  where
> it is preceded by . The approach is unfortunate as
> some headers don't have any , so the unfolded header
> is different from the original (signed) version. So if one of these
> headers is included in the signature it will fail.

> AFAICT, Postfix folds headers and, if there is not whitespaces
> typically puts  after a comma in the data structure
> of the header, thus breaking DKIM on unfolding (as there is an
> extra  character not in the original header used to construct
> the signature.

As mentioned above, Postfix may re-fold ADDRESS headers (see point
1 above) but not other headers (such as List-Unsubscribe). Additionally,
Postfix will 'break' long lines that cannot be sent within the SMTP
specification with  (see point 3 above).

> This raises a few questions:
> - is my analysis correct: postfix header folding can break DKIM as described

See points 1), 2) and 3) above.

> - is the problem common (not very, but it's often silent, so it
> could just be that it's not being found as it's not being looked
> for)

As mentioned before, DKIM is defined only for messages that are
fully RFC compliant before and after the mesage is signed. A message
that relies on UNDEFINED behavior may have its signatures broken
by any MTA that handles that message.

> - is there a pragmatic fix in Postfix configuration (e.g. to
> increase ths maximum size of a header line?)

See 3) above. If Postfix does not break too-long lines, then you
are depending on UNDEFINED behavior and other remote MTAs may still
break the signature. You just won't be able to verify that so easily
like you did with gmail. But, Postfix isn't changing the List-Unsubscribe
header in your later example.

To change the SMTP specification, you'd have to convince the IETF
community for SMTP to allow arbitrary line lengths.

> - is it legitimate to fold without  according to rfc 5322

Did you mean fold a line with  instead of 
?  Postfix does not do that. Other MTAs might depending
on how long the line is. See 3) above.

> - if there's an issue of incompatibility between folding/unfolding
> of headers, how will this be addressed in the longer term

Did you mean 1) folding, 2) canonicalization, or 3) breaking long
lines that violate the SMTP specification with ?

The longer term fix is to send messages that meet RFC recommendations
(lines SHOULD be <=78) and requirements (lines MUST be <= 998)
before and after they are signed. A proper base64 decoder will
ignore line breaks and whitespace, so there should be no need to
send jumbo header or body lines at all.

In later email I see that you have a long List-Unsubscribe header,
but its total length is well under the Postfix smtp_line_length_limit,
so if it is folded after 

[pfx] Re: ipv6 connection

[Wietse Venema via Postfix-users]

Jack Raats:
> Wietse,
> 
> I run the script every five minutes for more than 13 hours to the DNS 
> server of Cloudflare (2620:fe::fe).
> Four times I had some packet drops (about 25%).

Was that network path in any way similar to the path to your MX
checker? You can check that with mtr (or traceroute which is much
less informative)

> I think that cann't explain why postfix is not reachable on ipv6.

I think you have proven that some MX checker over some unknown path
cannot always reach your Postfix system. I routinely run ping and
mtr to monitor my connectivity. If there is packet loss, them mtr
will tell me where on the network path that happens.

> Can postscreen drop an ipv6 connection?

Postfix does not care if a connection is over IPv4 or IPv6.  Really.
If there is a difference, then that happens at a lower level in the
stack. It just happens to be more visible at the application level.
Don not shoot the messenger of the bad news.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Tim Coote via Postfix-users]

I mostly agree - I’ve been using Postfix for a long while now. But something is 
folding headers in my domain and failing DKIM that don’t get folded by gmail 
and which, if I manually unfold and remove the extra space do get signature 
agreement.

Here’s an example:
List-Unsubscribe: 
,
 



These are quite rare emails, typically machine generated and including odd 
headers like this.  fwiw, I’m validating the signatures with dkimpy (and it 
wasn’t trivial to pin it down to this issue, so I could well be wrong).


> On 29 Apr 2024, at 10:59, David Bürgin via Postfix-users 
>  wrote:
> 
> Remember that Postfix has supported DKIM via various milters for
> 15+ years without issues. So no, practically there is no problem with
> DKIM and header folding in Postfix.
> 
> ___
> Postfix-users mailing list -- postfix-users@postfix.org
> To unsubscribe send an email to postfix-users-le...@postfix.org


___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[David Bürgin via Postfix-users]

Remember that Postfix has supported DKIM via various milters for
15+ years without issues. So no, practically there is no problem with
DKIM and header folding in Postfix.

___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] long header folding and DKIM fails

[Tim Coote via Postfix-users]

Hullo

I’ve recently stumbled across this issue and wondered if it’s a/ common, b/ how 
it can be addressed.

SMTP headers are often ‘folded’ as they flow through MTAs. The standard 
approach to folding and unfolding is covered in rfcs 5322 and is relied on in 
6377 (DKIM). Message signing (DKIM) is increasingly used to avoid spam/phishing 
and relies on consistent header formats when the signature is generated and 
validated. 

It is known that of the two folding mechanisms, one, called 'simpled', just 
assumes that the messages are not changed in any way, which is not, in general, 
practical.
 
Unfortunately, the other folding mechanism (relaxed) is based on replacing 
 with  (thus turning one line into two), and 
unfolding consists of deleting  where it is preceded by . The 
approach is unfortunate as some headers don’t have any , so the 
unfolded header is different from the original (signed) version. So if one of 
these headers is included in the signature it will fail.

AFAICT, Postfix folds headers and, if there is not whitespaces typically puts 
 after a comma in the data structure of the header, thus breaking 
DKIM on unfolding (as there is an extra  character not in the original 
header used to construct the signature.

This raises a few questions:
- is my analysis correct: postfix header folding can break DKIM as described
- is the problem common (not very, but it’s often silent, so it could just be 
that it’s not being found as it’s not being looked for)
- is there a pragmatic fix in Postfix configuration (e.g. to increase ths 
maximum size of a header line?)
- is it legitimate to fold without  according to rfc 5322
- if there’s an issue of incompatibility between folding/unfolding of headers, 
how will this be addressed in the longer term

Tim
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: Fun with line endings, was Re: Mail text wrapping

[Matus UHLAR - fantomas via Postfix-users]

On Apr 24, 2024, at 09:05, John Levine via Postfix-users 
 wrote:
I suppose, but sending bare LF in SMTP is definitely wrong, so he needs to
fix that first.


On 28.04.24 19:15, Doug Hardie via Postfix-users wrote:
Well, the header lines are properly terminated by CRLF.  However, the text 
lines are whatever I get from postfix.


Get from postfix how?

If you receive mail via program or mailbox delivery from local(8), line 
endings on UNIX system are converted to LF (\n).  The MTA donverts them to 
this state.


If you receive mail via SMTP/LMTP, you should get "CR+LF" (\r\n) and when 
feeding to program/mailbox you should convert them yourself.


Sending mail is the opposite: when sending via SMTP, you should end lines 
using CR+LF.


When using postfix' sendmail, it usually does autodetection/conversion:
http://www.postfix.org/postconf.5.html#sendmail_fix_line_endings

Generally that is just a LF.  I 
copied the text and inserted the CRs and sent it to see what happens.  I 
get the same result: = signs at each fold point.


The solution was to replace all the =LFs with a couple spaces.


so, you discard the notion of "continued line" here.

Postfix/receiving MUA wraps the text of long lines properly (e.g., without 
and =).  There was one other = occurance that I found a replaced with 
spaces.  This solution was easy to accomplish since the text of the 
message is accessed via mmap.  I expected to see some extraneous spaces in 
the message, but something is optimizing them out.


There are MUAs (outlook) which tend to ignore line breaks and reformat the 
mail as they wish, which may look awfully (you can luckily turn this off).  

However, even these MUAs tend to understand final "=" as a sign 
"this line doesn't have to be wrapped".


Do not rely on these MUAs behaviour by dropping the final "=".

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Due to unexpected conditions Windows 2000 will be released
in first quarter of year 1901
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org