[pfx] Re: long header folding and DKIM fails

[Tim Coote via Postfix-users]

Further investigation showed that the issue is in Python 2.7’s `email` module.  
Although this is out of support, I’d expect some to be lying around and thought 
it worth mentioning to this group. Specifically, `email.Message.__str__()`. It 
seems ok in python3



> On 2 May 2024, at 12:53, Tim Coote via Postfix-users 
>  wrote:
> 
> I think that I’ve now fixed this in my domain, so I thought I’d just note the 
> route to finding it, more as a comment on the complexity of working out 
> what’s going on.
> 
> After making a simple robot to send emails with long headers and 
> demonstrating how they broke in my production environment, I rebuilt that env 
> on a Vagrant box. It worked properly (ie it didn’t replace “,” with 
> “,”) until I’d added in the old spambayes filter that I’d been 
> ‘using’.
> 
> Removing Spambayes fixed the issue. Dunno whether this will have an effect on 
> the amount of spam, but I suspect that it hadn’t done anything useful for 
> some time.
> 
> What would have helped - and I’ve no idea how feasible this is - would be 
> some tooling to pull out different versions of the message as they flow 
> through the queues.
> 
> Interesting how long latent bugs can lie around, isn’t it?
> 
> Tim



___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Bill Cole via Postfix-users]

On 2024-05-02 at 07:53:15 UTC-0400 (Thu, 2 May 2024 12:53:15 +0100)
Tim Coote via Postfix-users 
is rumored to have said:

What would have helped - and I’ve no idea how feasible this is - 
would be some tooling to pull out different versions of the message as 
they flow through the queues.


This can be done with a milter like MIMEDefang or MailMunge which let 
you do arbitrary things to messages at each step in the mail flow. I 
have used this to debug similar problems with signing and Sendmail's 
not-so-obvious mods to messages based on mailer flags.



--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo@toad.social and many *@billmail.scconsult.com 
addresses)

Not Currently Available For Hire
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Matus UHLAR - fantomas via Postfix-users]

On 02.05.24 12:53, Tim Coote via Postfix-users wrote:
I think that I’ve now fixed this in my domain, so I thought I’d just note 
the route to finding it, more as a comment on the complexity of working 
out what’s going on.


After making a simple robot to send emails with long headers and 
demonstrating how they broke in my production environment, I rebuilt that 
env on a Vagrant box.  It worked properly (ie it didn’t replace “,” with 
“,”) until I’d added in the old spambayes filter that I’d 
been ‘using’.


Removing Spambayes fixed the issue.  Dunno whether this will have an effect 
on the amount of spam, but I suspect that it hadn’t done anything useful 
for some time.


This looks like someone is signing headers they should not sign. Can you 
look at that?


Is fo, you can also check DKIM signature before you sign or before you check 
for spam.


What would have helped - and I’ve no idea how feasible this is - would be 
some tooling to pull out different versions of the message as they flow 
through the queues.


Interesting how long latent bugs can lie around, isn’t it?

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Honk if you love peace and quiet.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Tim Coote via Postfix-users]

I think that I’ve now fixed this in my domain, so I thought I’d just note the 
route to finding it, more as a comment on the complexity of working out what’s 
going on.

After making a simple robot to send emails with long headers and demonstrating 
how they broke in my production environment, I rebuilt that env on a Vagrant 
box. It worked properly (ie it didn’t replace “,” with “,”) 
until I’d added in the old spambayes filter that I’d been ‘using’.

Removing Spambayes fixed the issue. Dunno whether this will have an effect on 
the amount of spam, but I suspect that it hadn’t done anything useful for some 
time.

What would have helped - and I’ve no idea how feasible this is - would be some 
tooling to pull out different versions of the message as they flow through the 
queues.

Interesting how long latent bugs can lie around, isn’t it?

Tim
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Wietse Venema via Postfix-users wrote in
 <4vtb9v00wbzj...@spike.porcupine.org>:
 |Steffen Nurpmeso via Postfix-users:
 |> But one thing is plain, if lines get folded "artificially" to
 |> satisfy line length limits, then this is a whitespace that DKIM
 |> will see, and if it was not in the original message, the signature
 |> will break.
 |
 |After the DKIM signature is generated, the artificial line break
 |already invalidates the signature. The purpose of the artificial
 |space after this line break is to preserve the MIME structure, so
 |that humans aren't exposed to raw html, q-p, or base64 etc.

Maybe we are running off-bounds.

 |> The only possibility to get over this hurdle is to apply RFC 2047
 |> (or RFC 2231, in a parameter) MIME encoding, and then [...]
 |
 |Any such actions need to be taken sender-side, before the DKIM
 |signature is generated.

I agree.  That much is plain.

 --End of <4vtb9v00wbzj...@spike.porcupine.org>

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Wietse Venema via Postfix-users]

Steffen Nurpmeso via Postfix-users:
> But one thing is plain, if lines get folded "artificially" to
> satisfy line length limits, then this is a whitespace that DKIM
> will see, and if it was not in the original message, the signature
> will break.

After the DKIM signature is generated, the artificial line break
already invalidates the signature. The purpose of the artificial
space after this line break is to preserve the MIME structure, so
that humans aren't exposed to raw html, q-p, or base64 etc.

> The only possibility to get over this hurdle is to apply RFC 2047
> (or RFC 2231, in a parameter) MIME encoding, and then [...]

Any such actions need to be taken sender-side, before the DKIM
signature is generated.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Wietse Venema via Postfix-users wrote in
 <4vtlbf3vz0zj...@spike.porcupine.org>:
 |Postfix does not store line endings internally, because different
 |environments have different line ending conventions (for example
 |SMTP has  while UNIX has ). Postfix strips line endings
 |on input, and adds them on output. Postfix was modeled after routers
 |with different kinds of network interfaces.

'has nothing to do with the line endings.  On some lonely IETF
place i posted code snippets of opendkim's header parsing, he
knows that very well.

 |The Postfix sendmail command allows both  and , because
 |too many app developers were messing up.
 |
 |As of January 2024, Postfix replaces any  and  that are not
 |part of a line ending with space. This prevents outbound SMTP
 |smuggling, and makes DKIM signatures more likely to verify.  The
 |default setting "cleanup_replace_stray_cr_lf = yes" was added in
 |Postfix 3.9, 3.8.5, 3.7.10, 3.6.14, and 3.5.24 later.
 |
 |Returning to the problem at hand, the options that I see are:
 |
 |- Forward the message including all its MIME headers, and leave
 |  the decoding to the recipient's mail software.
 |
 |- Use a real MIME parser and decoder to extract the plain text
 |  that you want to forward.
 |
 |The trick to strip " =" works only for quoted-printable encoding,
 |but does not handle messages encoded in base64.

But one thing is plain, if lines get folded "artificially" to
satisfy line length limits, then this is a whitespace that DKIM
will see, and if it was not in the original message, the signature
will break.  The only possibility to get over this hurdle is to
apply RFC 2047 (or RFC 2231, in a parameter) MIME encoding, and
then (for RFC 2047) the software needs to be able to understand
the character set of the data in order not to break up multibyte
characters (not possible with ISO C, iconv can this, a bit),
because 2047 generates "atomic units" instead of a stream that is
united like 2231.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

John Levine wrote in
 <20240430015342.8DF9C89B9BE7@ary.local>:
 |It appears that Steffen Nurpmeso via Postfix-users  \
 |said:
 |W> |I did not want to insult you!
 |>|In mind i had these canon..py snippets
 |>|
 |>|  def strip_trailing_whitespace(content):
 |>|return re.sub(b"[\t ]+\r\n", b"\r\n", content)
 |>|
 |>|
 |>|  def compress_whitespace(content):
 |>|return re.sub(b"[\t ]+", b" ", content)
 |>|
 |>|  ...
 |>|
 |>|(x[0].lower().rstrip(),
 |>|  compress_whitespace(unfold_header_value(x[1])).strip() + b"\r\n")
 |>|  for x in headers]
 |>|
 |>|These strip()s remove any whitespace, as opposed to [\t ] only.
 |
 |If you are complaining that it removes bare \r and \n, those are invalid,
 |we've repeatedly told you why they're invalid and you're not doing yourself
 |any favors by continued digging.

No, as i said on the IETF list and now here is that isspace(3)
includes several US-ASCII aka lowermost 7-bit of ISO 10646
whitespace characters aside from SPace and HTabulator.
You may find it nitpicking, and as i also said dkimpy is the best
with that strip() above compared to plain usage of
[ascii_]isspace() (rspamd, opendkim) or \s regex (perl thing used
by amavisd: last step only on minimal subrange, too).  But it is
not the "hardcore literal RFC 6376" that the dkim library exim
uses implements, or what my thing does.
Much worse than the above whitespace thingy is anyway the RFC 5322
parser support of python as such, etc etc.

 |>In general "all the interoperability problems" are only (as far as
 |>*i* flew above it) about such whitespace issues.  Most dramatical,
 |>still, by opendkim and rspamd (git repository source).

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Wietse Venema via Postfix-users]

Postfix does not store line endings internally, because different
environments have different line ending conventions (for example
SMTP has  while UNIX has ). Postfix strips line endings
on input, and adds them on output. Postfix was modeled after routers
with different kinds of network interfaces.

The Postfix sendmail command allows both  and , because
too many app developers were messing up.

As of January 2024, Postfix replaces any  and  that are not
part of a line ending with space. This prevents outbound SMTP
smuggling, and makes DKIM signatures more likely to verify.  The
default setting "cleanup_replace_stray_cr_lf = yes" was added in
Postfix 3.9, 3.8.5, 3.7.10, 3.6.14, and 3.5.24 later.

Returning to the problem at hand, the options that I see are:

- Forward the message including all its MIME headers, and leave
  the decoding to the recipient's mail software.

- Use a real MIME parser and decoder to extract the plain text
  that you want to forward.

The trick to strip " =" works only for quoted-printable encoding,
but does not handle messages encoded in base64.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[John Levine via Postfix-users]

It appears that Steffen Nurpmeso via Postfix-users  said:
W> |I did not want to insult you!
> |In mind i had these canon..py snippets
> |
> |  def strip_trailing_whitespace(content):
> |return re.sub(b"[\t ]+\r\n", b"\r\n", content)
> |
> |
> |  def compress_whitespace(content):
> |return re.sub(b"[\t ]+", b" ", content)
> |
> |  ...
> |
> |(x[0].lower().rstrip(),
> |  compress_whitespace(unfold_header_value(x[1])).strip() + b"\r\n")
> |  for x in headers]
> |
> |These strip()s remove any whitespace, as opposed to [\t ] only.

If you are complaining that it removes bare \r and \n, those are invalid,
we've repeatedly told you why they're invalid and you're not doing yourself
any favors by continued digging.

R's,
JOhn
































































a
>a

>In general "all the interoperability problems" are only (as far as
>*i* flew above it) about such whitespace issues.  Most dramatical,
>still, by opendkim and rspamd (git repository source).
>
>--steffen
>|
>|Der Kragenbaer,The moon bear,
>|der holt sich munter   he cheerfully and one by one
>|einen nach dem anderen runter  wa.ks himself off
>|(By Robert Gernhardt)
>___
>Postfix-users mailing list -- postfix-users@postfix.org
>To unsubscribe send an email to postfix-users-le...@postfix.org


___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Steffen Nurpmeso via Postfix-users wrote in
 <20240429215451.hPgOZwzc@steffen%sdaoden.eu>:
 |Scott Kitterman via Postfix-users wrote in
 | <53d75fd8-e109-4712-ba9c-4ea07aa2b...@kitterman.com>:
 ||On April 29, 2024 9:27:20 PM UTC, Steffen Nurpmeso via Postfix-users \
 || wrote:
 ||>Tim Coote via Postfix-users wrote in
 ||> :
 | ...
 ||>|That’s why I formed a hypothesis that (my) Postfix had changed this \
 ||>|long header - as you note this is < 998 stmp limit. I presume it’s \
 ||>|not directly involved now, as smtp_line_length_limit is the default \
 ||>|(998)
 | ...
 ||>Note that in practice only exim and the DKIM library it uses (from
 ||>2009) produce really standard-conforming results, all other
 ||>implementations i have tested have some edge cases.  For example,
 ||>opendkim and rspamd (as of git) use (at least in pars, or always)
 ||>the C function isspace(3) to normalize whitespace, which is not
 ||>allowed according to the DKIM RFC.  They all do it wrong.  The
 ||>most minimum and in practice very likely non-failing (or not at
 ||>all) is dkimpy (which uses the python function trim(), but only as
 ||>the very last step).  My upcoming sign-only milter will only
 ||>support relaxed/relaxed, but is hard-conforming, too.
 ||
 ||If you have examples of cases where dkimpy is wrong, please file bugs \
 ||or email me off list.  It aims to support the RFC requirements with \
 ||only minimal tweaks for interoperability.
 |
 |I did not want to insult you!
 |In mind i had these canon..py snippets
 |
 |  def strip_trailing_whitespace(content):
 |return re.sub(b"[\t ]+\r\n", b"\r\n", content)
 |
 |
 |  def compress_whitespace(content):
 |return re.sub(b"[\t ]+", b" ", content)
 |
 |  ...
 |
 |(x[0].lower().rstrip(),
 |  compress_whitespace(unfold_header_value(x[1])).strip() + b"\r\n")
 |  for x in headers]
 |
 |These strip()s remove any whitespace, as opposed to [\t ] only.

In general "all the interoperability problems" are only (as far as
*i* flew above it) about such whitespace issues.  Most dramatical,
still, by opendkim and rspamd (git repository source).

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Wietse Venema via Postfix-users wrote in
 <4vstkr2gkhzj...@spike.porcupine.org>:
 |Steffen Nurpmeso via Postfix-users:
 |> Wietse Venema via Postfix-users wrote in
 |>  <4vsq5f6q3nzj...@spike.porcupine.org>:
 |>|Tim Coote via Postfix-users:
 |>  ..
 |>|> SMTP headers are often 'folded' as they flow through MTAs. The
 |>|> standard approach to folding and unfolding is covered in rfcs 5322
 |>  ...
 |>|3) Lines that exceed 998 bytes (not including ) cannot be
 |>|   sent in SMTP. The result of sending such text is UNDEFINED.
 |>  ...
 |>|When a line is too long, the Postfix SMTP client inserts
 |>| (controlled by smtp_line_length_limit).  
 |> 
 |> This is a deficit of the entire RFC *822 series that a superficial
 |> whitespace is necessary at that point.  It was simply not on the
 |> table, and i got not even an answer on that in private
 |> communication (but maybe because of my way of speaking things out,
 |> i mean, isn't that just a desaster: this *breaks* the protocol).
 |> (On the other hand the dinosaur nmh just got (partially) proper
 |> line folding in September last year:
 |> 
 |>   http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=542cb12b6d0646b7117\
 |>   72ee97c1e2aacf2bada86
 |> 
 |> and they will fail for the artificial case as i said in
 |> 
 |>   https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00010.html
 |>   [https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00011.htm\
 |>   l])
 |> 
 |>|Before Postfix inserted , some MTA would insert
 |>|a line break after ~1024 without adding a space. This would
 |>|"terminate" the message header, destroy the MIME structure, and
 |> 
 |> This is really what they do?  Then .. my "artificial whitespace"
 |> thought of a decade ago would be wrong.  I would have thought they
 |> possibly reformat with RFC 2047 (if they can), as i said in the
 |> msg00011 thing above.
 |
 |Yes this was a real problem.
 |
 |For example, an email message like this:
 |
 |label: value
 |label: value-that exceeds-smtp-limit
 |...zero or more label: value...
 |MIME-Version-1.0
 |Content-type: multipart/mixed, boundary="foobar"
 |END_OF_PRIMARY_HEADER
 |
 |This is a multi-part message in MIME format.
 |
 |--foobar
 |first body part 
 | 
 |--foobar
 | first body part
 |
 |--foobar--
 | 
 |Was received as:
 |
 |label: value
 |label: value
 |label: first-part-of-value-that-exceeds-smtp-limit
 |END_OF_PRIMARY_HEADER
 | 
 |first body line with remainder-of-value-that-exceeds-smtp-limit
 |
 |>From here on all remaining content including headers and MIME
 |boundaries was received as one large non-MIME message body and was
 |displayed to the user in its encoded form includeing MIME boundaries
 |encoded text (quoted-printable or base62 encoded).
 |
 |Why did that happen? Because some MTA's input reader did not know
 |if it was reading a header or body line.

Very interesting.  All i knew until know is that there were quite
some messages without separating empty line in between headers and
body, "we" (the MUA i maintain) do a "find :" dance thus, and in
the worst case add one.  (This MUA comes from the 70s and still
mostly thinks email==(single-message) MBOX.)

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Scott Kitterman via Postfix-users wrote in
 <53d75fd8-e109-4712-ba9c-4ea07aa2b...@kitterman.com>:
 |On April 29, 2024 9:27:20 PM UTC, Steffen Nurpmeso via Postfix-users \
 | wrote:
 |>Tim Coote via Postfix-users wrote in
 |> :
 ...
 |>|That’s why I formed a hypothesis that (my) Postfix had changed this \
 |>|long header - as you note this is < 998 stmp limit. I presume it’s \
 |>|not directly involved now, as smtp_line_length_limit is the default (998)
 ...
 |>Note that in practice only exim and the DKIM library it uses (from
 |>2009) produce really standard-conforming results, all other
 |>implementations i have tested have some edge cases.  For example,
 |>opendkim and rspamd (as of git) use (at least in pars, or always)
 |>the C function isspace(3) to normalize whitespace, which is not
 |>allowed according to the DKIM RFC.  They all do it wrong.  The
 |>most minimum and in practice very likely non-failing (or not at
 |>all) is dkimpy (which uses the python function trim(), but only as
 |>the very last step).  My upcoming sign-only milter will only
 |>support relaxed/relaxed, but is hard-conforming, too.
 |
 |If you have examples of cases where dkimpy is wrong, please file bugs \
 |or email me off list.  It aims to support the RFC requirements with \
 |only minimal tweaks for interoperability.

I did not want to insult you!
In mind i had these canon..py snippets

  def strip_trailing_whitespace(content):
return re.sub(b"[\t ]+\r\n", b"\r\n", content)


  def compress_whitespace(content):
return re.sub(b"[\t ]+", b" ", content)

  ...

(x[0].lower().rstrip(),
  compress_whitespace(unfold_header_value(x[1])).strip() + b"\r\n")
  for x in headers]

These strip()s remove any whitespace, as opposed to [\t ] only.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Scott Kitterman via Postfix-users]

On April 29, 2024 9:27:20 PM UTC, Steffen Nurpmeso via Postfix-users 
 wrote:
>Tim Coote via Postfix-users wrote in
> :
> |Thanks very much for the detailed response. My original issue was why \
> |dkim signatures were failing on some emails from email lists when arriving \
> |at my Postfix based domain (postfix-3.4.10-1.fc30.x86_64 - I know it \
> |needs updating: and that may be the only reasonable answer). I have \
> |only seen this sort of issue from two sources. So I tried subscribing \
> |to one of them with a gmail address and found the folded header differen\
> |ce, which, if the lines were merged meant that dkim signature verification \
> |worked (using dkimpy).  (tbh, I thought I’d done very well at this \
> |point as I’d spotted what had gone wrong for DKIM).
> |
> |That’s why I formed a hypothesis that (my) Postfix had changed this \
> |long header - as you note this is < 998 stmp limit. I presume it’s \
> |not directly involved now, as smtp_line_length_limit is the default (998)
> |
> |So, I tried creating an email using python and sending that to an account \
> |in my domain - that had the same  inserted after the \
> |‘,’ in the ‘List-Unsubscribe’ header. But if I use the python to send \
> |to an account in gmail, the behaviour is different from the externally \
> |sourced email as the 'List-Unsubscribe’, when observed with the gmail \
> |client in a browser does have an extra , but no .
> |
> |I clearly need to do several more experiments to work out what is going \
> |on.  It’s non-trivial when you control so little of the whole chain.
>
>Note that in practice only exim and the DKIM library it uses (from
>2009) produce really standard-conforming results, all other
>implementations i have tested have some edge cases.  For example,
>opendkim and rspamd (as of git) use (at least in pars, or always)
>the C function isspace(3) to normalize whitespace, which is not
>allowed according to the DKIM RFC.  They all do it wrong.  The
>most minimum and in practice very likely non-failing (or not at
>all) is dkimpy (which uses the python function trim(), but only as
>the very last step).  My upcoming sign-only milter will only
>support relaxed/relaxed, but is hard-conforming, too.


If you have examples of cases where dkimpy is wrong, please file bugs or email 
me off list.  It aims to support the RFC requirements with only minimal tweaks 
for interoperability.

Scott K
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Tim Coote via Postfix-users wrote in
 :
 |Thanks very much for the detailed response. My original issue was why \
 |dkim signatures were failing on some emails from email lists when arriving \
 |at my Postfix based domain (postfix-3.4.10-1.fc30.x86_64 - I know it \
 |needs updating: and that may be the only reasonable answer). I have \
 |only seen this sort of issue from two sources. So I tried subscribing \
 |to one of them with a gmail address and found the folded header differen\
 |ce, which, if the lines were merged meant that dkim signature verification \
 |worked (using dkimpy).  (tbh, I thought I’d done very well at this \
 |point as I’d spotted what had gone wrong for DKIM).
 |
 |That’s why I formed a hypothesis that (my) Postfix had changed this \
 |long header - as you note this is < 998 stmp limit. I presume it’s \
 |not directly involved now, as smtp_line_length_limit is the default (998)
 |
 |So, I tried creating an email using python and sending that to an account \
 |in my domain - that had the same  inserted after the \
 |‘,’ in the ‘List-Unsubscribe’ header. But if I use the python to send \
 |to an account in gmail, the behaviour is different from the externally \
 |sourced email as the 'List-Unsubscribe’, when observed with the gmail \
 |client in a browser does have an extra , but no .
 |
 |I clearly need to do several more experiments to work out what is going \
 |on.  It’s non-trivial when you control so little of the whole chain.

Note that in practice only exim and the DKIM library it uses (from
2009) produce really standard-conforming results, all other
implementations i have tested have some edge cases.  For example,
opendkim and rspamd (as of git) use (at least in pars, or always)
the C function isspace(3) to normalize whitespace, which is not
allowed according to the DKIM RFC.  They all do it wrong.  The
most minimum and in practice very likely non-failing (or not at
all) is dkimpy (which uses the python function trim(), but only as
the very last step).  My upcoming sign-only milter will only
support relaxed/relaxed, but is hard-conforming, too.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Tim Coote via Postfix-users]

Thanks very much for the detailed response. My original issue was why dkim 
signatures were failing on some emails from email lists when arriving at my 
Postfix based domain (postfix-3.4.10-1.fc30.x86_64 - I know it needs updating: 
and that may be the only reasonable answer). I have only seen this sort of 
issue from two sources. So I tried subscribing to one of them with a gmail 
address and found the folded header difference, which, if the lines were merged 
meant that dkim signature verification worked (using dkimpy).  (tbh, I thought 
I’d done very well at this point as I’d spotted what had gone wrong for DKIM).

That’s why I formed a hypothesis that (my) Postfix had changed this long header 
- as you note this is < 998 stmp limit. I presume it’s not directly involved 
now, as smtp_line_length_limit is the default (998)

So, I tried creating an email using python and sending that to an account in my 
domain - that had the same  inserted after the ‘,’ in the 
‘List-Unsubscribe’ header. But if I use the python to send to an account in 
gmail, the behaviour is different from the externally sourced email as the 
'List-Unsubscribe’, when observed with the gmail client in a browser does have 
an extra , but no .

I clearly need to do several more experiments to work out what is going on.  
It’s non-trivial when you control so little of the whole chain.

Tim

> On 29 Apr 2024, at 17:48, Wietse Venema via Postfix-users 
>  wrote:
> 
> Tim Coote via Postfix-users:
>> Hullo
>> 
>> I've recently stumbled across this issue and wondered if it's a/
>> common, b/ how it can be addressed.
>> 
>> SMTP headers are often 'folded' as they flow through MTAs. The
>> standard approach to folding and unfolding is covered in rfcs 5322
>> and is relied on in 6377 (DKIM). Message signing (DKIM) is
>> increasingly used to avoid spam/phishing and relies on consistent
>> header formats when the signature is generated and validated.
>> 
>> It is known that of the two folding mechanisms, one, called
>> 'simpled', just assumes that the messages are not changed in any
>> way, which is not, in general, practical.
> 
> You appear to be confusing:
> 
> 1) Header folding, which involves folding whitespace as described
>   in RFC 5322.
> 
>   When Postfix is allowed to rewrite ADDRESS headers
>   (controlled with local_header_rewrite_clients) then it may re-fold
>   address headers but NOT OTHER headers (such as List-Unsubscribe).
> 
> 2) DKIM header or body canonicalization, either simple or relaxed,
>   which is a way to process message content before computing or
>   verifying its DKIM signature as described in RFC 6376.
> 
> 3) Lines that exceed 998 bytes (not including ) cannot be
>   sent in SMTP. The result of sending such text is UNDEFINED.
> 
>But this is unlikely the cause of a problem with a List-Unsubscribe
>header that you show in a later email message. That header was
>only 756 bytes before it was broken up.
> 
>When a line is too long, the Postfix SMTP client inserts
> (controlled by smtp_line_length_limit).  
> 
>Before Postfix inserted , some MTA would insert
>a line break after ~1024 without adding a space. This would
>"terminate" the message header, destroy the MIME structure, and
>users would see message attachments as raw base64 content that
>had become part of the main message body.
> 
> For DKIM signatures to work across multiple email systems, a message
> has to be already in RFC compliant form before it is signed, and
> has to remain RFC compliant after signing. Messages that rely on
> UNDEFINED behavior may have DKIM signature failures.
> 
>> Unfortunately, the other folding mechanism (relaxed) is based on
>> replacing  with  (thus turning one
>> line into two), and unfolding consists of deleting  where
>> it is preceded by . The approach is unfortunate as
>> some headers don't have any , so the unfolded header
>> is different from the original (signed) version. So if one of these
>> headers is included in the signature it will fail.
> 
>> AFAICT, Postfix folds headers and, if there is not whitespaces
>> typically puts  after a comma in the data structure
>> of the header, thus breaking DKIM on unfolding (as there is an
>> extra  character not in the original header used to construct
>> the signature.
> 
> As mentioned above, Postfix may re-fold ADDRESS headers (see point
> 1 above) but not other headers (such as List-Unsubscribe). Additionally,
> Postfix will 'break' long lines that cannot be sent within the SMTP
> specification with  (see point 3 above).
> 
>> This raises a few questions:
>> - is my analysis correct: postfix header folding can break DKIM as described
> 
> See points 1), 2) and 3) above.
> 
>> - is the problem common (not very, but it's often silent, so it
>> could just be that it's not being found as it's not being looked
>> for)
> 
> As mentioned before, DKIM is defined only for messages that are
> fully RFC compliant before and after the mesage is 

[pfx] Re: long header folding and DKIM fails

[Wietse Venema via Postfix-users]

Steffen Nurpmeso via Postfix-users:
> Wietse Venema via Postfix-users wrote in
>  <4vsq5f6q3nzj...@spike.porcupine.org>:
>  |Tim Coote via Postfix-users:
>  ..
>  |> SMTP headers are often 'folded' as they flow through MTAs. The
>  |> standard approach to folding and unfolding is covered in rfcs 5322
>  ...
>  |3) Lines that exceed 998 bytes (not including ) cannot be
>  |   sent in SMTP. The result of sending such text is UNDEFINED.
>  ...
>  |When a line is too long, the Postfix SMTP client inserts
>  | (controlled by smtp_line_length_limit).  
> 
> This is a deficit of the entire RFC *822 series that a superficial
> whitespace is necessary at that point.  It was simply not on the
> table, and i got not even an answer on that in private
> communication (but maybe because of my way of speaking things out,
> i mean, isn't that just a desaster: this *breaks* the protocol).
> (On the other hand the dinosaur nmh just got (partially) proper
> line folding in September last year:
> 
>   
> http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=542cb12b6d0646b711772ee97c1e2aacf2bada86
> 
> and they will fail for the artificial case as i said in
> 
>   https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00010.html
>   [https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00011.html])
> 
>  |Before Postfix inserted , some MTA would insert
>  |a line break after ~1024 without adding a space. This would
>  |"terminate" the message header, destroy the MIME structure, and
> 
> This is really what they do?  Then .. my "artificial whitespace"
> thought of a decade ago would be wrong.  I would have thought they
> possibly reformat with RFC 2047 (if they can), as i said in the
> msg00011 thing above.

Yes this was a real problem.

For example, an email message like this:

label: value
label: value-that exceeds-smtp-limit
...zero or more label: value...
MIME-Version-1.0
Content-type: multipart/mixed, boundary="foobar"
END_OF_PRIMARY_HEADER

This is a multi-part message in MIME format.

--foobar
first body part 
 
--foobar
 first body part

--foobar--

Was received as:

label: value
label: value
label: first-part-of-value-that-exceeds-smtp-limit
END_OF_PRIMARY_HEADER
 
first body line with remainder-of-value-that-exceeds-smtp-limit

>From here on all remaining content including headers and MIME
boundaries was received as one large non-MIME message body and was
displayed to the user in its encoded form includeing MIME boundaries
encoded text (quoted-printable or base62 encoded).

Why did that happen? Because some MTA's input reader did not know
if it was reading a header or body line.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Steffen Nurpmeso via Postfix-users]

Wietse Venema via Postfix-users wrote in
 <4vsq5f6q3nzj...@spike.porcupine.org>:
 |Tim Coote via Postfix-users:
 ..
 |> SMTP headers are often 'folded' as they flow through MTAs. The
 |> standard approach to folding and unfolding is covered in rfcs 5322
 ...
 |3) Lines that exceed 998 bytes (not including ) cannot be
 |   sent in SMTP. The result of sending such text is UNDEFINED.
 ...
 |When a line is too long, the Postfix SMTP client inserts
 | (controlled by smtp_line_length_limit).  

This is a deficit of the entire RFC *822 series that a superficial
whitespace is necessary at that point.  It was simply not on the
table, and i got not even an answer on that in private
communication (but maybe because of my way of speaking things out,
i mean, isn't that just a desaster: this *breaks* the protocol).
(On the other hand the dinosaur nmh just got (partially) proper
line folding in September last year:

  
http://git.savannah.gnu.org/cgit/nmh.git/commit/?id=542cb12b6d0646b711772ee97c1e2aacf2bada86

and they will fail for the artificial case as i said in

  https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00010.html
  [https://lists.nongnu.org/archive/html/nmh-workers/2023-08/msg00011.html])

 |Before Postfix inserted , some MTA would insert
 |a line break after ~1024 without adding a space. This would
 |"terminate" the message header, destroy the MIME structure, and

This is really what they do?  Then .. my "artificial whitespace"
thought of a decade ago would be wrong.  I would have thought they
possibly reformat with RFC 2047 (if they can), as i said in the
msg00011 thing above.

  ...

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[Wietse Venema via Postfix-users]

Tim Coote via Postfix-users:
> Hullo
> 
> I've recently stumbled across this issue and wondered if it's a/
> common, b/ how it can be addressed.
> 
> SMTP headers are often 'folded' as they flow through MTAs. The
> standard approach to folding and unfolding is covered in rfcs 5322
> and is relied on in 6377 (DKIM). Message signing (DKIM) is
> increasingly used to avoid spam/phishing and relies on consistent
> header formats when the signature is generated and validated.
> 
> It is known that of the two folding mechanisms, one, called
> 'simpled', just assumes that the messages are not changed in any
> way, which is not, in general, practical.

You appear to be confusing:

1) Header folding, which involves folding whitespace as described
   in RFC 5322.

   When Postfix is allowed to rewrite ADDRESS headers
   (controlled with local_header_rewrite_clients) then it may re-fold
   address headers but NOT OTHER headers (such as List-Unsubscribe).

2) DKIM header or body canonicalization, either simple or relaxed,
   which is a way to process message content before computing or
   verifying its DKIM signature as described in RFC 6376.

3) Lines that exceed 998 bytes (not including ) cannot be
   sent in SMTP. The result of sending such text is UNDEFINED.

But this is unlikely the cause of a problem with a List-Unsubscribe
header that you show in a later email message. That header was
only 756 bytes before it was broken up.

When a line is too long, the Postfix SMTP client inserts
 (controlled by smtp_line_length_limit).  

Before Postfix inserted , some MTA would insert
a line break after ~1024 without adding a space. This would
"terminate" the message header, destroy the MIME structure, and
users would see message attachments as raw base64 content that
had become part of the main message body.

For DKIM signatures to work across multiple email systems, a message
has to be already in RFC compliant form before it is signed, and
has to remain RFC compliant after signing. Messages that rely on
UNDEFINED behavior may have DKIM signature failures.

> Unfortunately, the other folding mechanism (relaxed) is based on
> replacing  with  (thus turning one
> line into two), and unfolding consists of deleting  where
> it is preceded by . The approach is unfortunate as
> some headers don't have any , so the unfolded header
> is different from the original (signed) version. So if one of these
> headers is included in the signature it will fail.

> AFAICT, Postfix folds headers and, if there is not whitespaces
> typically puts  after a comma in the data structure
> of the header, thus breaking DKIM on unfolding (as there is an
> extra  character not in the original header used to construct
> the signature.

As mentioned above, Postfix may re-fold ADDRESS headers (see point
1 above) but not other headers (such as List-Unsubscribe). Additionally,
Postfix will 'break' long lines that cannot be sent within the SMTP
specification with  (see point 3 above).

> This raises a few questions:
> - is my analysis correct: postfix header folding can break DKIM as described

See points 1), 2) and 3) above.

> - is the problem common (not very, but it's often silent, so it
> could just be that it's not being found as it's not being looked
> for)

As mentioned before, DKIM is defined only for messages that are
fully RFC compliant before and after the mesage is signed. A message
that relies on UNDEFINED behavior may have its signatures broken
by any MTA that handles that message.

> - is there a pragmatic fix in Postfix configuration (e.g. to
> increase ths maximum size of a header line?)

See 3) above. If Postfix does not break too-long lines, then you
are depending on UNDEFINED behavior and other remote MTAs may still
break the signature. You just won't be able to verify that so easily
like you did with gmail. But, Postfix isn't changing the List-Unsubscribe
header in your later example.

To change the SMTP specification, you'd have to convince the IETF
community for SMTP to allow arbitrary line lengths.

> - is it legitimate to fold without  according to rfc 5322

Did you mean fold a line with  instead of 
?  Postfix does not do that. Other MTAs might depending
on how long the line is. See 3) above.

> - if there's an issue of incompatibility between folding/unfolding
> of headers, how will this be addressed in the longer term

Did you mean 1) folding, 2) canonicalization, or 3) breaking long
lines that violate the SMTP specification with ?

The longer term fix is to send messages that meet RFC recommendations
(lines SHOULD be <=78) and requirements (lines MUST be <= 998)
before and after they are signed. A proper base64 decoder will
ignore line breaks and whitespace, so there should be no need to
send jumbo header or body lines at all.

In later email I see that you have a long List-Unsubscribe header,
but its total length is well under the Postfix smtp_line_length_limit,
so if it is folded after 

[pfx] Re: long header folding and DKIM fails

[Tim Coote via Postfix-users]

I mostly agree - I’ve been using Postfix for a long while now. But something is 
folding headers in my domain and failing DKIM that don’t get folded by gmail 
and which, if I manually unfold and remove the extra space do get signature 
agreement.

Here’s an example:
List-Unsubscribe: 
,
 



These are quite rare emails, typically machine generated and including odd 
headers like this.  fwiw, I’m validating the signatures with dkimpy (and it 
wasn’t trivial to pin it down to this issue, so I could well be wrong).


> On 29 Apr 2024, at 10:59, David Bürgin via Postfix-users 
>  wrote:
> 
> Remember that Postfix has supported DKIM via various milters for
> 15+ years without issues. So no, practically there is no problem with
> DKIM and header folding in Postfix.
> 
> ___
> Postfix-users mailing list -- postfix-users@postfix.org
> To unsubscribe send an email to postfix-users-le...@postfix.org


___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: long header folding and DKIM fails

[David Bürgin via Postfix-users]

Remember that Postfix has supported DKIM via various milters for
15+ years without issues. So no, practically there is no problem with
DKIM and header folding in Postfix.

___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org