[issue44637] Quoting issue on header Reply-To and other address headers

[Julien Castiaux]

Julien Castiaux  added the comment:

Hello there,

Friendly reminder that this issue is still open and that there is a pull 
request ready. We continue to face the issue in production and our customers 
are getting upset.

Can you provide us a schedule when this issue will be addressed? So that we can 
decide either to wait our to start thinking about possible mitigations our side?

Regards,
Julien

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To and other address headers

[Julien Castiaux]

Julien Castiaux  added the comment:

Hello there,

There is a pull-request on github, had to modify `_refold_parse_tree` but I 
could keep the diff quite small. It is properly tested and it is waiting a 
review :)

We have a patch at work so it is *absolutely not* urgent, feel free to review 
it *anytime*. Since we are using the Ubuntu LTS version of python, we might be 
interested by a backport till 3.7, quite honestly I'm happy it was flag as a 
security issue :D

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To and other address headers

[Alexander Mohr]

Alexander Mohr  added the comment:

btw my work-around was to set maxheaderlen=sys.maxsize, worked for AWS SES at 
least

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To and other address headers

[Julien Castiaux]

Change by Julien Castiaux :


--
keywords: +patch
pull_requests: +28107
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/29881

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To and other address headers

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To and other address headers

[R. David Murray]

Change by R. David Murray :


--
nosy: +thehesiod
title: Quoting issue on header Reply-To -> Quoting issue on header Reply-To and 
other address headers

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[Julien Castiaux]

Julien Castiaux  added the comment:

Hello David, Victor,

Thank you for the triage, it reminds me about this issue. David, the 
solution I tried last month was wrong, it was breaking (for good 
reasons) tons of unittests. It seems to me that there is indeed no other 
solution than to bloat the re-folding function a bit more and to fix the 
dbquotes there as your last email suggested.

I agree with you that the code will be even messier, honestly I spent 
quite some time understanding the _refold_parse_tree function and I 
don't feel like patching it.

Regards,

On 11.08.21 14:57, STINNER Victor wrote:
> STINNER Victor  added the comment:
>
> I change the issue type to security. The bug can be abused to send emails to 
> the wrong email address.
>
> --
> nosy: +vstinner
> type: behavior -> security
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[STINNER Victor]

STINNER Victor  added the comment:

I change the issue type to security. The bug can be abused to send emails to 
the wrong email address.

--
nosy: +vstinner
type: behavior -> security

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[R. David Murray]

R. David Murray  added the comment:

Yes, compat32 uses a different parser and folder (the legacy ones), that have a 
lot of small bugs relative to the RFCs (which is why I rewrote it).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[Julien Castiaux]

Julien Castiaux  added the comment:

Update, it works fine with the compat32 policy

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[Julien Castiaux]

Julien Castiaux  added the comment:

Hello David,

I'm working in the same company as Baptiste and I'm trying to solve the 
problem. The issue is indeed related to the folding algorithm, the DBQUOTE 
character is lost in the parse_tree AST thus when the folding algo split the 
children to find a sweat spot to split the line it doesn't re-introduce the 
DBQUOTE and instead inject the content of the BareQuotedString right away.

I'm working on a fix which consist of adding two DBQUOTE, one at the beginning 
and one at the end, of the BareQuotedString token when it is created 
(_header_value_parser.py@get_bare_quoted_string()). I was inspired by how the 
angles < and > are injected around the AddrSpec token in a AngleAddr token.

Right now my fix isn't correct, there are some unittest falling. I'm trying to 
get it working and hopefully get back to you with a nice pull-request :)

Regards,
Julien

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[R. David Murray]

R. David Murray  added the comment:

Forget what I said about my different error, I made a mistake running the test 
script.

Interesting.  If it is related to the length of the name, then the problem is 
most likely in the folding algorithm, specifically in what happens when the 
"display-name" token is wrapped across lines.  And indeed, if we clone the SMTP 
policy and set the max_line_len to 1000 in your sample script. it renders the 
header correctly.

The problem here is that the surrounding quotation marks are added by the 
'value' property of DisplayName, but that property isn't invoked when handling 
parts of the display name separately during mulit-line folding.  I was always 
bothered by the handling of the quotation marks in the part of the parser and 
folder dealing with quoted strings, but I never hit on a better way to do it.  
This, unfortunately, is going to be non-trivial problem to solve.  It is 
probably going to require an ugly hack in the folding code :(

Really, the handling of quoted strings throughout the _header_value_parser code 
is...a hack :(  There are probably other places where it breaks down during 
multi-line folding.  If we are lucky the hack can just add special handling for 
the quoted-string token type in the folder.  If we aren't it will get messier :(

Glancing at the folder code (it's been a long time since I worked on it), one 
possible approach (not necessarily the best one) would be to mark the first and 
last sub-tokens in a quoted-string so that folder knows to put in a leading or 
trailing quote mark, respectively, during folding.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[Baptiste]

Baptiste  added the comment:

Thanks David,

Here is some other tests I ran
Issuing: 
- msg['Reply-To'] = '"foo Research Inc Foofoo BarBar on Summer Special Friday 
050 days (2021-02-31" '

- msg['Reply-To'] = '"foo Research Inc Foofoo BarBar on Summer Special Friday 
050 days 20210231   " '

But:
msg['Reply-To'] = '"foo Research Inc Foofoo BarBar on Summer Special Friday 050 
days 20210231  " '

worked. It looks more related to the length of the name than the character used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[R. David Murray]

R. David Murray  added the comment:

There is definitely a problem here, though I see a different problem when I run 
it (AttributeError: 'Group' object has no attribute 'local_part', presumably 
because of the ':' not getting escaped correctly).  I believe it applies to any 
address header, not just Reply-To.  Unfortunately I don't have time to 
investigate the cause, at least right now.  An interesting first step on 
diagnosing it might be to produce a minimal example: start deleting special 
characters from inside that quoted string until you find the one (or ones) that 
is triggering it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44637] Quoting issue on header Reply-To

[Baptiste]

New submission from Baptiste :

Hello,

When using as_string() on a Reply-To header like the following:
msg['Reply-To'] = '"foo Research, Inc. Foofoo BarBar on Summer Special Friday: 
0.50 days (2021-02-31)" '

The double quote disappear, which lead to wrong header value

See attached file for example

--
components: email
files: Reply-To.py
messages: 397478
nosy: Abridbus, Julien Castiaux, barry, r.david.murray
priority: normal
severity: normal
status: open
title: Quoting issue on header Reply-To
type: behavior
versions: Python 3.9
Added file: https://bugs.python.org/file50149/Reply-To.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com