[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Łukasz Langa added the comment: New changeset 634da2de88af06eb8c6ebdb90d8c5847063d by Senthil Kumaran in branch '3.8': [3.8] bpo-43882 - Mention urllib.parse changes in Whats new section. (#26277) https://github.com/python/cpython/commit/634da2de88af06eb8c6ebdb90d8c5847063d

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Gregory P. Smith : -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: New changeset 0593ae84af9e0e8332644e7ed13d7fd8306c4e1a by Senthil Kumaran in branch '3.9': [3.9] bpo-43882 - Mention urllib.parse changes in Whats new section. (GH-26276) https://github.com/python/cpython/commit/0593ae84af9e0e8332644e7ed13d7fd8306c4e1a

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: New changeset f14015adf52014c2345522fe32d43f15f001c986 by Senthil Kumaran in branch '3.10': [3.10] bpo-43882 - Mention urllib.parse changes in Whats new section. (GH-26275) https://github.com/python/cpython/commit/f14015adf52014c2345522fe32d43f15f001c986

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- pull_requests: +24883 pull_request: https://github.com/python/cpython/pull/26277 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- pull_requests: +24882 pull_request: https://github.com/python/cpython/pull/26276 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- pull_requests: +24881 stage: commit review -> patch review pull_request: https://github.com/python/cpython/pull/26275 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: Lets get equivalent whatsnew text into the 3.8 and 3.9 and 3.10 branches before closing it. -- ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: Thanks, Senthil and Greg! The updates for 3.7 and 3.6 are now merged. Is there anything else that needs to be done for this issue or can it now be closed? -- priority: release blocker -> high resolution: fixed -> stage: patch review -> commit review

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: New changeset 6f743e7a4da904f61dfa84cc7d7385e4dcc79ac5 by Senthil Kumaran in branch '3.6': [3.6] bpo-43882 - Mention urllib.parse changes in Whats New section for 3.6.14 (GH-26268) https://github.com/python/cpython/commit/6f743e7a4da904f61dfa84cc7d7385e4dcc79ac5

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: New changeset c723d5191110f99849f7b0944820f6c3cd5f7747 by Senthil Kumaran in branch '3.7': [3.7] bpo-43882 - Mention urllib.parse changes in Whats New section for 3.7.11 (GH-26267) https://github.com/python/cpython/commit/c723d5191110f99849f7b0944820f6c3cd5f7747

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- pull_requests: +24872 pull_request: https://github.com/python/cpython/pull/26268 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- pull_requests: +24871 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/26267 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: > I will include an additional blurb for this change for security fix versions. Ping. This issue is still blocking 3.7 and 3.6 security releases. -- nosy: +pablogsal priority: normal -> release blocker ___ Python

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: Ned wrote: > Senthil, I am not sure which previous message you are referring to but. I meant, the messages from other developers who raised that change broke certain test cases. Ned, but I got little concerned, if we planned to revert the change. > the

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: There is no less intrusive fix as far as I can see. I believe we're down to either stick with what we've done, or do nothing. It doesn't have to be the same choice in all release branches, being more conservative with changes the older the stable branch

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: > My reading of the previous message was, even if we raised exception > or gave as a parameter, it wont be any better for certain downstream > users, as we let the security problem open, and have it only as opt-in fix. Senthil, I am not sure which previous message

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: Hello All, I think, the current striping of ASCII newline and tab is a _reasonable_ solution given it was a security issue. It also follows the guidelines of "WHATWG" (Specifically Point 3) > 2. If input contains any ASCII tab or newline, validation

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: > Unfortunately you already have to deal with the existence of 3.9.5 having the > new behavior but not having a control. I have been holding off on 3.7.x and 3.6.x security releases pending resolutions of this and other open security issues. But based on the

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: FWIW, if we were to add a parameter, I'd lean towards a name of "invalid_url_characters = None" defaulting to using what's in our private _UNSAFE_URL_BYTES_TO_REMOVE global when None but otherwise letting the user specify a sequence of characters.

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: Of note: If we had chosen to raise a ValueError (or similar) for these characters by default, the cloud-init code would also fail to behave as intended today (based on what I see in

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Mike Lissner added the comment: > With the fix for this bug, urlsplit silently removes (some of) those > characters before we can replace them, modifying the output of our > sanitisation code I don't have any good solutions for 3.9.5, but going forward, this feels like another example of

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: We try to not add a new parameter in a bugfix release as that can be difficult to use. That said, adding a new bool keyword only parameter to control this behavior seems feasible. Unfortunately you already have to deal with the existence of 3.9.5 having

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Daniel Watkins added the comment: (Accidentally dropped Ned from nosy list; apologies!) -- nosy: +ned.deily ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Daniel Watkins added the comment: Hey folks, Thanks for all the work on this: I really appreciate the efforts to keep Python as secure as possible! This change _is_ causing us problems in the cloud-init codebase, which thankfully have been caught by our testing in Ubuntu's development

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: For completeness reference, the 'main' branch after the master->main rename also got fixed to check it early the same was as the release branches via: https://github.com/python/cpython/commit/985ac016373403e8ad41f8d563c4355ffa8d49ff our robot updating

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Ned Deily : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: New changeset 6c472d3a1d334d4eeb4a25eba7bf3b01611bf667 by Miss Islington (bot) in branch '3.6': [3.6] bpo-43882 - urllib.parse should sanitize urls containing ASCII newline and tabs (GH-25924)

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Ned Deily added the comment: New changeset f4dac7ec55477a6c5d965e594e74bd6bda786903 by Miss Islington (bot) in branch '3.7': [3.7] bpo-43882 - urllib.parse should sanitize urls containing ASCII newline and tabs. (GH-25923)

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: New changeset 24f1d1a8a2c4aa58a606b4b6d5fa4305a3b91705 by Miss Islington (bot) in branch '3.10': bpo-43882 Remove the newline, and tab early. From query and fragments. (GH-25936)

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by miss-islington : -- pull_requests: +24603 pull_request: https://github.com/python/cpython/pull/25936 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: Mike: There may be multiple ways to read that WHATWG recommendation? The linked to section is about implementing a state machine for parsing a URL into parts safely. But that may not imply that anything that passed through that state machine should be

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Mike Lissner added the comment: > I'd wonder how to pass through valid exceptions without urlparse raising > something. Oops, meant to say "valid URLs", not valid exceptions, sorry. -- ___ Python tracker

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Mike Lissner added the comment: > Instead of the patches as you see them, we could've raised an exception. In my mind the definition of a valid URL is what browsers recognize. They're moving towards the WHATWG definition, and so too must we. If we make python raise an exception when a URL

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: Thanks Florian! Indeed, I'm glad you have tests for this. (I expect anyone writing their own validation code will have such tests) Making urlsplit raise an exception where it never has before has other consequences: In CPython's own test suite

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by miss-islington : -- pull_requests: +24590 pull_request: https://github.com/python/cpython/pull/25923 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by miss-islington : -- pull_requests: +24591 pull_request: https://github.com/python/cpython/pull/25924 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Łukasz Langa added the comment: New changeset 515a7bc4e13645d0945b46a8e1d9102b918cd407 by Miss Islington (bot) in branch '3.8': [3.8] bpo-43882 - urllib.parse should sanitize urls containing ASCII newline and tabs. (GH-25595) (#25726)

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- pull_requests: +24589 pull_request: https://github.com/python/cpython/pull/25921 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Florian Apolloner added the comment: Thank you for the kind words Michał. We (Django) are exactly in the position that you describe. Our validation, at least for now has to stay strict, exactly to prevent fallout further down the road (see

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Michał Górny added the comment: In my opinion, raising an exception would have been safer. Botocore and django do precisely what you say — provide a validator. To make this validator easier, they do the validation on splitted up URL parts. I disagree with the premise that they were stupid

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: Both Django and Botocore issues appear to be in the category of: "depending on invalid data being passed through our urlsplit API so that they could look for it later" Not much sympathy. We never guaranteed we'd pass invalid data through. They're

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Mike Lissner added the comment: I haven't watched that Blackhat presentation yet, but from the slides, it seems like the fix is to get all languages parsing URLs the same as the browsers. That's what @orsenthil has been doing here and plans to do in https://bugs.python.org/issue43883.

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Seth Michael Larson added the comment: Leaving a thought here, I'm highlighting that we're now implementing two different standards, RFC 3986 with hints of WHATWG-URL. There are pitfalls to doing so as now a strict URL parser for RFC 3986 (like the one used by urllib3/requests) will give

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Michał Górny added the comment: I hate to be the bearer of bad news but I've already found this change to be breaking tests of botocore and django. In both cases, the test failure is apparently because upstream used to reject URLs after finding newlines in the split components, and now

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: New changeset 8a595744e696a0fb92dccc5d4e45da41571270a1 by Senthil Kumaran in branch '3.9': [3.9] bpo-43882 Remove the newline, and tab early. From query and fragments. (#25853)

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: Based on Greg's review comment, I have pushed the fix for 3.9, and 3.8 - [3.9] https://github.com/python/cpython/pull/25853 - [3.8] https://github.com/python/cpython/pull/25726 There is no need to hold off releases for these alone. If we get it merged

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- pull_requests: +24537 pull_request: https://github.com/python/cpython/pull/25853 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Łukasz Langa added the comment: Good catch, Greg. Since it's not merged already, this change will miss 3.8.10 but as a security fix will be included in 3.8.11 later in the year. The partial fix already landed in 3.9 will be released in 3.9.5 later today unless it's amended or reverted in a

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Gregory P. Smith added the comment: I think there's still a flaw in the fixes implemented in 3.10 and 3.9 so far. We're closer, but probably not quite good enough yet. why? We aren't stripping the newlines+tab early enough. I think we need to do the stripping *right after* the

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: New changeset 491fde0161d5e527eeff8586dd3972d7d3a631a7 by Miss Islington (bot) in branch '3.9': [3.9] bpo-43882 - urllib.parse should sanitize urls containing ASCII newline and tabs. (GH-25595) (GH-25725)

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by miss-islington : -- nosy: +miss-islington nosy_count: 4.0 -> 5.0 pull_requests: +24417 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/25725 ___ Python tracker

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by miss-islington : -- pull_requests: +24420 pull_request: https://github.com/python/cpython/pull/25728 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by miss-islington : -- pull_requests: +24419 pull_request: https://github.com/python/cpython/pull/25727 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by miss-islington : -- pull_requests: +24418 pull_request: https://github.com/python/cpython/pull/25726 ___ Python tracker ___

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: New changeset 76cd81d60310d65d01f9d7b48a8985d8ab89c8b4 by Senthil Kumaran in branch 'master': bpo-43882 - urllib.parse should sanitize urls containing ASCII newline and tabs. (GH-25595)

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Senthil Kumaran added the comment: I have added a PR to remove ascii newlines and tabs from URL input. It is as per the WHATWG spec. However, I still like to research more and find out if this isn't introducing behavior that will break existing systems. It should also be aligned the

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by Senthil Kumaran : -- keywords: +patch pull_requests: +24315 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/25595 ___ Python tracker

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

STINNER Victor added the comment: See also bpo-43883. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by STINNER Victor : -- components: +Library (Lib) ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.

Change by STINNER Victor : -- title: urllib.parse should sanitize urls containing ASCII newline and tabs. -> [security] urllib.parse should sanitize urls containing ASCII newline and tabs. ___ Python tracker