[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Serhiy Storchaka added the comment: New changeset c5734998d91e9953fd179ba6ed7015b6343e8191 by Serhiy Storchaka in branch 'master': bpo-33720: Refactor marshalling/unmarshalling floats. (GH-8071) https://github.com/python/cpython/commit/c5734998d91e9953fd179ba6ed7015b6343e8191 --

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Serhiy Storchaka added the comment: New changeset 9720f60f2aba457121bfe42d09aa3ed91f28b86f by Serhiy Storchaka in branch '2.7': [2.7] bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336). (GH-8107)

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by Serhiy Storchaka : -- pull_requests: +7701 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

miss-islington added the comment: New changeset 878c4fe40e8954372e9bf80ae555045ecae68e73 by Miss Islington (bot) in branch '3.6': bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336) https://github.com/python/cpython/commit/878c4fe40e8954372e9bf80ae555045ecae68e73

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

miss-islington added the comment: New changeset 3f121a40c7526aedb2a40a1be8f0ae96b267bf26 by Miss Islington (bot) in branch '3.7': bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336) https://github.com/python/cpython/commit/3f121a40c7526aedb2a40a1be8f0ae96b267bf26

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by miss-islington : -- pull_requests: +7700 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by miss-islington : -- pull_requests: +7699 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Serhiy Storchaka added the comment: New changeset fc05e68d8fac70349b7ea17ec14e7e0cfa956121 by Serhiy Storchaka in branch 'master': bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336) https://github.com/python/cpython/commit/fc05e68d8fac70349b7ea17ec14e7e0cfa956121

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Serhiy Storchaka added the comment: Unfortunately PR 8071 doesn't fix the crash on PGO builds. :-( Initially I tested it only on the debug build, and now have tested on the PGO build. -- ___ Python tracker

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: I added the backport to 3.7 label to your PR, as I want to restore the marshal stack depth limit in 3.7.1. -- ___ Python tracker ___

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: Thanks for writing that second PR, Serhiy! I'd already spotted that as the likely fix on our side, though I do want to complete the investigation into why MSVC generated unnecessarily sparse frames. But that is basically an internal discussion for me to follow

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Serhiy Storchaka added the comment: The alternate solution is provided by PR 8071. The culprit of the main stack consumption is the code for unmarshalling float and complex in protocols 0 and 1 which uses 256-bytes buffers on the stack. Moving it to the separate function allows to decrease

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by Serhiy Storchaka : -- pull_requests: +7681 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Serhiy Storchaka added the comment: What are your thoughts about rewriting the test (PR 7336)? -- ___ Python tracker ___ ___

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

STINNER Victor added the comment: > Thanks. Agreed this should stay open, but it is blocked on a compiler fix. Do you want to have a different limit depending on the compiler and the compiler version? I'm not sure that it's worth it. I'm happy with the heuristic "use lower limit on

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: Thanks. Agreed this should stay open, but it is blocked on a compiler fix. -- ___ Python tracker ___

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Ned Deily added the comment: With the workaround merged for 3.7.0rc1, I'm downgrading from "release blocker" to "high" since I believe Steve wants to keep this issue open for a possible compiler resolution. -- priority: release blocker -> high

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

miss-islington added the comment: New changeset 103058e19bea48f4a80afde51df04338a753e952 by Miss Islington (bot) in branch '3.7': bpo-33720: Reduces maximum marshal recursion depth on release builds. (GH-7401) https://github.com/python/cpython/commit/103058e19bea48f4a80afde51df04338a753e952

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: Thanks. I merged, and miss-islington is backporting it. I guess at this stage you're manually cherry-picking into your own 3.7 branch for release? I'll merge for 3.7.1 though. -- ___ Python tracker

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Ned Deily added the comment: > I guess at this stage you're manually cherry-picking into your own 3.7 branch > for release? No, not until 3.7.0rc1 next week. No need to cherry pick. Thanks! -- ___ Python tracker

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: New changeset 2a4a62ba4ae770bbc7b7fdec0760031c83fe1f7b by Steve Dower in branch 'master': bpo-33720: Reduces maximum marshal recursion depth on release builds. (GH-7401) https://github.com/python/cpython/commit/2a4a62ba4ae770bbc7b7fdec0760031c83fe1f7b

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by miss-islington : -- pull_requests: +7031 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Ned Deily added the comment: Steve, I'm OK with the proposed reduction as a workaround. We have other platform- and build-specific limits based on stack sizes etc. It's hard to have one-size-fits-all. -- ___ Python tracker

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: Ned - PR 7401 fixes the crash at the cost of reducing the permitted recursion depth in marshal. Your call whether we take this fix or disable PGO, but I'd much rather take this fix (PGO has full-interpreter effects whereas this is very narrow). When I hear

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by Steve Dower : -- pull_requests: +7027 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Serhiy Storchaka added the comment: As for the test itself, the original test was added in dc78cc6f7cea2040114a350aa9db835cc433ae05. It tested that the stack overflow is not happen when unmarshal a fake code object with a deeply nested dict instead of co_code. It was renamed to

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by Serhiy Storchaka : -- keywords: +patch pull_requests: +6967 stage: -> patch review ___ Python tracker ___ ___

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: FYI, I posted the problem at https://developercommunity.visualstudio.com/content/problem/265778/pgo-generates-large-stack-frame.html (I _think_ it's public). Until then, it's probably best to reduce the recursion limit. Going to a depth of 1000 seems okay,

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: I need to stop working on this right now, but here's the locals layout in a normal release build in r_object: @rdi @rdip = 0x0034`655ea3d0 0034`65403f60 @rsp+0x0080 v =

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: A crash in the test suite should be fixed, especially since we have protection against this crash (and a test that validates it). In this case, apparently the stack allocation for each frame of r_object grew and now there isn't room for 2000 calls (the value

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

STINNER Victor added the comment: > priority: normal -> release blocker I don't think that it's a release blocker. test_marshal does only crash on corner cases which should not occur on usual "valid" data. -- ___ Python tracker

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: Ned, FYI -- nosy: +ned.deily priority: normal -> release blocker versions: +Python 3.8 ___ Python tracker ___

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Steve Dower added the comment: The uploaded binary is compiled with PGO enabled (and trained on most of the test suite). I'll check it out - hopefully we don't need to do anything drastic and can get away with either a compiler update or disabling optimizations on a single function.

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

STINNER Victor added the comment: > I compiled the master branch of Python in release mode using VS2015 (MSC > v.1912 64 bit) and I failed to reproduce the crash I also failed to reproduce the crash in the 3.7 branch. I guess that the python.org binary has been compiled differently.

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by STINNER Victor : -- versions: +Python 3.7 -Python 3.8 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

Change by STINNER Victor : -- type: -> crash ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

STINNER Victor added the comment: I compiled the master branch of Python in release mode using VS2015 (MSC v.1912 64 bit) and I failed to reproduce the crash: * PCbuild/build.bat -e -p x64 * python -m test -v test_marshal * no crash -- ___ Python

[issue33720] test_marshal: crash in Python 3.7b5 on Windows 10

New submission from STINNER Victor : Follow-up of bpo-33719. C:\Users\vstinner\AppData\Local\Programs\Python\Python37>python.exe -m test test_marshal -v == CPython 3.7.0b5 (v3.7.0b5:abb8802389, May 31 2018, 01:54:01) [MSC v.1913 64 bit (AMD64)] == Windows-10-10.0.16299-SP0 little-endian ==