[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Guido van Rossum added the comment: If there are overspecified tests we can debate them of course. On Wed, Apr 4, 2018, 11:55 Brett Cannon wrote: > > Brett Cannon added the comment: > > And mostly for completeness, I know Thomas

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Brett Cannon added the comment: And mostly for completeness, I know Thomas has maligned the fact that zipimport has tests for some crazy things in its semantics which made it difficult for him to improve the module. -- ___ Python

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Guido van Rossum added the comment: OK, this sounds much better: zipimport has problems and we should fix them, presumably by getting Thomas's fixes (sans Google-isms, which may require some effort). I don't believe Łukasz's recommendation "don't use it" is the right attitude

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Barry A. Warsaw added the comment: Take a look at https://bugs.python.org/issue25711 which wants to rewrite zipimport from scratch. I took Serhiy's C->Python patch and turned it into a PR, but it would need a lot of love to get it into shape. (Let's not even talk about

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Brett Cannon added the comment: The other problem is zipimport is written in C and no one wants to deal with that headache if possible. ;) We actually tweaked the importlib.resources API specifically to avoid having to muck with zipimport's C code as it would have required

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Steve Dower added the comment: I recommend and use it without trouble, but then I rarely defer imports to worker threads. Barry may have some ideas, but I think this really shouldn't be difficult to do. importlib is pretty extensible, and it's not difficult to read

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Guido van Rossum added the comment: That's rather disturbing, and none of that is reflected in the zipimport docs. Is there at least a bpo issue about it? I may be the last person in the world to still recommend putting a zipfile on sys.path -- I used to believe it's faster

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Łukasz Langa added the comment: In the context of this issue: zipimport doesn't properly lock on imports happening in multiple threads at the same time. This causes one thread to receive a partially imported module while the other thread is still performing the import.

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Brett Cannon added the comment: There's a bunch of things wrong with zipimport, just ask Thomas. ;) -- nosy: +twouters ___ Python tracker

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Guido van Rossum added the comment: Yeah, what's wrong with zipimport? -- nosy: +gvanrossum ___ Python tracker ___

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: Łukasz Langa: "If you're looking at this issue because it still happens to you, you are probably using zipimport. Stop using zipimport." Hum, do you mean that zipimport also has an issue? Maybe open a new issue? --

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Łukasz Langa added the comment: If you're looking at this issue because it still happens to you, you are probably using zipimport. Stop using zipimport. -- nosy: +lukasz.langa ___ Python tracker

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: Victor: > I'm not sure that backporting the unit test is interesting. If the test > fails, we will be unable to fix it. Serhiy closed his PR: https://github.com/python/cpython/pull/2645#event-1181495893 So now I close this issue. -- resolution: ->

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: With Python 3.5.4rc1 released (3.5.4 will be the last 3.5 release accepting bug fixes), I'm not sure that it's still worth it to try to backport such changes to Python 3.5. The importlib is very important, any regression can be very important. I'm not sure

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: New changeset bb323b261d0f41ad6998d3a16c41b29b3bb8e982 by Victor Stinner in branch '3.6': bpo-30891: Fix again importlib _find_and_load() (#2665) (#2801) https://github.com/python/cpython/commit/bb323b261d0f41ad6998d3a16c41b29b3bb8e982 --

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Changes by STINNER Victor : -- pull_requests: +2852 ___ Python tracker ___ ___

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: New changeset e72b1359f81d1dd42bd8a5c5cc2b3928b74f8023 by Victor Stinner in branch 'master': bpo-30891: Fix again importlib _find_and_load() (#2665) https://github.com/python/cpython/commit/e72b1359f81d1dd42bd8a5c5cc2b3928b74f8023 --

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Changes by STINNER Victor : -- pull_requests: -2734 ___ Python tracker ___ ___

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Changes by Cooper Lees : -- pull_requests: +2734 ___ Python tracker ___ ___

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Changes by STINNER Victor : -- pull_requests: +2730 ___ Python tracker ___ ___

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: Backporting the fix ot Python 3.5 doesn't seem trivial, 3.5 lacks many changes. -- ___ Python tracker ___

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: New changeset fe6e686c27ce3a3ecdf03803cff7e230dee5530d by Victor Stinner in branch '3.6': bpo-30891: Fix importlib _find_and_load() race condition (#2646) (#2651) https://github.com/python/cpython/commit/fe6e686c27ce3a3ecdf03803cff7e230dee5530d --

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Changes by STINNER Victor : -- pull_requests: +2717 ___ Python tracker ___ ___

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

STINNER Victor added the comment: New changeset 4f9a446f3fb42f800e73cd9414dd1eccb3ca4fa7 by Victor Stinner in branch 'master': bpo-30891: Fix importlib _find_and_load() race condition (#2646) https://github.com/python/cpython/commit/4f9a446f3fb42f800e73cd9414dd1eccb3ca4fa7 --

[issue30891] importlib: _find_and_load() race condition on sys.modules[name] check

Changes by STINNER Victor : -- title: Sometimes, test_concurrency() of test_import fails with AttributeError: module 'package' has no attribute 'submodule' on AMD64 Windows8.1 Refleaks 3.6 -> importlib: _find_and_load() race condition on sys.modules[name]