[issue29694] race condition in pathlib mkdir with flags parents=True

Mariatta Wijaya added the comment: I merged Armin's PR and backported tp 3.5 and 3.6. Closing this now. Thanks all :) -- resolution: -> fixed stage: backport needed -> resolved status: open -> closed ___ Python tracker

[issue29694] race condition in pathlib mkdir with flags parents=True

Mariatta Wijaya added the comment: New changeset d7abeb7024b9755c291c29bdc8c4494246e975ad by Mariatta in branch '3.5': [3.5] bpo-29694: race condition in pathlib mkdir with flags parents=True (GH-1089). (GH-1127)

[issue29694] race condition in pathlib mkdir with flags parents=True

Mariatta Wijaya added the comment: New changeset cbc46afa59dcc43c2c8c90ae7a0a0dc404325a89 by Mariatta in branch '3.6': [3.6] bpo-29694: race condition in pathlib mkdir with flags parents=True (GH-1089). (GH-1126)

[issue29694] race condition in pathlib mkdir with flags parents=True

Changes by Mariatta Wijaya : -- pull_requests: +1262 ___ Python tracker ___ ___

[issue29694] race condition in pathlib mkdir with flags parents=True

Changes by Mariatta Wijaya : -- pull_requests: +1261 ___ Python tracker ___ ___

[issue29694] race condition in pathlib mkdir with flags parents=True

Changes by Mariatta Wijaya : -- stage: test needed -> backport needed ___ Python tracker ___

[issue29694] race condition in pathlib mkdir with flags parents=True

Mariatta Wijaya added the comment: New changeset 22a594a0047d7706537ff2ac676cdc0f1dcb329c by Mariatta (Armin Rigo) in branch 'master': bpo-29694: race condition in pathlib mkdir with flags parents=True (GH-1089) https://github.com/python/cpython/commit/22a594a0047d7706537ff2ac676cdc0f1dcb329c

[issue29694] race condition in pathlib mkdir with flags parents=True

Armin Rigo added the comment: https://github.com/python/cpython/pull/1089 (I fixed the problem with my CLA check. Now https://cpython-devguide.readthedocs.io/pullrequest.html#licensing says "you can ask for the CLA check to be run again" but doesn't tell how to do that, so as far as I can

[issue29694] race condition in pathlib mkdir with flags parents=True

Changes by Roundup Robot : -- pull_requests: +1232 ___ Python tracker ___

[issue29694] race condition in pathlib mkdir with flags parents=True

Armin Rigo added the comment: Update: a review didn't show any other similar problems (pathlib.py is a thin layer after all). Applied the fix and test (x2.diff) inside PyPy. -- ___ Python tracker

[issue29694] race condition in pathlib mkdir with flags parents=True

Armin Rigo added the comment: Maybe we should review pathlib.py for this kind of issues and first apply the fixes and new tests inside PyPy. That sounds like a better way to get things done for these rare issues, where CPython is understandably reluctant to do much changes. Note that the

[issue29694] race condition in pathlib mkdir with flags parents=True

Christoph Böddeker added the comment: I had a problem that can be solved with the presented change. But I had also problems to reproduce it in a small example. I am not sure if a test is allowed to depend on external libraries. The code at the end executed with mpirun -np 3 python test.py

[issue29694] race condition in pathlib mkdir with flags parents=True

Armin Rigo added the comment: Changes including a test. The test should check all combinations of "concurrent" creation of the directory. It hacks around at pathlib._normal_accessor.mkdir (patching "os.mkdir" has no effect, as the built-in function was already extracted and stored inside

[issue29694] race condition in pathlib mkdir with flags parents=True

Serhiy Storchaka added the comment: You can monkey-patch os.mkdir and pathlib._NormalAccessor.mkdir. Without tests we can't be sure that the patch fixes the issue and that the bug will be not reintroduced by future changes. Tests are required for this change. --

[issue29694] race condition in pathlib mkdir with flags parents=True

Armin Rigo added the comment: It's a mess to write a test, because the exact semantics of .mkdir() are not defined as far as I can tell. This patch is a best-effort attempt at making .mkdir() work in the presence of common parallel filesystem changes, that is, other processes that would

[issue29694] race condition in pathlib mkdir with flags parents=True

Serhiy Storchaka added the comment: Can you provide tests? -- nosy: +serhiy.storchaka stage: patch review -> test needed ___ Python tracker ___

[issue29694] race condition in pathlib mkdir with flags parents=True

Changes by Berker Peksag : -- components: +Library (Lib) nosy: +berker.peksag stage: -> patch review ___ Python tracker ___

[issue29694] race condition in pathlib mkdir with flags parents=True

Armin Rigo added the comment: A different bug in the same code: if someone creates the directory itself between the two calls to ``self._accessor.mkdir(self, mode)``, then the function will fail with an exception even if ``exist_ok=True``. Attached is a patch that fixes both cases.

[issue29694] race condition in pathlib mkdir with flags parents=True

New submission from whitespacer: When pathlib mkdir is called with parents=True and some parent doesn't exists it recursively calls self.parent.mkdir(parents=True) after catching OSError. However after catching of OSError and before call to self.parent.mkdir(parents=True) somebody else can