[issue29694] race condition in pathlib mkdir with flags parents=True

[Mariatta Wijaya]

Mariatta Wijaya added the comment:

I merged Armin's PR and backported tp 3.5 and 3.6.
Closing this now.
Thanks all :)

--
resolution:  -> fixed
stage: backport needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Mariatta Wijaya]

Mariatta Wijaya added the comment:


New changeset d7abeb7024b9755c291c29bdc8c4494246e975ad by Mariatta in branch 
'3.5':
[3.5] bpo-29694: race condition in pathlib mkdir with flags parents=True 
(GH-1089). (GH-1127)
https://github.com/python/cpython/commit/d7abeb7024b9755c291c29bdc8c4494246e975ad


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Mariatta Wijaya]

Mariatta Wijaya added the comment:


New changeset cbc46afa59dcc43c2c8c90ae7a0a0dc404325a89 by Mariatta in branch 
'3.6':
[3.6] bpo-29694: race condition in pathlib mkdir with flags parents=True 
(GH-1089). (GH-1126)
https://github.com/python/cpython/commit/cbc46afa59dcc43c2c8c90ae7a0a0dc404325a89


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
pull_requests: +1262

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
pull_requests: +1261

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
stage: test needed -> backport needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Mariatta Wijaya]

Mariatta Wijaya added the comment:


New changeset 22a594a0047d7706537ff2ac676cdc0f1dcb329c by Mariatta (Armin Rigo) 
in branch 'master':
bpo-29694: race condition in pathlib mkdir with flags parents=True (GH-1089)
https://github.com/python/cpython/commit/22a594a0047d7706537ff2ac676cdc0f1dcb329c


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Armin Rigo]

Armin Rigo added the comment:

https://github.com/python/cpython/pull/1089

(I fixed the problem with my CLA check.  Now 
https://cpython-devguide.readthedocs.io/pullrequest.html#licensing says "you 
can ask for the CLA check to be run again" but doesn't tell how to do that, so 
as far as I can tell, I have to ask e.g. here.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Roundup Robot]

Changes by Roundup Robot :


--
pull_requests: +1232

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Armin Rigo]

Armin Rigo added the comment:

Update: a review didn't show any other similar problems (pathlib.py is a thin 
layer after all).  Applied the fix and test (x2.diff) inside PyPy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Armin Rigo]

Armin Rigo added the comment:

Maybe we should review pathlib.py for this kind of issues and first apply the 
fixes and new tests inside PyPy.  That sounds like a better way to get things 
done for these rare issues, where CPython is understandably reluctant to do 
much changes.

Note that the PyPy version of the stdlib already contains fixes that have not 
been merged back to CPython (or only very slowly), though so far they are the 
kind of issues that trigger more often on PyPy than on CPython, like GC issues.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Christoph Böddeker]

Christoph Böddeker added the comment:

I had a problem that can be solved with the presented change.  
But I had also problems to reproduce it in a small example.
I am not sure if a test is allowed to depend on external libraries.

The code at the end executed with
mpirun -np 3 python test.py
always breaks with the current code of pathlib and works with the fix.

Maybe this helps to write a test and shows a usecase where this fix is 
necessary.

P.S.: I was not able to produce the error with multiprosessing.

from mpi4py import MPI
from pathlib import Path
import tempfile

comm = MPI.COMM_WORLD
rank = comm.rank
size = comm.size

with tempfile.TemporaryDirectory() as tmp_dir:

tmp_dir = comm.bcast(tmp_dir, root=0)
p = Path(tmp_dir) / 'a' / 'b'
comm.barrier()
p.mkdir(parents=True, exist_ok=True)

--
nosy: +Christoph Böddeker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Armin Rigo]

Armin Rigo added the comment:

Changes including a test.  The test should check all combinations of 
"concurrent" creation of the directory.  It hacks around at 
pathlib._normal_accessor.mkdir (patching "os.mkdir" has no effect, as the 
built-in function was already extracted and stored inside 
pathlib._normal_accessor).

--
Added file: http://bugs.python.org/file46764/x2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

You can monkey-patch os.mkdir and pathlib._NormalAccessor.mkdir.

Without tests we can't be sure that the patch fixes the issue and that the bug 
will be not reintroduced by future changes. Tests are required for this change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Armin Rigo]

Armin Rigo added the comment:

It's a mess to write a test, because the exact semantics of .mkdir() are not 
defined as far as I can tell.  This patch is a best-effort attempt at making 
.mkdir() work in the presence of common parallel filesystem changes, that is, 
other processes that would create the same directories at the same time.

This patch is by no means an attempt at being a complete solution for similar 
problems.  The exact semantics have probably never been discussed at all.  For 
example, what should occur if a parent directory is removed just after .mkdir() 
created it?

I'm not suggesting to discuss these issues now, but to simply leave them open.  
I'm trying instead to explain why writing a test is a mess (more than "just" 
creating another thread and creating/removing directories very fast while the 
main thread calls .mkdir()), because we have no exact notion of what should 
work and what shouldn't.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Can you provide tests?

--
nosy: +serhiy.storchaka
stage: patch review -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Berker Peksag]

Changes by Berker Peksag :


--
components: +Library (Lib)
nosy: +berker.peksag
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[Armin Rigo]

Armin Rigo added the comment:

A different bug in the same code: if someone creates the directory itself 
between the two calls to ``self._accessor.mkdir(self, mode)``, then the 
function will fail with an exception even if ``exist_ok=True``.

Attached is a patch that fixes both cases.

--
keywords: +patch
nosy: +arigo
Added file: http://bugs.python.org/file46707/x1.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29694] race condition in pathlib mkdir with flags parents=True

[whitespacer]

New submission from whitespacer:

When pathlib mkdir is called with parents=True and some parent doesn't exists 
it recursively calls self.parent.mkdir(parents=True) after catching OSError. 
However after catching of OSError and before call to 
self.parent.mkdir(parents=True) somebody else can create parent dir, which will 
lead to FileExistsError exception.

--
messages: 288801
nosy: whitespacer
priority: normal
severity: normal
status: open
title: race condition in pathlib mkdir with flags parents=True
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com