[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[coroa]

coroa  added the comment:

Just realised that the "race condition" is triggered consistently on Mac OSX 
machines working on non-Mac filesystems, where they store extended attributes 
like last access time in meta files with a `._` prefix. These files are listed 
by `os.scandir` after their original files and are removed, as soon as the 
original file is unlinked.

Ie. a test.txt, ._test.txt pair will always raise FileNotFoundError for 
._test.txt since the file system driver itself already removed it when the 
unlink on test.txt was executed.

Refer also to https://stackoverflow.com/a/70355470/2873952.

Current status seems to be:
1. https://bugs.python.org/issue29699 and https://bugs.python.org/issue37260 
are duplicates.
2. https://github.com/python/cpython/pull/13580 linked from here has been 
closed due to lack of unit tests (but with the possibility to re-open after 
providing those)
3. https://github.com/python/cpython/pull/14064 is still open and seems to be 
mostly the same patch, does have unit tests and awaiting a second review.

What is the next step?

--
nosy: +coroa
versions:  -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Irit Katriel]

Irit Katriel  added the comment:

PR13580 was not declined, the reviewer requested unit tests. If you want to 
advance this, perhaps you can help fill that in.

The other PR looks like a duplicate that should be closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Noam]

Noam  added the comment:

Hi Irit, 

Sorry, I'm still not following, 

The other issue you stated, states that PR is ready(06-2019), and this was in 
2019, is this going to be fixed? or declined (and both should be closed)?

What is the current updated status?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Irit Katriel]

Irit Katriel  added the comment:

seel also issue37260

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Noam]

Noam  added the comment:

Is this still alive? If decided to decline PR why is this still open?

--
nosy: +noamda

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Jeffrey Kintscher]

Jeffrey Kintscher  added the comment:

I created pull request bpo-29699 to fix this issue. It adds an additional 
exception handler to ignore FileNotFoundError for most of the try blocks that 
already handle OSError.

I decided not to add it to the initial os.open() call. This should provide the 
same semantics as the "rm -r" shell command. It will fail with 
FileNotFoundError when foo is missing, which is the same behavior as "rm -r 
foo" returning "rm: foo: No such file or directory" when foo is missing. 
Similarly, "rm -rf foo" always succeeds and is equivalent to setting 
"ignore_errors=true" in the shutil.rmtree() call.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
keywords: +patch
pull_requests: +13487
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/13580

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
nosy: +websurfer5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Jesse Farnham]

Change by Jesse Farnham :


--
nosy: +jesse.farnham

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Shouldn't this be considered as a new feature? There are no guaranties that 
shutil.rmtree() should work if files or directories are concurrently removed, 
or created, or made read-only in other threads or processes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

+1. It looks reasonable to ignore FileNotFoundError on os.rmdir(),  os.unlink() 
and also os.open() and os.scandir().

--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Daniel Kahn Gillmor]

New submission from Daniel Kahn Gillmor:

There is a race condition in shutil.rmtree, where if a file gets removed 
between when rmtree plans to remove it and when it gets around to removing it, 
a FileNotFound exception gets raised.

The expected semantics of rmtree imply that if the filesystem tree is removed, 
then the command has succeeded, so it doesn't make sense for rmtree to raise a 
FileNotFound error if someone else happened to have deleted the file before 
rmtree gets to it.

I'm attaching a C program (for GNU/Linux) which uses inotify to remove the 
other file in a directory when either file is removed.  This triggers the 
rmtree failure.

This behavior has caused a number of workarounds in external projects, like:

  
https://bitbucket.org/vinay.sajip/python-gnupg/commits/492fd45ca073a90aac434320fb0c8fe8d01f782b
  
https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gpgme.git;a=commitdiff;h=de8494b16bc50c60a8438f2cae1f8c88e8949f7a

It would be better for shutil.rmtree to ignore this particular exception 
(FileNotFoundError).

Another option for users is to set ignore_errors=True, but this ends up 
ignoring *all* errors, which doesn't seem like the right decision.

Finally, of course, a user could specify some sort of onerror function that 
explictly ignores FileNotFoundError, but this seems pretty complicated for the 
common pattern.

It's possible that shutil.rmtree() wants to raise FileNotFoundError if the 
actual argument passed by the user does not itself exist, but it really doesn't 
make sense to raise that error for any of the elements further down in the tree.

--
components: Library (Lib)
files: breaker.c
messages: 288822
nosy: dkg
priority: normal
severity: normal
status: open
title: shutil.rmtree should not fail with FileNotFoundError (race condition)
type: crash
versions: Python 3.5
Added file: http://bugs.python.org/file46687/breaker.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29699] shutil.rmtree should not fail with FileNotFoundError (race condition)

[Daniel Kahn Gillmor]

Daniel Kahn Gillmor added the comment:

and here is python demonstration script that will build breaker.c and then use 
it to cause the error to be raised from shutils.rmtree.

the output of demo.py looks like this:

make: 'breaker' is up to date.
Traceback (most recent call last):
  File "./demo.py", line 14, in 
shutil.rmtree('xx')
  File "/usr/lib/python3.5/shutil.py", line 480, in rmtree
_rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.5/shutil.py", line 438, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'b'

--
Added file: http://bugs.python.org/file46688/demo.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com