[issue24564] shutil.copytree fails when copying NFS to NFS

[Michael Burt]

Michael Burt  added the comment:

This is still a problem when shutil gets a errno.ENOSYS

I hit this bug on Microsoft Azure when I mount an Azure File (managed NFS) into 
an AKS cluster (managed Kubernetes offering) and try to copy a file from the 
NFS over to the local disk on the node using shutil.copytree().

The workaround I am using came from this StackOverflow answer: 
https://stackoverflow.com/a/51635427/3736286

--
nosy: +Michael Burt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Giampaolo Rodola']

Change by Giampaolo Rodola' :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions:  -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:


New changeset f1487b323549e2360460383b4304f6592fb38e27 by Giampaolo Rodola 
(Miss Islington (bot)) in branch '3.7':
bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() (GH-13369)
https://github.com/python/cpython/commit/f1487b323549e2360460383b4304f6592fb38e27


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[miss-islington]

Change by miss-islington :


--
pull_requests: +13562
pull_request: https://github.com/python/cpython/pull/13673

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:


New changeset a16387ab2d85f19665920bb6ff91a7e57f59dd2a by Giampaolo Rodola 
(Ying Wang) in branch 'master':
bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() (GH-13369)
https://github.com/python/cpython/commit/a16387ab2d85f19665920bb6ff91a7e57f59dd2a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Cheryl Sabella]

Cheryl Sabella  added the comment:

Nosying Hynek and Larry, as they had done the original code.

--
nosy: +cheryl.sabella, hynek, larry
versions:  -Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Ying Wang]

Change by Ying Wang :


--
keywords: +patch
pull_requests: +13279

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[hans.meine]

hans.meine  added the comment:

I completely agree with msg322848, both with the suggested fix and with its 
rationale.  A future API improvement could be to have a "strict" mode where 
errors copying xattr are not silently ignored anymore (if there are src xattr, 
but they cannot be written, or if src xattr cannot be read, for instance), but 
the current functionality seems to be "copy xattr if possible", and that should 
be fixed accordingly.

I also opened #36850 arguing that in any case, EINVAL should be ignored when 
*reading* extended attributes, acknowledging that people might have more 
problems with ignoring errors on *writing* them when they were present on the 
source.  However, as stated above, I personally vote for the fix suggested in 
msg322848.

--
nosy: +hans-meine

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Charlie Dyson]

Change by Charlie Dyson :


--
nosy: +cdyson37

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[desbma]

desbma  added the comment:

> Since the `copy_function` is customizable to switch between `copy` and 
> `copy2`, making copystat optional on files, perhaps the `copystat` should be 
> optional on directories, as well.

Related: https://bugs.python.org/issue32073

--
nosy: +desbma

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

[ https://bugs.python.org/issue24564#msg246278 ]

> Adding `EINVAL` to the ignored errnos would fix the problem, but might hide 
> real failures (I'm not sure about the real failures, but it seems logical).

I think this is an acceptable compromise considering that:

1) because of this the copy operation will ultimately fail (despite file will 
be copied)
2) there is nothing we can do except emit a warning somehow 
3) EPERM and ENODATA are already silently ignored (also ENOTSUP but that is 
more legit)

For completeness, other than copytree() also copy2() and move() are affected. 
I submitted a PR: https://github.com/python/cpython/pull/8601.

--
nosy: +giampaolo.rodola
pull_requests: +8109
stage:  -> patch review
versions: +Python 2.7, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components: +Library (Lib)
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Albert Zeyer]

Albert Zeyer  added the comment:

I'm also affected by this, with Python 3.6. My home directory is on a 
ZFS-backed NFS share.
See here for details:
https://github.com/Linuxbrew/homebrew-core/issues/4799

Basically:
Copying setuptools.egg-info to 
/u/zeyer/.linuxbrew/lib/python3.6/site-packages/setuptools-36.5.0-py3.6.egg-info
error: [Errno 5] Input/output error: 
'/u/zeyer/.linuxbrew/lib/python3.6/site-packages/setuptools-36.5.0-py3.6.egg-info/zip-safe'

Note that also by other tools, such as `mv` and `cp`, I get errors about 
setting `system.nfs4_acl`. But they just seem to ignore that and go on. I think 
this is the right thing to do here. You can print a warning about that, but 
then just go on. Maybe esp. just for `system.nfs4_acl`.

--
nosy: +Albert.Zeyer
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Gabriel Devenyi]

Gabriel Devenyi added the comment:

I'm running into this issue with python 3.5 and a ZFS-backed NFS4 mount.

Strace of a setuptools install:
chmod("/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg",
 0644) = 0
listxattr("dist/pydpiper-2.0-py3.4.egg", "system.nfs4_acl\0", 256) = 16
getxattr("dist/pydpiper-2.0-py3.4.egg", "system.nfs4_acl", 
"\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16\x01\x87\x00\x00\x00\x06OWNER@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x06GROUP@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x09EVERYONE@\x00\x00",
 128) = 80
setxattr("/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg",
 "system.nfs4_acl", 
"\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16\x01\x87\x00\x00\x00\x06OWNER@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x06GROUP@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x09EVERYONE@\x00\x00",
 80, 0) = -1 EIO (Input/output error)
stat("/tmp/easy_install-41z79x8r", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/tmp/easy_install-41z79x8r", 
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
getdents(4, /* 2 entries */, 32768) = 48
getdents(4, /* 0 entries */, 32768) = 0
close(4)= 0
rmdir("/tmp/easy_install-41z79x8r") = 0
write(2, "error: [Errno 5] Input/output er"..., 125error: [Errno 5] 
Input/output error: 
'/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg'
) = 125

--
nosy: +Gabriel Devenyi
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Min RK]

Min RK added the comment:

On a bit of further investigation, the NFS files have an xattr 
`system.nfs4_acl`. This can be read, but attempting to write it fails with 
EINVAL. Attempting to copy from NFS to non-NFS fails with ENOTSUP, which is 
caught and ignored, but copying from NFS to NFS raises EINVAL, which raises.

Adding `EINVAL` to the ignored errnos would fix the problem, but might hide 
real failures (I'm not sure about the real failures, but it seems logical).

Since the `copy_function` is customizable to switch between `copy` and `copy2`, 
making copystat optional on files, perhaps the `copystat` should be optional on 
directories, as well.

--
nosy: +minrk

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24564
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Jess Hamrick]

New submission from Jess Hamrick:

shutil.copytree seems to fail when copying files across NFS filesystems. In 
this example (see bug_demo.py), /tmp is a normal ext4 filesystem and the 
current working directory is NFS (version 4). Interestingly, it works fine to 
to copy between ext4 and NFS, just not NFS and NFS (even if it's the same NFS 
mount).

Depending on the specific version of 3.4, there are slightly different errors. 
For 3.4.0:

$ uname -a
Linux compmodels-node 3.13.0-55-generic #94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 
2015 x86_64 x86_64 x86_64 GNU/Linux
$ python3 bug_demo.py

--
Python version info:
3.4.0 (default, Jun 19 2015, 14:20:21)
[GCC 4.8.2]
--

Copying NFS -- /tmp
Copying /tmp -- NFS
Copying NFS -- NFS
Traceback (most recent call last):
  File /usr/lib/python3.4/shutil.py, line 336, in copytree
copystat(src, dst)
  File /usr/lib/python3.4/shutil.py, line 212, in copystat
_copyxattr(src, dst, follow_symlinks=follow)
  File /usr/lib/python3.4/shutil.py, line 152, in _copyxattr
os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
OSError: [Errno 22] Invalid argument: 'demo_files3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File bug_demo.py, line 27, in module
shutil.copytree('demo_files', 'demo_files3')
  File /usr/lib/python3.4/shutil.py, line 339, in copytree
if why.winerror is None:
AttributeError: 'OSError' object has no attribute 'winerror'


And for 3.4.3:

$ uname -a
Linux compmodels-node-02 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 
UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ python3 bug_demo.py

--
Python version info:
3.4.3 (default, Mar 26 2015, 22:03:40)
[GCC 4.9.2]
--

Copying NFS -- /tmp
Copying /tmp -- NFS
Copying NFS -- NFS
Traceback (most recent call last):
  File bug_demo.py, line 27, in module
shutil.copytree('demo_files', 'demo_files3')
  File /usr/lib/python3.4/shutil.py, line 343, in copytree
raise Error(errors)
shutil.Error: [('demo_files', 'demo_files3', [Errno 22] Invalid argument: 
'demo_files3')]


This is probably related to https://bugs.python.org/issue17076 and my guess is 
that's also why there is a difference in the error messages between 3.4.0 and 
3.4.3.

--
files: bug_demo.py
messages: 246272
nosy: jhamrick
priority: normal
severity: normal
status: open
title: shutil.copytree fails when copying NFS to NFS
type: crash
versions: Python 3.4
Added file: http://bugs.python.org/file39864/bug_demo.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24564
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[Jess Hamrick]

Jess Hamrick added the comment:

Some further information: if I run copystat directly on 3.4.3, I get 
essentially the same error as on 3.4.0. So really it only looks like the 
difference is just in how the error is reported:


Traceback (most recent call last):
  File bug_demo.py, line 31, in module
shutil.copystat('demo_files', 'demo_files3')
  File /usr/lib/python3.4/shutil.py, line 213, in copystat
_copyxattr(src, dst, follow_symlinks=follow)
  File /usr/lib/python3.4/shutil.py, line 153, in _copyxattr
os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
OSError: [Errno 22] Invalid argument: 'demo_files3'


Also, I should mention that I did test this on 2.7 as well, and it is not an 
issue there.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24564
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24564] shutil.copytree fails when copying NFS to NFS

[R. David Murray]

R. David Murray added the comment:

There are a number of open issues with copytree originating from copystat.  It 
would be great if someone could pull them all together and propose a solution.  
Making it optional might indeed be the best solution.

--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24564
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com