[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: > About changing copyfileobj(), here are some test cases which may help explain > the compatibility problems: I see, I was expecting something like that after reading https://bugs.python.org/issue25063 's conclusions. I removed that part. > Perhaps we could

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Martin Panter added the comment: FYI Martin Panter and vadmium are both me :) I’m not a big fan or expert on configure.ac and Autoconf, but I guess in this case it is the simplest solution. Maybe look at some of the existing AC_CHECK_DECL and AC_COMPILE_IFELSE invocations. I guess you need to

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: Updated the patch withe most of Martin Panter's and all vadium's comments. -- Added file: http://bugs.python.org/file42628/copy_file_range.diff ___ Python tracker

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: > I didn’t see any changes to the configure script in your patches. Did you > make that change to define HAVE_COPY_FILE_RANGE yet? I'm not really sure how to make the test for configure.ac. Other functions are checked differently (availability of header files),

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Martin Panter added the comment: Yes, the RST documentation has to be done by hand. It usually has more detail than the doc strings. I didn’t see any changes to the configure script in your patches. Did you make that change to define HAVE_COPY_FILE_RANGE yet? In /Modules/posixmodule.c (all

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Changes by Antoine Pitrou : -- nosy: -pitrou ___ Python tracker ___ ___ Python-bugs-list

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Changes by Marcos Dione : Removed file: http://bugs.python.org/file42617/copy_file_range.diff ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: New version. Factoring the old method in a nested function also paves the way to implement https://bugs.python.org/issue25156 . -- Added file: http://bugs.python.org/file42619/copy_file_range.diff ___ Python tracker

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: Hmm, I just noticed that it doesn't fallback to normal copy if the arguments are not valid (EBADF, EXDEV). Back to the drawing board... -- ___ Python tracker

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: Ok, I have a preliminary version of the patch. It has several parts: * Adding the functionality to the os module, with docstring. * Make shutil.copyfileobj() to use it if available. * Modify the docs (this has to be done by hand, right?). * Modify NEWS and ACKS.

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: Version without the NEWS and ACKS change. -- Added file: http://bugs.python.org/file42617/copy_file_range.diff ___ Python tracker

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Changes by Marcos Dione : Removed file: http://bugs.python.org/file42616/copy_file_range.diff ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

STINNER Victor added the comment: If we add a new private function to the os module (ex: os._copy_file_range), it can be used in shutil if available. But it would be a temporary solution until we declare the API stable. Maybe it's ok to add the API as public today. --

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: Then I don't know. My only worry is whether having the method local to os would allow shutils.copy() to use it or not. I will start by looking at other similar functions there to see to do it. urandom() looks like a good starting point. --

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

STINNER Victor added the comment: That's the manual page of the Linux kernel, not the glibc. It doesn't mean that the glibc implemented it. -- ___ Python tracker

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: Already there: http://man7.org/linux/man-pages/man2/copy_file_range.2.html -- ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

STINNER Victor added the comment: Kerbel support ok, but what about the libc support? Do you know if it is planned? Or do you want to use the syscall() low-level API. If we take the syscall() path, I suggest to make the function private in the os module. Wait until the API is standardized in the

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Marcos Dione added the comment: Debian Sid, arch Linux and Fedora FC24 (alpha) already have linux-4.5, others would certainly follow soon. Meanwhile, I could start developing the patch and we could review it when you think it's appropriate. -- ___

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Martin Panter added the comment: Tangentially related: Issue 25156, about using sendfile() to copy files in shutil. -- nosy: +martin.panter ___ Python tracker

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

STINNER Victor added the comment: > We usually wait until syscalls are generally available in common distros and > have bindings in glibc. It makes it easier to test the feature. Usually, yeah. os.urandom() uses syscall() to use the new getrandom() of Linux since it's still not exposed in the

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Christian Heimes added the comment: Thanks, looks interesting. We usually wait until syscalls are generally available in common distros and have bindings in glibc. It makes it easier to test the feature. -- nosy: +christian.heimes ___ Python

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

Changes by STINNER Victor : -- nosy: +haypo, neologix, pitrou ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

New submission from Marcos Dione: copy_file_range() has been introduced in the Linux kernel since version 4.5 (mid march 2016). This new syscall allows to copy data from one fd to another without passing by user space, improving speed in most cases. You can read more about it here: