[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[STINNER Victor]

STINNER Victor  added the comment:

32-bit buildbots are back to green. Thanks for the fix!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 938c00ca9e4207a2531041edff2e82490b02047f by Łukasz Langa (Miss 
Islington (bot)) in branch '3.8':
bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size 
(GH-16491) (#16506)
https://github.com/python/cpython/commit/938c00ca9e4207a2531041edff2e82490b02047f


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

Looks like it worked:
https://buildbot.python.org/all/#/builders/176/builds/1383

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[Giampaolo Rodola']

Change by Giampaolo Rodola' :


--
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[miss-islington]

Change by miss-islington :


--
pull_requests: +16093
pull_request: https://github.com/python/cpython/pull/16506

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:


New changeset 94e165096fd65e8237e60de570fb609604ab94c9 by Giampaolo Rodola in 
branch 'master':
bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size 
(GH-16491)
https://github.com/python/cpython/commit/94e165096fd65e8237e60de570fb609604ab94c9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[Giampaolo Rodola']

Change by Giampaolo Rodola' :


--
keywords: +patch
pull_requests: +16077
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/16491

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[STINNER Victor]

STINNER Victor  added the comment:

Similar failure on ARMv7 Debian buster 3.x:
https://buildbot.python.org/all/#/builders/176/builds/1372

pythoninfo:
sys.maxsize: 2147483647

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[STINNER Victor]

STINNER Victor  added the comment:

Oh, it's likely a regression caused by:

commit 5bcc6d89bcb622a6786fff632fabdcaf67dbb4e2
Author: Giampaolo Rodola 
Date:   Mon Sep 30 12:51:55 2019 +0800

bpo-37096: Add large-file tests for modules using sendfile(2) (GH-13676)


> https://buildbot.python.org/all/#/builders/103/builds/3162

configure:

checking for sendfile... yes
checking whether to enable large file support... yes

pythoninfo:

platform.libc_ver: glibc 2.29
platform.platform: 
Linux-4.19.72-gentoo-i686-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_5000+-with-glibc2.29

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38319] shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system

[STINNER Victor]

New submission from STINNER Victor :

Error on a 32-bit buildbot worker where ssize_t maximum = 2,147,483,647 
(2**31-1) bytes = ~2.0 GiB.

test_largefile uses:

# size of file to create (>2 GiB; 2 GiB == 2,147,483,648 bytes)
size = 2_500_000_000

x86 Gentoo Installed with X 3.x:
https://buildbot.python.org/all/#/builders/103/builds/3162

==
ERROR: test_it (test.test_largefile.TestCopyfile)
--
Traceback (most recent call last):
  File 
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_largefile.py",
 line 160, in test_it
shutil.copyfile(TESTFN, TESTFN2)
  File 
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/shutil.py",
 line 266, in copyfile
_fastcopy_sendfile(fsrc, fdst)
  File 
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/shutil.py",
 line 145, in _fastcopy_sendfile
sent = os.sendfile(outfd, infd, offset, blocksize)
OverflowError: Python int too large to convert to C ssize_t

On Linux (Fedora 30), man sendfile shows me the prototype:

   ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

Extract of Lib/shutil.py:

# Hopefully the whole file will be copied in a single call.
# sendfile() is called in a loop 'till EOF is reached (0 return)
# so a bufsize smaller or bigger than the actual file size
# should not make any difference, also in case the file content
# changes while being copied.
try:
blocksize = max(os.fstat(infd).st_size, 2 ** 23)  # min 8MB
except Exception:
blocksize = 2 ** 27  # 128MB

offset = 0
while True:
try:
sent = os.sendfile(outfd, infd, offset, blocksize)
except OSError as err:
...
else:
if sent == 0:
break  # EOF
offset += sent

Extract of the Linux implementation of os.sendfile():

int in, out;
Py_ssize_t ret;
off_t offset;
...
Py_ssize_t count;
PyObject *offobj;
static char *keywords[] = {"out", "in",
"offset", "count", NULL};
if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiOn:sendfile",
keywords, &out, &in, &offobj, &count))
return NULL;
...
if (!Py_off_t_converter(offobj, &offset))
return NULL;

do {
Py_BEGIN_ALLOW_THREADS
ret = sendfile(out, in, &offset, count);
Py_END_ALLOW_THREADS
} while (ret < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));

with:

static int
Py_off_t_converter(PyObject *arg, void *addr)
{
#ifdef HAVE_LARGEFILE_SUPPORT
*((Py_off_t *)addr) = PyLong_AsLongLong(arg);
#else
*((Py_off_t *)addr) = PyLong_AsLong(arg);
#endif
if (PyErr_Occurred())
return 0;
return 1;
}


I understand that the error comes from the 4th sendfile() parameter: "count". 
The C code (of the Linux implementation) uses the "n" format for Py_ssize_t: 
Python/getargs.c calls PyLong_AsSsize_t().

On a 64-bit system, it's less likely to reach Py_ssize_t maximum value (max = 
2**63-1), but it's easy to reach on a 32-bit system (max = 2**31-1).

--
components: Library (Lib)
messages: 353549
nosy: giampaolo.rodola, nanjekyejoannah, pablogsal, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit 
system
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com