New submission from STINNER Victor <vstin...@python.org>:
Error on a 32-bit buildbot worker where ssize_t maximum = 2,147,483,647 (2**31-1) bytes = ~2.0 GiB. test_largefile uses: # size of file to create (>2 GiB; 2 GiB == 2,147,483,648 bytes) size = 2_500_000_000 x86 Gentoo Installed with X 3.x: https://buildbot.python.org/all/#/builders/103/builds/3162 ====================================================================== ERROR: test_it (test.test_largefile.TestCopyfile) ---------------------------------------------------------------------- Traceback (most recent call last): File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_largefile.py", line 160, in test_it shutil.copyfile(TESTFN, TESTFN2) File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/shutil.py", line 266, in copyfile _fastcopy_sendfile(fsrc, fdst) File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/shutil.py", line 145, in _fastcopy_sendfile sent = os.sendfile(outfd, infd, offset, blocksize) OverflowError: Python int too large to convert to C ssize_t On Linux (Fedora 30), man sendfile shows me the prototype: ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count); Extract of Lib/shutil.py: # Hopefully the whole file will be copied in a single call. # sendfile() is called in a loop 'till EOF is reached (0 return) # so a bufsize smaller or bigger than the actual file size # should not make any difference, also in case the file content # changes while being copied. try: blocksize = max(os.fstat(infd).st_size, 2 ** 23) # min 8MB except Exception: blocksize = 2 ** 27 # 128MB offset = 0 while True: try: sent = os.sendfile(outfd, infd, offset, blocksize) except OSError as err: ... else: if sent == 0: break # EOF offset += sent Extract of the Linux implementation of os.sendfile(): int in, out; Py_ssize_t ret; off_t offset; ... Py_ssize_t count; PyObject *offobj; static char *keywords[] = {"out", "in", "offset", "count", NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiOn:sendfile", keywords, &out, &in, &offobj, &count)) return NULL; ... if (!Py_off_t_converter(offobj, &offset)) return NULL; do { Py_BEGIN_ALLOW_THREADS ret = sendfile(out, in, &offset, count); Py_END_ALLOW_THREADS } while (ret < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals())); with: static int Py_off_t_converter(PyObject *arg, void *addr) { #ifdef HAVE_LARGEFILE_SUPPORT *((Py_off_t *)addr) = PyLong_AsLongLong(arg); #else *((Py_off_t *)addr) = PyLong_AsLong(arg); #endif if (PyErr_Occurred()) return 0; return 1; } I understand that the error comes from the 4th sendfile() parameter: "count". The C code (of the Linux implementation) uses the "n" format for Py_ssize_t: Python/getargs.c calls PyLong_AsSsize_t(). On a 64-bit system, it's less likely to reach Py_ssize_t maximum value (max = 2**63-1), but it's easy to reach on a 32-bit system (max = 2**31-1). ---------- components: Library (Lib) messages: 353549 nosy: giampaolo.rodola, nanjekyejoannah, pablogsal, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system versions: Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue38319> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
