[issue22526] file iteration SystemError for huge lines (2GiB+)
Roundup Robot added the comment: New changeset be600ea4ad13 by Serhiy Storchaka in branch '2.7': Fixed and optimized a test of issue #22526. https://hg.python.org/cpython/rev/be600ea4ad13 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Changes by Benjamin Peterson benja...@python.org: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Serhiy Storchaka added the comment: Could anyone please test it on Windows? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Martin Pitt added the comment: I now raised it to 4 GB for python2.7 This is *still* not enough; I got a success with 6 GB. But this is really demanding.. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
STINNER Victor added the comment: On Fedora 20/x86_64, running test_file2k takes up to 4.8 GB (5114316 kB) of RSS memory (VmPeak in /proc/pid/status). It looks like readahead_get_line_skip() has an efficient code to handle buffer. It uses recursive calls: --- readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.2 MB readahead: allocate 0.2 MB readahead: allocate 0.3 MB readahead: allocate 0.3 MB readahead: allocate 0.4 MB readahead: allocate 0.5 MB readahead: allocate 0.7 MB readahead: allocate 0.8 MB readahead: allocate 1.1 MB readahead: allocate 1.3 MB readahead: allocate 1.7 MB readahead: allocate 2.1 MB readahead: allocate 2.6 MB readahead: allocate 3.2 MB readahead: allocate 4.0 MB readahead: allocate 5.0 MB readahead: allocate 6.3 MB readahead: allocate 7.9 MB readahead: allocate 9.9 MB readahead: allocate 12.3 MB readahead: allocate 15.4 MB readahead: allocate 19.3 MB readahead: allocate 24.1 MB readahead: allocate 30.1 MB readahead: allocate 37.6 MB readahead: allocate 47.0 MB readahead: allocate 58.8 MB readahead: allocate 73.5 MB readahead: allocate 91.8 MB readahead: allocate 114.8 MB readahead: allocate 143.5 MB readahead: allocate 179.4 MB readahead: allocate 224.2 MB readahead: allocate 280.2 MB readahead: allocate 350.3 MB readahead: allocate 437.9 MB readahead: allocate 547.3 MB Breakpoint 2, PyObject_Malloc (nbytes=2147483733) at Objects/obmalloc.c:792 792 if (nbytes PY_SSIZE_T_MAX) (gdb) where #0 PyObject_Malloc (nbytes=2147483733) at Objects/obmalloc.c:792 #1 0x00464af1 in _PyObject_DebugMallocApi (id=111 'o', nbytes=2147483701) at Objects/obmalloc.c:1474 #2 0x00464a48 in _PyObject_DebugMalloc (nbytes=2147483701) at Objects/obmalloc.c:1441 #3 0x0046efdc in PyString_FromStringAndSize (str=0x0, size=2147483648) at Objects/stringobject.c:88 #4 0x00436c30 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=2147483648, bufsize=573933340) at Objects/fileobject.c:2291 #5 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1836553986, bufsize=459146672) at Objects/fileobject.c:2311 #6 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1469236648, bufsize=367317338) at Objects/fileobject.c:2311 #7 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1175382777, bufsize=293853871) at Objects/fileobject.c:2311 #8 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=940299680, bufsize=235083097) at Objects/fileobject.c:2311 #9 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=752233202, bufsize=188066478) at Objects/fileobject.c:2311 #10 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=601780019, bufsize=150453183) at Objects/fileobject.c:2311 #11 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=481417472, bufsize=120362547) at Objects/fileobject.c:2311 #12 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=385127434, bufsize=96290038) at Objects/fileobject.c:2311 #13 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=308095403, bufsize=77032031) at Objects/fileobject.c:2311 #14 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=246469778, bufsize=61625625) at Objects/fileobject.c:2311 #15 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=197169278, bufsize=49300500) at Objects/fileobject.c:2311 #16 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=157728878, bufsize=39440400) at Objects/fileobject.c:2311 #17 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=126176558, bufsize=31552320) at Objects/fileobject.c:2311 #18 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=100934702, bufsize=25241856) at Objects/fileobject.c:2311 #19 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=80741217, bufsize=20193485) at Objects/fileobject.c:2311 #20 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=64586429, bufsize=16154788) at Objects/fileobject.c:2311 #21 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=51662598, bufsize=12923831) at Objects/fileobject.c:2311 #22 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=41323533, bufsize=10339065) at Objects/fileobject.c:2311 #23 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=33052281, bufsize=8271252) at Objects/fileobject.c:2311 #24 0x00436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=26435279, bufsize=6617002) at Objects/fileobject.c:2311 #25
[issue22526] file iteration SystemError for huge lines (2GiB+)
Antoine Pitrou added the comment: Serhiy's patch looks ok to me (haven't tested it). -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Martin Pitt added the comment: How much memory does that whatever is running that test have? Our default is 1 GB for our test runner VMs. I now raised it to 4 GB for python2.7, but we can only do that for our x86 VMs. For other architectures (ppc64el and ARM) the test VMs just don't have that much memory. So indeed it would be nice to skip this test if the machine has less than 4 GB of RAM. Thanks! -- nosy: +pitti ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Serhiy Storchaka added the comment: Test should be marked with dry_run=False, otherwise it is false passed. Allocation of growing buffer needs extra memory, in my experiments memuse=2.5 is enough. And I think this test should require the largefile resource. Here is a patch. It also significantly speeds up a test on Linux. -- keywords: +patch nosy: +serhiy.storchaka Added file: http://bugs.python.org/file36797/issue22526_test.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Matthias Klose added the comment: no, it doesn't. at least when testing the installed python installation, it just fails: https://jenkins.qa.ubuntu.com/job/utopic-adt-python2.7/39/? -- nosy: +doko resolution: fixed - status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Benjamin Peterson added the comment: For insufficient memory not an incorrect fix, though? -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Matthias Klose added the comment: maybe, but then you should skip the test, or expect at least a MemoryError. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Benjamin Peterson added the comment: How much memory does that whatever is running that test have? On Fri, Oct 3, 2014, at 17:17, Matthias Klose wrote: Matthias Klose added the comment: maybe, but then you should skip the test, or expect at least a MemoryError. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Changes by R. David Murray rdmur...@bitdance.com: -- title: file iteration crashes for huge lines (2GiB+) - file iteration SystemError for huge lines (2GiB+) type: crash - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22526] file iteration SystemError for huge lines (2GiB+)
Roundup Robot added the comment: New changeset beadb3e1dc81 by Benjamin Peterson in branch '2.7': use Py_ssize_t for file offset and length computations in iteration (closes #22526) https://hg.python.org/cpython/rev/beadb3e1dc81 -- nosy: +python-dev resolution: - fixed stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22526 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
