[issue4533] 3.0 file.read dreadfully slow
Antoine Pitrou pit...@free.fr added the comment: This has been fixed as part of the big IO update in trunk. I assume nobody really cares about making a separate patch for 2.6, please re-open if you are interested! -- resolution: accepted - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Antoine Pitrou pit...@free.fr added the comment: Since it is solved for 3.x and only needs to be bacported to 2.x (where the io module isn't the default), downgrading to critical. -- nosy: +pitrou priority: release blocker - critical ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Changes by Martin v. Löwis mar...@v.loewis.de: -- priority: deferred blocker - release blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Changes by Martin v. Löwis [EMAIL PROTECTED]: -- priority: release blocker - deferred blocker ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Christian Heimes [EMAIL PROTECTED] added the comment: This needs definitely some testing! -- components: +Extension Modules nosy: +christian.heimes priority: - release blocker stage: - test needed versions: +Python 3.1 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Christian Heimes [EMAIL PROTECTED] added the comment:
The small buffer size in Modules/_fileio.c is one reason for the slowness.
$ dd if=/dev/zero of=zeros bs=1MB count=50
$ cat testread.py
open(zeros, rb).read()
$ ./python -m cProfile testread.py
40 function calls (39 primitive calls) in 4.246 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
10.0160.0164.2464.246 string:1(module)
10.0000.0000.0000.000 io.py:277(__new__)
20.0000.0000.0000.000 io.py:355(flush)
20.0000.0000.0000.000 io.py:364(close)
20.0000.0000.0000.000 io.py:376(__del__)
10.0000.0000.0000.000 io.py:413(_checkReadable)
10.0000.0000.0000.000 io.py:614(__init__)
20.0000.0000.0000.000 io.py:618(close)
10.0000.0000.0000.000 io.py:708(__init__)
10.0000.0000.0000.000 io.py:733(flush)
10.0000.0000.0000.000 io.py:736(close)
10.0000.0000.0000.000 io.py:755(closed)
10.0000.0000.0000.000 io.py:82(open)
10.0000.0000.0000.000 io.py:896(__init__)
20.0000.0000.0000.000 io.py:905(_reset_read_buf)
10.0210.0214.2304.230 io.py:909(read)
10.0000.0004.2094.209 io.py:920(_read_unlocked)
10.0000.0000.0000.000 {built-in method
allocate_lock}
2/10.0000.0004.2464.246 {built-in method exec}
10.0000.0000.0000.000 {built-in method fstat}
20.0000.0000.0000.000 {built-in method
isinstance}
30.0000.0000.0000.000 {built-in method len}
10.0000.0000.0000.000 {method '__enter__' of
'_thread.lock' objects}
10.0000.0000.0000.000 {method 'append' of 'list'
objects}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
10.0000.0000.0000.000 {method 'fileno' of
'_FileIO' objects}
10.0000.0000.0000.000 {method 'isatty' of
'_FileIO' objects}
10.8250.8250.8250.825 {method 'join' of 'bytes'
objects}
23.3841.6923.3841.692 {method 'read' of
'_FileIO' objects}
10.0000.0000.0000.000 {method 'readable' of
'_FileIO' objects}
$ vi Modules/_fileio.c
-#define DEFAULT_BUFFER_SIZE (8*1024)
+#define DEFAULT_BUFFER_SIZE (80*1024)
$ ./python -m cProfile testread.py
40 function calls (39 primitive calls) in 1.273 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
10.0190.0191.2731.273 string:1(module)
10.0000.0000.0000.000 io.py:277(__new__)
20.0000.0000.0000.000 io.py:355(flush)
20.0000.0000.0000.000 io.py:364(close)
20.0000.0000.0000.000 io.py:376(__del__)
10.0000.0000.0000.000 io.py:413(_checkReadable)
10.0000.0000.0000.000 io.py:614(__init__)
20.0000.0000.0000.000 io.py:618(close)
10.0000.0000.0000.000 io.py:708(__init__)
10.0000.0000.0000.000 io.py:733(flush)
10.0000.0000.0000.000 io.py:736(close)
10.0000.0000.0000.000 io.py:755(closed)
10.0000.0000.0000.000 io.py:82(open)
10.0000.0000.0000.000 io.py:896(__init__)
20.0000.0000.0000.000 io.py:905(_reset_read_buf)
10.0160.0161.2541.254 io.py:909(read)
10.0000.0001.2381.238 io.py:920(_read_unlocked)
10.0000.0000.0000.000 {built-in method
allocate_lock}
2/10.0000.0001.2731.273 {built-in method exec}
10.0000.0000.0000.000 {built-in method fstat}
20.0000.0000.0000.000 {built-in method isinstance}
30.0000.0000.0000.000 {built-in method len}
10.0000.0000.0000.000 {method '__enter__' of
'_thread.lock' objects}
10.0000.0000.0000.000 {method 'append' of 'list'
objects}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
10.0000.0000.0000.000 {method 'fileno' of
'_FileIO' objects}
10.0000.0000.0000.000
[issue4533] 3.0 file.read dreadfully slow
Christian Heimes [EMAIL PROTECTED] added the comment: The fileio_buffer.patch implements the same progressive buffer as Python 2.x' Object/fileobject.c. -- keywords: +patch stage: test needed - patch review Added file: http://bugs.python.org/file12227/fileio_buffer.patch ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Gregory P. Smith [EMAIL PROTECTED] added the comment: patch looks good to me. nitpick comments: use += instead of = and + in: newsize = newsize + newsize and newsize = newsize + BIGCHUNK. As for the XXX about overflow, so long as BUFSIZ is not defined to be an insanely large number (it should never be) this will be fine. add a preprocessor test for that in. #if (BUFSIZ = 2**30) #error unreasonable BUFSIZ defined #endif -- nosy: +gregory.p.smith ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Christian Heimes [EMAIL PROTECTED] added the comment: The preprocessor doesn't handle power. 2 24 (64MB) sounds sufficient for me. Added file: http://bugs.python.org/file12228/fileio_buffer2.patch ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4533] 3.0 file.read dreadfully slow
Gregory P. Smith [EMAIL PROTECTED] added the comment: fileio_buffer2.patch looks good other than minor touchups: Turn the XXX comment into: /* NOTE: overflow impossible due to limits on BUFSIZ * Also, 2 24 is 32MB yet your error message test says = 64MB. I think you meant 1 26. fix those and commit. :) ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4533 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
