[issue11051] system calls per import
Changes by Éric Araujo mer...@netwok.org: -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Charles-François Natali neolo...@free.fr added the comment:
As noted by Antoine somewhere else (don't remember if it was on the bug tracker
or mailing list), most of the startup time is due to site import:
cf@neobox:~/python/cpython$ time ./python -c ''
[44249 refs]
real0m0.445s
user0m0.376s
sys 0m0.032s
cf@neobox:~/python/cpython$ time ./python -S -c ''
[35211 refs]
real0m0.181s
user0m0.144s
sys 0m0.016s
And that's mostly user time, the stat() calls don't weight much here.
Here's the output of a profiling:
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
2/10.0060.0030.3030.303 {built-in method exec}
10.0040.0040.3030.303 test_startup.py:1(module)
10.0000.0000.2670.267 site.py:509(main)
10.0000.0000.2370.237 site.py:254(addusersitepackages)
10.0000.0000.2360.236 site.py:231(getusersitepackages)
10.0060.0060.2260.226 site.py:217(getuserbase)
10.0110.0110.2140.214 sysconfig.py:1(module)
10.0040.0040.1220.122 configparser.py:120(module)
300.0060.0000.1200.004 {built-in method __build_class__}
90.0000.0000.1160.013 re.py:204(compile)
90.0020.0000.1160.013 functools.py:185(wrapper)
90.0000.0000.1130.013 re.py:256(_compile)
90.0010.0000.1130.013 sre_compile.py:488(compile)
10.0010.0010.0960.096
configparser.py:555(RawConfigParser)
On my (somewhat old) laptop, sysconfig import consumes 1/3 of the
total startup time.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11051
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset a541bda2f5e2 by Charles-François Natali in branch 'default': Issue #11051: Reduce the number of syscalls per import. http://hg.python.org/cpython/rev/a541bda2f5e2 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Martin v. Löwis mar...@v.loewis.de added the comment: I would have appreciated had you considered my review before pushing that change. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Charles-François Natali neolo...@free.fr added the comment: I would have appreciated had you considered my review before pushing that change. Sorry, I didn't receive an email notification for your review, so I didn't know you had done one. I'll add a comment. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Terry J. Reedy tjre...@udel.edu added the comment: I am a little puzzled by the patch. In logic, 'A and B' is equivalent to 'not A or not B'. But in the patch, -if (_Py_stat(filename, statbuf) == 0 /it exists */ -S_ISDIR(statbuf.st_mode)) /* it's a directory */ +if (_Py_stat(filename, statbuf) != 0 || S_ISDIR(statbuf.st_mode)) you seem to change to 'not A or B', without negating B. Is this intentional? What am I missing? Some subtle effect of lazy evaluation? Or an intentional change in the logic? -- nosy: +terry.reedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
STINNER Victor victor.stin...@haypocalc.com added the comment: By the way, _Py_stat() can fail because of a Python error: -1 result is not handled in import.c :-( It may leak to the XXX undetected error message. -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Terry J. Reedy tjre...@udel.edu added the comment: OK, I gather from the comment added in the second patch that you intentionally changed the logic to restart the loop more often. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Charles-François Natali neolo...@free.fr added the comment: Here's a trivial patch reducing the number of calls to open. before: $ strace -c -e open ./python -c % time seconds usecs/call callserrors syscall -- --- --- - - 100.000.49 0 392 306 open -- --- --- - - 100.000.49 392 306 total after: $ strace -c -e open ./python -c % time seconds usecs/call callserrors syscall -- --- --- - - 100.000.24 086 open -- --- --- - - 100.000.2486 total As for the flury of tentative locations, I don't feel like modifying this since I'm not familiar enough with the import machinery. -- keywords: +patch nosy: +neologix Added file: http://bugs.python.org/file23858/import_stat.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Martin v. Löwis mar...@v.loewis.de added the comment: First, I don't understand why we need to check both foo.so and foomodule.so. Because we always did, i.e. changing it now may break backwards compatibility. Now, as for why we always had *module.so also: it may be that calling an extension module foo.so might not be feasible if it needs to link with a system library called foo.so (rather than libfoo.so). I don't know how serious this problem is in practice - we may consider deprecating-then-removing *module.so from the import machinery. As for checking both foo.so and foo.cpython-32m.so: that's for backwards compatibility also. Existing build processes may produce foo.so, as they did in previous Python versions (in particular, if they are not based on distutils, but, say, make). Unfortunately, PEP 3149 did not specify a deprecation scheduling. You could ask Barry - I suppose that was an oversight, and not actually deliberate. Of course, with .cpython-32m.so being present, removing *module.so would be easy enough, except again for breaking existing projects which might use that name. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Antoine Pitrou pit...@free.fr added the comment: Here's a trivial patch reducing the number of calls to open. Looks good to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Changes by Nadeem Vawda nadeem.va...@gmail.com: -- nosy: +nvawda ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Martin v. Löwis mar...@v.loewis.de added the comment: Something has gone out of control here. Why do we need to check so many alternative locations? What change do you propose? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Changes by Raymond Hettinger rhettin...@users.sourceforge.net: -- nosy: +rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Changes by Georg Brandl ge...@python.org: -- nosy: +georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Antoine Pitrou pit...@free.fr added the comment: Something has gone out of control here. Why do we need to check so many alternative locations? What change do you propose? First, I don't understand why we need to check both foo.so and foomodule.so. Second, I don't understand why we need to check both foo.cpython32.so and foo.so. If you fix both these, the number of stat() calls per would-be extension module goes down from 6 to 2 per directory path. Then there seems to be a couple of redundant stat/fstat calls on some files. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11051] system calls per import
New submission from Antoine Pitrou pit...@free.fr:
Here are the system calls when importing a single pure Python module:
stat(/home/antoine/py3k/py3k/Lib/copyreg, 0x7fff1ed1f740) = -1 ENOENT (No
such file or directory)
open(/home/antoine/py3k/py3k/Lib/copyreg.cpython-32m.so, O_RDONLY) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/copyregmodule.cpython-32m.so, O_RDONLY) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/copyreg.abi3.so, O_RDONLY) = -1 ENOENT (No
such file or directory)
open(/home/antoine/py3k/py3k/Lib/copyregmodule.abi3.so, O_RDONLY) = -1 ENOENT
(No such file or directory)
open(/home/antoine/py3k/py3k/Lib/copyreg.so, O_RDONLY) = -1 ENOENT (No such
file or directory)
open(/home/antoine/py3k/py3k/Lib/copyregmodule.so, O_RDONLY) = -1 ENOENT (No
such file or directory)
open(/home/antoine/py3k/py3k/Lib/copyreg.py, O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0664, st_size=6633, ...}) = 0
open(/home/antoine/py3k/py3k/Lib/__pycache__/copyreg.cpython-32.pyc,
O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0664, st_size=5595, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f28e6381000
read(6, l\f\r\n\0\24\212Hc\0\0\0\0\0\0\0\0\0\0\0\0\f\0\0\0@\0\0\0s\307\0...,
4096) = 4096
fstat(6, {st_mode=S_IFREG|0664, st_size=5595, ...}) = 0
This is per possible directory, which means that it can be repeated for each
dir on sys.path:
stat(readline, 0x7fff1ed28760)= -1 ENOENT (No such file or directory)
open(readline.cpython-32m.so, O_RDONLY) = -1 ENOENT (No such file or
directory)
open(readlinemodule.cpython-32m.so, O_RDONLY) = -1 ENOENT (No such file or
directory)
open(readline.abi3.so, O_RDONLY) = -1 ENOENT (No such file or directory)
open(readlinemodule.abi3.so, O_RDONLY) = -1 ENOENT (No such file or directory)
open(readline.so, O_RDONLY) = -1 ENOENT (No such file or directory)
open(readlinemodule.so, O_RDONLY) = -1 ENOENT (No such file or directory)
open(readline.py, O_RDONLY) = -1 ENOENT (No such file or directory)
open(readline.pyc, O_RDONLY) = -1 ENOENT (No such file or directory)
stat(/home/antoine/py3k/py3k/Lib/readline, 0x7fff1ed28760) = -1 ENOENT (No
such file or directory)
open(/home/antoine/py3k/py3k/Lib/readline.cpython-32m.so, O_RDONLY) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/readlinemodule.cpython-32m.so, O_RDONLY) =
-1 ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/readline.abi3.so, O_RDONLY) = -1 ENOENT (No
such file or directory)
open(/home/antoine/py3k/py3k/Lib/readlinemodule.abi3.so, O_RDONLY) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/readline.so, O_RDONLY) = -1 ENOENT (No such
file or directory)
open(/home/antoine/py3k/py3k/Lib/readlinemodule.so, O_RDONLY) = -1 ENOENT (No
such file or directory)
open(/home/antoine/py3k/py3k/Lib/readline.py, O_RDONLY) = -1 ENOENT (No such
file or directory)
open(/home/antoine/py3k/py3k/Lib/readline.pyc, O_RDONLY) = -1 ENOENT (No such
file or directory)
stat(/home/antoine/py3k/py3k/Lib/plat-linux2, {st_mode=S_IFDIR|0775,
st_size=4096, ...}) = 0
stat(/home/antoine/py3k/py3k/Lib/plat-linux2, {st_mode=S_IFDIR|0775,
st_size=4096, ...}) = 0
stat(/home/antoine/py3k/py3k/Lib/plat-linux2/readline, 0x7fff1ed28760) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readline.cpython-32m.so,
O_RDONLY) = -1 ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readlinemodule.cpython-32m.so,
O_RDONLY) = -1 ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readline.abi3.so, O_RDONLY) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readlinemodule.abi3.so,
O_RDONLY) = -1 ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readline.so, O_RDONLY) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readlinemodule.so, O_RDONLY) =
-1 ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readline.py, O_RDONLY) = -1
ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/Lib/plat-linux2/readline.pyc, O_RDONLY) = -1
ENOENT (No such file or directory)
stat(/home/antoine/py3k/py3k/build/lib.linux-x86_64-3.2,
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat(/home/antoine/py3k/py3k/build/lib.linux-x86_64-3.2,
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat(/home/antoine/py3k/py3k/build/lib.linux-x86_64-3.2/readline,
0x7fff1ed28760) = -1 ENOENT (No such file or directory)
open(/home/antoine/py3k/py3k/build/lib.linux-x86_64-3.2/readline.cpython-32m.so,
O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0775, st_size=59970, ...}) = 0
futex(0x7f28e605e0ec, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open(/home/antoine/py3k/py3k/build/lib.linux-x86_64-3.2/readline.cpython-32m.so,
O_RDONLY) = 4
Something has gone out of control here. Why
[issue11051] system calls per import
Changes by Eric Smith e...@trueblade.com: -- nosy: +eric.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
