[issue13150] Most of Python's startup time is sysconfig
Roundup Robot added the comment: New changeset be3b4aa2ad28 by doko in branch '2.7': - Issue #13150, #17512: sysconfig no longer parses the Makefile and config.h http://hg.python.org/cpython/rev/be3b4aa2ad28 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Roundup Robot added the comment: New changeset d174cb3f5b9e by Benjamin Peterson in branch '2.7': backout 66e30c4870bb for breaking OSX (#13150) http://hg.python.org/cpython/rev/d174cb3f5b9e -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Roundup Robot added the comment: New changeset 66e30c4870bb by doko in branch '2.7': - Issue #13150: sysconfig no longer parses the Makefile and config.h files http://hg.python.org/cpython/rev/66e30c4870bb -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Matthias Klose d...@debian.org added the comment: the current ability to cross-build python now relies on being able to run the build python with the host library, using the _sysconfigdata.py from the host. if somebody decides to implement _sysconfigdata as a C extension, please ensure that this information still can be passed to the build python. -- nosy: +doko ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 677e625e2ef1 by Victor Stinner in branch 'default': Issue #13150: Add a comment in _sysconfigdata to explain the origin of this file http://hg.python.org/cpython/rev/677e625e2ef1 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 70160b53117f by Antoine Pitrou in branch 'default': Issue #13150: sysconfig no longer parses the Makefile and config.h files http://hg.python.org/cpython/rev/70160b53117f -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Antoine Pitrou pit...@free.fr added the comment: Done! If someone wants to give life to the C approach, they are welcome :) -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Éric Araujo mer...@netwok.org added the comment: BTW, distutils2 backports the sysconfig module and cfg file from 3.3, so now the two versions will diverge. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Roumen Petrov bugtr...@roumenpetrov.info added the comment: 10x for solution, 10x for commit . Good bye cross compilation! Any attempt to improve python build system to support cross-build, multilib build, build outside source tree with different options is useless. -- nosy: +rpetrov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Éric Araujo mer...@netwok.org added the comment: Since #9878 proposes an *alternate* solution to *part* of the sysconfig problem, I disagree with 'supersede'. It’s also an older issue. A Python solution would be more useful for other implementations if enough of the sysconfig info is not CPython specific. That’s the point: the info currently parsed at runtime by sysconfig is specific to CPython (Makefile and pyconfig.h), so adding a CPython-specific C module was thought the way to go. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Antoine Pitrou pit...@free.fr added the comment: A Python solution would be more useful for other implementations if enough of the sysconfig info is not CPython specific. That’s the point: the info currently parsed at runtime by sysconfig is specific to CPython (Makefile and pyconfig.h), so adding a CPython-specific C module was thought the way to go. A module doesn't have to be written in C to be CPython-specific. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Éric Araujo mer...@netwok.org added the comment: #9878 should also help with start-up time. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Changes by STINNER Victor victor.stin...@haypocalc.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Éric Araujo mer...@netwok.org added the comment: Actually, #9878 should supersede this bug: it proposes to generate a C module to avoid parsing Makefile and pyconfig.h, and your patch proposes to generate a Python module with the same goal. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Antoine Pitrou pit...@free.fr added the comment: Actually, #9878 should supersede this bug: it proposes to generate a C module to avoid parsing Makefile and pyconfig.h, and your patch proposes to generate a Python module with the same goal. Well, #9878 doesn't have a patch, but perhaps Barry is willing to work on one. Also, if we have a pure Python solution, perhaps a C module isn't needed. The main advantage of the C solution, though, would be to avoid dubious parsing altogher, even at build time. -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Terry J. Reedy tjre...@udel.edu added the comment: Since #9878 proposes an *alternate* solution to *part* of the sysconfig problem, I disagree with 'supersede'. A Python solution would be more useful for other implementations if enough of the sysconfig info is not CPython specific. A CPython design feature is that it parses and compiles Python code just once per run, and imported modules just once until the code changes (or might have). For functions, everything possible is put into a behind-the-scenes code object. So even inner functions are parsed and compiled just once. The problem with sysconfig, it appears, is that lacks the equivalent design feature but instead does the equivalent of re-parsing and re-compiling inner functions with each outer function call. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Changes by John O'Connor tehj...@gmail.com: -- nosy: +jcon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Changes by Nadeem Vawda nadeem.va...@gmail.com: -- nosy: +nadeem.vawda ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Antoine Pitrou pit...@free.fr added the comment: Actually, a big part of that is compiling some regexes in the tokenize module. Just relying on the re module's internal caching shaves off 20% of total startup time. Before: $ time ./python -S -c 'import tokenize' real0m0.034s user0m0.030s sys 0m0.003s $ time ./python -c '' real0m0.055s user0m0.050s sys 0m0.005s After: $ time ./python -S -c 'import tokenize' real0m0.021s user0m0.019s sys 0m0.001s $ time ./python -c '' real0m0.044s user0m0.038s sys 0m0.006s -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset df950158dc33 by Antoine Pitrou in branch 'default': Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. http://hg.python.org/cpython/rev/df950158dc33 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Tarek Ziadé ziade.ta...@gmail.com added the comment: I am curious: wouldn't be a way of keeping the compiled expressions in a static cache somewhere, so we would compile them just once and have both import time and runtime fast ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset ed0bc92fed68 by Antoine Pitrou in branch 'default': Use a dict for faster sysconfig startup (issue #13150) http://hg.python.org/cpython/rev/ed0bc92fed68 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Antoine Pitrou pit...@free.fr added the comment: I am curious: wouldn't be a way of keeping the compiled expressions in a static cache somewhere, so we would compile them just once and have both import time and runtime fast ? Runtime shouldn't be affected. The re module has its own LRU caching. That said, it seems regular expressions are pickleable: b'\x80\x03cre\n_compile\nq\x00X\x00\x00\x00\x00q\x01K \x86q\x02Rq\x03.' -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Antoine Pitrou pit...@free.fr added the comment:
Arg damn roundup e-mail gateway.
I wanted to paste:
pickle.dumps(re.compile(''))
b'\x80\x03cre\n_compile\nq\x00X\x00\x00\x00\x00q\x01K \x86q\x02Rq\x03.'
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13150
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Antoine Pitrou pit...@free.fr added the comment: Pre-parsing and building a cached module of built-time variables (from Makefile and pyconfig.h) under POSIX also removes more than 15% of startup time. Patch attached. -- keywords: +patch Added file: http://bugs.python.org/file23377/sysconfigdata.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Ross Lagerwall rosslagerw...@gmail.com added the comment: #11454 is another case where pre-parsing and pickling the regular expressions in the email module may improve import time considerably. -- nosy: +rosslagerwall ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
New submission from Antoine Pitrou pit...@free.fr: sysconfig is imported and used by site.py. $ time ./python -S -c '' real0m0.019s user0m0.013s sys 0m0.005s $ time ./python -S -c 'import sysconfig' real0m0.047s user0m0.046s sys 0m0.002s $ time ./python -S -c 'import sysconfig; sysconfig.get_path(purelib)' real0m0.053s user0m0.047s sys 0m0.005s $ time ./python -c '' real0m0.058s user0m0.054s sys 0m0.003s -- messages: 145328 nosy: eric.araujo, ncoghlan, pitrou, tarek, terry.reedy priority: normal severity: normal status: open title: Most of Python's startup time is sysconfig type: performance versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13150] Most of Python's startup time is sysconfig
Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13150 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
