Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset bbaab666e6c7 by Antoine Pitrou in branch 'default':
Issue #14043: Speed up importlib's _FileFinder by at least 8x, and add a new
importlib.invalidate_caches() function.
http://hg.python.org/cpython/rev/bbaab666e6c7
Antoine Pitrou pit...@free.fr added the comment:
Committed now (with invalidate_caches() as well).
--
resolution: - fixed
stage: commit review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
Antoine Pitrou pit...@free.fr added the comment:
Another possibility would be to include an explicit invalidate_caches()
function in importlib.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
Antoine Pitrou pit...@free.fr added the comment:
This is a patch demonstrating invalidate_caches().
As for test_runpy, it seems runpy uses imp.get_loader(), so I don't see how it
can remain synchronized with importlib's state.
--
Added file:
Brett Cannon br...@python.org added the comment:
I wouldn't worry too much about test_runpy as it has always been a touchy test
for me.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
New submission from Antoine Pitrou pit...@free.fr:
This patch makes importlib's _FileFinder more than 10x faster by keeping a
cache of directory entries. It's actually faster than imp.find_module()!
- imp.find_module:
$ ./python -m timeit -s import imp imp.find_module('logging', None)
1
Changes by Antoine Pitrou pit...@free.fr:
Removed file: http://bugs.python.org/file24547/find_module_cache.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
___
Changes by Antoine Pitrou pit...@free.fr:
Added file: http://bugs.python.org/file24548/find_module_cache.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
___
Antoine Pitrou pit...@free.fr added the comment:
Note that the cost of filling the cache itself isn't trivial:
$ ./python -m timeit -s import sys; from importlib import _bootstrap
_bootstrap._file_path_hook('Lib')._fill_cache()
1000 loops, best of 3: 1.88 msec per loop
--
Antoine Pitrou pit...@free.fr added the comment:
Updated patch with faster cache filling.
--
Added file: http://bugs.python.org/file24549/find_module_cache2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
Brett Cannon br...@python.org added the comment:
I have not had a chance to deep-dive in the patch, but I just wanted to
double-check that this (a) implemented the idea PJE and I discussed on
python-dev, and (b) you re-generate the patch as I pushed some changes to
importlib today.
Antoine Pitrou pit...@free.fr added the comment:
I have not had a chance to deep-dive in the patch, but I just wanted
to double-check that this (a) implemented the idea PJE and I discussed
on python-dev
I haven't followed this discussion.
, and (b) you re-generate the patch as I pushed
Antoine Pitrou pit...@free.fr added the comment:
Regenerated patch.
--
Added file: http://bugs.python.org/file24550/find_module_cache3.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
Brett Cannon br...@python.org added the comment:
Just to fill you in, the discussion centred on the idea of doing a listdir() of
the directory the FileFinder was in charge of watching and caching that. Then,
when it had to look up a file all it had to do was stat the directory to look
for a
Antoine Pitrou pit...@free.fr added the comment:
Just to fill you in, the discussion centred on the idea of doing a
listdir() of the directory the FileFinder was in charge of watching
and caching that. Then, when it had to look up a file all it had to do
was stat the directory to look for a
Brett Cannon br...@python.org added the comment:
OK, I have gone ahead and done a review over on rietveld for the code as-is.
But I do have a design question.
Why pre-calculate everything? In the most common case any single module will be
imported once, if at all. And once it is imported it
Antoine Pitrou pit...@free.fr added the comment:
Why pre-calculate everything? In the most common case any single
module will be imported once, if at all. And once it is imported it
will get cached in sys.modules, alleviating the need to hit the finder
again. So from a performance standpoint
Antoine Pitrou pit...@free.fr added the comment:
Patch with light-weight alternative.
--
Added file: http://bugs.python.org/file24551/listdir_cache.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
Brett Cannon br...@python.org added the comment:
On Fri, Feb 17, 2012 at 14:31, Antoine Pitrou rep...@bugs.python.orgwrote:
Antoine Pitrou pit...@free.fr added the comment:
Why pre-calculate everything? In the most common case any single
module will be imported once, if at all. And once
Brett Cannon br...@python.org added the comment:
And while I'm thinking about it, the hybrid caching approach of caching just
the directory contents to start and then caching full details on successful
searches (and failures) would also let you cache the full file path as the
strings will be
Antoine Pitrou pit...@free.fr added the comment:
And while I'm thinking about it, the hybrid caching approach of
caching just the directory contents to start and then caching full
details on successful searches (and failures) would also let you cache
the full file path as the strings will be
Antoine Pitrou pit...@free.fr added the comment:
Updated light-weight caching patch (now checks PYTHONCASEOK dynamically).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
___
Changes by Antoine Pitrou pit...@free.fr:
Added file: http://bugs.python.org/file24552/listdir_cache2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
___
Brett Cannon br...@python.org added the comment:
The lastest patch (listdir_cache2) LGTM. Have you by any chance patched it in
and run ./python -m importlib.test.regrtest (runs the entire test suite with
builtins.__import__() substituted with importlib.__import__())? If not I can do
it on my
Antoine Pitrou pit...@free.fr added the comment:
The lastest patch (listdir_cache2) LGTM. Have you by any chance
patched it in and run ./python -m importlib.test.regrtest (runs the
entire test suite with builtins.__import__() substituted with
importlib.__import__())?
There are a couple of
Brett Cannon br...@python.org added the comment:
Well, it's the reason you added file size to .pyc files. =) I assume if you
store the system clock as well to see if no time has changed it would kill the
performance gain. We could also have two versions of _FileFinder such that
people could
Antoine Pitrou pit...@free.fr added the comment:
Well, it's the reason you added file size to .pyc files. =)
pyc files only store the integer timestamp. Here I store the original
floating-point timestamp, and I assumed it would be sufficient :)
I assume if you store the system clock as well
Changes by Eric Snow ericsnowcurren...@gmail.com:
--
nosy: +eric.snow
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14043
___
___
Python-bugs-list
28 matches
Mail list logo
