[issue20321] ImportError when a module is created after a catched ImportError
Ezio Melotti added the comment: Because you inserted LIBDIR before it existed, import noticed it didn't exist and so put None into sys.path_importer_cache[LIBDIR] I'm not familiar with sys.path_importer_cache, but what happens if instead of putting None, sys.path_importer_cache[LIBDIR] is not created at all? Will it check for LIBDIR next time an import is performed (possibly finding it if the dir has been created in the meanwhile)? If this check happens every time, I guess it will defeat the purpose of the cache in case of missing dirs (especially if the dir is never created -- but maybe this is an uncommon case and the optimization can be removed?). Another option could be to recheck empty dirs in the cache in case of ImportError, and then update the cache entry and retry the import if a new dir that didn't exist before is found. This would still have an overhead, but even in this case the slow down might be acceptable. If we can't find a compromise I guess we could add a note to the documentation, even though I'm not sure where. FTR in the original report the user was adding a dir to sys.path before extracting the content of a tar file (thus creating the added dir), and adding the dir after the extraction fixes the problem. I'm also attaching my version of the test case, that shows two possible fixes to the problem and is closer to the original report. -- Added file: http://bugs.python.org/file33615/issue20321.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
Brett Cannon added the comment: So those semantics have existed as long as PEP 302 has been around, which is Python 2.3 (that PEP itself is over a decade old), so changing them now would break code. And honestly I wouldn't change it anyway. On some filesystems, stat calls are extremely costly (e.g. NFS). Even if it was only on failed imports it would still have a cost. And considering a way to stay compatible between Python 2 and 3 is to catch ImportError and then import a module whose named changed, it would still be a costly change. Now if you personally really want the semantics you are after you could have a sys.meta_path importer which cleared out sys.path_importer_cache and tried the import again. As for documentation, it's explained in the language reference: http://docs.python.org/3/reference/import.html#path-entry-finders . But otherwise there isn't another place unless someone writes a HOWTO on this, but that probably isn't a good thing as import is something you really should be weary of mucking with. For someone trying to import the contents of a tarfile, they would be better served by a tarfile importer than unpacking the tarfile and then adding a path to sys.path. But someone has to write that tarfile importer first. =) Maybe some day: issue #17630. -- resolution: - invalid status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
Brett Cannon added the comment:
I'm not sure why you think the example code as-is should work. The first entry
on sys.path is the current directory ('' or the absolute path, depending if you
are running from the interpreter prompt or specifying a file on the
command-line). Stripping off sys.path[0] guarantees the example code will not
work.
And as for why adding in '.' works on PyPy and not Python 3.3, it's because you
didn't call importlib.invalidate_caches() to clear out the directory
modification, so Python didn't notice that the file was added because the mtime
granularity for directories it larger than the time it took to have the import
for it_does_not_exist fail, write the impfile.py file, and to try importing
again.
--
assignee: - brett.cannon
resolution: - invalid
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20321
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
berdario added the comment: yes, sorry... I tried to simplify and generalize it too much (I tried to avoid creating a new directory in the test, assuming that the same behavior could make sense by only creating a new module in the current directory) I'll reupload the correct version of the failing code, and I'll try to understand importlib.invalid_caches before eventually reopening it -- Added file: http://bugs.python.org/file33592/bbug.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
Changes by berdario berda...@gmail.com: Removed file: http://bugs.python.org/file33582/bbug.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
berdario added the comment: Ok, the bug is unrelated with timings and the finder caches apparently -- resolution: invalid - status: closed - open versions: -3rd party, Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
Changes by berdario berda...@gmail.com: Added file: http://bugs.python.org/file33596/bug_with_invalidatecaches.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
Brett Cannon added the comment: It actually is a caching issue, but not with the caches in the finder but the cache *of* finders. Because you inserted LIBDIR before it existed, import noticed it didn't exist and so put None into sys.path_importer_cache[LIBDIR] (or imp.NullImporter prior to Python 3.3). If you del sys.path_importer_cache[LIBDIR] just before trying to import impfile then it works. If you leave the directory around but clear out its contents then importlib.invalidate_caches() would have been needed. As you have noticed, dynamically mucking around import is rather delicate. There are various caches and tricks used in order to speed it up since it is such a common operation. If you are trying to just load a single file that you dynamically wrote you can load the file directly using http://docs.python.org/3/library/importlib.html#importlib.machinery.SourceFileLoader (or if you need to support Python 2.7 as well, http://docs.python.org/2.7/library/imp.html#imp.load_module). Do let me know if you are trying to just load a single file. I'm contemplating adding a utility function to help with that use-case in importlib. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
New submission from berdario: This small script errors on both Python2.7, Python3.3 and Pypy2.0 (I just reduced the test case supplied to me by ezio) -- components: Interpreter Core files: bbug.py messages: 208608 nosy: berdario, ezio.melotti priority: normal severity: normal status: open title: ImportError when a module is created after a catched ImportError versions: 3rd party, Python 2.7, Python 3.3 Added file: http://bugs.python.org/file33582/bbug.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20321] ImportError when a module is created after a catched ImportError
Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +brett.cannon, eric.snow, ncoghlan stage: - test needed type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20321 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
