[issue19081] zipimport behaves badly when the zip file changes while the process is running

Change by Georg Brandl : -- nosy: -georg.brandl ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: On POSIX systems, keeping the file open means you will keep a handle to the original file in the case where something moves a new file into it's place (as is normal during software package updates) or otherwise unlinks the original. That is the situation

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Daniel Jewell added the comment: In playing with Lib/zipfile.py and Lib/zipimport.py, I noticed that zipfile has supported opportunistic loading of bz2/lzma for ~9 years. However, zipimport assumes only zlib will be used. (Yet, zipfile.PyZipFile will happily create zlib/bz2/lzma ZIP

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Philip Lee added the comment: and I got ZipImportError: bad local file header -- ___ Python tracker ___ ___ Python-bugs-list

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Philip Lee added the comment: The issue still remains in Python 3.8. -- nosy: +iMath versions: +Python 3.8 -Python 2.7, Python 3.4, Python 3.5, Python 3.6 ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Change by Tulir Asokan : -- nosy: +tulir ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Serhiy Storchaka added the comment: zipimport have been rewritten in pure Python (issue25711). -- ___ Python tracker ___ ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: I'm unassigning as i won't be figuring out how to hackishly fix this again. The best solution is a complete rewrite of zipimport. this remains a known issue: if you use zipimport, do not allow the zip file to change out from underneath your running process

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +superluser versions: +Python 3.6 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Florent Xicluna florent.xicl...@gmail.com: -- nosy: +flox ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___ ___ Python-bugs-list

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset 3dd8b0d31543 by Benjamin Peterson in branch '2.7': backout #19081 to fix #20621 http://hg.python.org/cpython/rev/3dd8b0d31543 -- ___ Python tracker rep...@bugs.python.org

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Benjamin Peterson added the comment: Note everything was backed out for #19081. -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset 10f09881320d by Benjamin Peterson in branch '2.7': finish backing out #19081 http://hg.python.org/cpython/rev/10f09881320d -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset 3350c6b7aa28 by Benjamin Peterson in branch '2.7': remove tests for #19081 http://hg.python.org/cpython/rev/3350c6b7aa28 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: What do you mean by everything? How much did you back out? I *ONLY* wanted the patches I posted in 20621 backed out as those were the source of the problem. The changes made on 2014-01-06 and 2014-01-07 were good even though they only addressed part of

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Benjamin Peterson added the comment: In 2.7, I backed out 8dbf8edb7128 - f9c54ada1b32 in zipimport. Most of the commits mentioned this issue, so I assumed they were related. I apologize if that was too much. It was quite messy with all the subsequent fixups with the original commit.

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Benjamin Peterson added the comment: Anyway, isn't the change 20b77ff040b6 on Jan 07 exactly what is backed out in the patch on #20621? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Benjamin Peterson added the comment: On 3.3/default, d28242a636c7, 2807a5f011e4, fafac90b69c4 were removed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: At this point i'll be reapplying things since 8dbf8edb7128 to 2.7 and trying to untangle where the problem came in. I confirmed with diff that you've backed everything out to the pre-January state in all branches. I've had part of those 2.7 patches

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset ecb272bae417 by Gregory P. Smith in branch '3.3': Remove issue 19081 Misc/NEWS entry since it isn't entirely fixed yet. http://hg.python.org/cpython/rev/ecb272bae417 New changeset 03fc7449f204 by Gregory P. Smith in branch 'default': Remove issue

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: Fixed in the 2.7 branch. The existing patch in the 3.3 and 3.4 branches is incomplete and does not actually fix the problem. Despite what the Misc/NEWS entry claims. The patch I am attaching now (applies to 3.3, I will forward port it to 3.4) fixes the

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset ca5431a434d6 by Gregory P. Smith in branch '2.7': Remove unneeded use of globals() and locals() in test on imports http://hg.python.org/cpython/rev/ca5431a434d6 -- ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Georg Brandl added the comment: Since this is a pretty big code churn, I'd prefer B) for 3.3.4. (3.3.5 will be soon anyway.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: Thanks Georg. I'll leave it until after the 3.3.4 release. For simplicity's sake I'll leave it for 3.4.1 as well unless Larry says otherwise. -- priority: release blocker - deferred blocker ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset 323cb02abfc6 by Gregory P. Smith in branch '2.7': Issue #19081: Remove the zipimporter.files reference as the zip TOC http://hg.python.org/cpython/rev/323cb02abfc6 -- ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Brett Cannon added the comment: Just to answer Nick's question: I read the extension import code just last month to see if I could update it for PEP 451 fast enough (obviously the answer was no =). As for zipimport, I read that (and the extension import code yet again) in the 3.3 timeframe

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Nick Coghlan added the comment: Heh, OK. I've managed to avoid learning the gory details of the zipimporter internals so far, and the details of Windows DLL loading is the gap on the extension module side :) -- ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset d140e3f41796 by Gregory P. Smith in branch '2.7': Refactor the new test for issue19081 to exec import statements into a http://hg.python.org/cpython/rev/d140e3f41796 -- ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: I refactored the unittests a bit and added another test for subimports when a directory within the .zip file is in sys.path as well. The quizzical wtf TODO I had in the code has been answered in that I am unable to trigger a situation where path != buf and

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: Updated to make that function (zipimporter_init) easier to follow by not repurposing the path variable mid-function and allocate the local path buffer rather than keeping it on the stack. -- Added file:

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Nick Coghlan added the comment: I dunno Brett - have you read the extension module import code and the zipimport code lately? I don't think I'll be willing to claim to fully understand even the default importers until we've rewritten those as PEP 451 compliant Python code with a couple of

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: The call stack for that code path failing is: #0 get_data (fp=0xf64920, archive=0xe77ecc /home/greg/sandbox/python/cpython/2.7/junk95142.zip, toc_entry=('/home/greg/sandbox/python/cpython/2.7/junk95142.zip/ziptestpackage/ziptestmodule.py', 0, 16, 16,

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: The problem appears to be that every zipimporter instance keeps its own reference to the zip files table of contents (self-files) instead of continually using the module wide zip_directory_cache but the zip_stat_cache is being maintained externally to that

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: Here's a fix that I believe works with one TODO in there that needs investigation due to a question about the current historical zipimport code. -- Added file: http://bugs.python.org/file33614/issue19081-subimport-fix-gps02.diff

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Brett Cannon added the comment: Can't you at least say you don't know how import works unless you're Canadian? =) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: Not all cases of this were fixed by the existing patch. subimports still trigger the bug via a different code path. attaching an updated unittest that demonstrates that. -- resolution: fixed - status: closed - open Added file:

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Brett Cannon added the comment: While replacing zipimport with a pure Python version using importlib would be a great feather to have in importlib's cap, the dependencies in zipfile would make that somewhat difficult: http://hg.python.org/cpython/file/f1f707dd7cae/Lib/zipfile.py --

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset e09644eb6b20 by Gregory P. Smith in branch '2.7': Should fix the issue19081 fix on Windows. Don't let the previous http://hg.python.org/cpython/rev/e09644eb6b20 -- ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: updated 3.3 patch based off the changes made to the 2.7 one. -- Added file: http://bugs.python.org/file9/issue19081-33-gps05.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: Removed file: http://bugs.python.org/file33323/issue19081-33-gps04.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: Added file: http://bugs.python.org/file33356/issue19081-33-gps06.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: Removed file: http://bugs.python.org/file9/issue19081-33-gps05.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset 2807a5f011e4 by Gregory P. Smith in branch '3.3': Fixes Issue #19081: When a zipimport .zip file in sys.path being imported from http://hg.python.org/cpython/rev/2807a5f011e4 New changeset 20b77ff040b6 by Gregory P. Smith in branch 'default': Fixes

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset 5609135c6e86 by Gregory P. Smith in branch '2.7': cleanup for the issue 19081 fix - pull the file open and close outside of the http://hg.python.org/cpython/rev/5609135c6e86 -- ___ Python tracker

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: I believe this is done. I'm opting not to go for a more complicated cache the open FILE* with the zip_directory_cache approach for 3.4 due to complexity and time. long term: It'd be ideal if zipimport weren't a pile of C code separate from the zipfile

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: attaching a patch for 3.3. up next, 3.4: So long as I get to it before the release candidates the approach can likely be improved for 3.4 to actually hold the zip file we're importing from open for the life of the process instead of doing all of these stat

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: Removed file: http://bugs.python.org/file31869/issue19081-gps02.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: Removed file: http://bugs.python.org/file31860/issue19081-gps01.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___ ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Roundup Robot added the comment: New changeset 8dbf8edb7128 by Gregory P. Smith in branch '2.7': Fixes issue19081: When a zipimport .zip file in sys.path being imported http://hg.python.org/cpython/rev/8dbf8edb7128 New changeset 90a99059aa36 by Gregory P. Smith in branch '2.7': news entry for

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: The Windows 7 buildbot is unhappy after that change and failing one of the new tests: ERROR: testZipFileChangesAfterFirstImport (test.test_zipimport.ZipFileModifiedAfterImportTestCase) Alter the zip file after caching its index and try an import.

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +brett.cannon, eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___ ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: Added file: http://bugs.python.org/file31869/issue19081-gps02.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Gregory P. Smith g...@krypto.org: Added file: http://bugs.python.org/file31871/issue19081-gps03.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Gregory P. Smith added the comment: Here's a fix (the test is now in the patch). -- keywords: +needs review stage: - patch review Added file: http://bugs.python.org/file31860/issue19081-gps01.diff ___ Python tracker rep...@bugs.python.org

[issue19081] zipimport behaves badly when the zip file changes while the process is running

Changes by Daniel Stutzbach stutzb...@google.com: -- nosy: +stutzbach ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19081 ___ ___ Python-bugs-list

[issue19081] zipimport behaves badly when the zip file changes while the process is running

New submission from Gregory P. Smith: If you are 1) using zipimport 2) zipimport.c has cached the zip file's central index during an import. 3) the .zip file is modified or replaced while the process is running 4) you try to import something new from that .zip file. you're gonna have a bad