[issue1060] zipfile cannot handle files larger than 2GB (inside archive)
Kevin Ar18 added the comment: Here's another bug report that talks about a 2GB file limit: http://bugs.python.org/issue1189216 The diff offered there does not solve the problem; actually it's possible that the diff may not have anything to do with fixing the problem (though I'm not certain), but may just be a readability change. I tried to program a solution based on other stuff I saw/read on the internet, but ran into different problems I took the line: bytes = self.fp.read(zinfo.compress_size) and made it read a little bit at a time and add the result to bytes as it went along. This was really slow (as it had to add the result to the larger and larger bytes string each time); I tried with a list, but I couldn't find how to join the list back together into a string when done (similar to the javascript join() method). However, even with the list method, I ran into an odd memory error, as it looped through into the higher numbers, that I have no idea why it was happening, so I gave up at that point. Also, I have no idea if this one line in the zipfile module is the only problem or if there are others that will pop up once you get that part fixed. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1060 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1060] zipfile cannot handle files larger than 2GB (inside archive)
Kevin Ar18 added the comment: Just some thoughts In posting about this problem elsewhere, it has been argued that you shouldn't be copying that much stuff into memory anyways (though there are possible cases for a need for that). However, the question is what should the zipfile module do. At the very least it should account for this 2GB limitation and say it can't do it. However, how it should interact with the programmer is another question. In one of the replies, I am told that strings have a 2GB limitation, which means the zipfile module can't be used in it's current form, even if fixed. Does this mean that the zipfile module needs to add some additional methods for incrementally getting data and writing data? Or does it mean that programmers should be the ones to program an incremental system when they need it... Or? __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1060 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1060] zipfile cannot handle files larger than 2GB (inside archive)
Kevin Ar18 added the comment: So, just add an error to the module (so it won't crash)? BTW, is Python 2.6 ready for use? I could use that feature now. :) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1060 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1060] zipfile cannot handle files larger than 2GB (inside archive)
Kevin Ar18 added the comment: Maybe a message that says that strings on 32-bit CPUs cannot handle more than 2GB of data; use the stream instead? __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1060 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1060] zipfile cannot handle files larger than 2GB (inside archive)
New submission from Kevin Ar18: Summary: If you have a zip file that contains a file inside of it greater than 2GB, then the zipfile module is unable to read that file. Steps to Reproduce: 1. Create a zip file several GB in size with a file inside of it that is over 2GB in size. 2. Attempt to read the large file inside the zip file. Here's some sample code: import zipfile import re dataObj = zipfile.ZipFile(zip.zip,r) for i in dataObj.namelist(): if(i[-1] == /): print dir else: fileName = re.split(r.*/,i,0)[1] fileData = dataObj.read(i) Result: Python returns the following error: File ...\zipfile.py, line 491, in read bytes = self.fp.read(zinfo.compress_size) OverflowError: long it too large to convert to int Expected Result: It should copy the data into the variable fileData... I'll try to post more info in a follow-up. -- components: Library (Lib) messages: 55444 nosy: Kevin Ar18 severity: normal status: open title: zipfile cannot handle files larger than 2GB (inside archive) type: compile error versions: Python 2.6 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1060 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
