Re: how can I put a 1Gb file in a zipfile??
Bennie wrote: How can I fix this? Since this is the question that you actually asked, there is an easy answer. Investigate the problem in detail, understand the source code of the zipfile module, and the format of zip files, and develop a change to the code to correct the behaviour. Then submit a patch to sf.net/projects/python. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: how can I put a 1Gb file in a zipfile??
On Sun, 20 Mar 2005 14:17:20 -0600, Jeff Epler [EMAIL PROTECTED] wrote:
--liOOAslEiF7prFVr
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
The limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/infozip/FAQ.html#limits
statistic limit
number of files65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length64 KB=20
I had no trouble creating a zip file from a 4GB file filled with '\0'
bytes:
$ python bennie.py
$ ls -ls test.zip big
12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big
4084 -rw-rw-r-- 1 jepler jepler4174545 Mar 20 14:14 test.zip
I'm using Python 2.3.3 on Fedora Core 2.
#
# bennie.py
def make_4gb_file(f):
f =3D open(f, w)
f.seek ( 4 * 1024 * 1024 * 1024 - 1)
f.write(\0)
f.close()
Not quite OT[?]:
This makes me think there ought to be a way of making at least python's builtin
open see virtual file system objects,
analogous to StringIO creating file objects.
If we had a mountvfs('/some/unix/and/or/maybe/win/style/path/vfs',
vfsclass(some, args, if_desired))
which would result in that
open(/some/unix/and/or/maybe/win/style/path/vfs/morepath/filename.ext', mode)
(where vfsclass(some, args, if_desired) = vsfclass_intance) would call
vfsclass_instance.open('morepath.filename.ext', mode) which could then return
an object that could support
file operations like returning 4gb of virtually read-by-read-method data, or
otherwise acting like an open file
object of a real file system that python programs and library function using
open and file would find
if given the mounted path. Subdirectories could be fixed for starters, but
virtualizing subdirectory creation
etc would be possible if you intercepted the right interface calls and
implemented it in the vfs.
This would let you define a virtual file in place of the real file above, and
also would allow a lot of transparent
testing of file-using software that takes paths and file names, not open file
objects.
Of course you can't affect what the underlying os sees without getting into its
file system machinery, but
being able to mount virtual file systems into what os.open sees would cover a
lot of ground ISTM. One
could argue pro and con about supporting virtual mounts into both unix and
windows-style paths.
import zipfile
z =3D zipfile.ZipFile(/tmp/test.zip, w, zipfile.ZIP_DEFLATED)
make_4gb_file(/tmp/big)
z.write(/tmp/big)
z.close()
#
Regards,
Bengt Richter
--
http://mail.python.org/mailman/listinfo/python-list
Re: how can I put a 1Gb file in a zipfile??
From what I understand, Winzip uses their own proprietary version of a Zip format. That means that you will only be able open those archives with Winzip. (IOW, you are locked in) I think someone mentioned having developed a Python module for the 7zip fomat, but I may be wrong. In any case, you could simply write a wrapper over the command line version of 7zip and your problem would be solved. There is a Linux version as well. (http://p7zip.sourceforge.net/) bennie wrote: Christos TZOTZIOY Georgiou wrote: AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip and to the total size of the zip file. This limit comes from the zip file specification (32 bit offsets). Can it be that you are creating a zip file that its total size exceeds the limit? That is possible. But with Winzip program it can. How come that is with ZipFile not works -- http://mail.python.org/mailman/listinfo/python-list
how can I put a 1Gb file in a zipfile??
Hi, I have a problem with ZipFile. It works okay untily I come across a file that is greater then 1Gb. Then it exit with the error: OverflowError: long int too large to convert to int How can I fix this? souce: zip = zipfile.ZipFile(file, 'w') for all in os.walk(os.getcwd()): path = all[0] for document in all[2]: zipaccview.write(path + os.sep + document) zip.close() Bennie -- http://mail.python.org/mailman/listinfo/python-list
Re: how can I put a 1Gb file in a zipfile??
On Sun, 20 Mar 2005 10:44:06 +0100, rumours say that Bennie [EMAIL PROTECTED] might have written: Hi, I have a problem with ZipFile. It works okay untily I come across a file that is greater then 1Gb. Then it exit with the error: OverflowError: long int too large to convert to int How can I fix this? AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip and to the total size of the zip file. This limit comes from the zip file specification (32 bit offsets). souce: zip = zipfile.ZipFile(file, 'w') for all in os.walk(os.getcwd()): path = all[0] for document in all[2]: zipaccview.write(path + os.sep + document) zip.close() Can it be that you are creating a zip file that its total size exceeds the limit? -- TZOTZIOY, I speak England very best. Be strict when sending and tolerant when receiving. (from RFC1958) I really should keep that in mind when talking with people, actually... -- http://mail.python.org/mailman/listinfo/python-list
Re: how can I put a 1Gb file in a zipfile??
Christos TZOTZIOY Georgiou wrote: On Sun, 20 Mar 2005 10:44:06 +0100, rumours say that Bennie [EMAIL PROTECTED] might have written: Hi, I have a problem with ZipFile. It works okay untily I come across a file that is greater then 1Gb. Then it exit with the error: OverflowError: long int too large to convert to int How can I fix this? AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip and to the total size of the zip file. This limit comes from the zip file specification (32 bit offsets). souce: zip = zipfile.ZipFile(file, 'w') for all in os.walk(os.getcwd()): path = all[0] for document in all[2]: zipaccview.write(path + os.sep + document) zip.close() Can it be that you are creating a zip file that its total size exceeds the limit? That is possible. But with Winzip program it can. How come that is with ZipFile not works -- http://mail.python.org/mailman/listinfo/python-list
Re: how can I put a 1Gb file in a zipfile??
The limits of ZIP files according to the folks who make info-zip: http://www.info-zip.org/pub/infozip/FAQ.html#limits statistic limit number of files65,536 uncompressed size of a single file 4 GB compressed size of a single file 4 GB total size of archive 256 TB maximum path/filename length64 KB I had no trouble creating a zip file from a 4GB file filled with '\0' bytes: $ python bennie.py $ ls -ls test.zip big 12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big 4084 -rw-rw-r-- 1 jepler jepler4174545 Mar 20 14:14 test.zip I'm using Python 2.3.3 on Fedora Core 2. # # bennie.py def make_4gb_file(f): f = open(f, w) f.seek ( 4 * 1024 * 1024 * 1024 - 1) f.write(\0) f.close() import zipfile z = zipfile.ZipFile(/tmp/test.zip, w, zipfile.ZIP_DEFLATED) make_4gb_file(/tmp/big) z.write(/tmp/big) z.close() # pgpZ211p2vOse.pgp Description: PGP signature -- http://mail.python.org/mailman/listinfo/python-list
