[issue14366] Supporting lzma compression in zip files
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset fccdcd83708a by Martin v. Löwis in branch 'default': Issue #14366: Support lzma compression in zip files. http://hg.python.org/cpython/rev/fccdcd83708a -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Martin v. Löwis mar...@v.loewis.de added the comment: Thanks for the patch! -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Serhiy Storchaka storch...@gmail.com added the comment: Thank you, Martin. Both of these patches basically your merit. Maybe take a look also at the small patches to issue14315 and issue10376? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Martin v. Löwis mar...@v.loewis.de added the comment: Serhiy, just to be sure we communicated well: I'm not asking that all the missing features for a 6.3 level zip library must be implemented. Instead, we need to detect whether an unsupported feature is used, and raise an exception reporting what the feature is that is unsupported. This is necessary as we otherwise may return incorrect data. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Serhiy Storchaka storch...@gmail.com added the comment:
I understand you, Martin. The time it took me to read the specification
and understand where should be the checks in the module. The patch
updated. The list of compression methods extended (in the future it can
be used for detailed output in the printdir()), flag of strict
encryption is checked directly, encrypted or compressed central
directory just will not be found (BadZipFile will be raised).
--
Added file: http://bugs.python.org/file25528/lzma_in_zip_3.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14366
___diff -r e08c3791f035 Doc/library/zipfile.rst
--- a/Doc/library/zipfile.rst Thu May 10 16:36:02 2012 +0200
+++ b/Doc/library/zipfile.rst Thu May 10 23:17:48 2012 +0300
@@ -97,12 +97,20 @@
.. versionadded:: 3.3
+.. data:: ZIP_LZMA
+
+ The numeric constant for the LZMA compression method. This requires the
+ lzma module.
+
+ .. versionadded:: 3.3
+
.. note::
The ZIP file format specification has included support for bzip2
compression
- since 2001. However, some tools (including older Python releases) do not
- support it, and may either refuse to process the ZIP file altogether, or
- fail to extract individual files.
+ since 2001, and for LZMA compression since 2006. However, some tools
+ (including older Python releases) do not support these compression
+ methods, and may either refuse to process the ZIP file altogether,
+ or fail to extract individual files.
.. seealso::
@@ -133,11 +141,11 @@
adding a ZIP archive to another file (such as :file:`python.exe`). If
*mode* is ``a`` and the file does not exist at all, it is created.
*compression* is the ZIP compression method to use when writing the archive,
- and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`; or
- :const:`ZIP_DEFLATED`; unrecognized
- values will cause :exc:`RuntimeError` to be raised. If
:const:`ZIP_DEFLATED` or
- :const:`ZIP_BZIP2` is specified but the corresponded module
- (:mod:`zlib` or :mod:`bz2`) is not available, :exc:`RuntimeError`
+ and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`,
+ :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized
+ values will cause :exc:`RuntimeError` to be raised. If
:const:`ZIP_DEFLATED`,
+ :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified but the corresponded
module
+ (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not available,
:exc:`RuntimeError`
is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is
``True`` zipfile will create ZIP files that use the ZIP64 extensions when
the zipfile is larger than 2 GB. If it is false (the default)
:mod:`zipfile`
@@ -161,7 +169,7 @@
Added the ability to use :class:`ZipFile` as a context manager.
.. versionchanged:: 3.3
- Added support for :mod:`bzip2` compression.
+ Added support for :mod:`bzip2` and :mod:`lzma` compression.
.. method:: ZipFile.close()
diff -r e08c3791f035 Lib/test/support.py
--- a/Lib/test/support.py Thu May 10 16:36:02 2012 +0200
+++ b/Lib/test/support.py Thu May 10 23:17:48 2012 +0300
@@ -45,6 +45,11 @@
except ImportError:
bz2 = None
+try:
+import lzma
+except ImportError:
+lzma = None
+
__all__ = [
Error, TestFailed, ResourceDenied, import_module,
verbose, use_resources, max_memuse, record_original_stdout,
@@ -62,7 +67,7 @@
get_attribute, swap_item, swap_attr, requires_IEEE_754,
TestHandler, Matcher, can_symlink, skip_unless_symlink,
import_fresh_module, requires_zlib, PIPE_MAX_SIZE, failfast,
-anticipate_failure, run_with_tz, requires_bz2
+anticipate_failure, run_with_tz, requires_bz2, requires_lzma
]
class Error(Exception):
@@ -513,6 +518,8 @@
requires_bz2 = unittest.skipUnless(bz2, 'requires bz2')
+requires_lzma = unittest.skipUnless(lzma, 'requires lzma')
+
is_jython = sys.platform.startswith('java')
# Filename used for testing
diff -r e08c3791f035 Lib/test/test_zipfile.py
--- a/Lib/test/test_zipfile.py Thu May 10 16:36:02 2012 +0200
+++ b/Lib/test/test_zipfile.py Thu May 10 23:17:48 2012 +0300
@@ -13,7 +13,7 @@
from random import randint, random
from unittest import skipUnless
-from test.support import TESTFN, run_unittest, findfile, unlink,
requires_zlib, requires_bz2
+from test.support import TESTFN, run_unittest, findfile, unlink,
requires_zlib, requires_bz2, requires_lzma
TESTFN2 = TESTFN + 2
TESTFNDIR = TESTFN + d
@@ -361,6 +361,55 @@
self.assertEqual(openobj.read(1), b'1')
self.assertEqual(openobj.read(1), b'2')
+@requires_lzma
+def test_lzma(self):
+for f in (TESTFN2, TemporaryFile(), io.BytesIO()):
+self.zip_test(f, zipfile.ZIP_LZMA)
+
+@requires_lzma
+def test_open_lzma(self):
+for f in (TESTFN2, TemporaryFile(),
[issue14366] Supporting lzma compression in zip files
Serhiy Storchaka storch...@gmail.com added the comment: The patch updated to use functions encode_filter_properties and decode_filter_properties from the lzma module. Thank you, Nadeem Vawda. -- Added file: http://bugs.python.org/file25488/lzma_in_zip_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file25430/lzma_in_zip.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Nadeem Vawda nadeem.va...@gmail.com added the comment: I've committed my patch as changeset 9118ef2b651a, adding functions encode_filter_properties and decode_filter_properties to the lzma module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Nadeem Vawda nadeem.va...@gmail.com added the comment: I've put together a patch for the lzma module adding functions for encoding and decoding filter properties (see issue 14736). It's a bit bulky, though, so I'd like to get a review before committing it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Martin v. Löwis mar...@v.loewis.de added the comment: I'm not sure where the test run time actually comes from. It's certainly desirable to reduce the test run time, as long as all test purposes are preserved. It's not necessary to do redundant tests, e.g. if some test isn't about compression, there is no point in running it with all compressors (again, I'm not sure whether this actually happens). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Antoine Pitrou pit...@free.fr added the comment: Note that the supporting of bzip2 increases the time of testing test_zipfile in 1.5x, and lzma -- 2x (total 3x). May be not all tests are needed? I think it's ok. Some of our tests run quite longer than that; and better to be exhaustive than fast (even though exhaustive *and* fast is better, of course). -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Changes by Martin v. Löwis mar...@v.loewis.de: -- title: Supporting bzip2 and lzma compression in zip files - Supporting lzma compression in zip files ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14366] Supporting lzma compression in zip files
Serhiy Storchaka storch...@gmail.com added the comment:
Here is the patch which adds support for lzma in zipfile. Later I'll
move `lzma_props_encode` and `lzma_props_decode` to lzma module.
--
Added file: http://bugs.python.org/file25430/lzma_in_zip.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14366
___diff -r 7bdac82c16ab Doc/library/zipfile.rst
--- a/Doc/library/zipfile.rst Tue May 01 09:00:59 2012 +0200
+++ b/Doc/library/zipfile.rst Tue May 01 11:29:44 2012 +0300
@@ -97,12 +97,20 @@
.. versionadded:: 3.3
+.. data:: ZIP_LZMA
+
+ The numeric constant for the LZMA compression method. This requires the
+ lzma module.
+
+ .. versionadded:: 3.3
+
.. note::
The ZIP file format specification has included support for bzip2
compression
- since 2001. However, some tools (including older Python releases) do not
- support it, and may either refuse to process the ZIP file altogether, or
- fail to extract individual files.
+ since 2001, and for LZMA compression since 2006. However, some tools
+ (including older Python releases) do not support these compression
+ methods, and may either refuse to process the ZIP file altogether,
+ or fail to extract individual files.
.. seealso::
@@ -133,11 +141,11 @@
adding a ZIP archive to another file (such as :file:`python.exe`). If
*mode* is ``a`` and the file does not exist at all, it is created.
*compression* is the ZIP compression method to use when writing the archive,
- and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`; or
- :const:`ZIP_DEFLATED`; unrecognized
- values will cause :exc:`RuntimeError` to be raised. If
:const:`ZIP_DEFLATED` or
- :const:`ZIP_BZIP2` is specified but the corresponded module
- (:mod:`zlib` or :mod:`bz2`) is not available, :exc:`RuntimeError`
+ and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`,
+ :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized
+ values will cause :exc:`RuntimeError` to be raised. If
:const:`ZIP_DEFLATED`,
+ :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified but the corresponded
module
+ (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not available,
:exc:`RuntimeError`
is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is
``True`` zipfile will create ZIP files that use the ZIP64 extensions when
the zipfile is larger than 2 GB. If it is false (the default)
:mod:`zipfile`
@@ -161,7 +169,7 @@
Added the ability to use :class:`ZipFile` as a context manager.
.. versionchanged:: 3.3
- Added support for :mod:`bzip2` compression.
+ Added support for :mod:`bzip2` and :mod:`lzma` compression.
.. method:: ZipFile.close()
diff -r 7bdac82c16ab Lib/test/support.py
--- a/Lib/test/support.py Tue May 01 09:00:59 2012 +0200
+++ b/Lib/test/support.py Tue May 01 11:29:44 2012 +0300
@@ -45,6 +45,11 @@
except ImportError:
bz2 = None
+try:
+import lzma
+except ImportError:
+lzma = None
+
__all__ = [
Error, TestFailed, ResourceDenied, import_module,
verbose, use_resources, max_memuse, record_original_stdout,
@@ -62,7 +67,7 @@
get_attribute, swap_item, swap_attr, requires_IEEE_754,
TestHandler, Matcher, can_symlink, skip_unless_symlink,
import_fresh_module, requires_zlib, PIPE_MAX_SIZE, failfast,
-anticipate_failure, run_with_tz, requires_bz2
+anticipate_failure, run_with_tz, requires_bz2, requires_lzma
]
class Error(Exception):
@@ -513,6 +518,8 @@
requires_bz2 = unittest.skipUnless(bz2, 'requires bz2')
+requires_lzma = unittest.skipUnless(lzma, 'requires lzma')
+
is_jython = sys.platform.startswith('java')
# Filename used for testing
diff -r 7bdac82c16ab Lib/test/test_zipfile.py
--- a/Lib/test/test_zipfile.py Tue May 01 09:00:59 2012 +0200
+++ b/Lib/test/test_zipfile.py Tue May 01 11:29:44 2012 +0300
@@ -13,7 +13,7 @@
from random import randint, random
from unittest import skipUnless
-from test.support import TESTFN, run_unittest, findfile, unlink,
requires_zlib, requires_bz2
+from test.support import TESTFN, run_unittest, findfile, unlink,
requires_zlib, requires_bz2, requires_lzma
TESTFN2 = TESTFN + 2
TESTFNDIR = TESTFN + d
@@ -361,6 +361,55 @@
self.assertEqual(openobj.read(1), b'1')
self.assertEqual(openobj.read(1), b'2')
+@requires_lzma
+def test_lzma(self):
+for f in (TESTFN2, TemporaryFile(), io.BytesIO()):
+self.zip_test(f, zipfile.ZIP_LZMA)
+
+@requires_lzma
+def test_open_lzma(self):
+for f in (TESTFN2, TemporaryFile(), io.BytesIO()):
+self.zip_open_test(f, zipfile.ZIP_LZMA)
+
+@requires_lzma
+def test_random_open_lzma(self):
+for f in (TESTFN2, TemporaryFile(), io.BytesIO()):
+self.zip_random_open_test(f, zipfile.ZIP_LZMA)
+
+
[issue14366] Supporting lzma compression in zip files
Serhiy Storchaka storch...@gmail.com added the comment: Note that the supporting of bzip2 increases the time of testing test_zipfile in 1.5x, and lzma -- 2x (total 3x). May be not all tests are needed? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14366 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
