[issue5689] Support xz compression in tarfile module
Changes by Nadeem Vawda nadeem.va...@gmail.com: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset b86b54fcb5c2 by Lars Gustäbel in branch 'default': Issue #5689: Avoid excessive memory usage by using the default lzma preset. http://hg.python.org/cpython/rev/b86b54fcb5c2 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Ping. Windows buildbots are still failing with MemoryError because of this preset=9. The patch looks good to me as well. -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Changes by Nikolaus Rath nikol...@rath.org: -- nosy: -Nikratio ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Lars Gustäbel l...@gustaebel.de added the comment: Wouldn't it be better then to use a default compresslevel of 6 in tarfile? I used level 9 in my patch without a particular reason, just because I thought 9 must be better than 6 ;-) -- Added file: http://bugs.python.org/file24084/lzma-preset.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Nadeem Vawda nadeem.va...@gmail.com added the comment: Yes, that's a good idea. I've been testing a similar change, and it seems to drop the peak memory usage for test_tarfile from around 810MB down to under 200MB. It looks like 2GB genuinely isn't enough to reliably use LZMA compression with preset=9. You might want to use preset=None instead of explicitly saying preset=6, though. This tells LZMAFile to use the default preset, and will allow you to get rid of the if-statement on lines 1821-1823. Something unrelated that I noticed in the surrounding code: gzopen and bz2open validate the mode by testing 'len(mode) 1 or mode not in rw'. This would be simpler as 'mode not in (r, w)' (like you've done in xzopen), and it would accept only r and w (but not or rw). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Changes by Lars Gustäbel l...@gustaebel.de: Removed file: http://bugs.python.org/file24084/lzma-preset.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Lars Gustäbel l...@gustaebel.de added the comment: Yes, that's much better. Thanks for the tip. -- Added file: http://bugs.python.org/file24086/lzma-preset.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Nadeem Vawda nadeem.va...@gmail.com added the comment: Patch looks good to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Nadeem Vawda nadeem.va...@gmail.com added the comment: This failure seems to crop up often, but not on every run: http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3941/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3940/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3937/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3929/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3921/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3916/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3914/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3906/steps/test/logs/stdio I've been able to reproduce the failure on my own XP machine; I'll investigate it over the weekend. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Antoine Pitrou pit...@free.fr added the comment: Perhaps Paul can try to reproduce and diagnose the issue directly on the buildbot? -- nosy: +pmoore ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Paul Moore p.f.mo...@gmail.com added the comment: A simple rebuild and test run of that test in debug mode didn't fail... I'll run the full test suite as a check, but that could take some time - that buildslave isn't the fastest in the world... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Nadeem Vawda nadeem.va...@gmail.com added the comment: Not to worry - as I said in my previous message, I can reproduce the error on my own XP machine. I also noticed that running test_tarfile alone doesn't trigger the errors, which leads me to suspect that the failure is due to some interaction with another test getting run before test_tarfile. I'm currently trying to determine what this test is. I suspect that the problem is at least partially caused by the fact that tarfile uses a default compresslevel of 9 for .tar.xz archives (rather than the recommended value of 6). According to the man page for the xz tool http://manpages.ubuntu.com/manpages/lucid/man1/xz.1.html, using a compresslevel of 9 can result in memory usage of up to 800MB during compression, which is a significant fraction of the bot's 2GB of RAM. (I suppose it would be a good idea to mention this in the documentation for the lzma module, so users won't get bitten by this...) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
STINNER Victor victor.stin...@haypocalc.com added the comment: There is failure on a XP buildbot. I don't know if it is a sporadic issue or not. http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3921/steps/test/logs/stdio == ERROR: test_append_lzma (test.test_tarfile.AppendTest) -- Traceback (most recent call last): File D:\Buildslave\3.x.moore-windows\build\lib\test\test_tarfile.py, line 1539, in test_append_lzma self._create_testtar(w:xz) File D:\Buildslave\3.x.moore-windows\build\lib\test\test_tarfile.py, line 1486, in _create_testtar with tarfile.open(self.tarname, mode) as tar: File D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py, line 1721, in open return func(name, filemode, fileobj, **kwargs) File D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py, line 1826, in xzopen mode=mode, fileobj=fileobj, preset=preset) File D:\Buildslave\3.x.moore-windows\build\lib\lzma.py, line 117, in __init__ preset=preset, filters=filters) MemoryError -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Éric Araujo mer...@netwok.org added the comment: Lars, as part of a small doc patch I want to change this in tarfile.rst: The :mod:`tarfile` module makes it possible to read and write tar archives, including those using gzip or bz2 compression. -(:file:`.zip` files can be read and written using the :mod:`zipfile` module.) +Use the :mod:`zipfile` module to read or write :file:`.zip` files, or the +higher-level functions in :ref:`shutil archiving-operations`. Any objection? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Lars Gustäbel l...@gustaebel.de added the comment: Please, go ahead! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 899a8c7b2310 by Lars Gustäbel in branch 'default': Issue #5689: Add support for lzma compression to the tarfile module. http://hg.python.org/cpython/rev/899a8c7b2310 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Lars Gustäbel l...@gustaebel.de added the comment: Thanks for the review, guys! I can't close this issue yet because it depends on #6715. -- resolution: - fixed stage: needs patch - committed/rejected ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Nadeem Vawda nadeem.va...@gmail.com added the comment: Great stuff! I'll close this issue along with issue 6715 once the buildbot stuff is all sorted out. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Lars Gustäbel l...@gustaebel.de added the comment: For those who want to test it first, I post the current state of the patch here. It is ready for commit, there are no failing tests. If nobody objects, I will apply it this weekend. -- Added file: http://bugs.python.org/file23880/2011-12-08-tarfile-lzma.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
STINNER Victor victor.stin...@haypocalc.com added the comment: Some comments about 2011-12-08-tarfile-lzma.diff: elif self.buf.startswith(b\x5d\x00\x00\x80) or self.buf.startswith(b... Micro-optimization: you can use self.buf.startswith((b\x5d\x00\x00\x80, b\xfd7zXZ)) here. raise ValueError(mode must be 'r' or 'w'.) Error messages usually don't end with a dot (or am I wrong?). It would be better to use a skip instead of just return here: def test_no_name_argument(self): if self.mode.endswith(bz2) or self.mode.endswith(xz): # BZ2File and LZMAFile have no name attribute. return In _Stream.__init__, for zlib: self.exception = zlib.error Could you add a test for this change? -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Éric Araujo mer...@netwok.org added the comment: Patch looks great. I did a review on Rietveld. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Éric Araujo mer...@netwok.org added the comment: I’m perfectly happy to let Lars do it next week or next month, there is no rush. The existing patch may even require very little or no change, as Nadeem’s module (in the stdlib) provides the same classes than the other lzma module which was used by the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Lars Gustäbel l...@gustaebel.de added the comment: I will be happy to, but my spare time is limited right now, so this could take about a week. If this is a problem, please go ahead. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Martin v. Löwis mar...@v.loewis.de added the comment: There is plenty of time until 3.3. OTOH, if Eric wants to work on it now: you got a week :-) Do recognize that there is a patch to start from already. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Éric Araujo mer...@netwok.org added the comment: Python now has an lzma module. Lars, do you have the time to update your patch or should I do it? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Changes by Éric Araujo mer...@netwok.org: -- components: +Library (Lib) -Extension Modules nosy: +nadeem.vawda title: please support lzma compression as an extension and in the tarfile module - Support xz compression in tarfile module versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5689] Support xz compression in tarfile module
Lars Gustäbel l...@gustaebel.de added the comment: Attached is a patch with the current state of my work on lzma integration into tarfile (17 test errors). -- assignee: - lars.gustaebel keywords: +patch Added file: http://bugs.python.org/file23162/2011-09-15-tarfile-lzma.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5689 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
