[issue5689] Support xz compression in tarfile module

[Nadeem Vawda]

Changes by Nadeem Vawda :


--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Roundup Robot]

Roundup Robot  added the comment:

New changeset b86b54fcb5c2 by Lars Gustäbel in branch 'default':
Issue #5689: Avoid excessive memory usage by using the default lzma preset.
http://hg.python.org/cpython/rev/b86b54fcb5c2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

Ping.  Windows buildbots are still failing with MemoryError because of this 
preset=9.
The patch looks good to me as well.

--
nosy: +amaury.forgeotdarc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Nikolaus Rath]

Changes by Nikolaus Rath :


--
nosy:  -Nikratio

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Nadeem Vawda]

Nadeem Vawda  added the comment:

Patch looks good to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Lars Gustäbel  added the comment:

Yes, that's much better. Thanks for the tip.

--
Added file: http://bugs.python.org/file24086/lzma-preset.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Changes by Lars Gustäbel :


Removed file: http://bugs.python.org/file24084/lzma-preset.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Nadeem Vawda]

Nadeem Vawda  added the comment:

Yes, that's a good idea. I've been testing a similar change, and it seems
to drop the peak memory usage for test_tarfile from around 810MB down to
under 200MB. It looks like 2GB genuinely isn't enough to reliably use LZMA
compression with preset=9.

You might want to use preset=None instead of explicitly saying preset=6,
though. This tells LZMAFile to use the default preset, and will allow you
to get rid of the if-statement on lines 1821-1823.

Something unrelated that I noticed in the surrounding code: gzopen and
bz2open validate the mode by testing 'len(mode) > 1 or mode not in "rw"'.
This would be simpler as 'mode not in ("r", "w")' (like you've done in
xzopen), and it would accept only "r" and "w" (but not "" or "rw").

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Lars Gustäbel  added the comment:

Wouldn't it be better then to use a default compresslevel of 6 in tarfile? I 
used level 9 in my patch without a particular reason, just because I thought 9 
must be better than 6 ;-)

--
Added file: http://bugs.python.org/file24084/lzma-preset.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Nadeem Vawda]

Nadeem Vawda  added the comment:

Not to worry - as I said in my previous message, I can reproduce the error
on my own XP machine.

I also noticed that running test_tarfile alone doesn't trigger the errors,
which leads me to suspect that the failure is due to some interaction with
another test getting run before test_tarfile. I'm currently trying to
determine what this test is.

I suspect that the problem is at least partially caused by the fact that
tarfile uses a default compresslevel of 9 for .tar.xz archives (rather
than the recommended value of 6). According to the man page for the xz
tool , using a
compresslevel of 9 can result in memory usage of up to 800MB during
compression, which is a significant fraction of the bot's 2GB of RAM.
(I suppose it would be a good idea to mention this in the documentation
for the lzma module, so users won't get bitten by this...)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Paul Moore]

Paul Moore  added the comment:

A simple rebuild and test run of that test in debug mode didn't fail...

I'll run the full test suite as a check, but that could take some time - that 
buildslave isn't the fastest in the world...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Perhaps Paul can try to reproduce and diagnose the issue directly on the 
buildbot?

--
nosy: +pmoore

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Nadeem Vawda]

Nadeem Vawda  added the comment:

This failure seems to crop up often, but not on every run:

  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3941/steps/test/logs/stdio
  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3940/steps/test/logs/stdio
  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3937/steps/test/logs/stdio
  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3929/steps/test/logs/stdio
  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3921/steps/test/logs/stdio
  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3916/steps/test/logs/stdio
  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3914/steps/test/logs/stdio
  
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3906/steps/test/logs/stdio

I've been able to reproduce the failure on my own XP machine;
I'll investigate it over the weekend.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[STINNER Victor]

STINNER Victor  added the comment:

There is failure on a XP buildbot. I don't know if it is a sporadic issue or 
not.

http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3921/steps/test/logs/stdio

==
ERROR: test_append_lzma (test.test_tarfile.AppendTest)
--
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_tarfile.py", line 
1539, in test_append_lzma
self._create_testtar("w:xz")
  File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_tarfile.py", line 
1486, in _create_testtar
with tarfile.open(self.tarname, mode) as tar:
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1721, in 
open
return func(name, filemode, fileobj, **kwargs)
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1826, in 
xzopen
mode=mode, fileobj=fileobj, preset=preset)
  File "D:\Buildslave\3.x.moore-windows\build\lib\lzma.py", line 117, in 
__init__
preset=preset, filters=filters)
MemoryError

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Lars Gustäbel  added the comment:

Please, go ahead!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Éric Araujo]

Éric Araujo  added the comment:

Lars, as part of a small doc patch I want to change this in tarfile.rst:

   The :mod:`tarfile` module makes it possible to read and write tar
   archives, including those using gzip or bz2 compression.
  -(:file:`.zip` files can be read and written using the :mod:`zipfile` module.)
  +Use the :mod:`zipfile` module to read or write :file:`.zip` files, or the
  +higher-level functions in :ref:`shutil `.

Any objection?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Nadeem Vawda]

Nadeem Vawda  added the comment:

Great stuff! I'll close this issue along with issue 6715 once the buildbot
stuff is all sorted out.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Lars Gustäbel  added the comment:

Thanks for the review, guys! I can't close this issue yet because it depends on 
#6715.

--
resolution:  -> fixed
stage: needs patch -> committed/rejected

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 899a8c7b2310 by Lars Gustäbel in branch 'default':
Issue #5689: Add support for lzma compression to the tarfile module.
http://hg.python.org/cpython/rev/899a8c7b2310

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Éric Araujo]

Éric Araujo  added the comment:

Patch looks great.  I did a review on Rietveld.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[STINNER Victor]

STINNER Victor  added the comment:

Some comments about 2011-12-08-tarfile-lzma.diff:

> elif self.buf.startswith(b"\x5d\x00\x00\x80") or self.buf.startswith(b"...

Micro-optimization: you can use self.buf.startswith((b"\x5d\x00\x00\x80", 
b"\xfd7zXZ")) here.

> raise ValueError("mode must be 'r' or 'w'.")

Error messages usually don't end with a dot (or am I wrong?).

It would be better to use a skip instead of just return here:

def test_no_name_argument(self):
  if self.mode.endswith("bz2") or self.mode.endswith("xz"):
# BZ2File and LZMAFile have no name attribute.
return

In _Stream.__init__, for zlib:

> self.exception = zlib.error

Could you add a test for this change?

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Lars Gustäbel  added the comment:

For those who want to test it first, I post the current state of the patch 
here. It is ready for commit, there are no failing tests. If nobody objects, I 
will apply it this weekend.

--
Added file: http://bugs.python.org/file23880/2011-12-08-tarfile-lzma.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Éric Araujo]

Éric Araujo  added the comment:

I’m perfectly happy to let Lars do it next week or next month, there is no 
rush.  The existing patch may even require very little or no change, as 
Nadeem’s module (in the stdlib) provides the same classes than the other lzma 
module which was used by the patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Martin v . Löwis]

Martin v. Löwis  added the comment:

There is plenty of time until 3.3. OTOH, if Eric wants to work on it now: you 
got a week :-) Do recognize that there is a patch to start from already.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Lars Gustäbel  added the comment:

I will be happy to, but my spare time is limited right now, so this could take 
about a week. If this is a problem, please go ahead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Éric Araujo]

Éric Araujo  added the comment:

Python now has an lzma module.  Lars, do you have the time to update your patch 
or should I do it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Lars Gustäbel]

Lars Gustäbel  added the comment:

Attached is a patch with the current state of my work on lzma integration into 
tarfile (17 test errors).

--
assignee:  -> lars.gustaebel
keywords: +patch
Added file: http://bugs.python.org/file23162/2011-09-15-tarfile-lzma.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5689] Support xz compression in tarfile module

[Éric Araujo]

Changes by Éric Araujo :


--
components: +Library (Lib) -Extension Modules
nosy: +nadeem.vawda
title: please support lzma compression as an extension and in   the tarfile 
module -> Support xz compression in tarfile module
versions: +Python 3.3 -Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com