Re: [Python-Dev] LZMA compression support in 3.3

2011-09-15 Thread Nadeem Vawda
Another update - I've added proper documentation. Now the code should be pretty much complete - all that's missing is the necessary bits and pieces to build it on Windows. Cheers, Nadeem ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] LZMA compression support in 3.3

2011-09-11 Thread Nadeem Vawda
I've posted an updated patch to the bug tracker, with a complete implementation of the lzma module, including 100% test coverage for the LZMAFile class (which is implemented entirely in Python). It doesn't include ReST documentation (yet), but the docstrings are quite detailed. Please take a look

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-29 Thread Barry Warsaw
On Aug 27, 2011, at 10:36 PM, Nadeem Vawda wrote: I talked to Antoine about this on IRC; he didn't seem to think a PEP would be necessary. But a summary of the discussion on the tracker issue might still be a useful thing to have, given how long it's gotten. I agree with Antoine - no PEP should

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-29 Thread Nadeem Vawda
I've updated the issue http://bugs.python.org/issue6715 with a patch containing my work so far - the LZMACompressor and LZMADecompressor classes, along with some tests. These two classes should provide a fairly complete interface to liblzma; it will be possible to implement LZMAFile on top of

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-28 Thread Martin v. Löwis
I just want to talk about it - for now. python-ideas is a better place to just talk than python-dev. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-28 Thread Stefan Behnel
Dan Stromberg, 27.08.2011 21:58: On Sat, Aug 27, 2011 at 9:04 AM, Nick Coghlan wrote: On Sun, Aug 28, 2011 at 1:58 AM, Nadeem Vawda wrote: On Sat, Aug 27, 2011 at 5:52 PM, Nick Coghlan wrote: It's acceptable for the Python version to use ctypes in the case of wrapping an existing library, but

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-28 Thread Guido van Rossum
On Sat, Aug 27, 2011 at 10:36 PM, Dan Stromberg drsali...@gmail.com wrote: On Sat, Aug 27, 2011 at 8:57 PM, Guido van Rossum gu...@python.org wrote: On Sat, Aug 27, 2011 at 3:14 PM, Dan Stromberg drsali...@gmail.com wrote: IMO, we really, really need some common way of accessing C libraries

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-28 Thread Greg Ewing
Guido van Rossum wrote: On Sat, Aug 27, 2011 at 3:14 PM, Dan Stromberg drsali...@gmail.com wrote: IMO, we really, really need some common way of accessing C libraries that works for all major Python variants. We have one. It's called writing an extension module. I think Dan means some way

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Ross Lagerwall
I'd like to propose the addition of a new module in Python 3.3. The 'lzma' module will provide support for compression and decompression using the LZMA algorithm, and the .xz and .lzma file formats. The matter has already been discussed on the tracker http://bugs.python.org/issue6715, where

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Martin v. Löwis
The implementation will also be similar to bz2 - basic compressor and decompressor classes written in C, with convenience functions and a file interface implemented on top of those in Python. When I reviewed lzma, I found that this approach might not be appropriate. lzma has many more options

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nadeem Vawda
On Sat, Aug 27, 2011 at 4:50 PM, Martin v. Löwis mar...@v.loewis.de wrote: The implementation will also be similar to bz2 - basic compressor and decompressor classes written in C, with convenience functions and a file interface implemented on top of those in Python. When I reviewed lzma, I

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Martin v. Löwis
As for file formats, these are handled by liblzma itself; the extension module just selects which compressor/decompressor initializer function to use depending on the value of the format argument. Our code won't contain anything along the lines of GzipFile; all of that work is done by the

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nadeem Vawda
On Sat, Aug 27, 2011 at 5:15 PM, Martin v. Löwis mar...@v.loewis.de wrote: As for file formats, these are handled by liblzma itself; the extension module just selects which compressor/decompressor initializer function to use depending on the value of the format argument. Our code won't

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nick Coghlan
On Sun, Aug 28, 2011 at 1:15 AM, Martin v. Löwis mar...@v.loewis.de wrote: This is exactly what I worry about. I think adding file I/O to bz2 was a mistake, as this doesn't integrate with Python's IO library (it used to, but now after dropping stdio, they were incompatible. Indeed, for Python

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Martin v. Löwis
It is not my intention for the _lzma C module to do I/O - that will be done by the LZMAFile class, which will be written in Python. My comparison with bz2 was in reference to the state of the module after it was rewritten for issue 5863. Ok. I'll defer my judgement then until actual code is

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Antoine Pitrou
On Sun, 28 Aug 2011 01:36:50 +1000 Nick Coghlan ncogh...@gmail.com wrote: On Sun, Aug 28, 2011 at 1:15 AM, Martin v. Löwis mar...@v.loewis.de wrote: This is exactly what I worry about. I think adding file I/O to bz2 was a mistake, as this doesn't integrate with Python's IO library (it used

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nick Coghlan
On Sun, Aug 28, 2011 at 1:40 AM, Antoine Pitrou solip...@pitrou.net wrote: On Sun, 28 Aug 2011 01:36:50 +1000 Nick Coghlan ncogh...@gmail.com wrote: On Sun, Aug 28, 2011 at 1:15 AM, Martin v. Löwis mar...@v.loewis.de wrote: This is exactly what I worry about. I think adding file I/O to bz2

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nadeem Vawda
On Sat, Aug 27, 2011 at 5:42 PM, Martin v. Löwis mar...@v.loewis.de wrote: Not sure whether you already have this: supporting the tarfile module would be nice. Yes, got that - issue 5689. Also of interest is issue 5411 - adding .xz support to distutils. But I think that these are separate

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nick Coghlan
On Sun, Aug 28, 2011 at 1:58 AM, Nadeem Vawda nadeem.va...@gmail.com wrote: On Sat, Aug 27, 2011 at 5:52 PM, Nick Coghlan ncogh...@gmail.com wrote: It's acceptable for the Python version to use ctypes in the case of wrapping an existing library, but the Python version should still exist. I'm

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Antoine Pitrou
On Sun, 28 Aug 2011 01:52:51 +1000 Nick Coghlan ncogh...@gmail.com wrote: The plausible story being that we basically wrap an existing library? I don't think PyPy et al have pure Python versions of the zlib or OpenSSL, do they? If we start taking PEP 399 conformance to such levels, we

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Martin v. Löwis
PEP 399 also comes into play - we need a pure Python version for PyPy et al (or a plausible story for why an exception should be granted). No, we don't. We can grant an exception, which I'm very willing to do. The PEP lists wrapping a specific C-based library as a plausible reason. It's

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Antoine Pitrou
On Sat, 27 Aug 2011 18:50:40 +0200 Antoine Pitrou solip...@pitrou.net wrote: On Sun, 28 Aug 2011 01:52:51 +1000 Nick Coghlan ncogh...@gmail.com wrote: The plausible story being that we basically wrap an existing library? I don't think PyPy et al have pure Python versions of the zlib or

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Terry Reedy
On 8/27/2011 9:47 AM, Nadeem Vawda wrote: I'd like to propose the addition of a new module in Python 3.3. The 'lzma' module will provide support for compression and decompression using the LZMA algorithm, and the .xz and .lzma file formats. The matter has already been discussed on the

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Dan Stromberg
On Sat, Aug 27, 2011 at 9:04 AM, Nick Coghlan ncogh...@gmail.com wrote: On Sun, Aug 28, 2011 at 1:58 AM, Nadeem Vawda nadeem.va...@gmail.com wrote: On Sat, Aug 27, 2011 at 5:52 PM, Nick Coghlan ncogh...@gmail.com wrote: It's acceptable for the Python version to use ctypes in the case of

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Martin v. Löwis
I'd like to better understand why ctypes is (sometimes) frowned upon. Is it the brittleness? Tendency to segfault? That, and Python should work completely if ctypes is not available. FWIW, I have a partial implementation of a module that does xz from Python using ctypes. So does it work

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nadeem Vawda
On Sat, Aug 27, 2011 at 9:47 PM, Terry Reedy tjre...@udel.edu wrote: On 8/27/2011 9:47 AM, Nadeem Vawda wrote: I'd like to propose the addition of a new module in Python 3.3. The 'lzma' module will provide support for compression and decompression using the LZMA algorithm, and the .xz and

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Dan Stromberg
On Sat, Aug 27, 2011 at 1:21 PM, Martin v. Löwis mar...@v.loewis.dewrote: I'd like to better understand why ctypes is (sometimes) frowned upon. Is it the brittleness? Tendency to segfault? That, and Python should work completely if ctypes is not available. What are the most major

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Nadeem Vawda
On Sat, Aug 27, 2011 at 10:41 PM, Dan Stromberg drsali...@gmail.com wrote: It seems like there should be some way of coming up with an xml file describing the types of the various bits of data and formal arguments - perhaps using gccxml or something like it. The problem is that you would need

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Dan Stromberg
On Sat, Aug 27, 2011 at 2:38 PM, Nadeem Vawda nadeem.va...@gmail.comwrote: On Sat, Aug 27, 2011 at 10:41 PM, Dan Stromberg drsali...@gmail.com wrote: It seems like there should be some way of coming up with an xml file describing the types of the various bits of data and formal arguments -

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Antoine Pitrou
On Sat, 27 Aug 2011 15:14:15 -0700 Dan Stromberg drsali...@gmail.com wrote: On Sat, Aug 27, 2011 at 2:38 PM, Nadeem Vawda nadeem.va...@gmail.comwrote: On Sat, Aug 27, 2011 at 10:41 PM, Dan Stromberg drsali...@gmail.com wrote: It seems like there should be some way of coming up with an

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Martin v. Löwis
Why -can't- we expect the user to have liblzma headers installed? Couldn't it just be a dependency in the package management system? Please give it up. You just won't convince that list that ctypes is a viable approach for the standard library. Regards, Martin

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Dan Stromberg
On Sat, Aug 27, 2011 at 3:26 PM, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 27 Aug 2011 15:14:15 -0700 Dan Stromberg drsali...@gmail.com wrote: On Sat, Aug 27, 2011 at 2:38 PM, Nadeem Vawda nadeem.va...@gmail.com wrote: On Sat, Aug 27, 2011 at 10:41 PM, Dan Stromberg

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Antoine Pitrou
On Sat, 27 Aug 2011 16:19:01 -0700 Dan Stromberg drsali...@gmail.com wrote: 2) It's a rather arbitrary distinction that's being drawn between dev and nondev today. There's no particular reason why the line couldn't be drawn somewhere else. Sure. Now please convince Linux distributions first,

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Dan Stromberg
On Sat, Aug 27, 2011 at 4:27 PM, Antoine Pitrou solip...@pitrou.net wrote: Sure. Now please convince Linux distributions first, because this particular subthread is going nowhere. I hope you're not a solipsist. Anyway, if the mere -discussion- of embracing a standard and safe way of making

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Dan Stromberg
On Sat, Aug 27, 2011 at 4:27 PM, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 27 Aug 2011 16:19:01 -0700 Dan Stromberg drsali...@gmail.com wrote: 2) It's a rather arbitrary distinction that's being drawn between dev and nondev today. There's no particular reason why the line couldn't

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Guido van Rossum
On Sat, Aug 27, 2011 at 3:14 PM, Dan Stromberg drsali...@gmail.com wrote: IMO, we really, really need some common way of accessing C libraries that works for all major Python variants. We have one. It's called writing an extension module. ctypes is a crutch because it doesn't realistically

Re: [Python-Dev] LZMA compression support in 3.3

2011-08-27 Thread Dan Stromberg
On Sat, Aug 27, 2011 at 8:57 PM, Guido van Rossum gu...@python.org wrote: On Sat, Aug 27, 2011 at 3:14 PM, Dan Stromberg drsali...@gmail.com wrote: IMO, we really, really need some common way of accessing C libraries that works for all major Python variants. We have one. It's called