[issue19395] unpickled LZMACompressor is crashy

2013-10-28 Thread Nadeem Vawda
Nadeem Vawda added the comment: The part of this issue specific to LZMACompressor should now be fixed; I've filed issue 19425 for the issue with Pool.map hanging. -- resolution: -> fixed stage: needs patch -> committed/rejected status: open -> closed ___

[issue19395] unpickled LZMACompressor is crashy

2013-10-28 Thread Roundup Robot
Roundup Robot added the comment: New changeset be363c1e94fe by Nadeem Vawda in branch '3.3': #19395: Raise exception when pickling a (BZ2|LZMA)(Compressor|Decompressor). http://hg.python.org/cpython/rev/be363c1e94fe New changeset b9df25608ad0 by Nadeem Vawda in branch 'default': #19395: Raise ex

[issue19395] unpickled LZMACompressor is crashy

2013-10-26 Thread Nadeem Vawda
Nadeem Vawda added the comment: It looks like there's also a separate problem in the multiprocessing module. The following code hangs after hitting a TypeError trying to pickle one of the TextIOWrapper objects: import multiprocessing def read(f): return f.read() files = [open(path)

[issue19395] unpickled LZMACompressor is crashy

2013-10-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: If it's not possible (or easily doable) to recreate the compressor's internal state, I agree it would be helpful for pickling to raise a TypeError. -- ___ Python tracker _

[issue19395] unpickled LZMACompressor is crashy

2013-10-25 Thread Tim Peters
Tim Peters added the comment: @cantor, this is a Python issue tracker, not a help desk. If you want advice about Python programming, please use the Python mailing list or any number of "help desk" web sites (e.g., stackoverflow). -- nosy: +tim.peters _

[issue19395] unpickled LZMACompressor is crashy

2013-10-25 Thread cantor
cantor added the comment: python 3.3 version - tried this code and got a sliglty faster processing time then when running lzma.compress() on its own. Could this be improved upon? import lzma from functools import partial from threading import Thread def split_len(seq, length): return [str.

[issue19395] unpickled LZMACompressor is crashy

2013-10-25 Thread cantor
cantor added the comment: in python 2.7.3 this kind of works however it is less efficient than the pure lzma.compress() from threading import Thread from backports import lzma from functools import partial import multiprocessing class CompressClass(Thread): def __init__ (self,data,c):

[issue19395] unpickled LZMACompressor is crashy

2013-10-25 Thread Nadeem Vawda
Nadeem Vawda added the comment: Yes, that's because the builtin map function doesn't handle each input in a separate process, so it uses the same LZMACompressor object everywhere. Whereas multiprocessing.Pool.map creates a new copy of the compressor object for each input, which is where the probl

[issue19395] unpickled LZMACompressor is crashy

2013-10-25 Thread cantor
cantor added the comment: just to mention that map() (i.e. the non parallel version) works: import lzma from functools import partial import multiprocessing def run_lzma(data,c): return c.compress(data) def split_len(seq, length): return [str.encode(seq[i:i+length]) for i in range(0,

[issue19395] unpickled LZMACompressor is crashy

2013-10-25 Thread Nadeem Vawda
Nadeem Vawda added the comment: As far as I can tell, liblzma provides no way to serialize a compressor's state, so the best we can do is raise a TypeError when attempting to pickle the LZMACompressor (and likewise for LZMADecompressor). Also, it's worth pointing out that the provided code would

[issue19395] unpickled LZMACompressor is crashy

2013-10-25 Thread Antoine Pitrou
Antoine Pitrou added the comment: The problem is that using an unpickled LZMACompressor crashes: $ ./python -c "import lzma, pickle; c = pickle.loads(pickle.dumps(lzma.LZMACompressor())); c.compress(b'')" Erreur de segmentation Here is the gdb backtrace: #0 0x77bcafc0 in sem_trywait