[issue19395] lzma hangs for a very long time when run in parallel using python's muptiprocessing module?

2013-10-25 Thread cantor 

New submission from cantor:

import lzma
from functools import partial
import multiprocessing


def run_lzma(data,c):
return c.compress(data)


def split_len(seq, length):
return [str.encode(seq[i:i+length]) for i in range(0, len(seq), length)]



def lzma_mp(sequence,threads=3):
  lzc = lzma.LZMACompressor()
  blocksize = int(round(len(sequence)/threads))
  strings = split_len(sequence, blocksize)
  lzc_partial = partial(run_lzma,c=lzc)
  pool=multiprocessing.Pool()
  lzc_pool = list(pool.map(lzc_partial,strings))
  pool.close()
  pool.join()
  out_flush = lzc.flush()
  return b.join(lzc_pool + [out_flush])

sequence = 'AJKGJFKSHFKLHALWEHAIHWEOIAH 
IOAHIOWEHIOHEIOFEAFEASFEAFWEWEWFQWEWQWQGEWQFEWFDWEWEGEFGWEG'


lzma_mp(sequence,threads=3)

--
components: ctypes
messages: 201278
nosy: cantor
priority: normal
severity: normal
status: open
title: lzma hangs for a very long time when run in parallel using python's 
muptiprocessing module?
type: behavior
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19395
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19395] lzma hangs for a very long time when run in parallel using python's muptiprocessing module?

2013-10-25 Thread cantor 

cantor added the comment:

lzma

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19395
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19395] lzma hangs for a very long time when run in parallel using python's muptiprocessing module?

2013-10-25 Thread cantor 

Changes by cantor cantorma...@gmail.com:


--
nosy: +nadeem.vawda

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19395
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19395] lzma hangs for a very long time when run in parallel using python's muptiprocessing module?

2013-10-25 Thread cantor 

Changes by cantor cantorma...@gmail.com:


--
components:  -ctypes

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19395
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com