Share as little as possible between your various processes - shared, mutable
state is a parallelism tragedy.
If you can avoid sharing an entire dictionary, do so. It'd probably be
better to dedicate one process to updating your dictionary, and then using a
multiprocessing.Queue to pass delta reco
On Tue, Jan 12, 2010 at 11:48 +0100, wiso wrote:
> They sent back the object filled with data. The problem is very simple: I
> have a container, the container has a method read(file_name) that read a
> huge file and fill the container with datas. I have more then 1 file to read
> so I want to pa
Robert Kern wrote:
> On 2010-01-11 17:50 PM, wiso wrote:
>
>> The problem now is this:
>> start reading file r1_200909.log
>> start reading file r1_200910.log
>> readen 488832 lines from file r1_200910.log
>> readen 517247 lines from file r1_200909.log
>>
>> with huge file (the real case) the pro
On 2010-01-11 17:50 PM, wiso wrote:
The problem now is this:
start reading file r1_200909.log
start reading file r1_200910.log
readen 488832 lines from file r1_200910.log
readen 517247 lines from file r1_200909.log
with huge file (the real case) the program freeze. Is there a solution to
avoid
Robert Kern wrote:
> On 2010-01-11 17:15 PM, wiso wrote:
>> I'm using a class to read some data from files:
>>
>> import multiprocessing
>> from collections import defaultdict
>>
>> def SingleContainer():
>> return list()
>>
>>
>> class Container(defaultdict):
>> """
>> this class s
On 2010-01-11 17:15 PM, wiso wrote:
I'm using a class to read some data from files:
import multiprocessing
from collections import defaultdict
def SingleContainer():
return list()
class Container(defaultdict):
"""
this class store odd line in self["odd"] and even line in self["
> Yes, to use the multiprocessing module, you must make your script
> importable, so runtime statements should go into a __main__
> conditional. This way, when multiprocessing imports the module for each
> of its threads, the actual runtime code only gets executed once in the
> parent thread, whic
On 09/02/2009 04:51 AM, Peter Otten wrote:
tleeuwenb...@gmail.com wrote:
I have a problem using multiprocessing in a simple way. I created a
file, testmp.py, with the following contents:
---
import multiprocessing as mp
p = mp.Pool(5)
def f(x):
tleeuwenb...@gmail.com wrote:
> I have a problem using multiprocessing in a simple way. I created a
> file, testmp.py, with the following contents:
>
> ---
> import multiprocessing as mp
>
> p = mp.Pool(5)
>
> def f(x):
> return x * x
>
> print
