I didn't have time yesterday but the attached illustrates what I mean
about putting the shared data in a module (it should work with the
previous myutil).
I don't get a big speed up but at least it is faster using multiple
subprocesses:
Not threaded: 0.450406074524
Using 8 processes: 0.282383
I
On 06/16/2011 02:05 PM, Brandt Belson wrote:
Hi all,
Thanks for the replies. As mentioned, I'm parallelizing so that I can
take many inner products simultaneously (which I agree is
embarrassingly parallel). The library I'm writing asks the user to
supply a function that takes two objects and r
The fact that you are still passing the myutil.arrayList argument to
pool.map means the data is still being pickled. All arguments to
pool.map are pickled to be passed to the subprocess. You need to
change the function so that it accesses the data directly, then just
pass indices to pool.map.
Chee
Hi all,
Thanks for the replies. As mentioned, I'm parallelizing so that I can take
many inner products simultaneously (which I agree is embarrassingly
parallel). The library I'm writing asks the user to supply a function that
takes two objects and returns their inner product. After all the discussi
On 06/16/2011 11:44 AM, Christopher Barker wrote:
NOTE: I'm only taking part in this discussion because it's interesting
and I hope to learn something. I do hope the OP chimes back in to
clarify his needs, but in the meantime...
Bruce Southey wrote:
Remember that is what the OP wanted to do
Sturla Molden wrote:
> Den 16.06.2011 19:55, skrev Sturla Molden:
>> The meta-data for the ndarray (strides, shape, dtype, etc.) and the name
>> of the shared memory segment. As there is no fork, we must tell the
>> other process where to find the shared memory and what to do with it.
got it, than
Den 16.06.2011 19:55, skrev Sturla Molden:
>
> The meta-data for the ndarray (strides, shape, dtype, etc.) and the name
> of the shared memory segment. As there is no fork, we must tell the
> other process where to find the shared memory and what to do with it.
>
Which by the way is what the share
Den 16.06.2011 19:47, skrev Christopher Barker:
>
> What do you need to pickle if you're using shared memory?
The meta-data for the ndarray (strides, shape, dtype, etc.) and the name
of the shared memory segment. As there is no fork, we must tell the
other process where to find the shared memory
Den 16.06.2011 19:23, skrev Robin:
>
> If you are on Linux or Mac then fork works nicely so you have read
> only shared memory you just have to put it in a module before the fork
> (so before pool = Pool() ) and then all the subprocesses can access it
> without any pickling required. ie
> myutil.da
Sturla Molden wrote:
> On Windows this sucks, because there is no fork system call.
Darn -- I do need to support Windows.
> Here we are
> stuck with multiprocessing and pickle, even if we use shared memory.
What do you need to pickle if you're using shared memory?
-Chris
--
Christopher Bar
Den 16.06.2011 18:44, skrev Christopher Barker:
>
> I'll bet this is a big issue, and one I'm curious about how to
> address, I have another problem where I need to multi-process, and I'd
> love to know a way to pass data to the other process and back
> *without* going through pickle. maybe memm
On Thu, Jun 16, 2011 at 6:44 PM, Christopher Barker
wrote:
>> 2. There is also the question of when the process pool is spawned. Though
>> I haven't checked, I suspect it happens prior to calling pool.map. But if it
>> does not, this is a factor as well, particularly on Windows (less so on
>> Linu
NOTE: I'm only taking part in this discussion because it's interesting
and I hope to learn something. I do hope the OP chimes back in to
clarify his needs, but in the meantime...
Bruce Southey wrote:
Remember that is what the OP wanted to do, not me.
Actually, I don't think that's what the
On 06/16/2011 09:30 AM, Sturla Molden wrote:
> Den 16.06.2011 16:16, skrev Bruce Southey:
>> The idea was to calculate:
>> innerProduct =numpy.sum(array1*array2) where array1 and array2 are, in
>> general, multidimensional.
> If you want to do that in parallel, fast an with minimal overhead, it's
>
Den 16.06.2011 16:16, skrev Bruce Southey:
> The idea was to calculate:
> innerProduct =numpy.sum(array1*array2) where array1 and array2 are, in
> general, multidimensional.
If you want to do that in parallel, fast an with minimal overhead, it's
a typical OpenMP problem. If the dimensions are unk
On 06/15/2011 07:25 PM, Sturla Molden wrote:
> Den 15.06.2011 23:22, skrev Christopher Barker:
>> I think the issue got confused -- the OP was not looking to speed up a
>> matrix multiply, but rather to speed up a whole bunch of independent
>> matrix multiplies.
> I would do it like this:
>
> 1. Wr
Den 15.06.2011 23:22, skrev Christopher Barker:
>
> I think the issue got confused -- the OP was not looking to speed up a
> matrix multiply, but rather to speed up a whole bunch of independent
> matrix multiplies.
I would do it like this:
1. Write a Fortran function that make multiple calls DGEM
Den 15.06.2011 23:22, skrev Christopher Barker:
>
> It would also would be great if someone that actually understands this
> stuff could look at his code and explain why the slowdown occurs (hint,
> hint!)
>
Not sure I qualify, but I think I notice several potential problems in
the OP's multiproc
Sturla Molden wrote:
> IMHO, trying to beat Intel or AMD performance library developers with
> Python, NumPy and multiprocessing is just silly. Nothing we do with
> array operator * and np.sum is ever going to compare with BLAS functions
> from these libraries.
I think the issue got confused --
Perhaps it is time to write somthing in the SciPy cookbook about
parallel computing with NumPy? It seems to be certain problems that are
discussed again and again. These are some issues that come to mind (I'm
sure there is more):
- The difference between I/O bound, memory bound, and CPU bound w
Den 13.06.2011 19:51, skrev srean:
> If you are on an intel machine and you have MKL libraries around I
> would strongly recommend that you use the matrix multiplication
> routine if possible. MKL will do the parallelization for you. Well,
> any good BLAS implementation would do the same, you dont
Looking at the code the arrays that you are multiplying seem fairly
small (300, 200) and you have 50 of them. So it might the case that
there is not enough computational work to compensate for the cost of
forking new processes and communicating the results. Have you tried
larger arrays and more of
2011 09:23:10 -0400
> From: Olivier Delalleau mailto:sh...@keba.be>>
> Subject: Re: [Numpy-discussion] Using multiprocessing (shared memory)
>with numpy array multiplication
> To: Discussion of Numerical Python <mailto:numpy-discussion@scipy.org>&g
derlying arrays.
Thanks,
Brandt
> Message: 2
> Date: Fri, 10 Jun 2011 09:23:10 -0400
> From: Olivier Delalleau
> Subject: Re: [Numpy-discussion] Using multiprocessing (shared memory)
>with numpy array multiplication
> To: Discussion of Numerical Python
> Message-ID:
>
res. I'm not using numpy.dot, if
> I'm not mistaken I don't think that would do what I need.
> Thanks again,
> Brandt
>
>
> Message: 1
>> Date: Thu, 09 Jun 2011 13:11:40 -0700
>> From: Christopher Barker
>> Subject: Re: [Numpy-discussion] Using
s. I'm not using numpy.dot, if
I'm not mistaken I don't think that would do what I need.
Thanks again,
Brandt
Message: 1
> Date: Thu, 09 Jun 2011 13:11:40 -0700
> From: Christopher Barker
> Subject: Re: [Numpy-discussion] Using multiprocessing (shared memory)
>
Not much time, here, but since you got no replies earlier:
> > I'm parallelizing some code I've written using the built in
> multiprocessing
> > module. In my application, I need to multiply many large arrays
> together
is the matrix multiplication, or element-wise? If matrix,
Hello all,
Since I haven't heard back, I decided to simplify my problem even more. I'm
attaching two files which demonstrate the problem very clearly. My original
message is below. The script can be run with "python shared_mem.py".
Thanks,
Brandt
> > Hello,
> > I'm parallelizing some code I've wr
many people do extensive testing.
>
> Bug fixes probably do not need further notification but feature
> additions or API/ABI changes should have wider notification. So an email
> to the list would be greatly appreciated so that interested people can
> track the request and any discussion
Hello,
I'm parallelizing some code I've written using the built in multiprocessing
module. In my application, I need to multiply many large arrays together and
sum the resulting product arrays (inner products). I noticed that when I
parallelized this with myPool.map(...) with 8 processes (on an 8-c
30 matches
Mail list logo
