A Wednesday 03 March 2010 15:31:29 Jesper Larsen escrigué:
> Hi people,
>
> I was wondering about the status of using the standard library
> multiprocessing module with numpy. I found a cookbook example last
> updated one year ago which states that:
>
> "This page was obsolete as multiprocessing's internals have changed.
> More information will come shortly; a link to this page will then be
> added back to the Cookbook."
>
> http://www.scipy.org/Cookbook/multiprocessing
>
> I also found the code that used to be on this page in the cookbook but
> it does not work any more. So my question is:
>
> Is it possible to use numpy arrays as shared arrays in an application
> using multiprocessing and how do you do it?
Yes, it is pretty easy if your problem can be vectorised. Just split your
arrays in chunks and assign the computation of each chunk to a different
process. I'm attaching a code that does this for computing a polynomial on a
certain range. Here it is the output (for a dual-core processor):
Serial computation...
10000000 0
Time elapsed in serial computation: 3.438
3333333 0
3333334 1
3333333 2
Time elapsed in parallel computation: 2.271 with 3 threads
Speed-up: 1.51x
--
Francesc Alted
import numpy
from multiprocessing import Pool
from time import time
import subprocess, os, sys
N = 1000*1000*10
expr = "a + b*x + c*x**2 + d*x**3 + e*x**4"
a, b, c, d, e = -1.1, 2.2, -3.3, 4.4, -5.5
xp = numpy.linspace(-1, 1, N)
parallel = True
NT = 3
global counter
counter = 0
def cb(r):
global counter
print r, counter
counter +=1
def compute(nt, i):
x = xp[i*N/nt:(i+1)*N/nt]
eval(expr)
return len(x)
if __name__ == '__main__':
print "Serial computation..."
t0 = time()
result = compute(1,0)
print result, 0
ts = round(time() - t0, 3)
print "Time elapsed in serial computation:", ts
if not parallel:
sys.exit()
t0 = time()
po = Pool(processes=NT)
for i in xrange(NT):
po.apply_async(compute, (NT,i), callback=cb)
po.close()
po.join()
tp = round(time() - t0, 3)
print "Time elapsed in parallel computation:", tp, "with %s threads" % NT
print "Speed-up: %sx" % round(ts/tp, 2)
_______________________________________________
NumPy-Discussion mailing list
[email protected]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
