Hi there,
I wonder how to parallelize the following scenario.
I have a method that initiates a (not very simple) data strucure and then
runs a for-loop (of, say, length 1,000-20,000) to populate that data
structure with data. The computations in each loop is not trivial, but
fairly optimized using cython. All iteration steps done serially take a few
secs (about 2 or 3). Nevertheless, the computations are fairly independent
and I would like to do them in parallel.
If I extract the content of the for-loop into an @parallel(2) decorated
function, it still seems to be using only one cpu to do the computation
(why?), but all the forking takes tons of time (i.e., including 80secs for
posix.wait and 15 for posix.fork). If I read the documentation right, this
is due to the issue that every computation is done in a subprocess itself
and the data structure is also forked and passed to the subprocess. Is that
correct?
If I use @parallel('reference',2) instead (without knowing what that
actually does), it is again as quick as in the beginning but also uses only
a single cpu.
What am I doing wrong here? Does anyone know how I should handle such a (I
suspect not very uncommon) situation?
Thanks, Christian
--
You received this message because you are subscribed to the Google Groups
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.
