This indeed seems like a good addition. I'm not aware yet of a place in
sage where something like this is done.
I would chose a different name though, something like "bundled_parallel",
because naming this one unordered seems to indirectly imply that the other
one is ordered, which it is absolutely not.
Le vendredi 21 juin 2013 14:28:14 UTC+2, Timo Kluck a écrit :
>
> Hi everyone,
>
> I just parallelized a for loop in a way that may be generally useful, and
> I'm wondering whether I should write an addition to sage.parallel. It's
> possible that this has been done already and in much better ways than my
> own, so I'd be very happy to hear your opinions.
>
> I had a for loop of the form
>
> for i in iterator:
> # do some stuff
> # eventually result <- f(i)
> yield result
>
> for some "long" iterator and a "short" computation i -> f(i).
>
> I tried to parallelize this using @parallel, but the result wasn't any
> quicker:
>
> @parallel()
> def f(i):
> # do some stuff
> return result
> for res in f(iterator):
> yield res[1]
>
> My guess about the lack of speed-up is that every iteration introduces
> pickling and forking overhead, which adds up to a lot, collectively.
>
> I tried to mitigate this overhead by not forking for every iteration, but
> by starting a fork for every cpu and then having that fork loop over its
> own slice of the iterator:
>
> @parallel('fork')
> def do_computation(start):
> result = []
> for args in itertools.islice(iterator, start, None,
> number_of_chunks):
> result.append(function(args))
> return result
>
> results = do_computation(range(number_of_chunks))
> for res in results:
> for r in res[1]:
> yield r
>
> This resulted in the expected speedup (~3x for ncpus=4).
>
> I collected this functionality in a decorator @unordered_parallel that
> lets you replace
>
> for i in iterator:
> yield f(i)
>
> by
>
> @unordered_parallel(iterator, number_of_chunks)
> def _(i):
> return f(i)
> for res in _: yield res
>
> (As a side note: I'm really happy with how the interface lets you
> parallelize your
> for loop while hardly changing its body.)
>
> Would there be any interest in an addition along these lines to the
> sage.parallel module?
>
> Best, Timo
>
>
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.
