This is not a good strategy since your code generates all the permutations explicitly in memory, and there are an exponentially large number of them. Instead you could loop though k=1:factorial(n) and generate the kth permutation programmatically using nthperm(the_keys, k). If your computation performs a reduction you can use the @parallel (+) construct (or something similar), otherwise you can distribute the work manually across the p processors and write a loop like
for p in 1:nprocs() #You'll have to do the rounding more carefully to avoid missing permutations on the edges @spawnat procs()[p] for k=iround((p-1)*n/nprocs())+1:iround(p*n/nprocs()) do_stuff() end end Thanks, Jiahao Chen Staff Research Scientist MIT Computer Science and Artificial Intelligence Laboratory On Mon, Oct 6, 2014 at 8:57 PM, Jason Solack <[email protected]> wrote: > Hello everyone, > > I'm trying to iterate through a collection of permutations in parallel and > i'm having trouble iterating through the collection. In the code below i'm > using "next(p)" in the place i'd like to grab the next permutation. This > is also the first bit of processing i've done in parallel in Julia so if > have any pointers on how i could do this more easily i'd appreciate any > advice. > > > np = nprocs() > output = Dict() > p = permutations(the_keys) > on_perm = 1 > @sync begin > for on_proc=1:np > if p != myid() || np == 1 > the_perm = next(p) > @async begin > while true > output[on_perm] = remotecall_fetch(on_proc, do_calcs, > the_perm) > on_perm += 1 > if on_perm > length(p) > break > end > end > end > end > end > end > return output > > Thank you for your help. > > Jason >
