Hi Matthew, ClusterUtils.jl looks very useful. I will definitely try it out. Am I correct in reading that the trick to moving input to the workers is here <https://github.com/pearcemc/ClusterUtils.jl/blob/ac5eb73bd565b43d0b05b9d8af1c930cef4088b7/src/ClusterUtils.jl#L157-L160> ?
You're also correct that write_results_to_disk does actually depend on myidx. I might have somewhat oversimplified the example. Thanks, Michael On Thursday, May 19, 2016 at 7:41:49 AM UTC-7, Matthew Pearce wrote: > > Hi Michael > > Your current code looks like will pull back the `coefficients` across the > network (500 gb transfer) and as you point out transfer `input` each time. > > I wrote a package ClusterUtils.jl > <https://github.com/pearcemc/ClusterUtils.jl> to handle my own problems > (MCMC sampling) which were somewhat similar. > > Roughly - given the available info - if I was trying to do something > similar I'd do: > > ```julia > using Compat > using ClusterUtils > > sow(pids, :input, input) > > @everywhere function dostuff(input, myidxs) > for myidx in myidxs > coefficients = spherical_harmonic_transforms(input[myidx]) > write_results_to_disk(coefficients) #needs myidx as arg too probably > end > end > > idxs = chunkit(limit, length(pids)) > sow(pids, :work, :(Dict(zip($pids, $idxs)))) > > reap(pids, :(dostuff(input, $work[myid()]))) > ``` > > This transfers `input` once, and writes something to disk from the remote > process. > > > >
