[julia-users] Re: moving data to workers for distributed workloads

[Matthew Pearce]

Hi Michael 

Your current code looks like will pull back the `coefficients` across the 
network (500 gb transfer) and as you point out transfer `input` each time.

I wrote a package ClusterUtils.jl 
<https://github.com/pearcemc/ClusterUtils.jl> to handle my own problems 
(MCMC sampling) which were somewhat similar.

Roughly - given the available info - if I was trying to do something 
similar I'd do:

```julia
using Compat
using ClusterUtils

sow(pids, :input, input)

@everywhere function dostuff(input, myidxs)
    for myidx in myidxs
        coefficients = spherical_harmonic_transforms(input[myidx])
    write_results_to_disk(coefficients) #needs myidx as arg too probably
  end
end

idxs = chunkit(limit, length(pids))
sow(pids, :work, :(Dict(zip($pids, $idxs))))

reap(pids, :(dostuff(input, $work[myid()])))
```

This transfers `input` once, and writes something to disk from the remote 
process.