[julia-users] Re: moving data to workers for distributed workloads

Greg, interesting suggestion about const - I should use that more. In that setup we'd still be transferring alot of big coefficient vectors around (workarounds possible).

[julia-users] Re: moving data to workers for distributed workloads

> > > It looks like that SO answer is moving data into the global scope of each > worker. It is probably worth experimenting with but I'd be worried about > performance implications of non-const global variables. It's probably the > case that this is still a win for my use case though. Thanks

[julia-users] Re: moving data to workers for distributed workloads

Michael That's right. With `sow` the mod.eval of the third argument gets bound to the second; where mod defaults to Main. Maybe someone could think of a cleaner way to make values available for later work, but it seems to do the trick. It seems best to avoid using the `sow` function heavily.

[julia-users] Re: moving data to workers for distributed workloads

Hi Fabian, It looks like that SO answer is moving data into the global scope of each worker. It is probably worth experimenting with but I'd be worried about performance implications of non-const global variables. It's probably the case that this is still a win for my use case though. Thanks

[julia-users] Re: moving data to workers for distributed workloads

Hi Matthew, ClusterUtils.jl looks very useful. I will definitely try it out. Am I correct in reading that the trick to moving input to the workers is here ? You're also

[julia-users] Re: moving data to workers for distributed workloads

Also... The above doesn't respect ordering of the `myidx` variable. Not sure how your problem domain operates, so it could get more complicated if things like order of execution matter. ;)

[julia-users] Re: moving data to workers for distributed workloads

Hi Michael Your current code looks like will pull back the `coefficients` across the network (500 gb transfer) and as you point out transfer `input` each time. I wrote a package ClusterUtils.jl to handle my own problems (MCMC sampling) which were

[julia-users] Re: moving data to workers for distributed workloads

Hi Michael, I recently had a similar problem and this SO thread helped me a lot: http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-another-processor-in-julia As a side question: Which code are you using to calculate spherical harmonics transforms. I was looking for a julia