[julia-users] passing data by remotecall_fetch slower than writing & loading from disk

[Andre Bieler]

I have an array (lets call it "data") containing composite types.
something like

type MyType
  x::Float64
  y::Float64
end

(basically just a collection of Floats and Ints..)


Now sending this data to a worker by

remotecall_fetch(iProc, someFunc, data)

is about 100 times slower than the following
3-step procedure:

a) decompose all composite types in "data" to arrays of Floats or Ints
(all MyType.x go into one array of Floats, etc.)

b) saving those arrays to disk with h5write()

c) loading the variables from disk again with h5read(),
re-build all the composite types and put it back into
an array of MyTypes.


I dont quite understand why this is the case / what I am missing here.
I expected that writing and reading from disk would be orders of magnitudes
slower.

I am aware of shared arrays. If 
it turns out that it is not possible to send custom type data "fast" between
procs, I ll implement shared arrays instead of my composite types.

I attached an example script with timings for those who care.

sending a RemoteRef to the worker and fetch it from there
gives me the same timings as for the remotecall_fetch() attempt.


Cheers,
Andre

parallel_send_test.jl
Description: Binary data