I'm not certain I understand the interaction between your functions and your
data. Let me make a guess: you're basically wanting to supply some parameters
as defaults? Then the strategy I'd recommend is the following:
module MyModule
export foo
foo(x, p) = println("Got ", x, " and parameter ", p)
end
julia> addprocs(1)
1-element Array{Int64,1}:
2
julia> push!(LOAD_PATH, pwd());
julia> using MyModule
julia> @everywhere using MyModule
julia> @everywhere foo1 = x -> foo(x, "hello")
julia> remotecall_fetch(2, foo1, 3)
From worker 2: Got 3 and parameter hello
Let me comment that I'm not sure it makes sense that one should have to say
@everywhere using MyModule. An alternative is to define the anonymous function
as
@everywhere foo1 = x -> MyModule.foo(x, "hello")
but, with this route, one wonders if this module-scoping could be added
automatically. If you do neither one of these things, you get the following
error:
julia> remotecall_fetch(2, foo1, 3)
ERROR: On worker 2:
UndefVarError: foo not defined
in anonymous at none:1
in anonymous at multi.jl:892
in run_work_thunk at multi.jl:645
[inlined code] from multi.jl:892
in anonymous at task.jl:63
in remotecall_fetch at multi.jl:731
in remotecall_fetch at multi.jl:734
which is quite confusing because this works:
julia> remotecall_fetch(2, foo, 3, "world")
From worker 2: Got 3 and parameter world
These seem like unnecessary bumps (it certainly cost me several minutes to
figure out), and anyone who ironed those out would be making a great
contribution! I just opened https://github.com/JuliaLang/julia/issues/13548
and marked it up-for-grabs.
Best,
--Tim
On Friday, October 09, 2015 07:35:20 AM Christopher Fisher wrote:
> Hi all-
>
> I am trying to load a file of functions on a cluster of computers. In the
> past, I used require() (now depreciated) and the sendto() function
> described here
> <http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-anoth
> er-processor-in-julia>to make a data variable available on all workers. (
> Note that I cannot simply load the data upon initializing the program
> because the data will change outside of the module, eventually receiving a
> stream of data from another program. So speed and flexibility is
> imperative). As recommended here
> <https://groups.google.com/forum/#!searchin/julia-users/$20require/julia-us
> ers/6zBKw4nd20I/5JLt7Ded0zkJ>, I defined a module containing the functions
> and used "using MyModule" to send it to the available workers. It seems
> that the major limitation of this approach is that data is not available to
> the functions within the module when using sendto(). I suspect this is
> because modules are
> encapsulated from other variables and functions. Bearing that in mind:
>
>
> 1. Is there a way around this problem using the module method?
>
> 2. Alternatively, is there a way I can make the functions and packages
> available to the workers without using modules? Perhaps something akin to
> the old require method?
>
> 3. Or is there a way to send the data via map() along with my function and
> distributed array? Essentially, my code loads stored inputs for numerous
> kernel density functions and converts them to a distributed array of
> arrays. For example:
>
> map(EvalKDFs,MyDistArray)
>
> Each time the above function is called, "MyData" needs to be available to
> the function EvalKDFs. However, map(EvalKDFs,MyDistArray,MyData) does not
> work because there is one array of data and many arrays within MyDistArray.
>
> I might be able to post a stripped down version of my code if my
> description does not suffice.
>
> Any help would be greatly appreciated.
