I find that putting everything required on workers into a module is the way
to go.
Then just use using Module (after adding workers)
This works for me (v0.4.5):
ProjectModule.jl:
module ProjectModule
using DataFrames
include("function1.jl")
export function1
end
function1.jl:
function function1(input::DataFrame)
# do something to input
println("function1 here")
end
Now test:
addprocs()
using DataFrames
using ProjectModule
df = DataFrame(A = 1:4, B = ["M", "F", "F", "M"])
pmap(function1, fill(df,10))
On Thursday, June 2, 2016 at 8:24:52 AM UTC+10, ABB wrote:
> Hello -
>
> I have the following problem: I would like to find a good way to import a
> collection of user-defined functions across several workers. Some of my
> functions are defined on DataFrames, but "using DataFrames" is not getting
> me anywhere on the other workers.
>
> I think the problem I am running into may result from some combination of
> the rules of scope and the command "using".
>
> Alternatively, given what I want to do, maybe running this with "julia -p
> N" is not the best way to make use of N workers in the way I want to.
>
> I open several workers: "julia -p 3"
>
> "using DataFrames" - this brings DataFrames into the 'main' worker's scope
> (I think), but not into the scope of subordinate workers.
>
> "using ProjectModule" - I am trying to load a module across all workers
> which contains several functions I have written (maybe this is not the best
> way to accomplish this task?)
>
> This error is returned:
>
> LoadError: LoadError: UndefVarError: DataFrame not defined
>
> ProjectModule looks something like
>
> module ProjectModule
> include("function1.jl")
>
> export function1
> end
>
> where function1 is defined as
>
> function1(input::DataFrame)
> #do something to input
> end
>
> I have tried a few things:
>
> - Running "@everywhere using DataFrames" from within the main worker (this
> has worked once or twice - that is, I can then use function1 on a different
> worker - but it isn't consistent)
>
> - Opening the workers at the outset using julia -p N -L
> ProjectModule.jl... (repeated N times) I get: "LoadError: UndefVarError:
> DataFrames not defined"
>
> - I also put "using DataFrames" into the ProjectModule.jl file. The
> program definitely hated that. (Specifically: I was warned that I was
> "overwriting DataFrames".)
>
> Is there a better way to load both the DataFrames package and the
> functions I have written across a couple of workers?
>
> Thanks!
>
> ABB
>
