Two main problems:
1. module KDFs doesn't know anything about variables stored in Main. (This has
nothing to do with parallel code, this is just a basic scoping issue.) It
should be
function KDFeval(KDFinputs, data)
...
L = pdf(f, data)
...
end
and then call it from MainScript as
output = map(x->KDFeval(x, MyData), KDFargs)
2. Add
@everywhere using KDFs
after you say `using KDFs`. See
https://github.com/JuliaLang/julia/issues/9245.
I think those are the only changes I had to make.
--Tim
On Sunday, October 11, 2015 07:22:27 AM Christopher Fisher wrote:
> Sorry for the confusion. I realized the problem with sendto() was that the
> argument workers() was missing and there was a minor indexing problem in
> MainScript.jl. Please see the updated code attached. I also included code
> that works in Julia .3 with the require method, just to show that the code
> is functioning.
>
>
>
> The include method is beginning to work but it results in a conflict of pdf
> between Distributions and KernelDensity:
>
> WARNING: using KernelDensity.UnivariateKDE in module Main conflicts with an
> existing identifier. WARNING: using KernelDensity.UnivariateKDE in module
> Main conflicts with an existing identifier. WARNING: using
> Distributions.pdf in module Main conflicts with an existing identifier.
> WARNING: using KernelDensity.UnivariateKDE in module Main conflicts with an
> existing identifier. WARNING: using KernelDensity.pdf in module Main
> conflicts with an existing identifier.
>
> LoadError: On worker 4:
> UndefVarError: pdf not defined
> in KDFeval at /Users/chrisfisher/Desktop/Example Updated/Include
> Method/KDFs.jl:10 in map at
> /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib
> in anonymous at
> /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:4
> 94 in anonymous at multi.jl:889
> in run_work_thunk at multi.jl:645
> in run_work_thunk at multi.jl:654
> in anonymous at task.jl:58
> in remotecall_fetch at multi.jl:731
> in call_on_owner at multi.jl:776
> in fetch at multi.jl:784
> in chunk at
> /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:2
> 57 in anonymous at task.jl:447
> while loading In[4], in expression starting on line 28
>
> in sync_end at
> /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib
> [inlined code] from task.jl:422
> in convert at
> /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:3
> 44 in convert at abstractarray.jl:421
> [inlined code] from In[4]:35
>
> in anonymous at no file:34
>
> On Sunday, October 11, 2015 at 7:14:27 AM UTC-4, Tim Holy wrote:
> > IIUC, there were two reasons for deprecating require:
> >
> > - many people complained about the slew of related concepts (include,
> > require,
> > reload, using, import). require seems like the easiest of these to
> > eliminate.
> >
> > - Package precompilation. It was quite ambiguous whether require(filename)
> > should defer to the precompiled file or the source code. However, now that
> > we
> > have timestamp checks to ensure these are in sync with each other, I'm not
> > sure that argument is relevant anymore.
> >
> > However, modules really should be "static" (containers of code, not of
> > data),
> > so perhaps use of modules indicates that the deprecation is a good idea on
> > its
> > own merits. Does my reply to Christopher about the anonymous functions
> > help
> > with your use case?
> >
> > Best,
> > --Tim
> >
> > On Saturday, October 10, 2015 04:08:36 AM Sara Freeman wrote:
> > > I've encountered a similar problem, but do not have a solution to
> >
> > report.
> >
> > > I'm not sure why require was depreciated. It worked quite well.
> > >
> > > On Friday, October 9, 2015 at 10:35:20 AM UTC-4, Christopher Fisher
> >
> > wrote:
> > > > Hi all-
> > > >
> > > > I am trying to load a file of functions on a cluster of computers. In
> >
> > the
> >
> > > > past, I used require() (now depreciated) and the sendto() function
> > > > described here
> > > > <
> >
> > http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-ano
> >
> > > > ther-processor-in-julia>to make a data variable available on all
> >
> > workers.
> >
> > > > ( Note that I cannot simply load the data upon initializing the
> >
> > program
> >
> > > > because the data will change outside of the module, eventually
> >
> > receiving
> >
> > > > a stream of data from another program. So speed and flexibility is
> > > > imperative). As recommended here
> > > > <
> >
> > https://groups.google.com/forum/#!searchin/julia-users/$20require/julia->
> >
> > > users/6zBKw4nd20I/5JLt7Ded0zkJ>, I defined a module containing the
> > >
> > > > functions and used "using MyModule" to send it to the available
> >
> > workers.
> >
> > > > It seems that the major limitation of this approach is that data is
> >
> > not
> >
> > > > available to the functions within the module when using sendto(). I
> > > > suspect this is because modules are
> > > > encapsulated from other variables and functions. Bearing that in mind:
> > > >
> > > >
> > > > 1. Is there a way around this problem using the module method?
> > > >
> > > > 2. Alternatively, is there a way I can make the functions and packages
> > > > available to the workers without using modules? Perhaps something akin
> >
> > to
> >
> > > > the old require method?
> > > >
> > > > 3. Or is there a way to send the data via map() along with my function
> >
> > and
> >
> > > > distributed array? Essentially, my code loads stored inputs for
> >
> > numerous
> >
> > > > kernel density functions and converts them to a distributed array of
> > > > arrays. For example:
> > > >
> > > > map(EvalKDFs,MyDistArray)
> > > >
> > > > Each time the above function is called, "MyData" needs to be available
> >
> > to
> >
> > > > the function EvalKDFs. However, map(EvalKDFs,MyDistArray,MyData) does
> >
> > not
> >
> > > > work because there is one array of data and many arrays within
> > > > MyDistArray.
> > > >
> > > > I might be able to post a stripped down version of my code if my
> > > > description does not suffice.
> > > >
> > > > Any help would be greatly appreciated.
