My appologies if the formatting was not readable. Essentially, I replaced
the for loop in MainScript.jl with:
MyData = rand(Normal(0,1),50)
output = map(x->KDFeval(x,MyData),KDFargs)
and the output was:
100-element DistributedArrays.DArray{Any,1,RemoteException}:
#undef
#undef
#undef...
On Sunday, October 11, 2015 at 2:52:36 PM UTC-4, Tim Holy wrote:
>
> Two main problems:
>
> 1. module KDFs doesn't know anything about variables stored in Main. (This
> has
> nothing to do with parallel code, this is just a basic scoping issue.) It
> should be
>
> function KDFeval(KDFinputs, data)
> ...
> L = pdf(f, data)
> ...
> end
>
> and then call it from MainScript as
>
> output = map(x->KDFeval(x, MyData), KDFargs)
>
>
> 2. Add
> @everywhere using KDFs
> after you say `using KDFs`. See
> https://github.com/JuliaLang/julia/issues/9245.
>
> I think those are the only changes I had to make.
>
> --Tim
>
>
> On Sunday, October 11, 2015 07:22:27 AM Christopher Fisher wrote:
> > Sorry for the confusion. I realized the problem with sendto() was that
> the
> > argument workers() was missing and there was a minor indexing problem in
> > MainScript.jl. Please see the updated code attached. I also included
> code
> > that works in Julia .3 with the require method, just to show that the
> code
> > is functioning.
> >
> >
> >
> > The include method is beginning to work but it results in a conflict of
> pdf
> > between Distributions and KernelDensity:
> >
> > WARNING: using KernelDensity.UnivariateKDE in module Main conflicts with
> an
> > existing identifier. WARNING: using KernelDensity.UnivariateKDE in
> module
> > Main conflicts with an existing identifier. WARNING: using
> > Distributions.pdf in module Main conflicts with an existing identifier.
> > WARNING: using KernelDensity.UnivariateKDE in module Main conflicts with
> an
> > existing identifier. WARNING: using KernelDensity.pdf in module Main
> > conflicts with an existing identifier.
> >
> > LoadError: On worker 4:
> > UndefVarError: pdf not defined
> > in KDFeval at /Users/chrisfisher/Desktop/Example Updated/Include
> > Method/KDFs.jl:10 in map at
> >
> /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib
> > in anonymous at
> >
> /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:4
> > 94 in anonymous at multi.jl:889
> > in run_work_thunk at multi.jl:645
> > in run_work_thunk at multi.jl:654
> > in anonymous at task.jl:58
> > in remotecall_fetch at multi.jl:731
> > in call_on_owner at multi.jl:776
> > in fetch at multi.jl:784
> > in chunk at
> >
> /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:2
> > 57 in anonymous at task.jl:447
> > while loading In[4], in expression starting on line 28
> >
> > in sync_end at
> >
> /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib
> > [inlined code] from task.jl:422
> > in convert at
> >
> /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:3
> > 44 in convert at abstractarray.jl:421
> > [inlined code] from In[4]:35
> >
> > in anonymous at no file:34
> >
> > On Sunday, October 11, 2015 at 7:14:27 AM UTC-4, Tim Holy wrote:
> > > IIUC, there were two reasons for deprecating require:
> > >
> > > - many people complained about the slew of related concepts (include,
> > > require,
> > > reload, using, import). require seems like the easiest of these to
> > > eliminate.
> > >
> > > - Package precompilation. It was quite ambiguous whether
> require(filename)
> > > should defer to the precompiled file or the source code. However, now
> that
> > > we
> > > have timestamp checks to ensure these are in sync with each other, I'm
> not
> > > sure that argument is relevant anymore.
> > >
> > > However, modules really should be "static" (containers of code, not of
> > > data),
> > > so perhaps use of modules indicates that the deprecation is a good
> idea on
> > > its
> > > own merits. Does my reply to Christopher about the anonymous functions
> > > help
> > > with your use case?
> > >
> > > Best,
> > > --Tim
> > >
> > > On Saturday, October 10, 2015 04:08:36 AM Sara Freeman wrote:
> > > > I've encountered a similar problem, but do not have a solution to
> > >
> > > report.
> > >
> > > > I'm not sure why require was depreciated. It worked quite well.
> > > >
> > > > On Friday, October 9, 2015 at 10:35:20 AM UTC-4, Christopher Fisher
> > >
> > > wrote:
> > > > > Hi all-
> > > > >
> > > > > I am trying to load a file of functions on a cluster of computers.
> In
> > >
> > > the
> > >
> > > > > past, I used require() (now depreciated) and the sendto() function
> > > > > described here
> > > > > <
> > >
> > >
> http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-ano
> > >
> > > > > ther-processor-in-julia>to make a data variable available on all
> > >
> > > workers.
> > >
> > > > > ( Note that I cannot simply load the data upon initializing the
> > >
> > > program
> > >
> > > > > because the data will change outside of the module, eventually
> > >
> > > receiving
> > >
> > > > > a stream of data from another program. So speed and flexibility is
> > > > > imperative). As recommended here
> > > > > <
> > >
> > >
> https://groups.google.com/forum/#!searchin/julia-users/$20require/julia->
> > >
> > > > users/6zBKw4nd20I/5JLt7Ded0zkJ>, I defined a module containing the
> > > >
> > > > > functions and used "using MyModule" to send it to the available
> > >
> > > workers.
> > >
> > > > > It seems that the major limitation of this approach is that data
> is
> > >
> > > not
> > >
> > > > > available to the functions within the module when using sendto().
> I
> > > > > suspect this is because modules are
> > > > > encapsulated from other variables and functions. Bearing that in
> mind:
> > > > >
> > > > >
> > > > > 1. Is there a way around this problem using the module method?
> > > > >
> > > > > 2. Alternatively, is there a way I can make the functions and
> packages
> > > > > available to the workers without using modules? Perhaps something
> akin
> > >
> > > to
> > >
> > > > > the old require method?
> > > > >
> > > > > 3. Or is there a way to send the data via map() along with my
> function
> > >
> > > and
> > >
> > > > > distributed array? Essentially, my code loads stored inputs for
> > >
> > > numerous
> > >
> > > > > kernel density functions and converts them to a distributed array
> of
> > > > > arrays. For example:
> > > > >
> > > > > map(EvalKDFs,MyDistArray)
> > > > >
> > > > > Each time the above function is called, "MyData" needs to be
> available
> > >
> > > to
> > >
> > > > > the function EvalKDFs. However, map(EvalKDFs,MyDistArray,MyData)
> does
> > >
> > > not
> > >
> > > > > work because there is one array of data and many arrays within
> > > > > MyDistArray.
> > > > >
> > > > > I might be able to post a stripped down version of my code if my
> > > > > description does not suffice.
> > > > >
> > > > > Any help would be greatly appreciated.
>
>
