Thanks for your help so far Tim. The behavior of output = map(x->KDFeval(x, MyData), KDFargs) is indeed strange and it happens with my code too. Do you have any idea why it works within a for loop but not outside of it? ~Sara
On Sunday, October 11, 2015 at 2:52:36 PM UTC-4, Tim Holy wrote: > > Two main problems: > > 1. module KDFs doesn't know anything about variables stored in Main. (This > has > nothing to do with parallel code, this is just a basic scoping issue.) It > should be > > function KDFeval(KDFinputs, data) > ... > L = pdf(f, data) > ... > end > > and then call it from MainScript as > > output = map(x->KDFeval(x, MyData), KDFargs) > > > 2. Add > @everywhere using KDFs > after you say `using KDFs`. See > https://github.com/JuliaLang/julia/issues/9245. > > I think those are the only changes I had to make. > > --Tim > > > On Sunday, October 11, 2015 07:22:27 AM Christopher Fisher wrote: > > Sorry for the confusion. I realized the problem with sendto() was that > the > > argument workers() was missing and there was a minor indexing problem in > > MainScript.jl. Please see the updated code attached. I also included > code > > that works in Julia .3 with the require method, just to show that the > code > > is functioning. > > > > > > > > The include method is beginning to work but it results in a conflict of > pdf > > between Distributions and KernelDensity: > > > > WARNING: using KernelDensity.UnivariateKDE in module Main conflicts with > an > > existing identifier. WARNING: using KernelDensity.UnivariateKDE in > module > > Main conflicts with an existing identifier. WARNING: using > > Distributions.pdf in module Main conflicts with an existing identifier. > > WARNING: using KernelDensity.UnivariateKDE in module Main conflicts with > an > > existing identifier. WARNING: using KernelDensity.pdf in module Main > > conflicts with an existing identifier. > > > > LoadError: On worker 4: > > UndefVarError: pdf not defined > > in KDFeval at /Users/chrisfisher/Desktop/Example Updated/Include > > Method/KDFs.jl:10 in map at > > > /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib > > in anonymous at > > > /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:4 > > 94 in anonymous at multi.jl:889 > > in run_work_thunk at multi.jl:645 > > in run_work_thunk at multi.jl:654 > > in anonymous at task.jl:58 > > in remotecall_fetch at multi.jl:731 > > in call_on_owner at multi.jl:776 > > in fetch at multi.jl:784 > > in chunk at > > > /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:2 > > 57 in anonymous at task.jl:447 > > while loading In[4], in expression starting on line 28 > > > > in sync_end at > > > /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib > > [inlined code] from task.jl:422 > > in convert at > > > /Users/chrisfisher/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:3 > > 44 in convert at abstractarray.jl:421 > > [inlined code] from In[4]:35 > > > > in anonymous at no file:34 > > > > On Sunday, October 11, 2015 at 7:14:27 AM UTC-4, Tim Holy wrote: > > > IIUC, there were two reasons for deprecating require: > > > > > > - many people complained about the slew of related concepts (include, > > > require, > > > reload, using, import). require seems like the easiest of these to > > > eliminate. > > > > > > - Package precompilation. It was quite ambiguous whether > require(filename) > > > should defer to the precompiled file or the source code. However, now > that > > > we > > > have timestamp checks to ensure these are in sync with each other, I'm > not > > > sure that argument is relevant anymore. > > > > > > However, modules really should be "static" (containers of code, not of > > > data), > > > so perhaps use of modules indicates that the deprecation is a good > idea on > > > its > > > own merits. Does my reply to Christopher about the anonymous functions > > > help > > > with your use case? > > > > > > Best, > > > --Tim > > > > > > On Saturday, October 10, 2015 04:08:36 AM Sara Freeman wrote: > > > > I've encountered a similar problem, but do not have a solution to > > > > > > report. > > > > > > > I'm not sure why require was depreciated. It worked quite well. > > > > > > > > On Friday, October 9, 2015 at 10:35:20 AM UTC-4, Christopher Fisher > > > > > > wrote: > > > > > Hi all- > > > > > > > > > > I am trying to load a file of functions on a cluster of computers. > In > > > > > > the > > > > > > > > past, I used require() (now depreciated) and the sendto() function > > > > > described here > > > > > < > > > > > > > http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-ano > > > > > > > > ther-processor-in-julia>to make a data variable available on all > > > > > > workers. > > > > > > > > ( Note that I cannot simply load the data upon initializing the > > > > > > program > > > > > > > > because the data will change outside of the module, eventually > > > > > > receiving > > > > > > > > a stream of data from another program. So speed and flexibility is > > > > > imperative). As recommended here > > > > > < > > > > > > > https://groups.google.com/forum/#!searchin/julia-users/$20require/julia-> > > > > > > > users/6zBKw4nd20I/5JLt7Ded0zkJ>, I defined a module containing the > > > > > > > > > functions and used "using MyModule" to send it to the available > > > > > > workers. > > > > > > > > It seems that the major limitation of this approach is that data > is > > > > > > not > > > > > > > > available to the functions within the module when using sendto(). > I > > > > > suspect this is because modules are > > > > > encapsulated from other variables and functions. Bearing that in > mind: > > > > > > > > > > > > > > > 1. Is there a way around this problem using the module method? > > > > > > > > > > 2. Alternatively, is there a way I can make the functions and > packages > > > > > available to the workers without using modules? Perhaps something > akin > > > > > > to > > > > > > > > the old require method? > > > > > > > > > > 3. Or is there a way to send the data via map() along with my > function > > > > > > and > > > > > > > > distributed array? Essentially, my code loads stored inputs for > > > > > > numerous > > > > > > > > kernel density functions and converts them to a distributed array > of > > > > > arrays. For example: > > > > > > > > > > map(EvalKDFs,MyDistArray) > > > > > > > > > > Each time the above function is called, "MyData" needs to be > available > > > > > > to > > > > > > > > the function EvalKDFs. However, map(EvalKDFs,MyDistArray,MyData) > does > > > > > > not > > > > > > > > work because there is one array of data and many arrays within > > > > > MyDistArray. > > > > > > > > > > I might be able to post a stripped down version of my code if my > > > > > description does not suffice. > > > > > > > > > > Any help would be greatly appreciated. > >
