I see. Thank you!
It's a pity that there is no explict way to just send a variable to other
process before starting the task, as what @parallel did. Sometimes I need
to read a large file and then do the calculation parallelly, like the Monte
Carlo method. Using pmap is the most explicit way, but the large file have
to be read many times using @everywhere, or the data have to be repeated
many times in the main process for the argument of pmap (The memory may not
sufficient to do it). The global method have some side-effect. Using
remote_call and fetch could be a good solusion, but the code is not so
elegent. It could be convient to have a function to "send" variable, or
pmap could support transfering a copy of the whole variable to every
processes as argument rather than "slice" it.
On Monday, September 15, 2014 6:41:39 PM UTC+2, John Drummond wrote:
>
> r = remote_call(2, rand)
> fetch(r)
> fetch(@spawnat 2 1+fetch(r)+myid())
>
> shows transferring data to another process and then running a query on it
> (iterate through the processes as per Strickland to do this everywhere)
>
> I didn't find a way to do this with pmap - you can transfer the data each
> time though
> pmap(x->myid()+x[1]+x[2], map(x->(x,a),1:3))
>
> On Monday, September 15, 2014 5:09:18 PM UTC+1, John Drummond wrote:
>>
>> @spawnat(2,whos()) is useful to know what's defined in other processes
>> (e.g. for process 2 in this case)
>>
>> if you want to gather the return values, use a reducer function
>> e.g.
>> let a = rand()
>> @parallel hcat for x in 1:3
>> foo(a,x)
>> end
>> end
>>
>> trying to remember the other approach which I thought I got working
>> rather than copying it over each run.
>>
>> One other thing is that requiring a file after adding the processes
>> should load it on all processes which is another approach to setting up the
>> environment for each process
>>
>>
>>
>>
>> On Monday, September 15, 2014 3:51:43 PM UTC+1, John Drummond wrote:
>>>
>>> with parallel
>>>
>>> addprocs(2)
>>> println("Number of processors", nprocs())
>>> @everywhere function foo(x,y)
>>> println(myid()," ", x, " ", y)
>>> return x + y
>>> end
>>> let a = rand()
>>> @sync @parallel for x in 1:3
>>> foo(a,x)
>>> end
>>> end
>>>
>>> On Monday, September 15, 2014 3:14:50 PM UTC+1, [email protected]
>>> wrote:
>>>>
>>>> "you could precreate A and then define it everywhere"
>>>> Could you please show me some code how to do it?
>>>>
>>>> On Monday, September 15, 2014 2:51:08 PM UTC+2, John Drummond wrote:
>>>>>
>>>>> Chris Strickland
>>>>> <https://groups.google.com/forum/#!msg/julia-users/jlKoEtErRL4/0ZcB_hxyJlYJ>
>>>>>
>>>>> lists one approach for the general problem
>>>>>
>>>>> you could precreate A and then define it everywhere, or send a copy
>>>>> over as a parameter to whatever function you use in pmap, similar but not
>>>>> the same as above
>>>>>
>>>>> Another approach which was useful to me was the @parallel for loops
>>>>> "Any variables used inside the parallel loop will be copied and
>>>>> broadcast to each process."
>>>>> <http://julia.readthedocs.org/en/latest/manual/parallel-computing/>
>>>>>
>>>>> and shared arrays
>>>>> <http://julia.readthedocs.org/en/latest/manual/parallel-computing/#shared-arrays-experimental-unix-only-feature>
>>>>>
>>>>> if using linux could be useful (I've not tried them).
>>>>>
>>>>>
>>>>>
>>>>> On Monday, September 15, 2014 10:52:33 AM UTC+1, [email protected]
>>>>> wrote:
>>>>>>
>>>>>> I want to transfer a variable to all parallel workers. However, if I
>>>>>> do:
>>>>>>
>>>>>> A=rand()
>>>>>> pmap(x->A+x,1:3)
>>>>>>
>>>>>> Return error:
>>>>>> exception on 2: ERROR: A not defined
>>>>>> in anonymous at none:1
>>>>>> in anonymous at multi.jl:855
>>>>>> in run_work_thunk at multi.jl:621
>>>>>> in anonymous at task.jl:855
>>>>>> exception on 3: ERROR: A not defined
>>>>>> in anonymous at none:1
>>>>>> in anonymous at multi.jl:855
>>>>>> in run_work_thunk at multi.jl:621
>>>>>> in anonymous at task.jl:855
>>>>>> 2-element Array{Any,1}:
>>>>>> UndefVarError(:A)
>>>>>> UndefVarError(:A)
>>>>>>
>>>>>> The result of
>>>>>> @everywhere A=rand()
>>>>>> pmap(x->A+x,1:3)
>>>>>> is not what I want, since I hope A in all mashines are the same.
>>>>>>
>>>>>> I know that pmap((x,y)->x+y,1:3,fill(A,3)) will be work, but I don't
>>>>>> think it is smart since A is expand in memery unnessarily. Is there any
>>>>>> simple way to just send a copy of A, or the reference of A, to all
>>>>>> parallel
>>>>>> mashines?
>>>>>>
>>>>>
