Hi There,
I just want to confirm this is a desired behavior of map/pmap functions.
What I am noticing is that in case of using pmap functions need to return data
(copy of input data), whereas map works fine with referenced values. I'd love
to have similiare behaviuor in pmap since it is much lightweight especially
when dealing with large datasets, but the more I think about it, the more it
makes sense to me that this is not going to happen due to the type of
parallelism Julia supports at teh moment (massage passing).
Could please someone more competent verify if the code attached below is
correct? What I was hoping for is to have a pmap working with the functions
where data is passed around as a reference (no return statmentts).
Unfortunately this is only true when using single threaded map() function.
I was wondering are there any alternatives at the moment? I know there is a
Shareded array options but so far I did't have a chance to fully comprehended
it. As far as I understand Distributed arrays will give me similar bottlneck as
pmap where data has to be copied back into original container...?
Thanks, any feedback is much appreciated.
kuba
##################################################
addprocs(2)
@everywhere function testRef(elem)
id = elem[1]
d = elem[2]
d[id]="___$id"
end
@everywhere function testCopy(elem)
id = elem[1]
d = elem[2]
d[id]="___$id"
return elem
end
ids = [[i,Dict()] for i=(1:3)]
pmap(testRef, ids) # pmap - Dictionary is empty
print("=== pmap ref:", ids)
map(testRef, ids) # map - works as expected
print("=== map ref:", ids)
ids_copy = pmap(testCopy, ids) # works - but creates a copy - slow
print("=== pmap copy:", ids_copy)
##################################################
Output:
=== pmap ref:1
Dict{Any,Any}()
2 Dict{Any,Any}()
3 Dict{Any,Any}()
=== map ref:1
{1=>"___1"}
2 {2=>"___2"}
3 {3=>"___3"}
=== pmap copy:1
{1=>"___1"}
2 {2=>"___2"}
3 {3=>"___3"}
