pmap always copies since the functions are executed in a different worker process. Do you want to collect the results of pmap execution in a dictionary? What is the actual problem you are trying to solve?
On Fri, May 30, 2014 at 12:53 AM, Kuba Roth <[email protected]> wrote: > Hi There, > I just want to confirm this is a desired behavior of map/pmap functions. > What I am noticing is that in case of using pmap functions need to return > data (copy of input data), whereas map works fine with referenced values. > I'd love to have similiare behaviuor in pmap since it is much lightweight > especially when dealing with large datasets, but the more I think about > it, the more it makes sense to me that this is not going to happen due to > the type of parallelism Julia supports at teh moment (massage passing). > > Could please someone more competent verify if the code attached below is > correct? What I was hoping for is to have a pmap working with the functions > where data is passed around as a reference (no return statmentts). > Unfortunately this is only true when using single threaded map() function. > > I was wondering are there any alternatives at the moment? I know there is > a Shareded array options but so far I did't have a chance to fully > comprehended it. As far as I understand Distributed arrays will give me > similar bottlneck as pmap where data has to be copied back into original > container...? > Thanks, any feedback is much appreciated. > > kuba > > ################################################## > addprocs(2) > > @everywhere function testRef(elem) > id = elem[1] > d = elem[2] > d[id]="___$id" > end > > > @everywhere function testCopy(elem) > id = elem[1] > d = elem[2] > d[id]="___$id" > return elem > end > > ids = [[i,Dict()] for i=(1:3)] > pmap(testRef, ids) # pmap - Dictionary is empty > print("=== pmap ref:", ids) > map(testRef, ids) # map - works as expected > print("=== map ref:", ids) > ids_copy = pmap(testCopy, ids) # works - but creates a copy - slow > print("=== pmap copy:", ids_copy) > > > ################################################## > Output: > === pmap ref:1 > Dict{Any,Any}() > 2 Dict{Any,Any}() > 3 Dict{Any,Any}() > === map ref:1 > {1=>"___1"} > 2 {2=>"___2"} > 3 {3=>"___3"} > === pmap copy:1 > {1=>"___1"} > 2 {2=>"___2"} > 3 {3=>"___3"} > >
