I think I realize now why pmap is so slow. It is because on each call, it is copying over a lot of information to send to each worker. Because I have many nested outer loops, this ends up calling pmap thousands of times. I would like to create and pass variables to pmap, that are not re-copied by pmap. As an abstract example, imagine that I have 4 cores, but want to make 4000 calls to a function called my_fun.
total = zeros(Float64, 100) for i = 1:1000 # pmap_answers is dimension 4 x 100, where my_fun returns a vector of information of length 100 pmap_answers = pmap(index -> my_fun(i, index), 1:4) # Sum across 4 parallel runs total += sum(pmap_answers,2) end This allocates memory for pmap_answers 1000 times. But, really, I could allocate this memory once outside of the loop and allow pmap to re-use that memory. I could pass in an array of size numCores x 100 to pmap. However, I know that pmap currently re-copies all variables that are passed to it. Is there a way to stop pmap from re-allocating memory, and instead just use pre-allocated memory? Or, any parallel functionality that allows this? On Thursday, June 2, 2016 at 10:37:15 AM UTC-4, Martha White wrote: > > I was printing information from each worker, and seeing the worker number > increase. But, when I actually check nworkers, the number stays at 3. So, I > was incorrect about the number of workers increasing. Rather, because I am > adding and removing workers in the outer loop, the worker id is increasing. > However, I do still have issues with speed, where it is slower to use pmap > and run in parallel. I am not currently seeing the open files issues, but > am running again to see if I can recreate that problem. In any case, for > speed, it might be that too much memory is being copied to pass to each > worker. Is there a way to restrict what is copied? For example, some values > are const; can I somehow give this information to pmap? > > On Wednesday, June 1, 2016 at 11:05:46 PM UTC-4, Greg Plowman wrote: >> >> You say you get a large number of workers. >> Without delving too deep, this seems pretty weird, regardless of other >> code. >> Have you checked the number of workers (using nworkers()) after call to >> addprocs()? >> If you are getting errors and re-run the script, is addprocs() just >> accumulating more workers? >> If so, perhaps try rmprocs(workers()) before addprocs() >> >>
