I haven't read this thread in detail, but are the answers from your calculation expressable as "bitstypes" (e.g., Float64, etc) and/or arrays of bitstypes? If so, perhaps you could make your function deposit its results in a SharedArray, and then return `nothing`?
Best, --Tim On Wednesday, June 8, 2016 8:17:47 PM CDT Martha White wrote: > I think I realize now why pmap is so slow. It is because on each call, it > is copying over a lot of information to send to each worker. Because I have > many nested outer loops, this ends up calling pmap thousands of times. I > would like to create and pass variables to pmap, that are not re-copied by > pmap. As an abstract example, imagine that I have 4 cores, but want to make > 4000 calls to a function called my_fun. > > total = zeros(Float64, 100) > for i = 1:1000 > # pmap_answers is dimension 4 x 100, where my_fun returns a vector of > information of length 100 > pmap_answers = pmap(index -> my_fun(i, index), 1:4) > # Sum across 4 parallel runs > total += sum(pmap_answers,2) > end > > This allocates memory for pmap_answers 1000 times. But, really, I could > allocate this memory once outside of the loop and allow pmap to re-use that > memory. I could pass in an array of size numCores x 100 to pmap. However, I > know that pmap currently re-copies all variables that are passed to it. Is > there a way to stop pmap from re-allocating memory, and instead just use > pre-allocated memory? Or, any parallel functionality that allows this? > > On Thursday, June 2, 2016 at 10:37:15 AM UTC-4, Martha White wrote: > > I was printing information from each worker, and seeing the worker number > > increase. But, when I actually check nworkers, the number stays at 3. So, > > I > > was incorrect about the number of workers increasing. Rather, because I am > > adding and removing workers in the outer loop, the worker id is > > increasing. > > However, I do still have issues with speed, where it is slower to use pmap > > and run in parallel. I am not currently seeing the open files issues, but > > am running again to see if I can recreate that problem. In any case, for > > speed, it might be that too much memory is being copied to pass to each > > worker. Is there a way to restrict what is copied? For example, some > > values > > are const; can I somehow give this information to pmap? > > > > On Wednesday, June 1, 2016 at 11:05:46 PM UTC-4, Greg Plowman wrote: > >> You say you get a large number of workers. > >> Without delving too deep, this seems pretty weird, regardless of other > >> code. > >> Have you checked the number of workers (using nworkers()) after call to > >> addprocs()? > >> If you are getting errors and re-run the script, is addprocs() just > >> accumulating more workers? > >> If so, perhaps try rmprocs(workers()) before addprocs()
