I was printing information from each worker, and seeing the worker number increase. But, when I actually check nworkers, the number stays at 3. So, I was incorrect about the number of workers increasing. Rather, because I am adding and removing workers in the outer loop, the worker id is increasing. However, I do still have issues with speed, where it is slower to use pmap and run in parallel. I am not currently seeing the open files issues, but am running again to see if I can recreate that problem. In any case, for speed, it might be that too much memory is being copied to pass to each worker. Is there a way to restrict what is copied? For example, some values are const; can I somehow give this information to pmap?
On Wednesday, June 1, 2016 at 11:05:46 PM UTC-4, Greg Plowman wrote: > > You say you get a large number of workers. > Without delving too deep, this seems pretty weird, regardless of other > code. > Have you checked the number of workers (using nworkers()) after call to > addprocs()? > If you are getting errors and re-run the script, is addprocs() just > accumulating more workers? > If so, perhaps try rmprocs(workers()) before addprocs() > >
