I have a function f(i) that operates on the i-th piece of a linear algebra heavy calculation I am doing, independently of the other pieces of the calculation. If I is the set of indices I need to work on, I can accomplish this with something like pmap(f, I).
How much memory should each worker process maintain after pmap(f, I) is done? presumably, just enough memory for holding Main and anything I allocated with an @everywhere, right? However, after pmap(f, I) is done, I still see nprocs()-1 julia workers using lots of memory, which seems wrong to me. I've coded up a MWE example here: https://gist.github.com/tcovert/3b9b982ba8b13f5984bebb63887b6def Here is what f() is supposed to do to each element i in I: 1) select a subset of a square, invertible matrix K. each i needs a different subset 2) select a subset of a rectangular matrix k whose column dimension is the same as the column dimension of the above subset of K 3) select a subset of a vector y 4) compute k_i * (K_i \ y_i), where _i indicates the subsets above I realize that the subsetting and linear algebra work should allocate memory along the way, but I would have thought that this memory would be freed up after pmap() is done. on my machine, this code leaves 4 worker processes, each using about 350 megabytes of RAM, even after an @everywhere gc(). With a problem this size, RAM isn't a problem, but ideally I am using more workers, on bigger problems, which eventually will become memory constrained. am I doing something wrong? or are my expectations wrong, and this kind of task should indeed involve that much memory after the task is over? -thom
