From the earlier example, whether the worker sees all the data or not depends
on whether it is in the environment of FUN, the object sent to the worker.
I don't really know about packages and forked processes. I'd bet that the
vector allocations are essentially constant, but that the
Can I get a indication here about what is expected to consume memory under fork
and socket models as well as patterns to mitigate excessive memory consumption?
When using sockets, the model is that of multiple communicating machines
running on their own memory, so it makes sense that memory
In one R session I did library(SummarizedExperiment) and then saved search().
In another R session I loaded the packages on the search path in reverse order,
recording pryr::mem_used() after each. I ended up with
mem_used
methods 25870312
datasets
Hi Martin,
Thanks for your explanation which make me understand BiocParallel
much better.
I compare memory usage in my code before packaged (using doSNOW) and after
packaged (using BiocParallel) and find the increased memory is caused by
the attached packages, especially 'SummarizedExperiment'.
Memory use can be complicated to understand.
library(BiocParallel)
v <- replicate(100, rnorm(1), simplify=FALSE)
bplapply(v, sum)
by default, bplapply splits 100 jobs (each element of the list) equally between
the number of cores available, and sends just the necessary data
