no, I'm not. mostly conventional use afaik. if this should not be
happening, I can trace it down to a small reproducible example to
figure it out.
--
Ivo Welch (ivo.we...@ucla.edu)
--
Ivo Welch (ivo.we...@ucla.edu)
http://www.ivo-welch.info/
J. Fred Weston Distinguished Professor of Finance, U
The list does strip html, but the quality of what remains varies greatly.
Are you using tidyverse functions in your workers? Sounds to me like you are
doing something in the workers that is triggering making copies of the input
data frame.
On July 7, 2020 10:15:15 PM PDT, ivo welch wrote:
>ugg
ugghhh---apologies. although in 2020, it would be nice if the mailing
list had an automatic html filter (or even bouncer!)
I am using macos. alas, my experiments suggest that `mclapply()` on a
32-core intel system with 64GB of RAM, where the input data frame is
8GB and the output is about 500MB
Use an operating system that supports forking, like Linux or MacOSX, and use
the parallel package mclapply function or similar to share memory for read
operations. [1]
And stop posting in HTML here.
[1] https://cran.r-project.org/web/views/HighPerformanceComputing.html
On July 7, 2020 9:20:39
if I understand correctly, R makes a copy of the full environment for each
process. thus, even if I have 32 processors, if I only have 64GB of RAM
and my R process holds about 10GB, I should probably not spawn 32 processes.
has anyone written a function that sets the number of cores for use (in
m
5 matches
Mail list logo
