Correction below.
On 07/15/2015 04:14 PM, Valerie Obenchain wrote:
Hi,
BiocParallel in release and devel are quite similar so I'd like to
narrow the focus to "before the changes to SnowParam" and after. This
means BiocParallel 1.0.3 (R 3.1) vs BiocParallel 1.3.34 (R 3.2.1) which
is the current
Hi,
BiocParallel in release and devel are quite similar so I'd like to
narrow the focus to "before the changes to SnowParam" and after. This
means BiocParallel 1.0.3 (R 3.1) vs BiocParallel 1.3.34 (R 3.2.1) which
is the current devel.
(1) master vs worker memory
I'm more concerned about mem
Hi Valerie,
I have re-run my two examples twice using "log = TRUE" and updated the
output at http://lcolladotor.github.io/SnowParam-memory/ As I was
writing this email (all morning...), I made a 4th run where I save the
gc() information to compare against R 3.1.x That fourth run kind of
debunked w
Vince,
On 07/10/2015 04:12 AM, Vincent Carey wrote:
I have had (potentially transient and environment-related) problems with
bplapply
in gQTLstats.
Was the problem during build or check where a man page example or unit
test could be isolated as the problem?
I substituted the foreach abst
Hi Leo,
Thanks for the sample code I'll take a look.
You're right, SnowParam has changed quite at bit - logging, error
handling etc. The memory use you're seeing is a concern - thanks for
reporting it.
As an fyi, the log output for SnowParam and MulticoreParam now includes
gc(), system.time
Hi,
I ran my example code with SerialParam() which had a negligible 4%
memory increase between R 3.2.x and 3.1.x This 4% could very well
fluctuate a little bit and might be non significantly different from 0
if I run the test more times.
I also added a second example using code based on my analys
I have had (potentially transient and environment-related) problems with
bplapply
in gQTLstats. I substituted the foreach abstractions and the code
worked. I still
have difficulty seeing how to diagnose the trouble I ran into.
I'd suggest that you code so that you can easily substitute parallel
Hi,
I have a script that at some point generates a list of DataFrame
objects which are rather large matrices. I then feed this list to
BiocParallel::bplapply() and process them.
Previously, I noticed that in our SGE managed cluster using
MulticoreParam() lead to 5 to 8 times higher memory usage a
