On 03/25/2015 07:48 PM, Simon Urbanek wrote:
On Mar 25, 2015, at 3:46 PM, Valerie Obenchain <voben...@fredhutch.org> wrote:
Hi Simon,
I'm having trouble with nested parallel workers, specifically, forking inside
socket connections.
You simply can't by definition - when you fork *all* the workers share the same
connection inherited from the parent, so you cannot use any I/O operations that
you didn't start in the worker since reading in one worker affects all the
workers.
Sorry if I'm missing the obvious here -
I thought since the fork workers were shut down by the time the SOCK
worker returned to its master conflicting I/O wouldn't be a problem.
There are quite a few examples floating around where SOCK workers are
spawned on a cluster and multicore workers are called within them. If I
understand correctly this should not be done (or at least not
encouraged). Instead, nested parallel should only be done with
distributed memory workers, SOCK, MPI etc.
Thanks.
Valerie
Cheers,
Simon
When mclapply is called inside a SOCK, PSOCK or FORK worker I get an
error in unserialize().
cl <- makeCluster(1, "SOCK")
fun = function(i) {
library(parallel)
mclapply(1:2, sqrt)
}
Failure occurs after multiple calls to clusterApply:
clusterApply(cl, 1, fun)
[[1]]
[[1]][[1]]
[1] 1
[[1]][[2]]
[1] 1.414214
clusterApply(cl, 1, fun)
[[1]]
[[1]][[1]]
[1] 1
[[1]][[2]]
[1] 1.414214
clusterApply(cl, 1, fun)
Error in unserialize(node$con) : error reading from connection
This example is from Martin and may be a different problem.
~/tmp >cat test1.R
## like mclapply
## should run 'forever' but terminates semi-randomly
library(parallel)
children <- parallel:::children
while (TRUE) {
n <- 8 ## n == dectectCores()
jobs <- lapply(seq_len(n), function(i) mcparallel(Sys.sleep(20)))
mccollect(children(jobs), FALSE)
parallel:::mckill(children(jobs), tools::SIGTERM)
leni <- length(mccollect(children(jobs)))
message("leni: ", leni)
}
~/tmp >R-dev --vanilla --slave -f test1.R
leni: 6
leni: 7
leni: 7
leni: 7
leni: 7
leni: 7
leni: 7
leni: 7
leni: 8
leni: 7
leni: 7
leni: 7
~/tmp >
Thanks.
Valerie
sessionInfo()
R Under development (unstable) (2015-03-18 r68009)
Platform: x86_64-unknown-linux-gnu (64-bit)
Running under: Fedora 21 (Twenty One)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
loaded via a namespace (and not attached):
[1] snow_0.3-13
--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, Seattle, WA 98109
Email: voben...@fredhutch.org
Phone: (206) 667-3158
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
