Hi Patrick, Thanks for posting. The wrapper function works as it should. If you do not specify return the function returns the last object declared, and this is exactly what I intended. Doing some research on my own I found out that the problem was here: x <- c(1: length(linH)) stat.matrix <- do.call(rbind, mclapply(x, wrapper, mc.cores= 6)) Since linH = readLines(conH, n= N.perm) and N.perm was set to 999, apparently mclapply sends _simultaneously_ the 999 jobs to the cpus specified in mc.cores. Even if you use more than 6, as in the example, the systems gets quickly overflowed. So the solution is to split length(linH) into blocks equal to the number of cpus and the message to end-users is "be careful with parallel processing functions in R". Cheers Juan Antonio Balbuena El 19/12/2013 22:03, p_connolly escribió:
On Wed, 18-Dec-2013 at 09:28AM +0100, Juan Antonio Balbuena wrote: |> |> Hello |> I am using package multicore for parallel computing in a Altix UltraViolet |> 1000 server with 64 CPUs and 960 GB of RAM memory. Access is managed by |> means of a SGE queue system. This is the first time I am using parallel |> computing and my experience with supercomputers is quite limited. So any |> help will be much, much appreciated. |> My experiment consists of a number of runs (N.runs) each involving a number |> of permutations (N. perms). (An excerpt of the code is included below.) The |> permutations are very time consuming and I am using mclapply to distribute |> the job among a given number of cpus (usually 12 to 24). The problem is that |> the system administrators notice that threads keep increasing as the program |> is executed to the point that they compromise the functioning of the whole |> system and have to abort the job. So, does that mean you get nothing written to 1MH_30.tre? I'd be surprised if you did get anything. Though there's lots of stuff happening I know nothing about, I'm pretty sure there's an issue with your wrapper() function. [...] |> wrapper <- function (x) { # THIS FUCNTION IS SUPPOSED TO BE |> PARALLELIZED (SEE BELOW) |> treeH <- read.tree(text=linH[x]) |> treeP <- read.tree(text=linP[x]) |> mrcaL <- MRCALink.simul (treeH, treeP, HP) |> stat.matrix[x,] <- three.stat(mrcaL) |> } Nothing is being returned, so your calls to rbind will have nothing to put together. So what they end up trying to do, I've no idea. That function probably needs a final line 'stat.matrix' before the '}' One thing I discovered with mclapply is that to use the browser() function, it's necessary to make what is x in your example of length 1 (and probably mc.cores to 1 also) before it's possible to examine what's happening at various parts of the function being debugged. Judicious use of cat() and Sys.time() to display what's happening at various stages is also helpful. HTH -- ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~. ___ Patrick Connolly {~._.~} Great minds discuss ideas _( Y )_ Middle minds discuss events (:_~*~_:) Small minds discuss people (_)-(_) ..... Anon ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~. -- Dr. Juan A. Balbuena Marine Zoology Unit Cavanilles Institute of Biodiversity and Evolutionary Biology University of Valencia [1]http://www.uv.es/~balbuena P.O. Box 22085 [2]http://www.uv.es/cavanilles/zoomarin/index.htm 46071 Valencia, Spain [3]http://cetus.uv.es/mullpardb/index.html e-mail: [4]j.a.balbu...@uv.es tel. +34 963 543 658 fax +34 963 543 733 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NOTE! For shipments by EXPRESS COURIER use the following street address: C/ Catedrático José Beltrán 2, 46980 Paterna (Valencia), Spain. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ References 1. http://www.uv.es/%7Ebalbuena 2. http://www.uv.es/cavanilles/zoomarin/index.htm 3. http://cetus.uv.es/mullpardb/index.html 4. mailto:j.a.balbu...@uv.es ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.