Thank you all for your replies. I will experiment with the list of BiocParallelParam objects and BatchJobs package.
Best wishes, Laurent > We discussed with Michael Lang at the Cambridge meeting and he agreed > to think about potential abstractions for nested parallelization > requests in BatchJobs. Not sure whether that ever took off. Ideally > one would be able to infer the number of requested nodes AND cpus > directly from the BiocParallel object. In BatchJobs this would then be > exposed as a variable that could be injected in the appropriate > template scripts. E.g., on our SGE I would ask for something like > this: #$ -pe smp <%=resources$cores%> > > Currently I am already doing this by manually providing the number of > requested cores in the resources list, but from an API point of view it > would be very beneficial to have that information in a standardized form > somewhere inside the BiocParallel object which can be queried in the > executing code. > Florian > > > > > On 2/14/14 7:23 PM, "Martin Morgan" <mtmor...@fhcrc.org> wrote: > >>On 02/14/2014 09:20 AM, Laurent Gatto wrote: >>> >>> Dear all, >>> >>> Looking back in my (sparse) notes from the last European Bioc meeting, I >>> saw two points that were mentioned during the BatchJobs and BiocParallel >>> talk that are of particular interest to me. >>> >>> - BatchJobs: is there support, or are there any plans to support the >>> HTCondor infrastructure [1], and will this back-end be supported in >>> BiocParallel? >>> >>> - BiocParallel: is there support, or are there any plans to support >>> nested parallelisation. The use case I have in mind is to have a >>>first >>> level of parallelisation to distribute n sets jobs on n different >>> machines using snow for example, and then each set of jobs would be >>> distributed on their respective machine via multicore? >>> >>> Any comments on the above solutions or their feasibility would be very >>> welcome. >> >>I opened a related issue on github from some off-hub commentary from >>Michael >>Lang (QuasaR). >> >> https://github.com/Bioconductor/BiocParallel/issues/36 >> >>and I think this is a reasonable request. At least sometimes it seems >>like there >>will be a natural transition from `bplapply` at the cluster level to >>`bpvec` at >>the machine level. >> >>Martin >> >>> >>> Thank you very much in advance. >>> >>> Best wishes, >>> >>> Laurent >>> >>> [1] http://research.cs.wisc.edu/htcondor/ >>> >>> >> >> >>-- >>Computational Biology / Fred Hutchinson Cancer Research Center >>1100 Fairview Ave. N. >>PO Box 19024 Seattle, WA 98109 >> >>Location: Arnold Building M1 B861 >>Phone: (206) 667-2793 >> >>_______________________________________________ >>Bioc-devel@r-project.org mailing list >>https://stat.ethz.ch/mailman/listinfo/bioc-devel _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
