Thanks for your quick reply, very much appreciated. I think I have it
working
correctly in parallel now, but I'm still a bit confused and would like to
confirm if possible. Here is my main function, which is similar to the
CRMAv2
vignette except modified to be run either in single CPU mode or by
multiple processors (i.e. if you pass arguments procNum and numProcs it
will only process a proportional subset of the arrays, otherwise it will
process
all the arrays):
x.aroma.run.parallel_aux <-
function (prj.name, reference, dsg.name, model, procNum, numProcs)
{
if (!missing(procNum) && !missing(numProcs) && (model != 'CRMAv2')) {
cat("Only CRMAv2 can be run in parallel mode.")
return
}
setOption(aromaSettings, "memory/ram", 500.0)
log <- verbose <- Arguments$getVerbose(-8, timestamp = TRUE)
options(digits = 4)
cat("Setting initial parameters\n")
cdf <- AffymetrixCdfFile$byChipType(dsg.name, tags = "Full")
gi <- getGenomeInformation(cdf)
si <- getSnpInformation(cdf)
acs <- AromaCellSequenceFile$byChipType(getChipType(cdf,
fullname = FALSE))
csR <- AffymetrixCelSet$byName(prj.name, cdf = cdf)
na <- nbrOfArrays(csR)
arrIdxs <- vector(mode="numeric", length=0)
if (!missing(numProcs) && (numProcs > na)) { numProcs = na }
if (!missing(procNum) && !missing(numProcs)) {
if (numProcs > 1) {
for (i in 1:na) {
if (((i - 1) %% numProcs) == procNum) {
arrIdxs <- c(arrIdxs, i);
}
}
cat(procNum+1,"/",numProcs,":")
cat(arrIdxs,"\n")
# Extract chunk to do
csR <- extract(csR, arrIdxs);
}
}
cat("Allele cross talk normalization\n")
acc <- AllelicCrosstalkCalibration(csR, model = model)
csC <- process(acc)
cat("Base Position normalization\n")
bpn <- BasePositionNormalization(csC, target = "zero")
csN <- process(bpn)
cat("Probe set summarization\n")
if (model == "CRMAv2") {
plm <- AvgCnPlm(csN, mergeStrands = TRUE, combineAlleles = TRUE)
}
else {
RmaCnPlm(csN, mergeStrands = TRUE, combineAlleles = TRUE)
}
cat("Get units\n")
if (length(findUnitsTodo(plm)) > 0) {
units <- fitCnProbes(plm, verbose = verbose)
units <- fit(plm, verbose = verbose)
}
cat("Get chip effects\n")
ces <- getChipEffectSet(plm)
cat("Fragment Length normalization\n")
fln <- FragmentLengthNormalization(ces, target = "zero")
cesN <- process(fln)
if (!missing(reference) && !is.null(reference)) {
cbs <- CbsModel(cesN, reference)
ce <- ChromosomeExplorer(cbs)
process(ce, chromosomes = 1:23, zooms = c(1, 16))
} else {
if (missing(procNum) && missing(numProcs)) {
cat("Reference set average\n")
reference <- getAverageFile(cesN)
return(reference)
}
}
}
Overall the way I use the above function is as follows. First, I call it in
single CPU mode for the reference set (i.e. arguments reference, procNum,
and
numProcs are not passed). This finishes very fast because we reuse the same
reference for all our projects and thus it has been pre-computed (i.e. all
the intermediate, on-disk memoized result files are there). This returns a
value for reference (i.e. the result of doing getAverageFile(cesN)). Next
I run this in parallel (using the mclapply function of the multicore
package)
for the main arrays to be analyzed (i.e. passing in values for reference,
procNum, and numProcs, in addition to all the other arguments). Finally, I
rerun this function a final time in single CPU mode for the main arrays to
be analyzed, in order to generate the final result (and this should return
quickly since the parallel run will have generated all the interemediate
result files). Does this seem reasonable and correct? If not, what should I
be doing differently? Do I really need to do the final run of the function
in single CPU mode, or just running in parallel will be sufficient?
I also have a question about the intermediate result files that Aroma
generates
on-disk. Another optimization I want to do is to reuse these when possible.
I.e. if a CEL file has been previously processed by Aroma as part of some
project, and then we also want to include that CEL file in another project,
I want to create symbolic links to all the Aroma intermediate result
files for that CEL file in the new project directory. I have set this up and
it seems to work well, but wanted to ask about it too. In particular, I have
found these intermediate result files for the CEL files in these directory
locations (e.g. for a CEL file MYARRAY.CEL in a project called MYPROJ):
probeData/MYPROJ,ACC,ra,-XY/GenomeWideSNP_6/MYARRAY.CEL
probeData/MYPROJ,ACC,ra,-XY/GenomeWideSNP_6/MYARRAY,fit.RData
probeData/MYPROJ,ACC,ra,-XY,BPN,-XY/GenomeWideSNP_6/MYARRAY.CEL
probeData/MYPROJ,ACC,ra,-XY,BPN,-XY/GenomeWideSNP_6/MYARRAY,fit.RData
plmData/MYPROJ,ACC,ra,-XY,BPN,-XY,AVG,A+B/GenomeWideSNP_6/MYARRAY,chipEffects.CEL
plmData/MYPROJ,ACC,ra,-XY,BPN,-XY,AVG,A+B,FLN,-XY/GenomeWideSNP_6/MYARRAY,chipEffects.CEL
cbsData/MYPROJ,ACC,ra,-XY,BPN,-XY,AVG,A+B,FLN,-XY,paired/GenomeWideSNP_6/MYARRAY,chrXX,....xdr
reports/MYPROJ/ACC,ra,-XY,BPN,-XY,AVG,A+B,FLN,-XY,paired/GenomeWideSNP_6/cbs/MYARRAY,chrXX,x0001.png
reports/MYPROJ/ACC,ra,-XY,BPN,-XY,AVG,A+B,FLN,-XY,paired/GenomeWideSNP_6/cbs/MYARRAY,chrXX,x0016.png
These were the intermediate result files that I was able to find (that are
directly associated with a CEL)
but is this all, or am I missing any that I could reuse? Is it okay to
reuse these result files like this?
I did notice that the final segmentation step in single CPU mode still
takes quite a while (and is the
majority of the overall processing time taken), which generates the
regions.xls file. Am I missing any
intermediate result files for this that could speed it up? Thanks again for
your help.
On Tuesday, March 4, 2014 8:40:26 PM UTC-5, Henrik Bengtsson wrote:
>
> On Mon, Mar 3, 2014 at 10:46 PM, Andrew Smith
> <[email protected]<javascript:>>
> wrote:
> > I would also like to run segmentation in parallel, but how do you do
> this if
> > you are using a separate reference set.
>
> It depends what type of reference set; if it's for a tumor-normal pair
> analysis nothing needs to be done, but if it's a pool of arrays for
> which the average should be calculated and used as a reference for all
> arrays, you need to calculate this average *before* launching any
> parallel processing, e.g.
>
> dfR <- getAverageFile(reference)
> cbs <- CbsModel(cesN, dfR)
>
> >
> > This is what I am running in a single process (that I would like to
> > parallelize):
> >
> > cbs <- CbsModel(cesN, reference)
> > ce <- ChromosomeExplorer(cbs)
> > process(ce, chromosomes = 1:23, zooms = c(1, 16))
> >
> > Could I do the same as your previous response, i.e. have each processor
> > extract out a subset of the arrays and process them
> > with the above, then run the above on a single CPU over all arrays to
> get
> > the final results (which should run fast since all the
> > computationally intensive intermediate results will have been memoized
> on
> > disk)?
>
> Yes, then you can simply do:
>
> process(ce, arrays=arraysForOneComputeNode, chromosomes = 1:23, zooms
> = c(1, 16))
>
> changing arraysForOneComputeNode for each compute node. This is
> actually ignoring the fact that each of the nodes will compete
> updating a few common files, but it should be alright. You can always
> refresh those at the end by running:
>
> updateSetupExplorerFile(ce)
> writeRegions(ce, chromosomes=1:23)
>
> at the very end.
>
> Hope this helps,
>
> /Henrik
>
> >
> > Overall, I am running the first 4 steps of the CRMAv2 vignette
> > (http://www.aroma-project.org/vignettes/CRMAv2) and then the above
> > for segmentation. I want to parallelize this, and my strategy is to have
> > each processor extract a subset of the arrays and run this on the
> subset,
> > then run it again on a single CPU for all the arrays. And I'm doing the
> > parallelization using the multicore package's mclapply.
> >
> > Does this seem okay or would you recommend any changes?
> >
> > thanks,
> > Andrew Smith
> >
> > On Friday, December 3, 2010 1:18:23 AM UTC-5, Henrik Bengtsson wrote:
> >>
> >> Hi.
> >>
> >> On Thu, Dec 2, 2010 at 10:24 AM, Kai <[email protected]> wrote:
> >> > Hi Henrik,
> >> >
> >> > I was trying to run segmentation on a large set of ~300 SNP
> genotyping
> >> > array profiles. Currently I am loading all the profiles into aroma
> and
> >> > run the segmenter on them one by one, which takes a really long time.
> >> > I was wondering whether there is a way to break the computation into
> >> > parts and run all the parts simultaneously. Specifically:
> >> >
> >> > 1) Is there a way to load only a subset of profiles in a project into
> >> > a "AromaUnitTotalCnBinarySet"?
> >>
> >> It is true that ds <- AromaUnitTotalCnBinarySet$byName(...) will setup
> >> a set containing all data files. However, after that you can always
> >> subset using extract(), e.g. ds <- extract(ds, 1:5);
> >>
> >> > 2) Is there a way to run segmentation on single, or a subset of
> >> > profiles in a "AromaUnitTotalCnBinarySet"?
> >>
> >> Yes, either by subsetting as above already from start or simply by
> >> specifying the 'arrays' argument to fit()/process() [below]. Note
> >> that the latter is a more convenient approach for various reasons.
> >> First, if the reference used to calculate CN ratios is the robust
> >> average across all samples, then you do not have to worry about
> >> getting it correct. (This is not a problem in your particular case
> >> because you use that cbs$.calculateRatios <- FALSE feature). Second,
> >> the final ChromosomeExplorer HTML page correctly list all samples. If
> >> you subset immediately after setting up the data set (as above), then
> >> you basically have to make sure to process() one ChromosomeExplorer on
> >> all arrays.
> >>
> >> > 3) Assuming that one can generate the CBS segmentation model on
> single
> >> > profiles, is there a way to load them together back in to aroma and
> >> > pass to the "ChromosomeExplorer"?
> >>
> >> Yes. The simplest way is to simply run what you are doing below ones
> >> at the very end. Already segmented and process samples will be
> >> skipped, also already generated image files etc.
> >>
> >> >
> >> > The current process I have (which performs segmentation one sample at
> >> > a time) is implemented as follows:
> >> >
> >> > # segment by CBS model
> >> > ds = AromaUnitTotalCnBinarySet
> >> > $byName("dataset,paired",chipType="HumanOmni1-Quad");
> >> >
> >> > cbs = CbsModel(ds);
> >> > cbs$.calculateRatios = FALSE;
> >> >
> >> > fit(cbs, chromosomes=c(1:23), min.width=5, undo.splits="sdundo",
> >> > undo.SD=1, verbose=2);
> >> >
> >> > # display data and segmentation in ChromosomeExplorer
> >> > ce = ChromosomeExplorer(cbs);
> >> > process(ce,chromosomes=c(1:23));
> >> >
> >> > Any other suggestion you may provide is also highly appreciated.
> Thank
> >> > you very much.
> >>
> >> library("aroma.core");
> >> verbose <- Arguments$getVerbose(-4, timestamp=TRUE);
> >>
> >> # Arrays that your host should process
> >> arrays <- 5:8; # Change for different hosts
> >>
> >> # Setup the complete data set
> >> ds <-
> >>
> AromaUnitTotalCnBinarySet$byName("dataset,paired",chipType="HumanOmni1-Quad");
>
>
> >>
> >> # Setup the "complete" segmentation model
> >> cbs <- CbsModel(ds, min.width=5, undo.splits="sdundo", undo.SD=1);
> >> cbs$.calculateRatios <- FALSE;
> >>
> >> # Fit a subset of the arrays
> >> fit(cbs, arrays=arrays, chromosomes=c(1:23), verbose=verbose);
> >>
> >> # Setup the "complete" ChromosomeExplorer, ...
> >> ce <- ChromosomeExplorer(cbs);
> >>
> >> # ...but generate PNG image files only for a subset
> >> process(ce, arrays=arrays, chromosomes=c(1:23), verbose=verbose);
> >>
> >> Note that you actually do not have to call fit() explicitly, because
> >> process() will do it implicitly (with the same arguments).
> >>
> >> If you want to call the above script from the command line, then
> >> replace the arrays <- ... line with
> >>
> >> arrays <- commandArgs(asValue=TRUE)$arrays;
> >> arrays <- eval(parse(text=arrays));
> >> arrays <- Arguments$getIndices(arrays);
> >>
> >> and then you can run the above by:
> >>
> >> R --args --arrays=1:5 < script.R
> >>
> >> Hope this helps
> >>
> >> Henrik
> >>
> >> >
> >> > Best,
> >> > Kai
> >> >
> >> > --
> >> > When reporting problems on aroma.affymetrix, make sure 1) to run the
> >> > latest version of the package, 2) to report the output of
> sessionInfo() and
> >> > traceback(), and 3) to post a complete code example.
> >> >
> >> >
> >> > You received this message because you are subscribed to the Google
> >> > Groups "aroma.affymetrix" group with website
> http://www.aroma-project.org/.
> >> > To post to this group, send email to [email protected]
> >>
> >> > To unsubscribe and other options, go to
> >> > http://www.aroma-project.org/forum/
> >> >
>
--
--
When reporting problems on aroma.affymetrix, make sure 1) to run the latest
version of the package, 2) to report the output of sessionInfo() and
traceback(), and 3) to post a complete code example.
You received this message because you are subscribed to the Google Groups
"aroma.affymetrix" group with website http://www.aroma-project.org/.
To post to this group, send email to [email protected]
To unsubscribe and other options, go to http://www.aroma-project.org/forum/
---
You received this message because you are subscribed to the Google Groups
"aroma.affymetrix" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.