I am putting the following exchange where it is better suited on the hcp-users list, rather than caret-users where it started. David
Begin forwarded message: > From: Timothy Coalson <[email protected]> > Subject: Re: [caret-users] cifti math mem > Date: October 5, 2013 5:06:54 PM CDT > To: "Caret, SureFit, and SuMS software users" <[email protected]> > Reply-To: "Caret, SureFit, and SuMS software users" > <[email protected]> > > It should compute it one row at a time, so as long as you have enough rows > that it divides the total cifti file into a managable row size, you should be > fine. You can always try it on your local machine, and see if it starts > allocating swap (cifti-math, along with the other math commands, aren't > parallel anyway, it should usually be IO bound since the amount of processing > to do is exactly proportional to the size of the data files, so unless you > are running multiple commands in parallel, it probably doesn't need to be on > a cluster). In general, I try to write cifti commands so that they > compute/write cifti files one row at a time during computation, so they don't > use as much memory. Correlation is a notable exception, since the output is > symmetric, retaining some/all of the results in memory allows it to finish > faster. > > Note, however, that metric/volume inputs/outputs always use the memory of the > uncompressed data (we don't do on-disk reading/writing for them), and most > cifti commands that do spatial things like smoothing or gradient separate the > cifti into in-memory metric/volume files one at a time, which may take a fair > amount of memory. > > I hadn't thought of making a "count nonzero" reduce operation, but it is > rather easy to add more reduce operations. > > Tim > > > > On Sat, Oct 5, 2013 at 11:07 AM, Colin Reveley <[email protected]> wrote: > Hi - > > I'm wondering (before I try and damage remote equipment) > > when running wb_command -cifti-math > > is it necessary that the cifti(s) fit into RAM memory? > > what I want to do is take a very large dconn.nii file, and binarize it such > that all non-zero entries are 1 and all 0 entries stay 0. > > this is, obviously, an intermediate step to something else (tracking stats > from vertex seeds). > > In an ideal world I'd also like to divide the value in each cell by another > cifti of the same dimensions. > > if -cifti-math is able to avoid RAM (like cifti-reduce) then great. If not I > need another way or I risk disrupting the university cluster. > > the basic task is to replace all non-zero entries with 1 without requiring > more than say 1/10 of the file size in RAM. > > if that is not possible then, if we call the binarised cifti above "A" then > the output of -cifti-reduce A.dconn.nii SUM > > obtained by any means would be great and a big help > > obviously there is more detail to the problem/questions that might help. But > I'm trying not to explicitly be "unsupported" at this point in case the > question can be answered in a "supported" way. > > best, > > Colin > > > _______________________________________________ > caret-users mailing list > [email protected] > http://brainvis.wustl.edu/mailman/listinfo/caret-users > > > _______________________________________________ > caret-users mailing list > [email protected] > http://brainvis.wustl.edu/mailman/listinfo/caret-users _______________________________________________ HCP-Users mailing list [email protected] http://lists.humanconnectome.org/mailman/listinfo/hcp-users
