The ciftiopen option of matlab cifti support should allow you to load the file, though I think it loads as double, and you'd need to convert it to single to reclaim some memory (which comes close to your memory+swap cap, as it needs the double and single versions allocated simultaneously in order to do the conversion).
One possibility, if you don't really need a 90k x 90k network, is downsampling or parcellating the dense connectome, see http://www.humanconnectome.org/software/workbench-command.php?function=-cifti-parcellate . If you do need the 90k x 90k resolution, and can work in c++ instead of matlab, you could use CiftiLib for reading the file (can read single rows from disk, rather than the entire matrix into memory), or code up a new command for wb_command (same support for cifti, but also support for other file types, access to other algorithms). Tim On Tue, Oct 25, 2016 at 3:21 PM, Harms, Michael <[email protected]> wrote: > > If you’re talking about some sort of community/modularity analysis, then, > no, there aren’t any tools for doing that directly in workbench. > > Maybe Tim C. will have some suggestions. I believe that the dconn is > stored as a single precision float. Have you tried using ‘ciftiopen’? I’m > not sure if that preserves the single precision format of the data, or if > it balloons by a factor of 2 due to conversion to matlab’s default double > precision format. If you can keep it as single precision in matlab, you > might be able to work with it provided Matlab has access to a lot of RAM. > > cheers, > -MH > > -- > Michael Harms, Ph.D. > ----------------------------------------------------------- > Conte Center for the Neuroscience of Mental Disorders > Washington University School of Medicine > Department of Psychiatry, Box 8134 > 660 South Euclid Ave. Tel: 314-747-6173 > St. Louis, MO 63110 Email: [email protected] > > From: "GLOMB, KATHARINA" <[email protected]> > Date: Tuesday, October 25, 2016 at 10:35 AM > To: Michael Harms <[email protected]> > Subject: Re: [HCP-Users] Problem with handling dense FC matrix > > Hi Michael, > > wow, thank you for this lightning speed reply! I want to run some network > analysis on this matrix.I don't need the whole matrix, I would just need > the unique entries (off-diagonal upper or lower triangular, squareform() in > Matlab) - maybe that would help? Can workbench help me in any way to do > this? > > Or do you have any other ideas? > > Thank you! > Katharina > > On Tue, Oct 25, 2016 at 5:21 PM, Harms, Michael <[email protected]> wrote: > >> >> What are you trying to do with the file in workbench? Have you reviewed >> the tutorial, because it includes examples (and a scene to help you along) >> of using that file. >> >> Trying to load/work with that file in matlab is probably going to cause >> loads of pain. It is just too large to load the entire thing into matlab. >> >> cheers, >> -MH >> >> -- >> Michael Harms, Ph.D. >> ----------------------------------------------------------- >> Conte Center for the Neuroscience of Mental Disorders >> Washington University School of Medicine >> Department of Psychiatry, Box 8134 >> 660 South Euclid Ave.Tel: 314-747-6173 >> St. Louis, MO 63110Email: [email protected] >> >> From: <[email protected]> on behalf of "GLOMB, >> KATHARINA" <[email protected]> >> Date: Tuesday, October 25, 2016 at 10:15 AM >> To: "[email protected]" <[email protected]> >> Subject: [HCP-Users] Problem with handling dense FC matrix >> >> Hello HCP users, >> >> I am new to HCP so please excuse me if I'm asking something that should >> be obvious. I've tried to find a solution but haven't been successful so >> far. I am trying to use the file >> HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500ROW_zcorr.dconn.nii >> which to my understanding should contain a full FC matrix that is an >> average of 820 subjects. I want to feed this matrix into a Matlab function. >> Of course, this is problematic because of the memory requirements. I have >> tried to read the file using ft_read_cifti from fieldtrip, but Matlab >> crashes at 112 GB of memory. So clearly, this is not the way. >> >> I know I am supposed to be able to use workbench for that, but can't make >> it work so far. Can someone point me in the right direction? Is what I'm >> trying to do impossible? >> >> I am using a cluster that can handle up to 132 GB of memory per cpu >> (RAM+swap) and runs Debian. >> >> Thank you!! >> Katharina >> >> -- >> PhD candidate <http://www.cns.upf.edu/katharina> >> Computational Neuroscience Group <http://www.cns.upf.edu/> >> Center for Brain and Cognition <http://cbc.upf.edu/> >> Universitat Pompeu Fabra <http://www.upf.edu/> >> Barcelona, Spain >> >> I am funded by the INDIREA <http://www.indirea.eu/> Marie Curie Initial >> Training Network. >> >> _______________________________________________ >> HCP-Users mailing list >> [email protected] >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >> >> >> ------------------------------ >> >> The materials in this message are private and may contain Protected >> Healthcare Information or other information of a sensitive nature. If you >> are not the intended recipient, be advised that any unauthorized use, >> disclosure, copying or the taking of any action in reliance on the contents >> of this information is strictly prohibited. If you have received this email >> in error, please immediately notify the sender via telephone or return mail. >> > > > > -- > PhD candidate <http://www.cns.upf.edu/katharina> > Computational Neuroscience Group <http://www.cns.upf.edu/> > Center for Brain and Cognition <http://cbc.upf.edu/> > Universitat Pompeu Fabra <http://www.upf.edu/> > Barcelona, Spain > > I am funded by the INDIREA <http://www.indirea.eu/> Marie Curie Initial > Training Network. > > > ------------------------------ > > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > > _______________________________________________ > HCP-Users mailing list > [email protected] > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > _______________________________________________ HCP-Users mailing list [email protected] http://lists.humanconnectome.org/mailman/listinfo/hcp-users
