Re: [HCP-Users] Question about dense functional connectome
Thanks Tim! I will consider your suggestions. Reza On Thu, Oct 25, 2018 at 11:51 PM Timothy Coalson wrote: > If you have their cifti indices then you can just make a new cifti dscalar > with only those grayordinates set to 1 (double check this in wb_view, > matlab uses 1-based indexing, and I don't remember which convention we use > for cifti indices). If you don't have the cifti indices, you can use > -cifti-export-dense-mapping output to make text files that translate > between them and vertex/voxel. I suggest loading some standard dscalar > file to use as the template for saving the data out (use > ciftiopen/ciftisavereset, the ft_read_cifti and related functions do some > unexpected things with the cifti matrix). > > Single points out of a dconn are likely to be noisy, you may also want to > consider -cifti-parcellate (if possible, -cifti-parcellate on the dtseries, > and then do the correlations on the parcellated timeseries, rather than > averaging correlations together). > > Tim > > > On Thu, Oct 25, 2018 at 10:20 AM, Reza Rajimehr > wrote: > >> Thanks Matt! Looks like -cifti-restrict-dense-map can do what I want if I >> run it twice (first run it along the COLUMN direction to get an output, >> then run it again on the output along the ROW direction). For this to work, >> I need to specify an roi file. In a .mat file, I have a list of >> grayordinate vertices/voxels (some in left hemi, some in right hemi, some >> in subcortex). How can I put all of them in one cifti roi file in Matlab? >> >> Reza >> >> >> On Thu, Oct 25, 2018 at 2:40 PM Glasser, Matthew >> wrote: >> >>> Yes that would be a better solution. I think the wb_command >>> -cifti-restrict-dense-map might work. >>> >>> Matt. >>> >>> From: Reza Rajimehr >>> Date: Thursday, October 25, 2018 at 3:16 AM >>> To: Matt Glasser >>> Cc: Timothy Coalson , "hcp-users@humanconnectome.org" < >>> hcp-users@humanconnectome.org> >>> >>> Subject: Re: [HCP-Users] Question about dense functional connectome >>> >>> We tried to open *.dconn.nii in Matlab using ft_read_cifti on an Ubuntu >>> system with 48 GB RAM. It used 48 GB RAM and 18 GB (out of 60 GB) swap, >>> then Matlab gave out of memory error! Will try to use a system with even >>> higher RAM. >>> >>> Tim: We have a list of 913 grayordinate voxels (we have their voxel >>> numbers/indices). These voxels are somewhat distributed in gray matter. We >>> are basically interested in the functional connectivity matrix just for >>> these 913 voxels (i.e. a 913 * 913 matrix). Is it possible to use >>> wb_command to select part of *.dconn.nii corresponding to the voxels of >>> interest? >>> >>> Best, >>> Reza >>> >>> >>> On Thu, Oct 25, 2018 at 3:49 AM Glasser, Matthew >>> wrote: >>> >>>> Use swap space. >>>> >>>> Matt. >>>> >>>> From: on behalf of Timothy >>>> Coalson >>>> Date: Wednesday, October 24, 2018 at 4:57 PM >>>> To: Reza Rajimehr >>>> Cc: "hcp-users@humanconnectome.org" >>>> Subject: Re: [HCP-Users] Question about dense functional connectome >>>> >>>> The cifti format doesn't support that, because it is designed to allow >>>> use without loading the entire file into memory - there is no obvious file >>>> organization that would allow efficient loading of a full row from a single >>>> triangular file, when seek times are nontrivial (rotating disks). I don't >>>> believe we have these files in any other format. I'm also not sure how >>>> easy it would be to work on such a matrix in matlab. If wb_command is >>>> capable of doing the operations you want, it is usually possible for it to >>>> do them without loading the entire cifti file into memory at once. wb_view >>>> will also display maps from it without loading the entire file into memory. >>>> >>>> DDR4 has 16GB modules available at reasonable prices, which should >>>> allow recent computers with even 4 slots for memory to expand to 64GB. You >>>> should also consider the "high-end desktop" platforms (threadripper, >>>> lga2066, lga2011), which typically offer 8 memory slots, and processors >>>> with more cores. >>>> >>>> Tim >>>> >>>> >>>> On Wed, Oct 24
Re: [HCP-Users] Question about dense functional connectome
Thanks Michael! Your suggestion was quite useful. I successfully loaded the connectivity matrix into Matlab using ciftiopen and did the operation that I wanted (i.e. selecting part of the matrix). Reza On Thu, Oct 25, 2018 at 6:41 PM Harms, Michael wrote: > > > As an aside, if ‘ft_read_cifti’ is expanding all the data to type double > (64 bit), then you are looking at needing > 66 GB to load a 91282x91282 > dconn.nii in matlab. I assume that is what is happening based on your > experience. > > > > If you want to see how far you can get in Matlab, I would at least try > switching to using ‘ciftiopen.m’ to load it into matlab, which uses the > gifti toolbox, which preserves the underlying data type of the .dconn.nii > (type single; i.e., 32 bit) upon load into matlab. > > > > I tested that just now on a system with 64 GB of RAM (and 64 GB of swap), > and after what must have been at least 5 min, it loaded successfully into > matlab (R2015a) with a size in matlab of 32 GB (and a type of ‘single’). > Note however that as that process was ongoing, the memory usage of the > overall system hit 56 GB (not counting cached memory), which decreased to > 33 GB when the load was completed (presumably that is reflective of there > essentially being two copies in memory during the loading process). > > > > Working with a dconn in matlab is inevitably going to be cumbersome and > slow, but by using this approach you might actually be able to get it > loaded and see where you can get from that point. Obviously, you would > have to be extremely conscious of memory issues for whatever operations you > try to perform in matlab. > > > > Cheers, > > -MH > > > > -- > > Michael Harms, Ph.D. > > --- > > Associate Professor of Psychiatry > > Washington University School of Medicine > > Department of Psychiatry, Box 8134 > > 660 South Euclid Ave > <https://maps.google.com/?q=660+South+Euclid+Ave=gmail=g>. > Tel: 314-747-6173 > > St. Louis, MO 63110 Email: mha...@wustl.edu > > > > *From: * on behalf of Reza > Rajimehr > *Date: *Thursday, October 25, 2018 at 3:17 AM > *To: *"Glasser, Matthew" > > > *Cc: *"hcp-users@humanconnectome.org" > *Subject: *Re: [HCP-Users] Question about dense functional connectome > > > > We tried to open *.dconn.nii in Matlab using ft_read_cifti on an Ubuntu > system with 48 GB RAM. It used 48 GB RAM and 18 GB (out of 60 GB) swap, > then Matlab gave out of memory error! Will try to use a system with even > higher RAM. > > > > Tim: We have a list of 913 grayordinate voxels (we have their voxel > numbers/indices). These voxels are somewhat distributed in gray matter. We > are basically interested in the functional connectivity matrix just for > these 913 voxels (i.e. a 913 * 913 matrix). Is it possible to use > wb_command to select part of *.dconn.nii corresponding to the voxels of > interest? > > > > Best, > > Reza > > > > > > On Thu, Oct 25, 2018 at 3:49 AM Glasser, Matthew > wrote: > > Use swap space. > > > > Matt. > > > > *From: * on behalf of Timothy > Coalson > *Date: *Wednesday, October 24, 2018 at 4:57 PM > *To: *Reza Rajimehr > *Cc: *"hcp-users@humanconnectome.org" > *Subject: *Re: [HCP-Users] Question about dense functional connectome > > > > The cifti format doesn't support that, because it is designed to allow use > without loading the entire file into memory - there is no obvious file > organization that would allow efficient loading of a full row from a single > triangular file, when seek times are nontrivial (rotating disks). I don't > believe we have these files in any other format. I'm also not sure how > easy it would be to work on such a matrix in matlab. If wb_command is > capable of doing the operations you want, it is usually possible for it to > do them without loading the entire cifti file into memory at once. wb_view > will also display maps from it without loading the entire file into memory. > > > > DDR4 has 16GB modules available at reasonable prices, which should allow > recent computers with even 4 slots for memory to expand to 64GB. You > should also consider the "high-end desktop" platforms (threadripper, > lga2066, lga2011), which typically offer 8 memory slots, and processors > with more cores. > > > > Tim > > > > > > On Wed, Oct 24, 2018 at 3:56 PM, Reza Rajimehr wrote: > > Hi, > > > > We are trying to unzip and load dense functional connectome > (HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500
Re: [HCP-Users] Question about dense functional connectome
If your swap space isn't on an SSD, then it will be extremely slow to do anything on it. Tim On Wed, Oct 24, 2018 at 7:19 PM, Glasser, Matthew wrote: > Use swap space. > > Matt. > > From: on behalf of Timothy > Coalson > Date: Wednesday, October 24, 2018 at 4:57 PM > To: Reza Rajimehr > Cc: "hcp-users@humanconnectome.org" > Subject: Re: [HCP-Users] Question about dense functional connectome > > The cifti format doesn't support that, because it is designed to allow use > without loading the entire file into memory - there is no obvious file > organization that would allow efficient loading of a full row from a single > triangular file, when seek times are nontrivial (rotating disks). I don't > believe we have these files in any other format. I'm also not sure how > easy it would be to work on such a matrix in matlab. If wb_command is > capable of doing the operations you want, it is usually possible for it to > do them without loading the entire cifti file into memory at once. wb_view > will also display maps from it without loading the entire file into memory. > > DDR4 has 16GB modules available at reasonable prices, which should allow > recent computers with even 4 slots for memory to expand to 64GB. You > should also consider the "high-end desktop" platforms (threadripper, > lga2066, lga2011), which typically offer 8 memory slots, and processors > with more cores. > > Tim > > > On Wed, Oct 24, 2018 at 3:56 PM, Reza Rajimehr wrote: > >> Hi, >> >> We are trying to unzip and load dense functional connectome >> (HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500ROW_zcorr.zip or >> HCP_S900_820_rfMRI_MSMSulc_groupPCA_d4500ROW_zcorr.zip from >> https://www.humanconnectome.org/study/hcp-young-adult/articl >> e/announcing-release-of-s900-ptn-and-other-group-average-data) in Matlab >> on a system that has 32 GB RAM. This is of course impossible due to the >> fact that the files are ~33 GB in size. Due to matrix symmetry, it would be >> sufficient to store only the upper or lower triangular part of a functional >> connectivity matrix, reducing memory occupancy by about 50% (Loewe et al., >> 2016). Does HCP provide such files that could be memory-friendly? >> >> Best, >> Reza >> >> ___ >> HCP-Users mailing list >> HCP-Users@humanconnectome.org >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >> > > ___ > HCP-Users mailing list > HCP-Users@humanconnectome.org > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > > > -- > > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] Question about dense functional connectome
As an aside, if ‘ft_read_cifti’ is expanding all the data to type double (64 bit), then you are looking at needing > 66 GB to load a 91282x91282 dconn.nii in matlab. I assume that is what is happening based on your experience. If you want to see how far you can get in Matlab, I would at least try switching to using ‘ciftiopen.m’ to load it into matlab, which uses the gifti toolbox, which preserves the underlying data type of the .dconn.nii (type single; i.e., 32 bit) upon load into matlab. I tested that just now on a system with 64 GB of RAM (and 64 GB of swap), and after what must have been at least 5 min, it loaded successfully into matlab (R2015a) with a size in matlab of 32 GB (and a type of ‘single’). Note however that as that process was ongoing, the memory usage of the overall system hit 56 GB (not counting cached memory), which decreased to 33 GB when the load was completed (presumably that is reflective of there essentially being two copies in memory during the loading process). Working with a dconn in matlab is inevitably going to be cumbersome and slow, but by using this approach you might actually be able to get it loaded and see where you can get from that point. Obviously, you would have to be extremely conscious of memory issues for whatever operations you try to perform in matlab. Cheers, -MH -- Michael Harms, Ph.D. --- Associate Professor of Psychiatry Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110 Email: mha...@wustl.edu From: on behalf of Reza Rajimehr Date: Thursday, October 25, 2018 at 3:17 AM To: "Glasser, Matthew" Cc: "hcp-users@humanconnectome.org" Subject: Re: [HCP-Users] Question about dense functional connectome We tried to open *.dconn.nii in Matlab using ft_read_cifti on an Ubuntu system with 48 GB RAM. It used 48 GB RAM and 18 GB (out of 60 GB) swap, then Matlab gave out of memory error! Will try to use a system with even higher RAM. Tim: We have a list of 913 grayordinate voxels (we have their voxel numbers/indices). These voxels are somewhat distributed in gray matter. We are basically interested in the functional connectivity matrix just for these 913 voxels (i.e. a 913 * 913 matrix). Is it possible to use wb_command to select part of *.dconn.nii corresponding to the voxels of interest? Best, Reza On Thu, Oct 25, 2018 at 3:49 AM Glasser, Matthew mailto:glass...@wustl.edu>> wrote: Use swap space. Matt. From: mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Timothy Coalson mailto:tsc...@mst.edu>> Date: Wednesday, October 24, 2018 at 4:57 PM To: Reza Rajimehr mailto:rajim...@gmail.com>> Cc: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] Question about dense functional connectome The cifti format doesn't support that, because it is designed to allow use without loading the entire file into memory - there is no obvious file organization that would allow efficient loading of a full row from a single triangular file, when seek times are nontrivial (rotating disks). I don't believe we have these files in any other format. I'm also not sure how easy it would be to work on such a matrix in matlab. If wb_command is capable of doing the operations you want, it is usually possible for it to do them without loading the entire cifti file into memory at once. wb_view will also display maps from it without loading the entire file into memory. DDR4 has 16GB modules available at reasonable prices, which should allow recent computers with even 4 slots for memory to expand to 64GB. You should also consider the "high-end desktop" platforms (threadripper, lga2066, lga2011), which typically offer 8 memory slots, and processors with more cores. Tim On Wed, Oct 24, 2018 at 3:56 PM, Reza Rajimehr mailto:rajim...@gmail.com>> wrote: Hi, We are trying to unzip and load dense functional connectome (HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500ROW_zcorr.zip or HCP_S900_820_rfMRI_MSMSulc_groupPCA_d4500ROW_zcorr.zip from https://www.humanconnectome.org/study/hcp-young-adult/article/announcing-release-of-s900-ptn-and-other-group-average-data) in Matlab on a system that has 32 GB RAM. This is of course impossible due to the fact that the files are ~33 GB in size. Due to matrix symmetry, it would be sufficient to store only the upper or lower triangular part of a functional connectivity matrix, reducing memory occupancy by about 50% (Loewe et al., 2016). Does HCP provide such files that could be memory-friendly? Best, Reza ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HC
Re: [HCP-Users] Question about dense functional connectome
Yes that would be a better solution. I think the wb_command -cifti-restrict-dense-map might work. Matt. From: Reza Rajimehr mailto:rajim...@gmail.com>> Date: Thursday, October 25, 2018 at 3:16 AM To: Matt Glasser mailto:glass...@wustl.edu>> Cc: Timothy Coalson mailto:tsc...@mst.edu>>, "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] Question about dense functional connectome We tried to open *.dconn.nii in Matlab using ft_read_cifti on an Ubuntu system with 48 GB RAM. It used 48 GB RAM and 18 GB (out of 60 GB) swap, then Matlab gave out of memory error! Will try to use a system with even higher RAM. Tim: We have a list of 913 grayordinate voxels (we have their voxel numbers/indices). These voxels are somewhat distributed in gray matter. We are basically interested in the functional connectivity matrix just for these 913 voxels (i.e. a 913 * 913 matrix). Is it possible to use wb_command to select part of *.dconn.nii corresponding to the voxels of interest? Best, Reza On Thu, Oct 25, 2018 at 3:49 AM Glasser, Matthew mailto:glass...@wustl.edu>> wrote: Use swap space. Matt. From: mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Timothy Coalson mailto:tsc...@mst.edu>> Date: Wednesday, October 24, 2018 at 4:57 PM To: Reza Rajimehr mailto:rajim...@gmail.com>> Cc: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] Question about dense functional connectome The cifti format doesn't support that, because it is designed to allow use without loading the entire file into memory - there is no obvious file organization that would allow efficient loading of a full row from a single triangular file, when seek times are nontrivial (rotating disks). I don't believe we have these files in any other format. I'm also not sure how easy it would be to work on such a matrix in matlab. If wb_command is capable of doing the operations you want, it is usually possible for it to do them without loading the entire cifti file into memory at once. wb_view will also display maps from it without loading the entire file into memory. DDR4 has 16GB modules available at reasonable prices, which should allow recent computers with even 4 slots for memory to expand to 64GB. You should also consider the "high-end desktop" platforms (threadripper, lga2066, lga2011), which typically offer 8 memory slots, and processors with more cores. Tim On Wed, Oct 24, 2018 at 3:56 PM, Reza Rajimehr mailto:rajim...@gmail.com>> wrote: Hi, We are trying to unzip and load dense functional connectome (HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500ROW_zcorr.zip or HCP_S900_820_rfMRI_MSMSulc_groupPCA_d4500ROW_zcorr.zip from https://www.humanconnectome.org/study/hcp-young-adult/article/announcing-release-of-s900-ptn-and-other-group-average-data) in Matlab on a system that has 32 GB RAM. This is of course impossible due to the fact that the files are ~33 GB in size. Due to matrix symmetry, it would be sufficient to store only the upper or lower triangular part of a functional connectivity matrix, reducing memory occupancy by about 50% (Loewe et al., 2016). Does HCP provide such files that could be memory-friendly? Best, Reza ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] Question about dense functional connectome
We tried to open *.dconn.nii in Matlab using ft_read_cifti on an Ubuntu system with 48 GB RAM. It used 48 GB RAM and 18 GB (out of 60 GB) swap, then Matlab gave out of memory error! Will try to use a system with even higher RAM. Tim: We have a list of 913 grayordinate voxels (we have their voxel numbers/indices). These voxels are somewhat distributed in gray matter. We are basically interested in the functional connectivity matrix just for these 913 voxels (i.e. a 913 * 913 matrix). Is it possible to use wb_command to select part of *.dconn.nii corresponding to the voxels of interest? Best, Reza On Thu, Oct 25, 2018 at 3:49 AM Glasser, Matthew wrote: > Use swap space. > > Matt. > > From: on behalf of Timothy > Coalson > Date: Wednesday, October 24, 2018 at 4:57 PM > To: Reza Rajimehr > Cc: "hcp-users@humanconnectome.org" > Subject: Re: [HCP-Users] Question about dense functional connectome > > The cifti format doesn't support that, because it is designed to allow use > without loading the entire file into memory - there is no obvious file > organization that would allow efficient loading of a full row from a single > triangular file, when seek times are nontrivial (rotating disks). I don't > believe we have these files in any other format. I'm also not sure how > easy it would be to work on such a matrix in matlab. If wb_command is > capable of doing the operations you want, it is usually possible for it to > do them without loading the entire cifti file into memory at once. wb_view > will also display maps from it without loading the entire file into memory. > > DDR4 has 16GB modules available at reasonable prices, which should allow > recent computers with even 4 slots for memory to expand to 64GB. You > should also consider the "high-end desktop" platforms (threadripper, > lga2066, lga2011), which typically offer 8 memory slots, and processors > with more cores. > > Tim > > > On Wed, Oct 24, 2018 at 3:56 PM, Reza Rajimehr wrote: > >> Hi, >> >> We are trying to unzip and load dense functional connectome >> (HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500ROW_zcorr.zip or >> HCP_S900_820_rfMRI_MSMSulc_groupPCA_d4500ROW_zcorr.zip from >> >> https://www.humanconnectome.org/study/hcp-young-adult/article/announcing-release-of-s900-ptn-and-other-group-average-data) >> in Matlab on a system that has 32 GB RAM. This is of course impossible due >> to the fact that the files are ~33 GB in size. Due to matrix symmetry, it >> would be sufficient to store only the upper or lower triangular part of a >> functional connectivity matrix, reducing memory occupancy by about 50% >> (Loewe et al., 2016). Does HCP provide such files that could be >> memory-friendly? >> >> Best, >> Reza >> >> ___ >> HCP-Users mailing list >> HCP-Users@humanconnectome.org >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >> > > ___ > HCP-Users mailing list > HCP-Users@humanconnectome.org > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > > > -- > > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you > are not the intended recipient, be advised that any unauthorized use, > disclosure, copying or the taking of any action in reliance on the contents > of this information is strictly prohibited. If you have received this email > in error, please immediately notify the sender via telephone or return mail. > ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] Question about dense functional connectome
Use swap space. Matt. From: mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Timothy Coalson mailto:tsc...@mst.edu>> Date: Wednesday, October 24, 2018 at 4:57 PM To: Reza Rajimehr mailto:rajim...@gmail.com>> Cc: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] Question about dense functional connectome The cifti format doesn't support that, because it is designed to allow use without loading the entire file into memory - there is no obvious file organization that would allow efficient loading of a full row from a single triangular file, when seek times are nontrivial (rotating disks). I don't believe we have these files in any other format. I'm also not sure how easy it would be to work on such a matrix in matlab. If wb_command is capable of doing the operations you want, it is usually possible for it to do them without loading the entire cifti file into memory at once. wb_view will also display maps from it without loading the entire file into memory. DDR4 has 16GB modules available at reasonable prices, which should allow recent computers with even 4 slots for memory to expand to 64GB. You should also consider the "high-end desktop" platforms (threadripper, lga2066, lga2011), which typically offer 8 memory slots, and processors with more cores. Tim On Wed, Oct 24, 2018 at 3:56 PM, Reza Rajimehr mailto:rajim...@gmail.com>> wrote: Hi, We are trying to unzip and load dense functional connectome (HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500ROW_zcorr.zip or HCP_S900_820_rfMRI_MSMSulc_groupPCA_d4500ROW_zcorr.zip from https://www.humanconnectome.org/study/hcp-young-adult/article/announcing-release-of-s900-ptn-and-other-group-average-data) in Matlab on a system that has 32 GB RAM. This is of course impossible due to the fact that the files are ~33 GB in size. Due to matrix symmetry, it would be sufficient to store only the upper or lower triangular part of a functional connectivity matrix, reducing memory occupancy by about 50% (Loewe et al., 2016). Does HCP provide such files that could be memory-friendly? Best, Reza ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] Question about dense functional connectome
The cifti format doesn't support that, because it is designed to allow use without loading the entire file into memory - there is no obvious file organization that would allow efficient loading of a full row from a single triangular file, when seek times are nontrivial (rotating disks). I don't believe we have these files in any other format. I'm also not sure how easy it would be to work on such a matrix in matlab. If wb_command is capable of doing the operations you want, it is usually possible for it to do them without loading the entire cifti file into memory at once. wb_view will also display maps from it without loading the entire file into memory. DDR4 has 16GB modules available at reasonable prices, which should allow recent computers with even 4 slots for memory to expand to 64GB. You should also consider the "high-end desktop" platforms (threadripper, lga2066, lga2011), which typically offer 8 memory slots, and processors with more cores. Tim On Wed, Oct 24, 2018 at 3:56 PM, Reza Rajimehr wrote: > Hi, > > We are trying to unzip and load dense functional connectome > (HCP_S900_820_rfMRI_MSMAll_groupPCA_d4500ROW_zcorr.zip or > HCP_S900_820_rfMRI_MSMSulc_groupPCA_d4500ROW_zcorr.zip from > https://www.humanconnectome.org/study/hcp-young-adult/ > article/announcing-release-of-s900-ptn-and-other-group-average-data) in > Matlab on a system that has 32 GB RAM. This is of course impossible due to > the fact that the files are ~33 GB in size. Due to matrix symmetry, it > would be sufficient to store only the upper or lower triangular part of a > functional connectivity matrix, reducing memory occupancy by about 50% > (Loewe et al., 2016). Does HCP provide such files that could be > memory-friendly? > > Best, > Reza > > ___ > HCP-Users mailing list > HCP-Users@humanconnectome.org > http://lists.humanconnectome.org/mailman/listinfo/hcp-users > ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
