Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while
Thanks for letting us know. Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Monday, April 8, 2019 at 6:34 AM To: Timothy Coalson mailto:tsc...@mst.edu>> Cc: Matt Glasser mailto:glass...@wustl.edu>>, Ido Tavor mailto:idota...@gmail.com>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while Hey Tim, Sorry for the belated response. The opm_num_threads solution worked great! It runs in a few minutes now. Thanks, Shachar On Fri, Apr 5, 2019, 21:12 Timothy Coalson mailto:tsc...@mst.edu> wrote: Have you had a chance to try this? If nothing else, you can export OMP_NUM_THREADS=1 and see if that command finishes in less than an hour. Tim On Tue, Apr 2, 2019 at 2:47 AM Shachar Gal mailto:gal.shac...@gmail.com>> wrote: sounds reasonable we'll try to run the process again and restrict it to one socket, and see if it speeds things up, and let you know thanks! Shachar On Mon, 1 Apr 2019 at 23:04, Timothy Coalson mailto:tsc...@mst.edu>> wrote: This finished in just over 2 minutes on my quad-core, so I am betting it is the multi-socket or other architecture issue: tim@timsdev:~/Downloads$ time nice released_wb_command -cifti-resample rfMRI_REST_AP_temp_subject_smooth.dtseries.nii COLUMN rfMRI_REST_AP_temp_template.dlabel.nii COLUMN ADAP_BARY_AREA CUBIC test.dtseries.nii -volume-predilate 10 real 2m12.248s user 16m42.659s sys 0m1.365s tim@timsdev:~/Downloads$ released_wb_command -version Connectome Workbench Type: Command Line Application Version: 1.3.2 Qt Compiled Version: 5.7.0 Qt Runtime Version: 5.7.0 Commit: 1c774d37d6cb5a179e0de3cbe1081cd1f698963a Commit Date: 2018-08-28 08:50:51 -0500 Compiled with OpenMP: YES Compiler: g++ (/home/caret/gcc/install/gcc-4.8.5/bin) Compiler Version: 4.8.5 Compiled Debug: NO Operating System: Linux Tim On Mon, Apr 1, 2019 at 2:01 PM Timothy Coalson mailto:tsc...@mst.edu>> wrote: Sorry, that is only one physical CPU per process - you can run one instance of wb_command on each CPU just fine, it is simply when a single wb_command process has threads on more than one physical CPU that things get very slow. Tim On Mon, Apr 1, 2019 at 1:05 PM Timothy Coalson mailto:tsc...@mst.edu>> wrote: Does the computer you were running this on have multiple CPU sockets? Synchronizing code across physical sockets is quite expensive in terms of time, and on such systems we strongly recommend using cpusets or restricting the number of threads to try to use only one physical CPU. Tim On Mon, Apr 1, 2019 at 2:15 AM Shachar Gal mailto:gal.shac...@gmail.com>> wrote: well, it took 13 hours, but it completed the execution, and the final atlas.dtseries output looks good.. so it just takes a REALLY long time, and it takes a crazy amount of CPU [image.png] let me know if anything else happens when you run it I your environment? Shachar On Mon, 1 Apr 2019 at 00:19, Glasser, Matthew mailto:glass...@wustl.edu>> wrote: We’ll have a look tomorrow and let you know what we find. Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 4:17 PM To: Matt Glasser mailto:glass...@wustl.edu>> Cc: Timothy Coalson mailto:tsc...@mst.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while It does, and I've uploaded it as well to the drive, so you could have a look yourself Shachar On Mon, Apr 1, 2019, 00:12 Glasser, Matthew mailto:glass...@wustl.edu> wrote: Well in that case this confirms that what Tim was thinking is not the issue and that we will need to look at the files themselves. Again, we didn’t really change stuff in the fMRISurface pipeline recently, so this is quite puzzling… Does the output of fMRIVolume look okay to you? Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 4:09 PM To: Matt Glasser mailto:glass...@wustl.edu>> Cc: Timothy Coalson mailto:tsc...@mst.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while sure this specific command is identical as far as i can tell, so maybe the difference is more upstream than this, and has to do with the way the input files were constructed (whether its the volume input to this pipeline, or upstream inside the fMRISurface pipeline)... #!/bin/bash set -e script_name="SubcorticalProcessing.sh" echo "${script_name}: START" AtlasSpaceFolder="$1" echo "${script_name}: AtlasSpaceFolder: ${AtlasSpaceFolder}" ROIFolder="$2" echo "${script_name}: ROIFolder: ${ROIFolder}" FinalfMRIResolution="$3" echo "${script_name}: FinalfMRIResolution: ${FinalfMRIResolution}" ResultsFolder="$4" echo "${script_name}: ResultsFolder: ${ResultsFolder}" NameOffMRI=&quo
Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while
I’d like Tim’s input on where to upload if you don’t have your own convenient spot. Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:58 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while yes, it happened for each of my subjects. these are the input files for the command - temp_subject_smooth.dtseries.nii, temp_template.dlabel.nii where should i upload them to? On Sun, 31 Mar 2019 at 18:56, Glasser, Matthew mailto:glass...@wustl.edu>> wrote: We need the input files to the command that is taking too long. Also, does the problem happen reproducibly? Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:53 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while so ive connected to our server and visually examined the rfMRI_REST_AP.nii.gz file, the output from the fMRIVolume pipeline, and it looks great. where would you like me to upload the files, and which files exactly, other than temp_subject_smooth.dtseries.nii, temp_template.dlabel.nii and the registered volume? thanks, Shachar On Sun, 31 Mar 2019 at 18:37, Glasser, Matthew mailto:glass...@wustl.edu>> wrote: Perhaps you should upload the various input files to the specific wb_command command somewhere so we can test them if this issue is reproducible (i.e. is not a random thing from bad RAM or something like that). Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:32 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while Its 500 time points. I'll have a look at the volume input tommorow when I get back to the office. I ran them serially, the fmri pipelines.. Cheers Shachar On Sun, Mar 31, 2019, 18:13 Glasser, Matthew mailto:glass...@wustl.edu> wrote: How many volumes and did you look at the data being input yet? This code didn’t change so we don’t expect any difference. Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:10 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while hey matt, this is the call to the pipeline: /root/hcppilelines/HCPpipelines-master/fMRISurface/GenericfMRISurfaceProcessingPipeline.sh --path=/root/hcppilelines/piano_hcp/working_directory --subject=101 --fmriname=rfMRI_REST_AP --lowresmesh=32 --fmrires=2 --smoothingFWHM=4 --grayordinatesres=2 --regname=MSMSulc the TR is 0.75, and the resolution is 2 mm isometric. as i wrote, this went smoothly with the previous release of the pipelines, and resulted in data that looked great thanks, Shachar On Sun, 31 Mar 2019 at 17:53, Glasser, Matthew mailto:glass...@wustl.edu>> wrote: How did you call fMRISurface and what kind of data is this in terms of spatial and temporal resolution? Do the input and template data line up and look reasonable? Matt From: mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 8:27 AM To: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while some further information that might be useful - the output file rfMRI_REST_AP_temp_atlas.dtseries.nii, has already been created. its in the target folder. but the command won't stop "running" On Sun, 31 Mar 2019 at 16:13, Shachar Gal mailto:gal.shac...@gmail.com>> wrote: Dear experts, while running the fMRISurface pipeline (using the latest pipeline release) I encountered an issue where the subcortical-processing script takes a really long while (now running for more than 6 hours). this is the command on which it is stuck ${CARET7DIR}/wb_command -cifti-resample ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii COLUMN ${ResultsFolder}/${NameOffMRI}_temp_template.dlabel.nii COLUMN ADAP_BARY_AREA CUBIC ${ResultsFolder}/${NameOffMRI}_temp_atlas.dtseries.nii -volume-predilate 10 rm -f ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii the wb_command version is 1.3.2 there is no error being raise or anything. its still running, but its ages... the thing is, when running the older version of the pipeline (but the same workbench version), there was no problem. any ideas? ill report later on if it completes the command eventually thanks, Shachar ___ HCP-
Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while
We need the input files to the command that is taking too long. Also, does the problem happen reproducibly? Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:53 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while so ive connected to our server and visually examined the rfMRI_REST_AP.nii.gz file, the output from the fMRIVolume pipeline, and it looks great. where would you like me to upload the files, and which files exactly, other than temp_subject_smooth.dtseries.nii, temp_template.dlabel.nii and the registered volume? thanks, Shachar On Sun, 31 Mar 2019 at 18:37, Glasser, Matthew mailto:glass...@wustl.edu>> wrote: Perhaps you should upload the various input files to the specific wb_command command somewhere so we can test them if this issue is reproducible (i.e. is not a random thing from bad RAM or something like that). Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:32 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while Its 500 time points. I'll have a look at the volume input tommorow when I get back to the office. I ran them serially, the fmri pipelines.. Cheers Shachar On Sun, Mar 31, 2019, 18:13 Glasser, Matthew mailto:glass...@wustl.edu> wrote: How many volumes and did you look at the data being input yet? This code didn’t change so we don’t expect any difference. Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:10 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while hey matt, this is the call to the pipeline: /root/hcppilelines/HCPpipelines-master/fMRISurface/GenericfMRISurfaceProcessingPipeline.sh --path=/root/hcppilelines/piano_hcp/working_directory --subject=101 --fmriname=rfMRI_REST_AP --lowresmesh=32 --fmrires=2 --smoothingFWHM=4 --grayordinatesres=2 --regname=MSMSulc the TR is 0.75, and the resolution is 2 mm isometric. as i wrote, this went smoothly with the previous release of the pipelines, and resulted in data that looked great thanks, Shachar On Sun, 31 Mar 2019 at 17:53, Glasser, Matthew mailto:glass...@wustl.edu>> wrote: How did you call fMRISurface and what kind of data is this in terms of spatial and temporal resolution? Do the input and template data line up and look reasonable? Matt From: mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 8:27 AM To: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while some further information that might be useful - the output file rfMRI_REST_AP_temp_atlas.dtseries.nii, has already been created. its in the target folder. but the command won't stop "running" On Sun, 31 Mar 2019 at 16:13, Shachar Gal mailto:gal.shac...@gmail.com>> wrote: Dear experts, while running the fMRISurface pipeline (using the latest pipeline release) I encountered an issue where the subcortical-processing script takes a really long while (now running for more than 6 hours). this is the command on which it is stuck ${CARET7DIR}/wb_command -cifti-resample ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii COLUMN ${ResultsFolder}/${NameOffMRI}_temp_template.dlabel.nii COLUMN ADAP_BARY_AREA CUBIC ${ResultsFolder}/${NameOffMRI}_temp_atlas.dtseries.nii -volume-predilate 10 rm -f ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii the wb_command version is 1.3.2 there is no error being raise or anything. its still running, but its ages... the thing is, when running the older version of the pipeline (but the same workbench version), there was no problem. any ideas? ill report later on if it completes the command eventually thanks, Shachar ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone
Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while
Perhaps you should upload the various input files to the specific wb_command command somewhere so we can test them if this issue is reproducible (i.e. is not a random thing from bad RAM or something like that). Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:32 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while Its 500 time points. I'll have a look at the volume input tommorow when I get back to the office. I ran them serially, the fmri pipelines.. Cheers Shachar On Sun, Mar 31, 2019, 18:13 Glasser, Matthew mailto:glass...@wustl.edu> wrote: How many volumes and did you look at the data being input yet? This code didn’t change so we don’t expect any difference. Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 10:10 AM To: Matt Glasser mailto:glass...@wustl.edu>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while hey matt, this is the call to the pipeline: /root/hcppilelines/HCPpipelines-master/fMRISurface/GenericfMRISurfaceProcessingPipeline.sh --path=/root/hcppilelines/piano_hcp/working_directory --subject=101 --fmriname=rfMRI_REST_AP --lowresmesh=32 --fmrires=2 --smoothingFWHM=4 --grayordinatesres=2 --regname=MSMSulc the TR is 0.75, and the resolution is 2 mm isometric. as i wrote, this went smoothly with the previous release of the pipelines, and resulted in data that looked great thanks, Shachar On Sun, 31 Mar 2019 at 17:53, Glasser, Matthew mailto:glass...@wustl.edu>> wrote: How did you call fMRISurface and what kind of data is this in terms of spatial and temporal resolution? Do the input and template data line up and look reasonable? Matt From: mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 8:27 AM To: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while some further information that might be useful - the output file rfMRI_REST_AP_temp_atlas.dtseries.nii, has already been created. its in the target folder. but the command won't stop "running" On Sun, 31 Mar 2019 at 16:13, Shachar Gal mailto:gal.shac...@gmail.com>> wrote: Dear experts, while running the fMRISurface pipeline (using the latest pipeline release) I encountered an issue where the subcortical-processing script takes a really long while (now running for more than 6 hours). this is the command on which it is stuck ${CARET7DIR}/wb_command -cifti-resample ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii COLUMN ${ResultsFolder}/${NameOffMRI}_temp_template.dlabel.nii COLUMN ADAP_BARY_AREA CUBIC ${ResultsFolder}/${NameOffMRI}_temp_atlas.dtseries.nii -volume-predilate 10 rm -f ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii the wb_command version is 1.3.2 there is no error being raise or anything. its still running, but its ages... the thing is, when running the older version of the pipeline (but the same workbench version), there was no problem. any ideas? ill report later on if it completes the command eventually thanks, Shachar ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. On Mar 31, 2019 18:13, "Glasser, Matthew" mailto:glass...@wustl.edu>> wrote: How many volumes and did you look at the data being input yet? This code didn’t change so we don’t expect any difference. Matt. From: Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 20
Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while
How did you call fMRISurface and what kind of data is this in terms of spatial and temporal resolution? Do the input and template data line up and look reasonable? Matt From: mailto:hcp-users-boun...@humanconnectome.org>> on behalf of Shachar Gal mailto:gal.shac...@gmail.com>> Date: Sunday, March 31, 2019 at 8:27 AM To: "hcp-users@humanconnectome.org<mailto:hcp-users@humanconnectome.org>" mailto:hcp-users@humanconnectome.org>> Subject: Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while some further information that might be useful - the output file rfMRI_REST_AP_temp_atlas.dtseries.nii, has already been created. its in the target folder. but the command won't stop "running" On Sun, 31 Mar 2019 at 16:13, Shachar Gal mailto:gal.shac...@gmail.com>> wrote: Dear experts, while running the fMRISurface pipeline (using the latest pipeline release) I encountered an issue where the subcortical-processing script takes a really long while (now running for more than 6 hours). this is the command on which it is stuck ${CARET7DIR}/wb_command -cifti-resample ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii COLUMN ${ResultsFolder}/${NameOffMRI}_temp_template.dlabel.nii COLUMN ADAP_BARY_AREA CUBIC ${ResultsFolder}/${NameOffMRI}_temp_atlas.dtseries.nii -volume-predilate 10 rm -f ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii the wb_command version is 1.3.2 there is no error being raise or anything. its still running, but its ages... the thing is, when running the older version of the pipeline (but the same workbench version), there was no problem. any ideas? ill report later on if it completes the command eventually thanks, Shachar ___ HCP-Users mailing list HCP-Users@humanconnectome.org<mailto:HCP-Users@humanconnectome.org> http://lists.humanconnectome.org/mailman/listinfo/hcp-users The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
Re: [HCP-Users] wb_command -cifti-resample in sub-cortical processing taking suspiciously long while
some further information that might be useful - the output file rfMRI_REST_AP_temp_atlas.dtseries.nii, has already been created. its in the target folder. but the command won't stop "running" On Sun, 31 Mar 2019 at 16:13, Shachar Gal wrote: > Dear experts, > > while running the fMRISurface pipeline (using the latest pipeline release) > I encountered an issue where the subcortical-processing script takes a > really long while (now running for more than 6 hours). > this is the command on which it is stuck > > ${CARET7DIR}/wb_command -cifti-resample > ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii COLUMN > ${ResultsFolder}/${NameOffMRI}_temp_template.dlabel.nii COLUMN > ADAP_BARY_AREA CUBIC ${ResultsFolder}/${NameOffMRI}_temp_atlas.dtseries.nii > -volume-predilate 10 > rm -f ${ResultsFolder}/${NameOffMRI}_temp_subject_smooth.dtseries.nii > > the wb_command version is 1.3.2 > there is no error being raise or anything. its still running, but its > ages... > the thing is, when running the older version of the pipeline (but the same > workbench version), there was no problem. > any ideas? > > ill report later on if it completes the command eventually > > thanks, > Shachar > > > ___ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users