Re: [Freesurfer] Recon-all on a cluster error
External Email - Use Caution Hi Doug, Ok I'll verify the creation of the symlink today. Thank you for following-up. Could you also maybe give me some details about what this termination status means? : nu_correct: crashed while running nu_estimate_np_and_em (termination status=256) Perhaps I can try to debug if I knew a little more. Sorry if this information is easily available. I couldn't find what the termination status means. Best, Michael -Original Message- From: freesurfer-boun...@nmr.mgh.harvard.edu On Behalf Of Greve, Douglas N.,Ph.D. Sent: Thursday, September 19, 2019 7:41 AM To: freesurfer@nmr.mgh.harvard.edu; Dicamillo, Robert Subject: Re: [Freesurfer] Recon-all on a cluster error Not sure then. I've seen cases where the creation of symbolic links was not possible on a cluster. It does not look like this is the problem, but it could be doing something behind the scenes. Can you verify that you can create a symlink from a program on the cluster? I've cc'ed Rob D who can help debug this problem if it persists. doug On 9/17/19 1:57 PM, Lauricella, Michael wrote: > External Email - Use Caution > > Hi Douglas, > > Yes it does fine. On the cluster, this same error happens for all subjects. > > Thank you for your help! > > Best, > Michael > > -Original Message- > From: freesurfer-boun...@nmr.mgh.harvard.edu > On Behalf Of Greve, Douglas N.,Ph.D. > Sent: Tuesday, September 17, 2019 10:50 AM > To: freesurfer@nmr.mgh.harvard.edu > Subject: Re: [Freesurfer] Recon-all on a cluster error > > Does it run properly off the cluster for this data set? > > On 9/17/19 1:45 PM, Lauricella, Michael wrote: >> External Email - Use Caution >> >> Dear Freesurfer Experts, >> >> I have a an error when running recon-all on a computing cluster. I've >> checked the environment and so have some of my more knowledgeable >> colleagues but we can't see anything wrong with the environment. I >> was wondering if anyone else has experienced difficulties like the >> one below when trying to run recon-all on a computing cluster and >> might have some advice. Error of my run is below and the full log >> file is attached. Thank you in advance for your help! >> >> "Iteration 1 Wed Aug 7 13:53:08 PDT 2019 >> >> nu_correct -clobber ./tmp.mri_nu_correct.mni.18980/nu0.mnc >> ./tmp.mri_nu_correct.mni.18980/nu1.mnc -tmpdir >> ./tmp.mri_nu_correct.mni.18980/0/ -iterations 1000 -distance 50 >> >> [mlauricella@qb3-id115:/mnt/freesurfer_subjs/SUBJECT_ID/mri/] >> [2019-08-07 13:53:08] running: >> >> /mnt/neuroimaging/FreeSurfer/V6.0.0/mni/bin/nu_estimate_np_and_em >> -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink >> 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute >> -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.18980/0/ >> ./tmp.mri_nu_correct.mni.18980/nu0.mnc >> ./tmp.mri_nu_correct.mni.18980/nu1.imp >> >> Assertion failed at line 742 in file templates/CachedArray.cc >> >> nu_estimate_np_and_em: crashed while running volume_stats >> (termination >> status=256) >> >> nu_correct: crashed while running nu_estimate_np_and_em (termination >> status=256) >> >> ERROR: nu_correct >> >> Linux qb3-id115 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 >> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux >> >> recon-all -s SUBJECT_ID exited with ERRORS at Wed Aug 7 13:53:09 PDT >> 2019 >> >> To report a problem, see >> https://urldefense.proofpoint.com/v2/url?u=http-3A__surfer.nmr.mgh.ha >> r >> vard.edu_fswiki_BugReporting=DwIF-g=iORugZls2LlYyCAZRB3XLg=LfvX >> l >> A0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=lOOasQKo-5smKR_WagoWBOFxCVZA >> 3 8xYpWHKJOqIVGU=hZ8c1_TEb9oDCsFUiTaioaVn7pRCbIkq9BrlyHA-jps= >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__surfer.nmr.mgh.harvard.edu_fswiki_BugReporting=DwIF-g=iORugZls2LlYyCAZRB3XLg=LfvXlA0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=lOOasQKo-5smKR_WagoWBOFxCVZA38xYpWHKJOqIVGU=hZ8c1_TEb9oDCsFUiTaioaVn7pRCbIkq9BrlyHA-jps= >> >" >> >> Best, >> >> Michael >> >> >> ___ >> Freesurfer mailing list >> Freesurfer@nmr.mgh.harvard.edu >> https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.nmr.mgh.har >> v >> ard.edu_mailman_listinfo_freesurfer=DwIF-g=iORugZls2LlYyCAZRB3XLg >> & >> r=LfvXlA0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=lOOasQKo-5smKR_WagoWB >> O >> FxCVZA38xYpWHKJOqIVGU=RJ3pNp6iSv3KoImUDqKyKKlfJax1r3VtjF-gcSU_E60 >> = >
Re: [Freesurfer] mris_sphere question
External Email - Use Caution Hi Bruce, Sometimes it completes. About 20% of the time I get this problem for anyone. I don't think it's specific to a certain subject. Best, Michael -Original Message- From: freesurfer-boun...@nmr.mgh.harvard.edu On Behalf Of Bruce Fischl Sent: Tuesday, September 17, 2019 10:53 AM To: Freesurfer support list Subject: Re: [Freesurfer] mris_sphere question hmmm, if you rerun does it complete? On Tue, 17 Sep 2019, Lauricella, Michael wrote: >External Email - Use Caution > > Hi Bruce, > > No error message, just runs forever and eventually breaks my SSH connection > to the cloud I'm running it on. But when I do "-make all" afterwards it > starts a few steps before mris_spheres, so I have to redo and often crashes > again. > > Thank you for your help! > > Best, > Michael > > -Original Message- > From: freesurfer-boun...@nmr.mgh.harvard.edu > On Behalf Of Bruce Fischl > Sent: Tuesday, September 17, 2019 10:16 AM > To: Freesurfer support list > Subject: Re: [Freesurfer] mris_sphere question > > Hi Michael > > I think that is the last message that mris_sphere prints so I believe it has > finished. Does it exit with an error message or just keep running forever? > Bruce > > > On Tue, 17 Sep 2019, Lauricella, Michael wrote: > >> >> External Email - Use Caution >> >> Dear Freesurfer Experts, >> >> >> >> I’m fairly new to freesurfer, but have a set-up in which I have been >> able to run many successful recon-all runs. One problem that I have >> sometimes, maybe about 20% of the time, is that the run will freeze and >> crash at this step: >> >> >> >> mris_sphere -rusage >> /mnt/freesurfer_subjs/SUBJECT_ID/touch/rusage.mris_sphere.lh.dat >> -seed >> 1234 ../surf/lh.inflated ../surf/lh.sphere >> >> >> >> here’s the end of the logfile: >> >> >> >> “== Number of threads available to mris_sphere for OpenMP = 2 == >> >> scaling brain by 0.287... >> >> MRISunfold() max_passes = 1 --- >> >> tol=5.0e-01, sigma=0.0, host=mgt-i, nav=1024, nbrs=2, l_area=1.000, >> l_dist=1.000 >> >> using quadratic fit line minimization >> >> complete_dist_mat 0 >> >> rms 0 >> >> smooth_averages 0 >> >> remove_neg 0 >> >> ico_order 0 >> >> which_surface 0 >> >> target_radius 0.00 >> >> nfields 0 >> >> scale 1.00 >> >> desired_rms_height -1.00 >> >> momentum 0.90 >> >> nbhd_size 7 >> >> max_nbrs 8 >> >> niterations 25 >> >> nsurfaces 0 >> >> SURFACES 3 >> >> flags 0 (0) >> >> use curv 0 >> >> no sulc 0 >> >> no rigid align 0 >> >> mris->nsize 2 >> >> mris->hemisphere 0 >> >> randomSeed 1234 >> >> >> >> >> >> mrisRemoveNegativeArea() >> >> pass 1: epoch 1 of 3 starting distance error %21.21 >> >> pass 1: epoch 2 of 3 starting distance error %20.95 >> >> unfolding complete - removing small folds... >> >> starting distance error %20.57 >> >> removing remaining folds... >> >> final distance error %20.59 >> >> MRISunfold() return, current seed 1234” >> >> >> >> >> >> Does anyone have any ideas why this might be happening? Maybe this is >> a computationally intensive step? If I run a few recon-alls >> concurrently it almost always crashes at this step. >> >> >> >> The command I run is: recon-all -all -openmp 2 -qcache >> -nuintensitycor-3T -s SUBJECT_ID -sd /mnt/freesurfer_subjs/ >> >> >> >> Thank you so much for your help! >> >> >> >> Best, >> >> Michael >> >> >> > > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.nmr.mgh.harv > ard.edu_mailman_listinfo_freesurfer=DwIDaQ=iORugZls2LlYyCAZRB3XLg& > r=LfvXlA0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=wDKDRoscofjbwXBayxe0Xl > 3ZJe_ADuGgm0djvLYBcNc=04816km8Ktp-KqrA37Scn_k4WuUW-xi0Ha0OnqlUrVw= > > > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Re: [Freesurfer] Recon-all on a cluster error
External Email - Use Caution Hi Douglas, Yes it does fine. On the cluster, this same error happens for all subjects. Thank you for your help! Best, Michael -Original Message- From: freesurfer-boun...@nmr.mgh.harvard.edu On Behalf Of Greve, Douglas N.,Ph.D. Sent: Tuesday, September 17, 2019 10:50 AM To: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Recon-all on a cluster error Does it run properly off the cluster for this data set? On 9/17/19 1:45 PM, Lauricella, Michael wrote: > > External Email - Use Caution > > Dear Freesurfer Experts, > > I have a an error when running recon-all on a computing cluster. I've > checked the environment and so have some of my more knowledgeable > colleagues but we can't see anything wrong with the environment. I was > wondering if anyone else has experienced difficulties like the one > below when trying to run recon-all on a computing cluster and might > have some advice. Error of my run is below and the full log file is > attached. Thank you in advance for your help! > > "Iteration 1 Wed Aug 7 13:53:08 PDT 2019 > > nu_correct -clobber ./tmp.mri_nu_correct.mni.18980/nu0.mnc > ./tmp.mri_nu_correct.mni.18980/nu1.mnc -tmpdir > ./tmp.mri_nu_correct.mni.18980/0/ -iterations 1000 -distance 50 > > [mlauricella@qb3-id115:/mnt/freesurfer_subjs/SUBJECT_ID/mri/] > [2019-08-07 13:53:08] running: > > /mnt/neuroimaging/FreeSurfer/V6.0.0/mni/bin/nu_estimate_np_and_em > -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 > -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute > -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.18980/0/ > ./tmp.mri_nu_correct.mni.18980/nu0.mnc > ./tmp.mri_nu_correct.mni.18980/nu1.imp > > Assertion failed at line 742 in file templates/CachedArray.cc > > nu_estimate_np_and_em: crashed while running volume_stats (termination > status=256) > > nu_correct: crashed while running nu_estimate_np_and_em (termination > status=256) > > ERROR: nu_correct > > Linux qb3-id115 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 > UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > > recon-all -s SUBJECT_ID exited with ERRORS at Wed Aug 7 13:53:09 PDT > 2019 > > To report a problem, see > https://urldefense.proofpoint.com/v2/url?u=http-3A__surfer.nmr.mgh.har > vard.edu_fswiki_BugReporting=DwIF-g=iORugZls2LlYyCAZRB3XLg=LfvXl > A0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=lOOasQKo-5smKR_WagoWBOFxCVZA3 > 8xYpWHKJOqIVGU=hZ8c1_TEb9oDCsFUiTaioaVn7pRCbIkq9BrlyHA-jps= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__surfer.nmr.mgh.harvard.edu_fswiki_BugReporting=DwIF-g=iORugZls2LlYyCAZRB3XLg=LfvXlA0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=lOOasQKo-5smKR_WagoWBOFxCVZA38xYpWHKJOqIVGU=hZ8c1_TEb9oDCsFUiTaioaVn7pRCbIkq9BrlyHA-jps= > >" > > Best, > > Michael > > > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.nmr.mgh.harv > ard.edu_mailman_listinfo_freesurfer=DwIF-g=iORugZls2LlYyCAZRB3XLg& > r=LfvXlA0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=lOOasQKo-5smKR_WagoWBO > FxCVZA38xYpWHKJOqIVGU=RJ3pNp6iSv3KoImUDqKyKKlfJax1r3VtjF-gcSU_E60= ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.nmr.mgh.harvard.edu_mailman_listinfo_freesurfer=DwIF-g=iORugZls2LlYyCAZRB3XLg=LfvXlA0kIui4WmLLCYdWlZdLsYFY_HKuPRMZoERJi_o=lOOasQKo-5smKR_WagoWBOFxCVZA38xYpWHKJOqIVGU=RJ3pNp6iSv3KoImUDqKyKKlfJax1r3VtjF-gcSU_E60= ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
[Freesurfer] Recon-all on a cluster error
External Email - Use Caution Dear Freesurfer Experts, I have a an error when running recon-all on a computing cluster. I've checked the environment and so have some of my more knowledgeable colleagues but we can't see anything wrong with the environment. I was wondering if anyone else has experienced difficulties like the one below when trying to run recon-all on a computing cluster and might have some advice. Error of my run is below and the full log file is attached. Thank you in advance for your help! "Iteration 1 Wed Aug 7 13:53:08 PDT 2019 nu_correct -clobber ./tmp.mri_nu_correct.mni.18980/nu0.mnc ./tmp.mri_nu_correct.mni.18980/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.18980/0/ -iterations 1000 -distance 50 [mlauricella@qb3-id115:/mnt/freesurfer_subjs/SUBJECT_ID/mri/] [2019-08-07 13:53:08] running: /mnt/neuroimaging/FreeSurfer/V6.0.0/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.18980/0/ ./tmp.mri_nu_correct.mni.18980/nu0.mnc ./tmp.mri_nu_correct.mni.18980/nu1.imp Assertion failed at line 742 in file templates/CachedArray.cc nu_estimate_np_and_em: crashed while running volume_stats (termination status=256) nu_correct: crashed while running nu_estimate_np_and_em (termination status=256) ERROR: nu_correct Linux qb3-id115 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux recon-all -s SUBJECT_ID exited with ERRORS at Wed Aug 7 13:53:09 PDT 2019 To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting; Best, Michael cluster_error_recon-all.log Description: cluster_error_recon-all.log ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Re: [Freesurfer] mris_sphere question
External Email - Use Caution Hi Bruce, No error message, just runs forever and eventually breaks my SSH connection to the cloud I'm running it on. But when I do "-make all" afterwards it starts a few steps before mris_spheres, so I have to redo and often crashes again. Thank you for your help! Best, Michael -Original Message- From: freesurfer-boun...@nmr.mgh.harvard.edu On Behalf Of Bruce Fischl Sent: Tuesday, September 17, 2019 10:16 AM To: Freesurfer support list Subject: Re: [Freesurfer] mris_sphere question Hi Michael I think that is the last message that mris_sphere prints so I believe it has finished. Does it exit with an error message or just keep running forever? Bruce On Tue, 17 Sep 2019, Lauricella, Michael wrote: > > External Email - Use Caution > > Dear Freesurfer Experts, > > > > I’m fairly new to freesurfer, but have a set-up in which I have been > able to run many successful recon-all runs. One problem that I have > sometimes, maybe about 20% of the time, is that the run will freeze and crash > at this step: > > > > mris_sphere -rusage > /mnt/freesurfer_subjs/SUBJECT_ID/touch/rusage.mris_sphere.lh.dat -seed > 1234 ../surf/lh.inflated ../surf/lh.sphere > > > > here’s the end of the logfile: > > > > “== Number of threads available to mris_sphere for OpenMP = 2 == > > scaling brain by 0.287... > > MRISunfold() max_passes = 1 --- > > tol=5.0e-01, sigma=0.0, host=mgt-i, nav=1024, nbrs=2, l_area=1.000, > l_dist=1.000 > > using quadratic fit line minimization > > complete_dist_mat 0 > > rms 0 > > smooth_averages 0 > > remove_neg 0 > > ico_order 0 > > which_surface 0 > > target_radius 0.00 > > nfields 0 > > scale 1.00 > > desired_rms_height -1.00 > > momentum 0.90 > > nbhd_size 7 > > max_nbrs 8 > > niterations 25 > > nsurfaces 0 > > SURFACES 3 > > flags 0 (0) > > use curv 0 > > no sulc 0 > > no rigid align 0 > > mris->nsize 2 > > mris->hemisphere 0 > > randomSeed 1234 > > > > > > mrisRemoveNegativeArea() > > pass 1: epoch 1 of 3 starting distance error %21.21 > > pass 1: epoch 2 of 3 starting distance error %20.95 > > unfolding complete - removing small folds... > > starting distance error %20.57 > > removing remaining folds... > > final distance error %20.59 > > MRISunfold() return, current seed 1234” > > > > > > Does anyone have any ideas why this might be happening? Maybe this is > a computationally intensive step? If I run a few recon-alls > concurrently it almost always crashes at this step. > > > > The command I run is: recon-all -all -openmp 2 -qcache > -nuintensitycor-3T -s SUBJECT_ID -sd /mnt/freesurfer_subjs/ > > > > Thank you so much for your help! > > > > Best, > > Michael > > > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
[Freesurfer] mris_sphere question
External Email - Use Caution Dear Freesurfer Experts, I'm fairly new to freesurfer, but have a set-up in which I have been able to run many successful recon-all runs. One problem that I have sometimes, maybe about 20% of the time, is that the run will freeze and crash at this step: mris_sphere -rusage /mnt/freesurfer_subjs/SUBJECT_ID/touch/rusage.mris_sphere.lh.dat -seed 1234 ../surf/lh.inflated ../surf/lh.sphere here's the end of the logfile: "== Number of threads available to mris_sphere for OpenMP = 2 == scaling brain by 0.287... MRISunfold() max_passes = 1 --- tol=5.0e-01, sigma=0.0, host=mgt-i, nav=1024, nbrs=2, l_area=1.000, l_dist=1.000 using quadratic fit line minimization complete_dist_mat 0 rms 0 smooth_averages 0 remove_neg 0 ico_order 0 which_surface 0 target_radius 0.00 nfields 0 scale 1.00 desired_rms_height -1.00 momentum 0.90 nbhd_size 7 max_nbrs 8 niterations 25 nsurfaces 0 SURFACES 3 flags 0 (0) use curv 0 no sulc 0 no rigid align 0 mris->nsize 2 mris->hemisphere 0 randomSeed 1234 mrisRemoveNegativeArea() pass 1: epoch 1 of 3 starting distance error %21.21 pass 1: epoch 2 of 3 starting distance error %20.95 unfolding complete - removing small folds... starting distance error %20.57 removing remaining folds... final distance error %20.59 MRISunfold() return, current seed 1234" Does anyone have any ideas why this might be happening? Maybe this is a computationally intensive step? If I run a few recon-alls concurrently it almost always crashes at this step. The command I run is: recon-all -all -openmp 2 -qcache -nuintensitycor-3T -s SUBJECT_ID -sd /mnt/freesurfer_subjs/ Thank you so much for your help! Best, Michael ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
