BTW, this should not be a problem in the next version On 10/07/2018 08:29 AM, Jasper van den Bosch wrote: > > > Michael, I can't thank you enough. > > Knowing that it writes to /tmp made all the difference. > In my case I mounted an external directory on /tmp instead of the > sessiondir approach, but it now runs fine. > Jasper > > On Fri, 5 Oct 2018 at 23:47, Michael Krause <kra...@mpib-berlin.mpg.de > <mailto:kra...@mpib-berlin.mpg.de>> wrote: > > Dear *, > > we also run Freesurfer on a cluster and are starting to use > singularity, so > naturally I'm curious about this issue. > > I was able to reproduce the error with a clean singularity run (-c > -e). > Then, I rebuilt a container to start nu_correct with "strace -f -e > open,write" and I believe I found the problem: > > [pid 35994] > open("/subs/testsub/mri/tmp.mri_nu_correct.mni.35410/nu1.imp", > O_RDONLY) = 3 > [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_EXCL, 0600) = 5 > [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_TRUNC, 0666) = 5 > [pid 35994] write(5, "\0\0\0\0\0\0\0\0", 8) = 8 > [pid 35994] write(5, > "CDF\1\0\0\0\0\0\0\0\n\0\0\0\3\0\0\0\6xspace\0\0\0\0\1\0"..., > 8192) = 8192 > [pid 35994] write(5, > > "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., > 8192) = 8192 > [...] > [pid 35994] write(5, > > "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., > 8192) = -1 ENOSPC (No space left on device) > [pid 35994] write(2, "ncendef: ", 9ncendef: ) = 9 > [pid 35994] write(2, "ncid 5", 6ncid 5) = 6 > [pid 35994] write(2, ": No space left on device", 25: No space left on > device) = 25 > > > So there are two issues here I think: > > 1) nu_correct is saving things to /tmp when it could just use ./tmp > 2) the singularity default for "sessiondir max size" is 16MB, > which includes > /tmp, is too small for the minc-tmp files > > I changed the value from 16 to 1024 and it passed the nu_correct step. > > cheers, > Michael > > On 10/5/18 5:57 PM, Greve, Douglas N.,Ph.D. wrote: > > Not that I know of, but this is not a native FS script, it comes > from the > > MNI, so it could be doing something we don't understand. Can you > do an > > experiment for us? Can you run it outside of the container using > reprozip > > (or something equivalent) to see all the files it touches? > > > > On 10/5/18 5:31 AM, Jasper van den Bosch wrote: > >> > >> External Email - Use Caution > >> > >> We are trying to run recon_all inside an fmriprep singularity > container, > >> but are running into an issue with nu_correct. It says it > cannot write to > >> disk, however there is plenty of space on the tmpdir and output > dir. Does > >> this script try to write to any other locations? Because this > would fail > >> inside the container, where each writable location has to be > 'mounted'. > >> > >> Error message: > >> > >> ncendef: ncid 5: No space left on device > >> Error outputting volume: possibly disk full? > >> nu_evaluate: crashed while running evaluate_field (termination > status=139) > >> nu_correct: crashed while running nu_evaluate (termination > status=65280) > >> ERROR: nu_correct > >> > >> Command ran: > >> > >> nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc > >> ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir > >> ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50 > >> [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/] > >> [2018-10-04 14:59:29] running: > >> /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log > -sharpen 0.15 > >> 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask > -nonotify -b_spline > >> 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir > >> ./tmp.mri_nu_correct.mni.1121/0/ > ./tmp.mri_nu_correct.mni.1121/nu0.mnc > >> ./tmp.mri_nu_correct.mni.1121/nu1.imp > >> > >> > >> this issue at the fmriprep > >> repo: https://github.com/poldracklab/fmriprep/issues/1308 > >> possibly related: > https://github.com/freesurfer/freesurfer/issues/462 > >> > >> Many thanks for any advice, > >> Jasper van den Bosch > >> > >> > >> > >> _______________________________________________ > >> Freesurfer mailing list > >> Freesurfer@nmr.mgh.harvard.edu > <mailto:Freesurfer@nmr.mgh.harvard.edu> > >> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > > > > _______________________________________________ > > Freesurfer mailing list > > Freesurfer@nmr.mgh.harvard.edu > <mailto:Freesurfer@nmr.mgh.harvard.edu> > > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > > External Email - Use Caution > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
