The fMRIVolume pipeline also takes advantage of SGE through fslsub, and we hacked it to use condor as well--that allows for significant time savings on our end
_________________________ gaurav patel gauravpa...@gmail.com www.neurofreak.net > On Apr 27, 2017, at 2:29 PM, Glasser, Matthew <glass...@wustl.edu> wrote: > > Hi Mike, > > The only thing that is parallelized in the minimal preprocessing pipelines > aside from specific FreeSurfer binaries is wb_command. wb_command > defaults to using as many cores as the system has. More advanced > pipelines that use matlab will adhere to matlab¹s parallelization > behaviors. > > Matt. > > On 4/27/17, 9:27 AM, "hcp-users-boun...@humanconnectome.org on behalf of > Harms, Michael" <hcp-users-boun...@humanconnectome.org on behalf of > mha...@wustl.edu> wrote: > >> >> Matt, Tim, >> What is parallelized in the fMRI pipelines? >> >> I would be good if we created a FAQ entry on the HCP Wiki detailing: >> (1) Which aspects of the pipelines are parallelized and the default number >> of cores used for each. >> (2) Which wb_commands are parallelized and the default number of cores >> used. >> (3) How to explicitly control the number of cores used in both the >> pipelines and wb_command. >> >> cheers, >> -MH >> >> -- >> Michael Harms, Ph.D. >> >> ----------------------------------------------------------- >> Conte Center for the Neuroscience of Mental Disorders >> Washington University School of Medicine >> Department of Psychiatry, Box 8134 >> 660 South Euclid Ave.Tel: 314-747-6173 >> St. Louis, MO 63110Email: mha...@wustl.edu >> >> >> >> >> On 4/27/17, 9:20 AM, "hcp-users-boun...@humanconnectome.org on behalf of >> Gaurav Patel" <hcp-users-boun...@humanconnectome.org on behalf of >> gauravpa...@gmail.com> wrote: >> >> For what its worth, on our workstations we run each subject's anatomy >> pipeline as a separate thread, which takes 8-12 hours, then use the >> parallelization to speed up the fMRI pipelines with either SGE/OGE or >> ht_condor, with a thread devoted to each bold run being processed >> >> __________________________ >> gaurav patel >> gauravpa...@gmail.com >> www.neurofreak.net >> >> >> >> >>> On Apr 27, 2017, at 9:44 AM, Harms, Michael wrote: >>> >>> >>> The number of windows you use to launch the processing is irrelevant. >>> But if you only have 8 cores on your machine, and only want to use 7 of >>> them, you can¹t run 50 subjects at once. >>> >>> In this case, where you have a bunch of subjects to process, I would run >>> 7 subjects at a time, while limiting each to just 1 core, by setting >>> whatever variable in the pipeline that is used to limit/control the >>> number of threads used for multi-thread capable binaries. >>> >>> cheers, >>> -MH >>> >>> -- >>> Michael Harms, Ph.D. >>> ----------------------------------------------------------- >>> Conte Center for the Neuroscience of Mental Disorders >>> Washington University School of Medicine >>> Department of Psychiatry, Box 8134 >>> 660 South Euclid Ave. Tel: 314-747-6173 >>> St. Louis, MO 63110 Email: mha...@wustl.edu >>> >>> From: <hcp-users-boun...@humanconnectome.org> on behalf of Vanessa K >>> <vanessakr1...@gmail.com> >>> Date: Thursday, April 27, 2017 at 4:30 AM >>> To: "hcp-users@humanconnectome.org" <hcp-users@humanconnectome.org> >>> Subject: [HCP-Users] Fwd: Using multiple cores while running Structural >>> Pipelines >>> >>> Alright, thanks! And just for clarification for dummies: if not >>> installing something like grid_engine, will running all 50 subjects in >>> one Terminal window or 10 subjects in each of five different windows be >>> more efficient? Or doesn't it make any difference? And I estimate around >>> 20 days for 50 subjects (for the structural pipelines), is it a normal >>> processing time? >>> >>> Thanks again! >>> >>> >>> 2017-04-26 16:20 GMT+02:00 Glasser, Matthew <glass...@wustl.edu>: >>>> I find it helps to install something like gridengine which interfaces >>>> with fsl_sub. You can then queue up the jobs and they will all run >>>> automatically. You can choose how many to run at a time. I believe you >>>> don¹t need to specify an openmp flag unless you want fewer than all of >>>> the cores. >>>> >>>> Peace, >>>> >>>> Matt. >>>> >>>> From: Vanessa K <vanessakr1...@gmail.com> >>>> Date: Wednesday, April 26, 2017 at 4:39 AM >>>> To: Matt Glasser <glass...@wustl.edu> >>>> Cc: "hcp-users@humanconnectome.org" <hcp-users@humanconnectome.org> >>>> Subject: Re: [HCP-Users] Using multiple cores while running Structural >>>> Pipelines >>>> >>>> Thanks for your quick answer! >>>> Just to make sure: it means I shouldn't specify -openmp 5 in the >>>> recon-all line and that the multi-threading happens automatically? >>>> So given that I have around 50 subjects who I need to run this on, >>>> would the best way to accelerate the performance be to just open several >>>> terminal windows (e.g. 5) and run ten subjects in each? Or would it make >>>> no difference from running all 50 in one window? I considered looking >>>> into GNU parallel, but not sure it would make sense. What would your >>>> advice for such a number of subjects be? >>>> >>>> Thanks so much for your help. >>>> >>>> V >>>> >>>> 2017-04-25 16:55 GMT+02:00 Glasser, Matthew <glass...@wustl.edu>: >>>>> PreFreeSurfer uses mostly single threaded tools. Some of FreeSurfer >>>>> is multi-threaded (and importantly one binary has some issues with >>>>> oversubscribed cores leading to much longer execution times). >>>>> PostFreeSurfer is largely multi-threaded. In all cases, the pipelines >>>>> use multiple threads by default if they can. >>>>> >>>>> Peace, >>>>> >>>>> Matt. >>>>> >>>>> From: <hcp-users-boun...@humanconnectome.org> on behalf of Vanessa K >>>>> <vanessakr1...@gmail.com> >>>>> Date: Tuesday, April 25, 2017 at 7:36 AM >>>>> To: "hcp-users@humanconnectome.org" <hcp-users@humanconnectome.org> >>>>> Subject: [HCP-Users] Using multiple cores while running Structural >>>>> Pipelines >>>>> >>>>> Dear users, >>>>> >>>>> it might be a silly question but I hope you can help. I have a >>>>> computer with 8 cores and I would like to use 7 of them while running >>>>> pipelines that take a lot of time (such as FreeSurfer). Is there any >>>>> way to specify the core number when running the script? Or are there >>>>> any tricks to speed up the processing time? >>>>> >>>>> Thanks a lot! >>>>> >>>>> V >>>>> >>>>> _______________________________________________ >>>>> HCP-Users mailing list >>>>> HCP-Users@humanconnectome.org >>>>> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >>>>> >>>>> >>>>> The materials in this message are private and may contain Protected >>>>> Healthcare Information or other information of a sensitive nature. If >>>>> you are not the intended recipient, be advised that any unauthorized >>>>> use, disclosure, copying or the taking of any action in reliance on the >>>>> contents of this information is strictly prohibited. If you have >>>>> received this email in error, please immediately notify the sender via >>>>> telephone or return mail. >>>> >>>> >>>> >>>> The materials in this message are private and may contain Protected >>>> Healthcare Information or other information of a sensitive nature. If >>>> you are not the intended recipient, be advised that any unauthorized >>>> use, disclosure, copying or the taking of any action in reliance on the >>>> contents of this information is strictly prohibited. If you have >>>> received this email in error, please immediately notify the sender via >>>> telephone or return mail. >>> >>> >>> _______________________________________________ >>> HCP-Users mailing list >>> HCP-Users@humanconnectome.org >>> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >>> >>> >>> The materials in this message are private and may contain Protected >>> Healthcare Information or other information of a sensitive nature. If you >>> are not the intended recipient, be advised that any unauthorized use, >>> disclosure, copying or the taking of any action in reliance on the >>> contents of this information is strictly prohibited. If you have received >>> this email in error, please immediately notify the sender via telephone >>> or return mail. >>> _______________________________________________ >>> HCP-Users mailing list >>> HCP-Users@humanconnectome.org >>> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >> >> >> _______________________________________________ >> HCP-Users mailing list >> HCP-Users@humanconnectome.org >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >> >> >> ________________________________ >> The materials in this message are private and may contain Protected >> Healthcare Information or other information of a sensitive nature. If you >> are not the intended recipient, be advised that any unauthorized use, >> disclosure, copying or the taking of any action in reliance on the >> contents of this information is strictly prohibited. If you have received >> this email in error, please immediately notify the sender via telephone >> or return mail. >> >> _______________________________________________ >> HCP-Users mailing list >> HCP-Users@humanconnectome.org >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users > > > ________________________________ > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you are > not the intended recipient, be advised that any unauthorized use, disclosure, > copying or the taking of any action in reliance on the contents of this > information is strictly prohibited. If you have received this email in error, > please immediately notify the sender via telephone or return mail. _______________________________________________ HCP-Users mailing list HCP-Users@humanconnectome.org http://lists.humanconnectome.org/mailman/listinfo/hcp-users
