Re: [Freesurfer] Open MP parallel processing scaling factor?

[Z K]

Andrew,

I basically agree with everything Satra is saying. Just as an added 
piece if info, Ive included a chart which did a systematic study of 
recon-all processing times with different number of cpus in parallel.

These results are a bit old (and might be for low-res data) but the 
general idea is that there is a point of diminishing returns after using 
4 processors in parallel. The gain from using more processors may be 
worth it for you.

There are other tricks to getting faster recons (like processing left 
and right hemishperes in parallel) but that is a bit more tricky and 
requires a custom script. I can provide additional information if you 
think you would like to pursue that avenue

-Zeke




On 08/20/2014 11:21 AM, Satrajit Ghosh wrote:
> hi andrew,
>
> i haven't done many systematic comparisons but there are a few practical
> considerations to take into account. in recon-all, i believe a few steps
> are affected by the openmp option and that creates resource
> underutilization. processors run idle when those steps are not being run.
>
> in my ad hoc analysis i have found i can process a single subject in
> about 5 hours using openmp 8, but that holds up 8 processors for that
> subject. the same subject can be processed in about 12 hours one 1
> processor. say i have 16 processors, i can process 16 subjects in say 12
> hours using 1 processor per recon. however, using 8 per recon would take
> about 40 hours, 2 subjects every 5 hours.
>
> so on our cluster, we tend to process with openmp 1 or 2 depending on
> the average load on the cluster. this is also dependent on your
> hardware, amount of memory, cluster scheduler, etc.,.
>
> if i really need speed on an individual case i go with -openmp 8.
>
> cheers,
>
> satra
>
> On Wed, Aug 20, 2014 at 11:04 AM, O'Shea,Andrew  > wrote:
>
> Hello all,
> Traditionally I have only processed FS data using a single core per
> person, but processing many people at once. Now we have caught up
> with the backlog of scans, we have a continuos trickle of scans
> coming in 1 by 1. I was wondering if anyone has tested how the
> speed-up of open-mp varies with number of cores used simultaneously.
> For example how much faster is using 100 cores versus 10? I am
> trying to find a sweet spot of resource usage and speed. Thanks!
> -Andrew
>
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu 
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
>
> The information in this e-mail is intended only for the person to
> whom it is
> addressed. If you believe this e-mail was sent to you in error and
> the e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to
> you in error
> but does not contain patient information, please contact the sender
> and properly
> dispose of the e-mail.
>
>
>
>
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

Re: [Freesurfer] Open MP parallel processing scaling factor?

[Satrajit Ghosh]

hi andrew,

i haven't done many systematic comparisons but there are a few practical
considerations to take into account. in recon-all, i believe a few steps
are affected by the openmp option and that creates resource
underutilization. processors run idle when those steps are not being run.

in my ad hoc analysis i have found i can process a single subject in about
5 hours using openmp 8, but that holds up 8 processors for that subject.
the same subject can be processed in about 12 hours one 1 processor. say i
have 16 processors, i can process 16 subjects in say 12 hours using 1
processor per recon. however, using 8 per recon would take about 40 hours,
2 subjects every 5 hours.

so on our cluster, we tend to process with openmp 1 or 2 depending on the
average load on the cluster. this is also dependent on your hardware,
amount of memory, cluster scheduler, etc.,.

if i really need speed on an individual case i go with -openmp 8.

cheers,

satra

On Wed, Aug 20, 2014 at 11:04 AM, O'Shea,Andrew  wrote:

>  Hello all,
> Traditionally I have only processed FS data using a single core per
> person, but processing many people at once. Now we have caught up with the
> backlog of scans, we have a continuos trickle of scans coming in 1 by 1. I
> was wondering if anyone has tested how the speed-up of open-mp varies with
> number of cores used simultaneously. For example how much faster is using
> 100 cores versus 10? I am trying to find a sweet spot of resource usage and
> speed. Thanks!
> -Andrew
>
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
>
> The information in this e-mail is intended only for the person to whom it
> is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
>
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

[Freesurfer] Open MP parallel processing scaling factor?

[O'Shea,Andrew]

Hello all,
Traditionally I have only processed FS data using a single core per person, but 
processing many people at once. Now we have caught up with the backlog of 
scans, we have a continuos trickle of scans coming in 1 by 1. I was wondering 
if anyone has tested how the speed-up of open-mp varies with number of cores 
used simultaneously. For example how much faster is using 100 cores versus 10? 
I am trying to find a sweet spot of resource usage and speed. Thanks!
-Andrew
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.