Re: [slurm-users] Automatically setting OMP_NUM_THREADS=SLURM_CPUS_PER_TASK?

[Bjørn-Helge Mevik]

We have the following code in our TaskProlog:

## Set $OMP_NUM_THREADS unless it was set when calling sbatch or in the job 
script:
if [[ -z $OMP_NUM_THREADS ]]; then
if [[ -n $SLURM_CPUS_PER_TASK ]]; then
echo export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
else
echo export OMP_NUM_THREADS=1
fi
fi

It sets OMP_NUM_THREADS to SLURM_CPUS_PER_TASK (or 1 if --cpus-per-task
is not specified), unless already set.  That way, users can override it
in their job script if they wish.  It seems to work fine.

-- 
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo


signature.asc
Description: PGP signature

Re: [slurm-users] Automatically setting OMP_NUM_THREADS=SLURM_CPUS_PER_TASK?

[Bill Barth]

We do the same at TACC in our base module (which happens to be called “TACC”), 
and then we document it.

Best,
Bill.

-- 
Bill Barth, Ph.D., Director, HPC
bba...@tacc.utexas.edu|   Phone: (512) 232-7069
Office: ROC 1.435|   Fax:   (512) 475-9445
 
 

On 3/6/18, 5:13 PM, "slurm-users on behalf of Ryan Novosielski" 
 wrote:

Thanks, Martin — I almost mentioned Utah in my original e-mail as I turned 
up your support page in a search.

It is good to know definitively that MKL honors that variable — would be 
preferable to having to know about various different ones.

> On Mar 6, 2018, at 6:07 PM, Martin Cuma  wrote:
> 
> Ryan,
> 
> we set OMP_NUM_THREADS=1 in the R and Python modules (MKL will honor 
that), and instruct those users that want to run multi-threaded to set 
OMP_NUM_THREADS themselves after loading the module - and make sure they don't 
oversubscribe the node.
> 
> In our experience majority of R and Python users run independent serial 
calculations so the default OMP_NUM_THREADS=1 is reasonable.

--

|| \\UTGERS, 
|---*O*---
||_// the State  | Ryan Novosielski - novos...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\of NJ  | Office of Advanced Research Computing - MSB C630, 
Newark
 `'

Re: [slurm-users] Automatically setting OMP_NUM_THREADS=SLURM_CPUS_PER_TASK?

[Ryan Novosielski]

Thanks again! I’d seen the second one but not the first one.

> On Mar 6, 2018, at 6:28 PM, Martin Cuma  wrote:
> 
> MKL is trying to be flexible as it has different potential levels of 
> parallelism inside. Having MKL_ and OMP_NUM_THREADS can be beneficial in 
> programs where you may want to use your own OpenMP but restrict MKL's or vice 
> versa.
> 
> A good article on the different options that MKL provides is here:
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.diracprogram.org%2Fdoc%2Frelease-12%2Finstallation%2Fmkl.html=02%7C01%7Cnovosirj%40rutgers.edu%7C1cd606b70e004f334bc408d583ba29be%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C1%7C636559757962555771=5D%2B3OJmw7pbNVYO45vF%2F%2B7Nvx29ewJnMH2thf4O2slo%3D=0
> 
> Official Intel guide is here:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsoftware.intel.com%2Fen-us%2Farticles%2Frecommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications=02%7C01%7Cnovosirj%40rutgers.edu%7C1cd606b70e004f334bc408d583ba29be%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C1%7C636559757962555771=FLYu9kXwIq9OW2%2BwHNNDEwXXy1UUV%2F0C35TmTH5HJEU%3D=0

--

|| \\UTGERS, |---*O*---
||_// the State  | Ryan Novosielski - novos...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\of NJ  | Office of Advanced Research Computing - MSB C630, Newark
 `'



signature.asc
Description: Message signed with OpenPGP

Re: [slurm-users] Automatically setting OMP_NUM_THREADS=SLURM_CPUS_PER_TASK?

[Martin Cuma]

MKL is trying to be flexible as it has different potential levels of 
parallelism inside. Having MKL_ and OMP_NUM_THREADS can be beneficial in 
programs where you may want to use your own OpenMP but restrict MKL's or 
vice versa.


A good article on the different options that MKL provides is here:
http://www.diracprogram.org/doc/release-12/installation/mkl.html

Official Intel guide is here:
https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications

MC
--
Martin Cuma
Center for High Performance Computing
Department of Geology and Geophysics
University of Utah

Re: [slurm-users] Automatically setting OMP_NUM_THREADS=SLURM_CPUS_PER_TASK?

[Ryan Novosielski]

Thanks, Martin — I almost mentioned Utah in my original e-mail as I turned up 
your support page in a search.

It is good to know definitively that MKL honors that variable — would be 
preferable to having to know about various different ones.

> On Mar 6, 2018, at 6:07 PM, Martin Cuma  wrote:
> 
> Ryan,
> 
> we set OMP_NUM_THREADS=1 in the R and Python modules (MKL will honor that), 
> and instruct those users that want to run multi-threaded to set 
> OMP_NUM_THREADS themselves after loading the module - and make sure they 
> don't oversubscribe the node.
> 
> In our experience majority of R and Python users run independent serial 
> calculations so the default OMP_NUM_THREADS=1 is reasonable.

--

|| \\UTGERS, |---*O*---
||_// the State  | Ryan Novosielski - novos...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\of NJ  | Office of Advanced Research Computing - MSB C630, Newark
 `'



signature.asc
Description: Message signed with OpenPGP

Re: [slurm-users] Automatically setting OMP_NUM_THREADS=SLURM_CPUS_PER_TASK?

[Martin Cuma]

Ryan,

we set OMP_NUM_THREADS=1 in the R and Python modules (MKL will honor 
that), and instruct those users that want to run multi-threaded to set 
OMP_NUM_THREADS themselves after loading the module - and make sure they 
don't oversubscribe the node.


In our experience majority of R and Python users run independent serial 
calculations so the default OMP_NUM_THREADS=1 is reasonable.


MC
--
Martin Cuma
Center for High Performance Computing
Department of Geology and Geophysics
University of Utah