I've found changes to limits.conf don't get applied to the slurm daemons when started via init.d scripts as the limits.conf is only used for PAM sessions. I ran into such an issue when debugging why VASP was failing [1].
We currently set all limits that need to be applied to jobs in /etc/sysconfig/slurm [2] which is sourced by /etc/init.d/slurm, at least is the case for RPM installs of SLURM. - Trey [1]: https://groups.google.com/d/msg/slurm-devel/Ibm2-fP-wUA/H34eKKLDzN0J [2]: ulimit -l unlimited ulimit -n 8192 ulimit -s unlimited CONFDIR="/etc/slurm" SLURMCTLD_OPTIONS="-f /etc/slurm/slurm.conf" SLURMD_OPTIONS="-f /etc/slurm/slurm.conf -M" export SLURM_CONF="/etc/slurm/slurm.conf" ============================= Trey Dockendorf Systems Analyst I Texas A&M University Academy for Advanced Telecommunications and Learning Technologies Phone: (979)458-2396 Email: [email protected] Jabber: [email protected] On Tue, Aug 11, 2015 at 1:36 PM, Blosch, Edwin L <[email protected]> wrote: > With PBS Pro, it was customary and necessary to modify /etc/init.d/pbs_mom > to add ulimit commands shown in (a) below, but I haven’t seen a need for > that with SLURM yet. Recently there are some users at my organization who > are focusing on process-limit issues as potential sources of application > failures. > > > > I was wondering if anyone has found need to be concerned with change (a) > below, if change (b) has already been taken care of? In the user reports, > I think they are using OpenMPI 1.6.5 and did not include the –without-slurm > configuration, so I’m not certain of the launching mechanism for the > children processes, but I think it may leverage SLURM as opposed to using > ssh. > > > > (a) adding the following lines to /etc/init.d/slurm > > > > ulimit -l unlimited > > ulimit -s unlimited > > > > (b) adding these lines to /etc/security/limits.conf > > > > * soft memlock unlimited > > * hard memlock unlimited > > * soft stack unlimited > > * hard stack unlimited > > * soft nofile 1000000 > > * hard nofile 1000000 > > > > Thanks for your help, > > > > Ed >
