Probably because of this: http://slurm.schedmd.com/faq.html#rlimit
The slurmd daemons may have also been started manually by someone with lower ulimits.
Quoting [email protected]:
Good Afternoon All,We ran into an interesting message on our compute nodes today from the SLURM daemon:Can't propagate RLIMIT_NOFILE of <some number> from submit host: Operation not permittedThe limits were set on the compute nodes properly utilizing /etc/security/limits.conf, but those limits were not respected by the SLURM daemon. The solution was to add a ulimit command to the slurm start script on the compute node.Curiosity is peaked as to why the SLURM daemon doesn't obtain the correct values in from the environment. Does anyone know?Kelly
-- Morris "Moe" Jette CTO, SchedMD LLC Commercial Slurm Development and Support
