Re: [slurm-users] ulimit in sbatch script

[Mahmood Naderan]

Great. Thank you very much. It passed the problematic point.



On Tue, Apr 17, 2018, 19:24 Ole Holm Nielsen 
wrote:

> On 04/17/2018 04:38 PM, Mahmood Naderan wrote:
> > That parameter is used in slurm.conf. Should I modify that only on the
> > head node? Or all nodes? Then should I restart slurm processes?
>
> Yes, definitely!  I collected the detailed instructions here:
>
> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#reconfiguration-of-slurm-conf
>
> /Ole
>
>

Re: [slurm-users] ulimit in sbatch script

[Ole Holm Nielsen]

On 04/17/2018 04:38 PM, Mahmood Naderan wrote:

That parameter is used in slurm.conf. Should I modify that only on the
head node? Or all nodes? Then should I restart slurm processes?


Yes, definitely!  I collected the detailed instructions here:
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#reconfiguration-of-slurm-conf

/Ole

Re: [slurm-users] ulimit in sbatch script

[Mahmood Naderan]

That parameter is used in slurm.conf. Should I modify that only on the
head node? Or all nodes? Then should I restart slurm processes?

Regards,
Mahmood




On Tue, Apr 17, 2018 at 4:18 PM, Chris Samuel  wrote:
> On Tuesday, 17 April 2018 7:23:40 PM AEST Mahmood Naderan wrote:
>
>> [hamid@rocks7 case1_source2]$  scontrol show config | fgrep VSizeFactor
>> VSizeFactor = 110 percent
>
> Great, I think that's the cause of the limit you are seeing..
>
>VSizeFactor
>   Memory  specifications in job requests apply to real memory size
>   (also known as resident set size). It  is  possible  to  enforce
>   virtual  memory  limits  for both jobs and job steps by limiting
>   their virtual memory to some percentage  of  their  real  memory
>   allocation. The VSizeFactor parameter specifies the job's or job
>   step's virtual memory limit as a percentage of its  real  memory
>   limit.  For  example,  if a job's real memory limit is 500MB and
>   VSizeFactor is set to 101 then the job will  be  killed  if  its
>   real  memory  exceeds  500MB or its virtual memory exceeds 505MB
>   (101 percent of the real memory limit).  The default value is 0,
>   which  disables enforcement of virtual memory limits.  The value
>   may not exceed 65533 percent.
>
> Setting it to 0 should make that limit go away.
>
> All the best,
> Chris
> --
>  Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC
>
>

Re: [slurm-users] ulimit in sbatch script

[Chris Samuel]

On Tuesday, 17 April 2018 5:08:09 PM AEST Mahmood Naderan wrote:

> So, UsePAM has not been set. So, slurm shouldn't limit anything. Is
> that correct? however, I see that slurm limits the virtual memory size

What does this say?

scontrol show config | fgrep VSizeFactor


-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: [slurm-users] ulimit in sbatch script

[Mahmood Naderan]

Hi Bill,
Sorry for the late reply. As I greped for pam_limits.so, I see

[root@rocks7 ~]# grep -r pam_limits.so /etc/pam.d/
/etc/pam.d/sudo:sessionrequired pam_limits.so
/etc/pam.d/runuser:session  requiredpam_limits.so
/etc/pam.d/sudo-i:sessionrequired pam_limits.so
/etc/pam.d/system-auth-ac:session required  pam_limits.so
/etc/pam.d/fingerprint-auth-ac:session required  pam_limits.so
/etc/pam.d/smartcard-auth-ac:session required  pam_limits.so
/etc/pam.d/password-auth-ac:session required  pam_limits.so
[root@rocks7 ~]# grep -r UsePAM /etc/slurm/
/etc/slurm/slurm.conf:#UsePAM=



So, UsePAM has not been set. So, slurm shouldn't limit anything. Is
that correct? however, I see that slurm limits the virtual memory size


[hamid@rocks7 case1_source2]$ cat slurm_script.sh
#!/bin/bash
#SBATCH --job-name=hvacSteadyFoam
#SBATCH --output=hvacSteadyFoam.log
#SBATCH --ntasks=32
#SBATCH --time=100:00:00
#SBATCH --mem=64000M
ulimit -a
mpirun hvacSteadyFoam -parallel

[hamid@rocks7 case1_source2]$ sbatch slurm_script.sh
Submitted batch job 55
[hamid@rocks7 case1_source2]$ ssh compute-0-3
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Last login: Sun Apr 15 23:11:15 2018 from rocks7.local
Rocks Compute Node
Rocks 7.0 (Manzanita)
Profile built 19:21 11-Apr-2018

Kickstarted 19:37 11-Apr-2018
[hamid@compute-0-3 ~]$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 256712
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 4096
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited
[hamid@compute-0-3 ~]$ exit
logout
Connection to compute-0-3 closed.
[hamid@rocks7 case1_source2]$ cat hvacSteadyFoam.log
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 256712
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) 65536000
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 4096
virtual memory  (kbytes, -v) 72089600
file locks  (-x) unlimited
[hamid@rocks7 case1_source2]$


Regards,
Mahmood




On Mon, Apr 16, 2018 at 12:02 AM, Bill Barth  wrote:
> Specifying --mem to Slurm only tells it to find a node that has that much, 
> not to enforce a limit as far as I know. That node has that much so it finds 
> it. You probably want to enable UsePAM and setup the pam.d slurm files and 
> /etc/security/limits.conf to keep users under the 64000MB physical memory 
> that the node has (minus some padding for the OS, etc.). IS UsePAM enabled in 
> your slurm.conf, maybe that’s doing it.
>
> Best,
> Bill.
>
> --
> Bill Barth, Ph.D., Director, HPC
> bba...@tacc.utexas.edu|   Phone: (512) 232-7069
> Office: ROC 1.435|   Fax:   (512) 475-9445
>

Re: [slurm-users] ulimit in sbatch script

[Bill Barth]

Specifying --mem to Slurm only tells it to find a node that has that much, not 
to enforce a limit as far as I know. That node has that much so it finds it. 
You probably want to enable UsePAM and setup the pam.d slurm files and 
/etc/security/limits.conf to keep users under the 64000MB physical memory that 
the node has (minus some padding for the OS, etc.). IS UsePAM enabled in your 
slurm.conf, maybe that’s doing it.

Best,
Bill.

-- 
Bill Barth, Ph.D., Director, HPC
bba...@tacc.utexas.edu|   Phone: (512) 232-7069
Office: ROC 1.435|   Fax:   (512) 475-9445
 
 

On 4/15/18, 2:28 PM, "slurm-users on behalf of Mahmood Naderan" 
 wrote:

Bill,
Thing is that both user and root see unlimited virtual memory when
they directly ssh to the node. However, when the job is submitted, the
user limits change. That means, slurm modifies something.

The script is

#SBATCH --job-name=hvacSteadyFoam
#SBATCH --output=hvacSteadyFoam.log
#SBATCH --ntasks=32
#SBATCH --time=100:00:00
#SBATCH --mem=64000M
ulimit -a
mpirun hvacSteadyFoam -parallel


The physical memory on the node is 64GB, therefore, I specified 64000M
for --mem. Is that correct? the only thing I am guessing is that --mem
also modifies virtual memory limit. Though I am not sure.


Regards,
Mahmood




On Sun, Apr 15, 2018 at 11:32 PM, Bill Barth  wrote:
> Mahmood, sorry to presume. I meant to address the root user and your ssh 
to the node in your example.
>
> At our site, we use UsePAM=1 in our slurm.conf, and our /etc/pam.d/slurm 
and slurm.pam files both contain pam_limits.so, so it could be that way for 
you, too. I.e. Slurm could be setting the limits for jobscripts for your users, 
but for root SSHes, where that’s being set by PAM through another config file. 
Also, root’s limits are potentially differently set by PAM (in 
/etc/security/limits.conf) or the kernel at boot time.
>
> Finally, users should be careful using ulimit in their job scripts b/c 
that can only change the limits for that shell script process and not across 
nodes. That jobscript appears to only apply to one node, but if they want 
different limits for jobs that span nodes, they may need to use other features 
of SLURM to get them across all  the nodes their job wants (cgroups, perhaps?).
>
> Best,
> Bill.
>
> --
> Bill Barth, Ph.D., Director, HPC
> bba...@tacc.utexas.edu|   Phone: (512) 232-7069
> Office: ROC 1.435|   Fax:   (512) 475-9445
>

Re: [slurm-users] ulimit in sbatch script

[Mahmood Naderan]

Bill,
Thing is that both user and root see unlimited virtual memory when
they directly ssh to the node. However, when the job is submitted, the
user limits change. That means, slurm modifies something.

The script is

#SBATCH --job-name=hvacSteadyFoam
#SBATCH --output=hvacSteadyFoam.log
#SBATCH --ntasks=32
#SBATCH --time=100:00:00
#SBATCH --mem=64000M
ulimit -a
mpirun hvacSteadyFoam -parallel


The physical memory on the node is 64GB, therefore, I specified 64000M
for --mem. Is that correct? the only thing I am guessing is that --mem
also modifies virtual memory limit. Though I am not sure.


Regards,
Mahmood




On Sun, Apr 15, 2018 at 11:32 PM, Bill Barth  wrote:
> Mahmood, sorry to presume. I meant to address the root user and your ssh to 
> the node in your example.
>
> At our site, we use UsePAM=1 in our slurm.conf, and our /etc/pam.d/slurm and 
> slurm.pam files both contain pam_limits.so, so it could be that way for you, 
> too. I.e. Slurm could be setting the limits for jobscripts for your users, 
> but for root SSHes, where that’s being set by PAM through another config 
> file. Also, root’s limits are potentially differently set by PAM (in 
> /etc/security/limits.conf) or the kernel at boot time.
>
> Finally, users should be careful using ulimit in their job scripts b/c that 
> can only change the limits for that shell script process and not across 
> nodes. That jobscript appears to only apply to one node, but if they want 
> different limits for jobs that span nodes, they may need to use other 
> features of SLURM to get them across all  the nodes their job wants (cgroups, 
> perhaps?).
>
> Best,
> Bill.
>
> --
> Bill Barth, Ph.D., Director, HPC
> bba...@tacc.utexas.edu|   Phone: (512) 232-7069
> Office: ROC 1.435|   Fax:   (512) 475-9445
>

Re: [slurm-users] ulimit in sbatch script

[Bill Barth]

Mahmood, sorry to presume. I meant to address the root user and your ssh to the 
node in your example. 

At our site, we use UsePAM=1 in our slurm.conf, and our /etc/pam.d/slurm and 
slurm.pam files both contain pam_limits.so, so it could be that way for you, 
too. I.e. Slurm could be setting the limits for jobscripts for your users, but 
for root SSHes, where that’s being set by PAM through another config file. 
Also, root’s limits are potentially differently set by PAM (in 
/etc/security/limits.conf) or the kernel at boot time. 

Finally, users should be careful using ulimit in their job scripts b/c that can 
only change the limits for that shell script process and not across nodes. That 
jobscript appears to only apply to one node, but if they want different limits 
for jobs that span nodes, they may need to use other features of SLURM to get 
them across all  the nodes their job wants (cgroups, perhaps?).

Best,
Bill.

-- 
Bill Barth, Ph.D., Director, HPC
bba...@tacc.utexas.edu|   Phone: (512) 232-7069
Office: ROC 1.435|   Fax:   (512) 475-9445
 
 

On 4/15/18, 1:41 PM, "slurm-users on behalf of Mahmood Naderan" 
 wrote:

Excuse me... I think the problem is not pam.d.
How do you interpret the following output?


[hamid@rocks7 case1_source2]$ sbatch slurm_script.sh
Submitted batch job 53
[hamid@rocks7 case1_source2]$ tail -f hvacSteadyFoam.log
max memory size (kbytes, -m) 65536000
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 4096
virtual memory  (kbytes, -v) 72089600
file locks  (-x) unlimited
^C
[hamid@rocks7 case1_source2]$ squeue
 JOBID PARTITION NAME USER ST   TIME  NODES
NODELIST(REASON)
53   CLUSTER hvacSteahamid  R   0:27  1 
compute-0-3
[hamid@rocks7 case1_source2]$ ssh compute-0-3
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Last login: Sun Apr 15 23:03:29 2018 from rocks7.local
Rocks Compute Node
Rocks 7.0 (Manzanita)
Profile built 19:21 11-Apr-2018

Kickstarted 19:37 11-Apr-2018
[hamid@compute-0-3 ~]$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 256712
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 4096
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited
[hamid@compute-0-3 ~]$



As you can see, the log file where I put  "ulimit -a" before the main
command says limited virtual memory. However, when I login to the
node, it says unlimited!

Regards,
Mahmood




On Sun, Apr 15, 2018 at 11:01 PM, Bill Barth  wrote:
> Are you using pam_limits.so in any of your /etc/pam.d/ configuration 
files? That would be enforcing /etc/security/limits.conf for all users which 
are usually unlimited for root. Root’s almost always allowed to do stuff bad 
enough to crash the machine or run it out of resources. If the /etc/pam.d/sshd 
file has pam_limits.so in it, that’s probably where the unlimited setting for 
root is coming from.
>
> Best,
> Bill.

Re: [slurm-users] ulimit in sbatch script

[Mahmood Naderan]

Excuse me... I think the problem is not pam.d.
How do you interpret the following output?


[hamid@rocks7 case1_source2]$ sbatch slurm_script.sh
Submitted batch job 53
[hamid@rocks7 case1_source2]$ tail -f hvacSteadyFoam.log
max memory size (kbytes, -m) 65536000
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 4096
virtual memory  (kbytes, -v) 72089600
file locks  (-x) unlimited
^C
[hamid@rocks7 case1_source2]$ squeue
 JOBID PARTITION NAME USER ST   TIME  NODES
NODELIST(REASON)
53   CLUSTER hvacSteahamid  R   0:27  1 compute-0-3
[hamid@rocks7 case1_source2]$ ssh compute-0-3
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Last login: Sun Apr 15 23:03:29 2018 from rocks7.local
Rocks Compute Node
Rocks 7.0 (Manzanita)
Profile built 19:21 11-Apr-2018

Kickstarted 19:37 11-Apr-2018
[hamid@compute-0-3 ~]$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 256712
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 4096
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited
[hamid@compute-0-3 ~]$



As you can see, the log file where I put  "ulimit -a" before the main
command says limited virtual memory. However, when I login to the
node, it says unlimited!

Regards,
Mahmood




On Sun, Apr 15, 2018 at 11:01 PM, Bill Barth  wrote:
> Are you using pam_limits.so in any of your /etc/pam.d/ configuration files? 
> That would be enforcing /etc/security/limits.conf for all users which are 
> usually unlimited for root. Root’s almost always allowed to do stuff bad 
> enough to crash the machine or run it out of resources. If the 
> /etc/pam.d/sshd file has pam_limits.so in it, that’s probably where the 
> unlimited setting for root is coming from.
>
> Best,
> Bill.

Re: [slurm-users] ulimit in sbatch script

[Mahmood Naderan]

BTW, the memory size of the node is 64GB.
Regards,
Mahmood




On Sun, Apr 15, 2018 at 10:56 PM, Mahmood Naderan  wrote:
> I actually have disabled the swap partition (!) since the system goes
> really bad and based on my experience I have to enter the room and
> reset the affected machine (!). Otherwise I have to wait for long
> times to see it get back to normal.
>
> When I ssh to the node with root user, the ulimit -a says unlimited
> virtual memory. So, it seems that the root have unlimited value while
> users have limited value.
>
> Regards,
> Mahmood
>
>
>
>
> On Sun, Apr 15, 2018 at 10:26 PM, Ole Holm Nielsen
>  wrote:
>> Hi Mahmood,
>>
>> It seems your compute node is configured with this limit:
>>
>> virtual memory  (kbytes, -v) 72089600
>>
>> So when the batch job tries to set a higher limit (ulimit -v 82089600) than
>> permitted by the system (72089600), this must surely get rejected, as you
>> have discovered!
>>
>> You may want to reconfigure your compute nodes' limits, for example by
>> setting the virtual memory limit to "unlimited" in your configuration. If
>> the nodes has a very small RAM memory + swap space size, you might encounter
>> Out Of Memory errors...
>>
>> /Ole

Re: [slurm-users] ulimit in sbatch script

[Bill Barth]

Are you using pam_limits.so in any of your /etc/pam.d/ configuration files? 
That would be enforcing /etc/security/limits.conf for all users which are 
usually unlimited for root. Root’s almost always allowed to do stuff bad enough 
to crash the machine or run it out of resources. If the /etc/pam.d/sshd file 
has pam_limits.so in it, that’s probably where the unlimited setting for root 
is coming from.

Best,
Bill.

-- 
Bill Barth, Ph.D., Director, HPC
bba...@tacc.utexas.edu|   Phone: (512) 232-7069
Office: ROC 1.435|   Fax:   (512) 475-9445
 
 

On 4/15/18, 1:26 PM, "slurm-users on behalf of Mahmood Naderan" 
 wrote:

I actually have disabled the swap partition (!) since the system goes
really bad and based on my experience I have to enter the room and
reset the affected machine (!). Otherwise I have to wait for long
times to see it get back to normal.

When I ssh to the node with root user, the ulimit -a says unlimited
virtual memory. So, it seems that the root have unlimited value while
users have limited value.

Regards,
Mahmood




On Sun, Apr 15, 2018 at 10:26 PM, Ole Holm Nielsen
 wrote:
> Hi Mahmood,
>
> It seems your compute node is configured with this limit:
>
> virtual memory  (kbytes, -v) 72089600
>
> So when the batch job tries to set a higher limit (ulimit -v 82089600) 
than
> permitted by the system (72089600), this must surely get rejected, as you
> have discovered!
>
> You may want to reconfigure your compute nodes' limits, for example by
> setting the virtual memory limit to "unlimited" in your configuration. If
> the nodes has a very small RAM memory + swap space size, you might 
encounter
> Out Of Memory errors...
>
> /Ole

[slurm-users] ulimit in sbatch script

[Mahmood Naderan]

Hi,
The user can run "ulimit -v VALUE" on the frontend. However, when I
put that command in a slurm script, it says that operation is not
permitted by the user!

[hamid@rocks7 case1_source2]$ ulimit -v 82089600
[hamid@rocks7 case1_source2]$ cat slurm_script.sh
#!/bin/bash
#SBATCH --job-name=hvacSteadyFoam
#SBATCH --output=hvac.log
#SBATCH --ntasks=32
#SBATCH --time=100:00:00
#SBATCH --mem=64000M
ulimit -v 82089600
ulimit -a
mpirun hvacSteadyFoam -parallel
[hamid@rocks7 case1_source2]$ sbatch slurm_script.sh
Submitted batch job 50
[hamid@rocks7 case1_source2]$ cat hvacSteadyFoam.log
/var/spool/slurmd/job00050/slurm_script: line 11: ulimit: virtual
memory: cannot modify limit: Operation not permitted
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 256712
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) 65536000
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 4096
virtual memory  (kbytes, -v) 72089600
file locks  (-x) unlimited
[hamid@rocks7 case1_source2]$




Any idea?


Regards,
Mahmood