Installed memory per node:
RAM 32 GB
SWAP 10 GB
#### slurm.conf ####
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
SelectTypeParameters=CR_Core_Memory
NodeName=... RealMemory=29000
####################
### cgroup.conf ####
AllowedRAMSpace=100
AllowedSwapSpace=30.0
ConstrainRAMSpace=YES
ConstrainSwapSpace=YES
MaxRAMPercent=100
MaxSwapPercent=100
MinRAMSpace=30
####################
This program just eats up the requested amount of memory:
### memoryHog.c ####
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define PAGE_SZ (1<<12)
int main(int argc, char **argv) {
int i;
int gb = atoi( (argv[1]) ); // memory to consume in GB
for (i = 0; i < ((unsigned long)gb<<30)/PAGE_SZ ; ++i) {
void *m = malloc(PAGE_SZ);
if (!m)
break;
memset(m, 0, 1);
}
printf("allocated %lu MB\n", ((unsigned long)i*PAGE_SZ)>>20);
sleep(10);
return 0;
}
####################
### TESTING ###
$> salloc --mem-per-cpu=9000
salloc: Granted job allocation 1503
$> srun memoryHog.x 8
allocated 8192 MB
$> srun memoryHog.x 9
allocated 9050 MB
slurmstepd: Exceeded step memory limit at some point. Step may have been
partially swapped out to disk.
### LOGS: /var/log/slurm/slurmd.log ###
[2014-04-15T18:58:26.212] [1503.0] task/cgroup: /slurm/uid_500/job_1503:
alloc=9000MB mem.limit=9000MB memsw.limit=11700MB
[2014-04-15T18:58:26.212] [1503.0] task/cgroup: /slurm/uid_500/job_1503/step_0:
alloc=9000MB mem.limit=9000MB memsw.limit=11700MB
[2014-04-15T18:58:39.961] [1503.0] done with job
..
..
[2014-04-15T18:58:45.916] [1503.1] task/cgroup: /slurm/uid_500/job_1503:
alloc=9000MB mem.limit=9000MB memsw.limit=11700MB
[2014-04-15T18:58:45.916] [1503.1] task/cgroup: /slurm/uid_500/job_1503/step_1:
alloc=9000MB mem.limit=9000MB memsw.limit=11700MB
[2014-04-15T18:59:01.087] [1503.1] Exceeded step memory limit at some point.
Step may have been partially swapped out to disk.
[2014-04-15T18:59:01.120] [1503.1] done with job
####################
Since slurm sets "memsw.limit=11700MB" I was expecting
the cgroup feature to start swapping out the exceeding
50 MB or so... they would actually fit in the swap area
and the job should not be killed...
What am I missing here?
Should the code itself be aware of the given "mem.limit=9000MB"?
Thanks for any explanation.
MG
