Hi,
i did some more tests in our testbed and i think this issue is caused by
MaxMemPerNode value.
In the testing environment we have Slurm version 14.11.8
With same configuration:
$ sbatch -t 60 -N 2 --ntasks-per-node=10 --mincpus=10
--mem-per-cpu=6000 --partition=multinode hello.sh
sbatch: error: Batch job submission failed: Memory required by task is
not available
$ sbatch -t 60 -N 2 --ntasks-per-node=10 --mincpus=10 --mem=60000
--partition=multinode hello.sh
Submitted batch job 121
In both cases im asking for 60000Mb per node.
If MaxMemPerNode is removed then the Job is accepted:
PartitionName=multinode Shared=EXCLUSIVE Nodes=node[001-008]
DefMemPerCPU=3200 MaxCPUsPerNode=20 DefaultTime=10 MaxTime=72:00:00
sbatch -t 60 -N 2 --ntasks-per-node=10 --mincpus=10 --mem-per-cpu=6000
--partition=multinode hello.sh
Submitted batch job 122
We also have "EnforcePartLimits=yes" set
Can anyone confirm this ?
best regards
M.Soysal
On 27.08.2015 21:42, Mehmet Soysal wrote:
Hi,
i have a question about DefMemCPU and MaxMemPerNode.
At the moment we are running Slurm 14.11.6 and we did not changed the
Config since a few slurm versions.
We have the following lines:
--------------------
SelectType=select/cons_res
SelectTypeParameters=CR_Core_Memory
NodeName=DEFAULT Sockets=2 CoresPerSocket=10 ThreadsPerCore=1
NodeName=node[001-512] RealMemory=64000
PartitionName=singlenode Shared=FORCE Nodes=node[001-512]
DefMemPerCPU=3200 MaxMemPerNode=64000 MaxCPUsPerNode=20 DefaultTime=31
MaxTime=72:00:00 MaxNodes=1
PartitionName=multinode Shared=EXCLUSIVE Nodes=node[001-512]
DefMemPerCPU=3200 MaxMemPerNode=64000 MaxCPUsPerNode=20 DefaultTime=10
MaxTime=72:00:00 MinNodes=2 MaxNodes=128
----------------------------------
The overlapping partitions are wanted and this worked fine for us.
But now we noticed that sbatch returns "Memory required by task is not
available",
even if the user asks for correct core/memory per job.
Example:
$ sbatch -t 60 -N 15 --ntasks-per-node=10 --mincpus=10
--mem-per-cpu=6000 --partition=multinode test.sh
sbatch: error: Batch job submission failed: Memory required by task is
not available
$ sbatch -t 60 -N 15 --ntasks-per-node=10 --mem-per-cpu=6000
--partition=multinode test.sh
sbatch: error: Batch job submission failed: Memory required by task is
not available
$ sbatch -t 60 -N 15 --mincpus=10 --mem-per-cpu=6000
--partition=multinode test.sh
sbatch: error: Batch job submission failed: Memory required by task is
not available
submitting with 3200mb is ok:
sbatch -t 60 -N 15 --mincpus=10 --mem-per-cpu=3200
--partition=multinode test.sh
Submitted batch job 62281
submitting with 3201mb is not ok:
sbatch -t 60 -N 15 --mincpus=10 --mem-per-cpu=3201
--partition=multinode test.sh
Submitted batch job 62284
but this issue is only happening with the multinode partition,
with the singlenode partiton we dont have this issue.
example:
$ sbatch -t 60 -N 1 --mincpus=10 --mem-per-cpu=6000
--partition=singlenode test.sh
Submitted batch job 62285
Im a little bit confused, cause this setup was working fine and we did
not changed anything in the slurm config.
Did the behavior for DefMemPerCPU changed or did i just a made a mistake ?
best regards
M.Soysal
