I could do (almost) what I want, I defined DefMemPerCPU parameter to a lower value than the real memory per core in slurm.conf and set AllowedRAMSpace=100 MaxRAMPercent=100. By this way, I garantue that memory usage of different jobs from the the same user (or different users) will never exceed the total memory, and there will always be some memory for system services. Also n * DefMemPerCPU amount of memory reserved for multicore jobs. Any user who need more memory may request more memory via CgroupAutomount=yes CgroupReleaseAgentDir="/etc/slurm/cgroup"
ConstrainCores=yes TaskAffinity=yes ConstrainRAMSpace=yes AllowedRAMSpace=10 ConstrainSwapSpace=yes AllowedSwapSpace=100 MaxRAMPercent=10 MaxSwapPercent=100 Our nodes have 128GB ram. I am testing the configuration via xmalloc program which takes the given parameter as MegaBytes $>salloc -n 1 --nodelist=mercan44 $> srun xmalloc 100000 According to my configuration, I hope that the program will start to use swap space when the memory usage reaches 13GB.. (10% of 128GB) But program continue to use the physical memory even if it reaches 13GB.. What is the problem with my configuration? slurm configuration: scontrol: show config Configuration data as of 2012-04-25T17:04:10 AccountingStorageBackupHost = (null) AccountingStorageEnforce = associations,limits AccountingStorageHost = mercan5 AccountingStorageLoc = N/A AccountingStoragePort = 6819 AccountingStorageType = accounting_storage/slurmdbd AccountingStorageUser = N/A AccountingStoreJobComment = YES AuthType = auth/munge BackupAddr = (null) BackupController = (null) BatchStartTimeout = 10 sec BOOT_TIME = 2012-04-25T16:33:50 CacheGroups = 1 CheckpointType = checkpoint/none ClusterName = linux CompleteWait = 0 sec ControlAddr = mercan5 ControlMachine = mercan5 CryptoType = crypto/munge DebugFlags = (null) DefMemPerNode = UNLIMITED DisableRootJobs = NO EnforcePartLimits = YES Epilog = (null) EpilogMsgTime = 2000 usec EpilogSlurmctld = (null) FastSchedule = 1 FirstJobId = 100000 GetEnvTimeout = 2 sec GresTypes = (null) GroupUpdateForce = 0 GroupUpdateTime = 600 sec HASH_VAL = Different Ours=0x8997cb77 Slurmctld=0x7742bf02 HealthCheckInterval = 0 sec HealthCheckProgram = (null) InactiveLimit = 0 sec JobAcctGatherFrequency = 30 sec JobAcctGatherType = jobacct_gather/linux JobCheckpointDir = /var/slurm/checkpoint JobCompHost = localhost JobCompLoc = /var/log/slurm/job_completions JobCompPort = 0 JobCompType = jobcomp/filetxt JobCompUser = root JobCredentialPrivateKey = (null) JobCredentialPublicCertificate = (null) JobFileAppend = 0 JobRequeue = 1 JobSubmitPlugins = (null) KillOnBadExit = 0 KillWait = 30 sec Licenses = (null) MailProg = /bin/mail MaxJobCount = 1000000 MaxJobId = 4294901760 MaxMemPerNode = UNLIMITED MaxStepCount = 40000 MaxTasksPerNode = 128 MessageTimeout = 10 sec MinJobAge = 300 sec MpiDefault = none MpiParams = (null) NEXT_JOB_ID = 100678 OverTimeLimit = 0 min PluginDir = /usr/lib64/slurm PlugStackConfig = /etc/slurm/plugstack.conf PreemptMode = OFF PreemptType = preempt/none PriorityDecayHalfLife = 00:00:00 PriorityCalcPeriod = 00:05:00 PriorityFavorSmall = 0 PriorityMaxAge = 1-00:00:00 PriorityUsageResetPeriod = NONE PriorityType = priority/multifactor PriorityWeightAge = 1000 PriorityWeightFairShare = 100000 PriorityWeightJobSize = 1000 PriorityWeightPartition = 10000 PriorityWeightQOS = 0 PrivateData = none ProctrackType = proctrack/cgroup Prolog = (null) PrologSlurmctld = (null) PropagatePrioProcess = 0 PropagateResourceLimits = ALL PropagateResourceLimitsExcept = (null) RebootProgram = (null) ReconfigFlags = (null) ResumeProgram = (null) ResumeRate = 300 nodes/min ResumeTimeout = 60 sec ResvOverRun = 0 min ReturnToService = 0 SallocDefaultCommand = (null) SchedulerParameters = (null) SchedulerPort = 7321 SchedulerRootFilter = 1 SchedulerTimeSlice = 30 sec SchedulerType = sched/backfill SelectType = select/cons_res SelectTypeParameters = CR_CORE_MEMORY SlurmUser = root(0) SlurmctldDebug = debug3 SlurmctldLogFile = /var/log/slurm/slurmctld.log SlurmSchedLogFile = (null) SlurmctldPort = 6817 SlurmctldTimeout = 300 sec SlurmdDebug = debug3 SlurmdLogFile = /var/log/slurm/slurmd.log SlurmdPidFile = /var/run/slurmd.pid SlurmdPort = 6818 SlurmdSpoolDir = /tmp/slurmd SlurmdTimeout = 300 sec SlurmdUser = root(0) SlurmSchedLogLevel = 0 SlurmctldPidFile = /var/run/slurmctld.pid SLURM_CONF = /etc/slurm/slurm.conf SLURM_VERSION = 2.4.0-pre4 SrunEpilog = (null) SrunProlog = (null) StateSaveLocation = /tmp SuspendExcNodes = (null) SuspendExcParts = (null) SuspendProgram = (null) SuspendRate = 60 nodes/min SuspendTime = NONE SuspendTimeout = 30 sec SwitchType = switch/none TaskEpilog = (null) TaskPlugin = task/none TaskPluginParam = (null type) TaskProlog = (null) TmpFS = /tmp TopologyPlugin = topology/none TrackWCKey = 0 TreeWidth = 50 UsePam = 0 UnkillableStepProgram = (null) UnkillableStepTimeout = 60 sec VSizeFactor = 0 percent WaitTime = 0 sec
