Sefa Arslan <[email protected]> writes: > after I set TaskPlugin=task/cgroup, the cgroup mechanism started to > work:) thanks alot. > But still I have some missing parameter on my configuration files I > think. > > I have following lines on my cgroup.conf files. > MaxRAMPercent=10 > MaxSwapPercent=100 > If only one user's job is > running on the worker node, the user is allowed to use only 10% of the > total real memory, and for remaning memory demand of the program, swap > is started to use.. That is ok.. > But when another user's job (or another job of the same user) start on > the same worker node, he is also allowed to use 10% of the total > Memory. Namely, when two users job are running on the same node, %20 of > the physical memory is used. We want to limit total memory usage of > all jobs (started via slurm, not the system services etc.) to a > certain amount.
If you want to limit the memory usage of all slurm jobs in total to 10% of RAM, then the SLURM cgroups support isn't really what you want to use. That code is for constraining each job to its allocated amount of RAM. One quick approach you can take is to run slurmd itself in a memcg, and then set ConstrainRAMSpace=no (to disable making per-job memory cgroups). Then all tasks will run within the cgroup of slurmd, and all slurm jobs on the node as a whole would be limited to the constraints of the slurmd cgroup. However, this would disable other features of per-job-step memcg -- for example a memcg OOM (if the memory cgroup goes above threshold) will invoke the OOM killer for all jobs instead just one job step. e.g., as root: - Edit cgroup.conf and set ConstrainRAMSpace=no - Move slurmd to its own memory cgroup: pgrep slurmd > /cgroup/memory/slurm/tasks - Constrain the slurm cgroup to whatever you want echo 100M > /cgroup/memory/slurm/memory.limit_in_bytes ... etc.. Now all running jobs + slurmd will be constrained to a total of 100M resident in this case: $ srun testing/usemem 1024 Zeroing 1024M slurmd[hype352]: task0: [usemem] invoked OOM killer: vsz=1091.9M rss=99.4M srun: error: hype352: task 0: Killed > > Which parameters I need to use in slurm.conf and cgroup.conf? > Also, I think MaxRAMPercent and MaxSwapPercent parameters work as > Amount/perJob. Is there a way to define MaxRAMPercent per process ?? The task/cgroup plugin could be extended to create a per-task memcg, placing individually exectuted tasks in their own memory cgroups -- but the limits will apply to those tasks and their children, not really to individual processes. mark > Danny Auble wrote: > Along with what Mark said you need to also inable the task/cgroup > plugin. According to your config output you are running task/none. > > Danny > > On 04/25/12 07:13, Sefa Arslan wrote: > Hii.. > I am trying to use cgroup with slurm. > > I have the following lines on my /etc/slurm/cgroup.conf file: > ### > # > # Slurm cgroup support configuration file > # > # See man slurm.conf and man cgroup.conf for further > # information on cgroup configuration parameters > # nodelist=mercan44 > $> srun xmalloc 100000 > > According to my configuration, I hope that the program will start to use > swap space when the memory usage reaches 13GB.. (10% of 128GB) > > But program continue to use the physical memory even if it reaches 13GB.. > > What is the problem with my configuration? > > slurm configuration: > > scontrol: show config > Configuration data as of 2012-04-25T17:04:10 > AccountingStorageBackupHost = (null) > AccountingStorageEnforce = associations,limits > AccountingStorageHost = mercan5 > AccountingStorageLoc = N/A > AccountingStoragePort = 6819 > AccountingStorageType = accounting_storage/slurmdbd > AccountingStorageUser = N/A > AccountingStoreJobComment = YES > AuthType = auth/munge > BackupAddr = (null) > BackupController = (null) > BatchStartTimeout = 10 sec > BOOT_TIME = 2012-04-25T16:33:50 > CacheGroups = 1 > CheckpointType = checkpoint/none > ClusterName = linux > CompleteWait = 0 sec > ControlAddr = mercan5 > ControlMachine = mercan5 > CryptoType = crypto/munge > DebugFlags = (null) > DefMemPerNode = UNLIMITED > DisableRootJobs = NO > EnforcePartLimits = YES > Epilog = (null) > EpilogMsgTime = 2000 usec > EpilogSlurmctld = (null) > FastSchedule = 1 > FirstJobId = 100000 > GetEnvTimeout = 2 sec > GresTypes = (null) > GroupUpdateForce = 0 > GroupUpdateTime = 600 sec > HASH_VAL = Different Ours=0x8997cb77 Slurmctld=0x7742bf02 > HealthCheckInterval = 0 sec > HealthCheckProgram = (null) > InactiveLimit = 0 sec > JobAcctGatherFrequency = 30 sec > JobAcctGatherType = jobacct_gather/linux > JobCheckpointDir = /var/slurm/checkpoint > JobCompHost = localhost > JobCompLoc = /var/log/slurm/job_completions > JobCompPort = 0 > JobCompType = jobcomp/filetxt > JobCompUser = root > JobCredentialPrivateKey = (null) > JobCredentialPublicCertificate = (null) > JobFileAppend = 0 > JobRequeue = 1 > JobSubmitPlugins = (null) > KillOnBadExit = 0 > KillWait = 30 sec > Licenses = (null) > MailProg = /bin/mail > MaxJobCount = 1000000 > MaxJobId = 4294901760 > MaxMemPerNode = UNLIMITED > MaxStepCount = 40000 > MaxTasksPerNode = 128 > MessageTimeout = 10 sec > MinJobAge = 300 sec > MpiDefault = none > MpiParams = (null) > NEXT_JOB_ID = 100678 > OverTimeLimit = 0 min > PluginDir = /usr/lib64/slurm > PlugStackConfig = /etc/slurm/plugstack.conf > PreemptMode = OFF > PreemptType = preempt/none > PriorityDecayHalfLife = 00:00:00 > PriorityCalcPeriod = 00:05:00 > PriorityFavorSmall = 0 > PriorityMaxAge = 1-00:00:00 > PriorityUsageResetPeriod = NONE > PriorityType = priority/multifactor > PriorityWeightAge = 1000 > PriorityWeightFairShare = 100000 > PriorityWeightJobSize = 1000 > PriorityWeightPartition = 10000 > PriorityWeightQOS = 0 > PrivateData = none > ProctrackType = proctrack/cgroup > Prolog = (null) > PrologSlurmctld = (null) > PropagatePrioProcess = 0 > PropagateResourceLimits = ALL > PropagateResourceLimitsExcept = (null) > RebootProgram = (null) > ReconfigFlags = (null) > ResumeProgram = (null) > ResumeRate = 300 nodes/min > ResumeTimeout = 60 sec > ResvOverRun = 0 min > ReturnToService = 0 > SallocDefaultCommand = (null) > SchedulerParameters = (null) > SchedulerPort = 7321 > SchedulerRootFilter = 1 > SchedulerTimeSlice = 30 sec > SchedulerType = sched/backfill > SelectType = select/cons_res > SelectTypeParameters = CR_CORE_MEMORY > SlurmUser = root(0) > SlurmctldDebug = debug3 > SlurmctldLogFile = /var/log/slurm/slurmctld.log > SlurmSchedLogFile = (null) > SlurmctldPort = 6817 > SlurmctldTimeout = 300 sec > SlurmdDebug = debug3 > SlurmdLogFile = /var/log/slurm/slurmd.log > SlurmdPidFile = /var/run/slurmd.pid > SlurmdPort = 6818 > SlurmdSpoolDir = /tmp/slurmd > SlurmdTimeout = 300 sec > SlurmdUser = root(0) > SlurmSchedLogLevel = 0 > SlurmctldPidFile = /var/run/slurmctld.pid > SLURM_CONF = /etc/slurm/slurm.conf > SLURM_VERSION = 2.4.0-pre4 > SrunEpilog = (null) > SrunProlog = (null) > StateSaveLocation = /tmp > SuspendExcNodes = (null) > SuspendExcParts = (null) > SuspendProgram = (null) > SuspendRate = 60 nodes/min > SuspendTime = NONE > SuspendTimeout = 30 sec > SwitchType = switch/none > TaskEpilog = (null) > TaskPlugin = task/none > TaskPluginParam = (null type) > TaskProlog = (null) > TmpFS = /tmp > TopologyPlugin = topology/none > TrackWCKey = 0 > TreeWidth = 50 > UsePam = 0 > UnkillableStepProgram = (null) > UnkillableStepTimeout = 60 sec > VSizeFactor = 0 percent > WaitTime = 0 sec
