after I set TaskPlugin=task/cgroup, the cgroup mechanism started to work:) thanks alot. But still I have some missing parameter on my configuration files I think.
I have following lines on my cgroup.conf files. MaxRAMPercent=10 MaxSwapPercent=100 If only one user's job is running on the worker node, the user is allowed to use only 10% of the total real memory, and for remaning memory demand of the program, swap is started to use.. That is ok.. But when another user's job (or another job of the same user) start on the same worker node, he is also allowed to use 10% of the total Memory. Namely, when two users job are running on the same node, %20 of the physical memory is used. We want to limit total memory usage of all jobs (started via slurm, not the system services etc.) to a certain amount. Which parameters I need to use in slurm.conf and cgroup.conf? Also, I think MaxRAMPercent and MaxSwapPercent parameters work as Amount/perJob. Is there a way to define MaxRAMPercent per process ?? Thanks. Danny Auble wrote: Along with what Mark said you need to also inable the task/cgroup plugin. According to your config output you are running task/none. Danny On 04/25/12 07:13, Sefa Arslan wrote: Hii.. I am trying to use cgroup with slurm. I have the following lines on my /etc/slurm/cgroup.conf file: ### # # Slurm cgroup support configuration file # # See man slurm.conf and man cgroup.conf for further # information on cgroup configuration parameters # nodelist=mercan44 $> srun xmalloc 100000 According to my configuration, I hope that the program will start to use swap space when the memory usage reaches 13GB.. (10% of 128GB) But program continue to use the physical memory even if it reaches 13GB.. What is the problem with my configuration? slurm configuration: scontrol: show config Configuration data as of 2012-04-25T17:04:10 AccountingStorageBackupHost = (null) AccountingStorageEnforce = associations,limits AccountingStorageHost = mercan5 AccountingStorageLoc = N/A AccountingStoragePort = 6819 AccountingStorageType = accounting_storage/slurmdbd AccountingStorageUser = N/A AccountingStoreJobComment = YES AuthType = auth/munge BackupAddr = (null) BackupController = (null) BatchStartTimeout = 10 sec BOOT_TIME = 2012-04-25T16:33:50 CacheGroups = 1 CheckpointType = checkpoint/none ClusterName = linux CompleteWait = 0 sec ControlAddr = mercan5 ControlMachine = mercan5 CryptoType = crypto/munge DebugFlags = (null) DefMemPerNode = UNLIMITED DisableRootJobs = NO EnforcePartLimits = YES Epilog = (null) EpilogMsgTime = 2000 usec EpilogSlurmctld = (null) FastSchedule = 1 FirstJobId = 100000 GetEnvTimeout = 2 sec GresTypes = (null) GroupUpdateForce = 0 GroupUpdateTime = 600 sec HASH_VAL = Different Ours=0x8997cb77 Slurmctld=0x7742bf02 HealthCheckInterval = 0 sec HealthCheckProgram = (null) InactiveLimit = 0 sec JobAcctGatherFrequency = 30 sec JobAcctGatherType = jobacct_gather/linux JobCheckpointDir = /var/slurm/checkpoint JobCompHost = localhost JobCompLoc = /var/log/slurm/job_completions JobCompPort = 0 JobCompType = jobcomp/filetxt JobCompUser = root JobCredentialPrivateKey = (null) JobCredentialPublicCertificate = (null) JobFileAppend = 0 JobRequeue = 1 JobSubmitPlugins = (null) KillOnBadExit = 0 KillWait = 30 sec Licenses = (null) MailProg = /bin/mail MaxJobCount = 1000000 MaxJobId = 4294901760 MaxMemPerNode = UNLIMITED MaxStepCount = 40000 MaxTasksPerNode = 128 MessageTimeout = 10 sec MinJobAge = 300 sec MpiDefault = none MpiParams = (null) NEXT_JOB_ID = 100678 OverTimeLimit = 0 min PluginDir = /usr/lib64/slurm PlugStackConfig = /etc/slurm/plugstack.conf PreemptMode = OFF PreemptType = preempt/none PriorityDecayHalfLife = 00:00:00 PriorityCalcPeriod = 00:05:00 PriorityFavorSmall = 0 PriorityMaxAge = 1-00:00:00 PriorityUsageResetPeriod = NONE PriorityType = priority/multifactor PriorityWeightAge = 1000 PriorityWeightFairShare = 100000 PriorityWeightJobSize = 1000 PriorityWeightPartition = 10000 PriorityWeightQOS = 0 PrivateData = none ProctrackType = proctrack/cgroup Prolog = (null) PrologSlurmctld = (null) PropagatePrioProcess = 0 PropagateResourceLimits = ALL PropagateResourceLimitsExcept = (null) RebootProgram = (null) ReconfigFlags = (null) ResumeProgram = (null) ResumeRate = 300 nodes/min ResumeTimeout = 60 sec ResvOverRun = 0 min ReturnToService = 0 SallocDefaultCommand = (null) SchedulerParameters = (null) SchedulerPort = 7321 SchedulerRootFilter = 1 SchedulerTimeSlice = 30 sec SchedulerType = sched/backfill SelectType = select/cons_res SelectTypeParameters = CR_CORE_MEMORY SlurmUser = root(0) SlurmctldDebug = debug3 SlurmctldLogFile = /var/log/slurm/slurmctld.log SlurmSchedLogFile = (null) SlurmctldPort = 6817 SlurmctldTimeout = 300 sec SlurmdDebug = debug3 SlurmdLogFile = /var/log/slurm/slurmd.log SlurmdPidFile = /var/run/slurmd.pid SlurmdPort = 6818 SlurmdSpoolDir = /tmp/slurmd SlurmdTimeout = 300 sec SlurmdUser = root(0) SlurmSchedLogLevel = 0 SlurmctldPidFile = /var/run/slurmctld.pid SLURM_CONF = /etc/slurm/slurm.conf SLURM_VERSION = 2.4.0-pre4 SrunEpilog = (null) SrunProlog = (null) StateSaveLocation = /tmp SuspendExcNodes = (null) SuspendExcParts = (null) SuspendProgram = (null) SuspendRate = 60 nodes/min SuspendTime = NONE SuspendTimeout = 30 sec SwitchType = switch/none TaskEpilog = (null) TaskPlugin = task/none TaskPluginParam = (null type) TaskProlog = (null) TmpFS = /tmp TopologyPlugin = topology/none TrackWCKey = 0 TreeWidth = 50 UsePam = 0 UnkillableStepProgram = (null) UnkillableStepTimeout = 60 sec VSizeFactor = 0 percent WaitTime = 0 sec
