Hello Everyone, I'm having a bit of a strange problem. Basically, it appears our test instance of Slurm is constraining cores even though it is explicitly disabled in the cgroups.conf file. I'm hoping you can help me figure out why.
I also can run this code on the test compute node outside of slurm and I get the expected behavior (runs at 200%). Attached is a copy of the configuration for this test cluster and a diff compared against our production configuration. Here is an example of what I'm seeing with a very simple 2 thread application. With the core constraints disabled on both systems, I'd expect both to run at 200% with ConstrainCores=no. ################################################################# Submit script ################################################################## #!/bin/bash # #SBATCH --job-name=stress #SBATCH --output=stress.output #SBATCH -p batch #SBATCH -n 1 #SBATCH --time=10:00 #SBATCH --mem=200 ./multithread ################################################################## ================================================================== Job Behavior (production cluster) ================================================================== PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9455 crutledg 20 0 10316 472 372 S 200.0 0.0 17:16.34 multithread ================================================================== ================================================================== Job Behavior (test cluster) ================================================================== PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10436 crutledg 20 0 26704 480 376 S 100.0 0.0 4:53.16 multithread ==================================================================
[crutledge@ctr-scratch ~]$ cat /etc/slurm/cgroup.conf ### # # Slurm cgroup support configuration file # # See man slurm.conf and man cgroup.conf for further # information on cgroup configuration parameters #-- CgroupAutomount=yes CgroupReleaseAgentDir="/etc/slurm/cgroup" ConstrainCores=no ConstrainRAMSpace=no [crutledge@ctr-scratch ~]$ scontrol sho conf Configuration data as of 2016-03-17T13:38:35 AccountingStorageBackupHost = (null) AccountingStorageEnforce = none AccountingStorageHost = ctr-scratch AccountingStorageLoc = N/A AccountingStoragePort = 6819 AccountingStorageTRES = cpu,mem,energy,node AccountingStorageType = accounting_storage/slurmdbd AccountingStorageUser = N/A AccountingStoreJobComment = Yes AcctGatherEnergyType = acct_gather_energy/rapl AcctGatherFilesystemType = acct_gather_filesystem/none AcctGatherInfinibandType = acct_gather_infiniband/none AcctGatherNodeFreq = 1 sec AcctGatherProfileType = acct_gather_profile/none AllowSpecResourcesUsage = 0 AuthInfo = (null) AuthType = auth/munge BackupAddr = (null) BackupController = (null) BatchStartTimeout = 10 sec BOOT_TIME = 2016-03-17T13:20:45 BurstBufferType = (null) CacheGroups = 0 CheckpointType = checkpoint/none ChosLoc = (null) ClusterName = home CompleteWait = 0 sec ControlAddr = ctr-scratch ControlMachine = ctr-scratch CoreSpecPlugin = core_spec/none CpuFreqDef = OnDemand CpuFreqGovernors = OnDemand CryptoType = crypto/munge DebugFlags = Gang,SelectType DefMemPerCPU = 900 DisableRootJobs = No EioTimeout = 60 EnforcePartLimits = No Epilog = (null) EpilogMsgTime = 2000 usec EpilogSlurmctld = (null) ExtSensorsType = ext_sensors/none ExtSensorsFreq = 0 sec FairShareDampeningFactor = 1 FastSchedule = 1 FirstJobId = 1 GetEnvTimeout = 2 sec GresTypes = (null) GroupUpdateForce = 0 GroupUpdateTime = 600 sec HASH_VAL = Match HealthCheckInterval = 0 sec HealthCheckNodeState = ANY HealthCheckProgram = (null) InactiveLimit = 0 sec JobAcctGatherFrequency = 30 JobAcctGatherType = jobacct_gather/linux JobAcctGatherParams = (null) JobCheckpointDir = /var/slurm/checkpoint JobCompHost = localhost JobCompLoc = /var/log/slurm_jobcomp.log JobCompPort = 0 JobCompType = jobcomp/none JobCompUser = root JobContainerType = job_container/none JobCredentialPrivateKey = (null) JobCredentialPublicCertificate = (null) JobFileAppend = 0 JobRequeue = 1 JobSubmitPlugins = (null) KeepAliveTime = SYSTEM_DEFAULT KillOnBadExit = 0 KillWait = 30 sec LaunchParameters = (null) LaunchType = launch/slurm Layouts = Licenses = (null) LicensesUsed = (null) MailProg = /bin/mail MaxArraySize = 10001 MaxJobCount = 50000 MaxJobId = 2147418112 MaxMemPerNode = UNLIMITED MaxStepCount = 40000 MaxTasksPerNode = 128 MemLimitEnforce = Yes MessageTimeout = 10 sec MinJobAge = 300 sec MpiDefault = none MpiParams = (null) MsgAggregationParams = (null) NEXT_JOB_ID = 22 OverTimeLimit = 0 min PluginDir = /usr/lib64/slurm PlugStackConfig = /etc/slurm/plugstack.conf PowerParameters = (null) PowerPlugin = PreemptMode = GANG,SUSPEND PreemptType = preempt/partition_prio PriorityParameters = (null) PriorityDecayHalfLife = 3-00:00:00 PriorityCalcPeriod = 00:05:00 PriorityFavorSmall = No PriorityFlags = PriorityMaxAge = 3-00:00:00 PriorityUsageResetPeriod = MONTHLY PriorityType = priority/multifactor PriorityWeightAge = 10000 PriorityWeightFairShare = 10000 PriorityWeightJobSize = 10000 PriorityWeightPartition = 10000 PriorityWeightQOS = 10000 PriorityWeightTRES = (null) PrivateData = none ProctrackType = proctrack/cgroup Prolog = (null) PrologEpilogTimeout = 65534 PrologSlurmctld = (null) PrologFlags = (null) PropagatePrioProcess = 0 PropagateResourceLimits = (null) PropagateResourceLimitsExcept = MEMLOCK RebootProgram = (null) ReconfigFlags = (null) RequeueExit = (null) RequeueExitHold = (null) ResumeProgram = (null) ResumeRate = 300 nodes/min ResumeTimeout = 60 sec ResvEpilog = (null) ResvOverRun = 0 min ResvProlog = (null) ReturnToService = 2 RoutePlugin = (null) SallocDefaultCommand = (null) SchedulerParameters = (null) SchedulerPort = 7321 SchedulerRootFilter = 1 SchedulerTimeSlice = 30 sec SchedulerType = sched/backfill SelectType = select/cons_res SelectTypeParameters = CR_CORE_MEMORY SlurmUser = slurm(20016) SlurmctldDebug = debug5 SlurmctldLogFile = /var/log/slurm/slurmctld.log SlurmctldPort = 6817 SlurmctldTimeout = 600 sec SlurmdDebug = unknown SlurmdLogFile = /var/log/slurm/slurmd.log SlurmdPidFile = /var/run/slurmd.pid SlurmdPlugstack = (null) SlurmdPort = 6818 SlurmdSpoolDir = /var/spool/slurmd SlurmdTimeout = 600 sec SlurmdUser = root(0) SlurmSchedLogFile = /var/log/slurm/slurmSched.log SlurmSchedLogLevel = 10 SlurmctldPidFile = /var/run/slurmctld.pid SlurmctldPlugstack = (null) SLURM_CONF = /etc/slurm/slurm.conf SLURM_VERSION = 15.08.8 SrunEpilog = (null) SrunPortRange = 0-0 SrunProlog = (null) StateSaveLocation = /tmp/slurm SuspendExcNodes = (null) SuspendExcParts = (null) SuspendProgram = (null) SuspendRate = 60 nodes/min SuspendTime = NONE SuspendTimeout = 30 sec SwitchType = switch/none TaskEpilog = (null) TaskPlugin = task/affinity TaskPluginParam = (null type) TaskProlog = (null) TmpFS = /tmp TopologyParam = (null) TopologyPlugin = topology/tree TrackWCKey = No TreeWidth = 50 UsePam = 0 UnkillableStepProgram = (null) UnkillableStepTimeout = 60 sec VSizeFactor = 0 percent WaitTime = 0 sec Slurmctld(primary/backup) at ctr-scratch/(NULL) are UP/DOWN
[crutledge@ctr-scratch ~]$ diff /tmp/slurm.conf.test /tmp/slurm.conf.prod 1,2c1,2 < Configuration data as of 2016-03-18T07:26:49 < AccountingStorageBackupHost = (null) --- > Configuration data as of 2016-03-17T10:56:56 > AccountingStorageBackupHost = htgw0 4c4 < AccountingStorageHost = ctr-scratch --- > AccountingStorageHost = htsn 6c6 < AccountingStoragePort = 6819 --- > AccountingStoragePort = 7031 13c13 < AcctGatherInfinibandType = acct_gather_infiniband/none --- > AcctGatherInfinibandType = acct_gather_infiniband/ofed 19,20c19,20 < BackupAddr = (null) < BackupController = (null) --- > BackupAddr = 10.4.0.2 > BackupController = htgw0 22c22 < BOOT_TIME = 2016-03-17T14:18:17 --- > BOOT_TIME = 2016-03-17T07:46:01 27c27 < ClusterName = home --- > ClusterName = hatteras 29,30c29,30 < ControlAddr = ctr-scratch < ControlMachine = ctr-scratch --- > ControlAddr = 10.4.0.1 > ControlMachine = htsn 35,36c35,36 < DebugFlags = Gang,SelectType < DefMemPerCPU = 900 --- > DebugFlags = (null) > DefMemPerNode = UNLIMITED 53c53 < HealthCheckInterval = 0 sec --- > HealthCheckInterval = 90 sec 55c55 < HealthCheckProgram = (null) --- > HealthCheckProgram = /usr/sbin/nhc 82c82 < MaxJobCount = 50000 --- > MaxJobCount = 100000 93c93 < NEXT_JOB_ID = 34 --- > NEXT_JOB_ID = 15944003 145c145 < SlurmctldDebug = debug5 --- > SlurmctldDebug = info 147,149c147,149 < SlurmctldPort = 6817 < SlurmctldTimeout = 600 sec < SlurmdDebug = unknown --- > SlurmctldPort = 6817-6899 > SlurmctldTimeout = 60 sec > SlurmdDebug = info 158c158 < SlurmSchedLogLevel = 10 --- > SlurmSchedLogLevel = 3 166c166 < StateSaveLocation = /tmp/slurm --- > StateSaveLocation = /var/slurm/spool 189c189,192 < Slurmctld(primary/backup) at ctr-scratch/(NULL) are UP/DOWN --- > Account Gather > InfinibandOFEDPort = 0 > > Slurmctld(primary/backup) at htsn/htgw0 are UP/UP
