Dear Developers,
We have the slurm 15.08.6 as the workload manager on our Cray XC30
system (12 core dual socket Ivy Bridge processors, each node has 24
cores, each core with 2 hardware threads). Currently the srun option,
--ntasks-per-socket=<ntasks>, does not work -m option. I would expect
the following result with the srun command *srun -n 8
--ntasks-per-socket=4 --cpu_bind=cores -m block:block:block**./a.out
*
Hello from rank 0, thread 0, on nid00292. (core affinity = 0,24)
Hello from rank 1, thread 0, on nid00292. (core affinity = 1,25)
Hello from rank 2, thread 0, on nid00292. (core affinity = 2,26)
Hello from rank 3, thread 0, on nid00292. (core affinity = 3,27)
Hello from rank 4, thread 0, on nid00292. (core affinity = 12,40)
Hello from rank 5, thread 0, on nid00292. (core affinity = 13,41)
Hello from rank 6, thread 0, on nid00292. (core affinity = 14,42)
Hello from rank 7, thread 0, on nid00292. (core affinity = 15,43)
i.e., the first 4 tasks to be allocated on to the first socket, and next
4 tasks (rank 4-7) to be allocated to the second socket, but I am
getting the following:
Hello from rank 0, thread 0, on nid00292. (core affinity = 0,24)
Hello from rank 1, thread 0, on nid00292. (core affinity = 1,25)
Hello from rank 2, thread 0, on nid00292. (core affinity = 2,26)
Hello from rank 3, thread 0, on nid00292. (core affinity = 3,27)
Hello from rank 4, thread 0, on nid00292. (core affinity = 4,28)
Hello from rank 5, thread 0, on nid00292. (core affinity = 5,29)
Hello from rank 6, thread 0, on nid00292. (core affinity = 6,30)
Hello from rank 7, thread 0, on nid00292. (core affinity = 7,31)
which means all 8 tasks are allocated on to the first socket. Could you
please advice how we can get this to work?
Thanks,
Zhengji
NERSC
Here is our configuration:
zz217@nid00292:~/tests/coreid> scontrol show config
Configuration data as of 2016-01-05T18:26:45
AccountingStorageBackupHost = edique02
AccountingStorageEnforce = associations,limits,qos,safe
AccountingStorageHost = edique01
AccountingStorageLoc = N/A
AccountingStoragePort = 6819
AccountingStorageTRES = cpu,mem,energy,node
AccountingStorageType = accounting_storage/slurmdbd
AccountingStorageUser = N/A
AccountingStoreJobComment = Yes
AcctGatherEnergyType = acct_gather_energy/cray
AcctGatherFilesystemType = acct_gather_filesystem/none
AcctGatherInfinibandType = acct_gather_infiniband/none
AcctGatherNodeFreq = 0 sec
AcctGatherProfileType = acct_gather_profile/none
AllowSpecResourcesUsage = 1
AuthInfo = (null)
AuthType = auth/munge
BackupAddr = 128.55.143.33
BackupController = edique01
BatchStartTimeout = 60 sec
BOOT_TIME = 2016-01-05T08:29:12
BurstBufferType = (null)
CacheGroups = 0
CheckpointType = checkpoint/none
ChosLoc = (null)
ClusterName = edison
CompleteWait = 0 sec
ControlAddr = nid01605
ControlMachine = nid01605
CoreSpecPlugin = core_spec/cray
CpuFreqDef = OnDemand
CpuFreqGovernors = OnDemand
CryptoType = crypto/munge
DebugFlags = (null)
DefMemPerNode = UNLIMITED
DisableRootJobs = Yes
EioTimeout = 60
EnforcePartLimits = Yes
Epilog = (null)
EpilogMsgTime = 2000 usec
EpilogSlurmctld = (null)
ExtSensorsType = ext_sensors/none
ExtSensorsFreq = 0 sec
FairShareDampeningFactor = 1
FastSchedule = 1
FirstJobId = 1
GetEnvTimeout = 2 sec
GresTypes = craynetwork
GroupUpdateForce = 0
GroupUpdateTime = 600 sec
HASH_VAL = Match
HealthCheckInterval = 0 sec
HealthCheckNodeState = ANY
HealthCheckProgram = (null)
InactiveLimit = 600 sec
JobAcctGatherFrequency = 0
JobAcctGatherType = jobacct_gather/cgroup
JobAcctGatherParams = (null)
JobCheckpointDir = /var/slurm/checkpoint
JobCompHost = localhost
JobCompLoc = /var/log/slurm_jobcomp.log
JobCompPort = 0
JobCompType = jobcomp/none
JobCompUser = root
JobContainerType = job_container/cncu
JobCredentialPrivateKey = (null)
JobCredentialPublicCertificate = (null)
JobFileAppend = 0
JobRequeue = 0
JobSubmitPlugins = cray,lua
KeepAliveTime = SYSTEM_DEFAULT
KillOnBadExit = 1
KillWait = 30 sec
LaunchParameters = (null)
LaunchType = launch/slurm
Layouts =
Licenses = (null)
LicensesUsed = (null)
MailProg = /bin/mail
MaxArraySize = 65000
MaxJobCount = 200000
MaxJobId = 2147418112
MaxMemPerNode = UNLIMITED
MaxStepCount = 40000
MaxTasksPerNode = 128
MemLimitEnforce = Yes
MessageTimeout = 60 sec
MinJobAge = 300 sec
MpiDefault = openmpi
MpiParams = ports=63001-64000
MsgAggregationParams = WindowMsgs=1,WindowTime=100
NEXT_JOB_ID = 3542
OverTimeLimit = 0 min
PluginDir = /opt/slurm/default/lib/slurm
PlugStackConfig = /opt/slurm/etc/plugstack.conf
PowerParameters = (null)
PowerPlugin =
PreemptMode = REQUEUE
PreemptType = preempt/qos
PriorityParameters = (null)
PriorityDecayHalfLife = 8-00:00:00
PriorityCalcPeriod = 00:05:00
PriorityFavorSmall = No
PriorityFlags = ACCRUE_ALWAYS
PriorityMaxAge = 21-00:00:00
PriorityUsageResetPeriod = NONE
PriorityType = priority/multifactor
PriorityWeightAge = 30240
PriorityWeightFairShare = 1440
PriorityWeightJobSize = 0
PriorityWeightPartition = 0
PriorityWeightQOS = 24480
PriorityWeightTRES = (null)
PrivateData = none
ProctrackType = proctrack/cray
Prolog = (null)
PrologEpilogTimeout = 65534
PrologSlurmctld = (null)
PrologFlags = Alloc
PropagatePrioProcess = 0
PropagateResourceLimits = ALL
PropagateResourceLimitsExcept = (null)
RebootProgram = (null)
ReconfigFlags = (null)
RequeueExit = (null)
RequeueExitHold = (null)
ResumeProgram = (null)
ResumeRate = 300 nodes/min
ResumeTimeout = 60 sec
ResvEpilog = (null)
ResvOverRun = 0 min
ResvProlog = (null)
ReturnToService = 1
RoutePlugin = (null)
SallocDefaultCommand = srun -n1 -N1 --mem-per-cpu=0 --pty
--preserve-env --gres=craynetwork:0 --mpi=none $SHELL
SchedulerParameters =
no_backup_scheduling,bf_window=5760,bf_resolution=120,bf_max_job_array_resv=20,default_queue_depth=400,bf_max_job_test=6000,bf_max_job_user=10,bf_continue,nohold_on_prolog_fail,kill_invalid_depend
SchedulerPort = 7321
SchedulerRootFilter = 1
SchedulerTimeSlice = 30 sec
SchedulerType = sched/backfill
SelectType = select/cray
SelectTypeParameters = CR_SOCKET_MEMORY,OTHER_CONS_RES
SlurmUser = root(0)
SlurmctldDebug = debug
SlurmctldLogFile = /var/tmp/slurm/slurmctld.log
SlurmctldPort = 6817
SlurmctldTimeout = 120 sec
SlurmdDebug = info
SlurmdLogFile = /var/spool/slurmd/%h.log
SlurmdPidFile = /var/run/slurmd.pid
SlurmdPlugstack = (null)
SlurmdPort = 6818
SlurmdSpoolDir = /var/spool/slurmd
SlurmdTimeout = 300 sec
SlurmdUser = root(0)
SlurmSchedLogFile = (null)
SlurmSchedLogLevel = 0
SlurmctldPidFile = /var/run/slurmctld.pid
SlurmctldPlugstack = (null)
SLURM_CONF = /opt/slurm/etc/slurm.conf
SLURM_VERSION = 15.08.6
SrunEpilog = (null)
SrunPortRange = 60001-63000
SrunProlog = (null)
StateSaveLocation = /global/syscom/sc/nsg/var/edison-slurm-state
SuspendExcNodes = (null)
SuspendExcParts = (null)
SuspendProgram = (null)
SuspendRate = 60 nodes/min
SuspendTime = NONE
SuspendTimeout = 30 sec
SwitchType = switch/cray
TaskEpilog = (null)
TaskPlugin = task/cgroup,task/cray
TaskPluginParam = (null type)
TaskProlog = (null)
TmpFS = /tmp
TopologyParam = (null)
TopologyPlugin = topology/none
TrackWCKey = No
TreeWidth = 18
UsePam = 0
UnkillableStepProgram = (null)
UnkillableStepTimeout = 60 sec
VSizeFactor = 0 percent
WaitTime = 0 sec
Slurmctld(primary/backup) at nid01605/edique01 are UP/UP
