Greetings - We have a shared memory system configured as a Slurm partition and an OpenMP job that needs a large amount of memory but is sensitive (performance wise) to memory latency.
The machine consists of "blades" connected by an interconnect in a enhanced 3D hypercube. Each blade contains two numa nodes (Intel processor). Each blade shows up as a board in slurmd. We are trying to figure out how to reliably pin a job to stay within the blade numa nodes, and if we need to go beyond a single blade get an adjacent blade allocated. We are using cgroups with : TaskPlugin = task/cgroup,task/affinity TaskPluginParam = cores SelectType = select/cons_res SelectTypeParameters = CR_CPU_MEMORY # slurmd -C ClusterName=(null) NodeName=machinename CPUs=256 Boards=16 SocketsPerBoard=32 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=4006379 TmpDisk=234080 UpTime=5-23:39:30 # slurmd -V slurm 14.03.11 This bug gets at what we're trying to figure out, but not exactly: http://bugs.schedmd.com/show_bug.cgi?id=2109 CR_Socket or CR_Board isn't necessarily what we are after - we do have a fair amount of 1 core high throughput jobs we want to handle also. Any pointers would be greatly appreciated! Jeff
