Evren, The -m option takes two values, separated by a ":". The first value controls the distribution of tasks to nodes. That is not relevant to your problem. The second value controls the distribution of allocated CPUs to tasks for binding when task/affinity is configured (as you have done). You should be able force each task to bind to CPUs on the same core by specifying "block" as the SECOND value for -m. I'm not sure how to interpret the output of your test program. Here is a simple example of what I mean from my system:
My node has hyperthreading, with two threads per core. It's configured like this: Sockets=2 CoresPerSocket=4 ThreadsPerCore=2 Procs=16 [sulu] (slurm)> srun -N 1 -n 2 -c 2 -m block:block -l cat /proc/self/status | grep Cpus_allowed_list | sort 0: Cpus_allowed_list: 6,14 1: Cpus_allowed_list: 7,15 The job has been allocated 4 logical CPUs (hyperthreads) on 2 cores. task 0 has been bound to logical CPUs 6 and 14. These two CPUs are on the same core. task 1 has been bound to CPUs 7 and 15. These CPUs are also on the same core. For more information, see the CPU management guide at http://lists.schedmd.com/cgi-bin/dada/mail.cgi/r/slurmdev/135323313686/. It has lots of examples. Hope this helps. Martin From: Evren Yurtesen IB <[email protected]> To: "slurm-dev" <[email protected]>, Date: 03/07/2013 03:35 PM Subject: [slurm-dev] Re: hyperthreading scheduling question I used a small code to see exactly what is happening. (below) compiled it: mpicc -fopenmp cpuid.c -o cpuid Then ran it: srun -c 2 -n 2 -N 1 -m block ./cpuid |sort -n It prints out: Rank 0 CPU4 is set Rank 0 CPU5 is set Rank 1 CPU16 is set Rank 1 CPU17 is set The box has 24 logical processors, so processor 4 and 16 are same physical core and 5 and 17 are same physical core. So I should have seen rank 0 having CPU4 & CPU16 I tested something else (thanks for the hint about -m :) -bash-4.1$ srun -c 2 -n 4 -N 1 -m plane=1 ./cpuid |sort -n srun: job 116180 queued and waiting for resources srun: job 116180 has been allocated resources Rank 0 CPU17 is set Rank 0 CPU5 is set Rank 1 CPU15 is set Rank 1 CPU3 is set Rank 2 CPU11 is set Rank 2 CPU23 is set Rank 3 CPU19 is set Rank 3 CPU7 is set -bash-4.1$ This seem to be working as I expect. Every rank is sharing 2 thread from a single core. Does this look correct? #define _GNU_SOURCE #include <sched.h> #include <stdio.h> #include <mpi.h> int main(int argc, char *argv[]) { int id; int rank, size; MPI_Init (&argc, &argv); /* starts MPI */ MPI_Comm_rank (MPI_COMM_WORLD, &rank); /* get current process id */ MPI_Comm_size (MPI_COMM_WORLD, &size); /* get number of processes */ cpu_set_t set; CPU_ZERO(&set); sched_getaffinity(0,sizeof(cpu_set_t),&set); int i; for (i=0; i<CPU_SETSIZE; i++) { if( CPU_ISSET(i, &set) ) { printf("Rank %d CPU%i is set\n",rank,i); } } return 0; } On Thu, 7 Mar 2013, [email protected] wrote: > Assuming you're allocating two logical cpus per task (-c 2) and your hyperthreading uses two threads per core (ThreadsPerCore=2), you should be able to > force each task to bind to two threads on the same core using -m xxxx:block. > --threads-per-core=2 will constrain node selection to nodes with at least two threads per core. > > Regards, > Martin Perry > > > > From: Aaron Knister <[email protected]> > To: "slurm-dev" <[email protected]>, > Date: 03/07/2013 04:56 AM > Subject: [slurm-dev] Re: hyperthreading scheduling question > > ____________________________________________________________________________________________________________________________________________________________ > > > > > As far as a users mailing list, this is it I believe. > > So, the desired result is 1 process with 2 hardware threads(logical cores)? You could try the -c 2 option but there's no guarantee that both allocated > logical cores will map to two hardware threads on the same physical core. You could also try --threads-per-core=2 perhaps in conjunction with the -c option. > > Sent from my iPhone > > On Mar 7, 2013, at 4:31 AM, Evren Yurtesen IB <[email protected]> wrote: > > > > > Hello, > > > > First question, dont you have a users mailing list? > > > > I am on a system with 2 sockets, 6 cores per socket and withhyper-threading. I have the following options set: > > > > TaskPlugin=task/affinity > > TaskPluginParam=threads > > SelectType=select/cons_res > > SelectTypeParameters=CR_Core_Memory > > > > I am trying to run a process and allocate 2 threads per core. However it looks like slurm is allocating this strangely and 2 processes end up in the > threads of 1 core. I am not 100% certain if this is the case but I thought I would ask how is the best way to accomplish this? > > > > Thanks, > > Evren > > [spacer.png] >
