I used a small code to see exactly what is happening. (below)
compiled it:
mpicc -fopenmp cpuid.c -o cpuid
Then ran it:
srun -c 2 -n 2 -N 1 -m block ./cpuid |sort -n
It prints out:
Rank 0 CPU4 is set
Rank 0 CPU5 is set
Rank 1 CPU16 is set
Rank 1 CPU17 is set
The box has 24 logical processors, so processor 4 and 16 are same physical
core and 5 and 17 are same physical core. So I should have seen rank 0
having CPU4 & CPU16
I tested something else (thanks for the hint about -m :)
-bash-4.1$ srun -c 2 -n 4 -N 1 -m plane=1 ./cpuid |sort -n
srun: job 116180 queued and waiting for resources
srun: job 116180 has been allocated resources
Rank 0 CPU17 is set
Rank 0 CPU5 is set
Rank 1 CPU15 is set
Rank 1 CPU3 is set
Rank 2 CPU11 is set
Rank 2 CPU23 is set
Rank 3 CPU19 is set
Rank 3 CPU7 is set
-bash-4.1$
This seem to be working as I expect. Every rank is sharing 2 thread from a
single core. Does this look correct?
#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char *argv[])
{
int id;
int rank, size;
MPI_Init (&argc, &argv); /* starts MPI */
MPI_Comm_rank (MPI_COMM_WORLD, &rank); /* get current process id */
MPI_Comm_size (MPI_COMM_WORLD, &size); /* get number of processes */
cpu_set_t set;
CPU_ZERO(&set);
sched_getaffinity(0,sizeof(cpu_set_t),&set);
int i;
for (i=0; i<CPU_SETSIZE; i++) {
if( CPU_ISSET(i, &set) ) {
printf("Rank %d CPU%i is set\n",rank,i);
}
}
return 0;
}
On Thu, 7 Mar 2013, [email protected] wrote:
Assuming you're allocating two logical cpus per task (-c 2) and your
hyperthreading uses two threads per core (ThreadsPerCore=2), you should be able
to
force each task to bind to two threads on the same core using -m xxxx:block.
--threads-per-core=2 will constrain node selection to nodes with at least two
threads per core.
Regards,
Martin Perry
From: Aaron Knister <[email protected]>
To: "slurm-dev" <[email protected]>,
Date: 03/07/2013 04:56 AM
Subject: [slurm-dev] Re: hyperthreading scheduling question
____________________________________________________________________________________________________________________________________________________________
As far as a users mailing list, this is it I believe.
So, the desired result is 1 process with 2 hardware threads(logical cores)? You
could try the -c 2 option but there's no guarantee that both allocated
logical cores will map to two hardware threads on the same physical core. You
could also try --threads-per-core=2 perhaps in conjunction with the -c option.
Sent from my iPhone
On Mar 7, 2013, at 4:31 AM, Evren Yurtesen IB <[email protected]> wrote:
>
> Hello,
>
> First question, dont you have a users mailing list?
>
> I am on a system with 2 sockets, 6 cores per socket and withhyper-threading.
I have the following options set:
>
> TaskPlugin=task/affinity
> TaskPluginParam=threads
> SelectType=select/cons_res
> SelectTypeParameters=CR_Core_Memory
>
> I am trying to run a process and allocate 2 threads per core. However it
looks like slurm is allocating this strangely and 2 processes end up in the
threads of 1 core. I am not 100% certain if this is the case but I thought I
would ask how is the best way to accomplish this?
>
> Thanks,
> Evren
[spacer.png]
