Hi Martin,
Thanks for the detailed information. I didnt know that you could just get
this information from /proc so my program does the same thing as your
cat/grep in C :) Your suggestion -m block:block seem to work in my system
as well. Thanks!
Thanks,
Evren
On Thu, 7 Mar 2013, [email protected] wrote:
Evren,
The -m option takes two values, separated by a ":". The first value controls
the distribution of tasks to
nodes. That is not relevant to your problem. The second value controls the
distribution of allocated CPUs to
tasks for binding when task/affinity is configured (as you have done). You
should be able force each task to
bind to CPUs on the same core by specifying "block" as the SECOND value for -m.
I'm not sure how to interpret
the output of your test program. Here is a simple example of what I mean from
my system:
My node has hyperthreading, with two threads per core. It's configured like
this:
Sockets=2 CoresPerSocket=4 ThreadsPerCore=2 Procs=16
[sulu] (slurm)> srun -N 1 -n 2 -c 2 -m block:block -l cat /proc/self/status |
grep Cpus_allowed_list | sort
0: Cpus_allowed_list: 6,14
1: Cpus_allowed_list: 7,15
The job has been allocated 4 logical CPUs (hyperthreads) on 2 cores. task 0
has been bound to logical CPUs 6
and 14. These two CPUs are on the same core. task 1 has been bound to CPUs 7
and 15. These CPUs are also on
the same core.
For more information, see the CPU management guide at
http://lists.schedmd.com/cgi-bin/dada/mail.cgi/r/slurmdev/569013258155/. It
has lots of examples.
Hope this helps.
Martin
From: Evren Yurtesen IB <[email protected]>
To: "slurm-dev" <[email protected]>,
Date: 03/07/2013 03:35 PM
Subject: [slurm-dev] Re: hyperthreading scheduling question
_______________________________________________________________________________________________________________
I used a small code to see exactly what is happening. (below)
compiled it:
mpicc -fopenmp cpuid.c -o cpuid
Then ran it:
srun -c 2 -n 2 -N 1 -m block ./cpuid |sort -n
It prints out:
Rank 0 CPU4 is set
Rank 0 CPU5 is set
Rank 1 CPU16 is set
Rank 1 CPU17 is set
The box has 24 logical processors, so processor 4 and 16 are same physical
core and 5 and 17 are same physical core. So I should have seen rank 0
having CPU4 & CPU16
I tested something else (thanks for the hint about -m :)
-bash-4.1$ srun -c 2 -n 4 -N 1 -m plane=1 ./cpuid |sort -n
srun: job 116180 queued and waiting for resources
srun: job 116180 has been allocated resources
Rank 0 CPU17 is set
Rank 0 CPU5 is set
Rank 1 CPU15 is set
Rank 1 CPU3 is set
Rank 2 CPU11 is set
Rank 2 CPU23 is set
Rank 3 CPU19 is set
Rank 3 CPU7 is set
-bash-4.1$
This seem to be working as I expect. Every rank is sharing 2 thread from a
single core. Does this look correct?
#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char *argv[])
{
int id;
int rank, size;
MPI_Init (&argc, &argv); /* starts MPI */
MPI_Comm_rank (MPI_COMM_WORLD, &rank); /* get current process id */
MPI_Comm_size (MPI_COMM_WORLD, &size); /* get number of processes */
cpu_set_t set;
CPU_ZERO(&set);
sched_getaffinity(0,sizeof(cpu_set_t),&set);
int i;
for (i=0; i<CPU_SETSIZE; i++) {
if( CPU_ISSET(i, &set) ) {
printf("Rank %d CPU%i is set\n",rank,i);
}
}
return 0;
}
On Thu, 7 Mar 2013, [email protected] wrote:
> Assuming you're allocating two logical cpus per task (-c 2) and your
hyperthreading uses two threads per core
(ThreadsPerCore=2), you should be able to
> force each task to bind to two threads on the same core using -m xxxx:block.
> --threads-per-core=2 will constrain node selection to nodes with at least two
threads per core.
>
> Regards,
> Martin Perry
>
>
>
> From: Aaron Knister <[email protected]>
> To: "slurm-dev" <[email protected]>,
> Date: 03/07/2013 04:56 AM
> Subject: [slurm-dev] Re: hyperthreading scheduling question
>
>______________________________________________________________________________________________________________
______________________________________________
>
>
>
>
> As far as a users mailing list, this is it I believe.
>
> So, the desired result is 1 process with 2 hardware threads(logical cores)?
You could try the -c 2 option but
there's no guarantee that both allocated
> logical cores will map to two hardware threads on the same physical core. You
could also try
--threads-per-core=2 perhaps in conjunction with the -c option.
>
> Sent from my iPhone
>
> On Mar 7, 2013, at 4:31 AM, Evren Yurtesen IB <[email protected]> wrote:
>
> >
> > Hello,
> >
> > First question, dont you have a users mailing list?
> >
> > I am on a system with 2 sockets, 6 cores per socket and
withhyper-threading. I have the following options
set:
> >
> > TaskPlugin=task/affinity
> > TaskPluginParam=threads
> > SelectType=select/cons_res
> > SelectTypeParameters=CR_Core_Memory
> >
> > I am trying to run a process and allocate 2 threads per core. However it
looks like slurm is allocating
this strangely and 2 processes end up in the
> threads of 1 core. I am not 100% certain if this is the case but I thought I
would ask how is the best way to
accomplish this?
> >
> > Thanks,
> > Evren
>
> [spacer.png]
>
[spacer.png]
