Nifty -- good to know. Thanks for looking into this!
Do any kernel-hacker types on this list know roundabout what version
thread-affinity was brought into the Linux kernel?
FWIW: all the same concepts here (using pid==0) should also work for
PLPA, so you can set via socket/core, etc.
On Dec 5, 2008, at 10:45 AM, Edgar Gabriel wrote:
ok, so I digged a little deeper, and have some good news. Let me
start with a set of routines, that we didn't even discuss yet, but
which works for setting thread affinity, and discuss then libnuma
and sched_setaffinity() again.
-----------
On linux systems, the pthread library has a set of routines to
modify and determine thread-affinity related information:
#define __USE_GNU
int pthread_setaffinity_np (pthread_t __th, size_t __cpusetsize,
const cpu_set_t *__cpuset);
int pthread_getaffinity_np (pthread_t __th, size_t __cpusetsize,
cpu_set_t *__cpuset)
These two routines can be used to modify the affinity of an existing
thread. If you would like to modify the affinity of a thread
*before* creating it, you can use a similar routines.
int pthread_attr_setaffinity_np (pthread_attr_t *__attr,
size_t __cpusetsize,
const cpu_set_t *__cpuset)
I tested the first two routines, and they did work for me.
-----------
Now to libnuma vs. sched_setaffinity: after digging a little deeper
in the libnuma sources, I realized that one of the differences on
what they do vs. what I did in my testcases was, that libnuma uses
the sched_setaffinity() calls with a pid of 0, instead of
determining the pid using the getpid() function. According to the
sched_setaffinity() manpages, pid of zero means 'apply the new rules
to the current process', but it does in fact mean 'to the current
task/thread'. I wrote a set of tests, where I used
sched_setaffinity() with the pid zero, and I was in fact able to
modify the affinity of an individual thread using
sched_setaffinity(). If you pass in the pid of the process, it will
affect the affinity of all threads of that process.
Bottom line is, you can modify the affinity of a thread using both
libnuma on a per socket basis and the sched_setaffinity() calls on a
per core basis. Alternatively, you can use the
pthread_setaffinity_np() function to modify the affinity of a thread
using a cpu_set_t similar to sched_setaffinity.
Thanks
Edgar
Jeff Squyres wrote:
Fair enough; let me know what you find. It would be good to
understand exactly why you're seeing what you're seeing...
On Dec 2, 2008, at 5:47 PM, Edgar Gabriel wrote:
its on OpenSuSE 11 with kernel 2.6.25.11. I don't know the libnuma
library version, but I suspect that its fairly new.
I will try to investigate that in the next days a little more. I
do think that they use sched_setaffinity() underneath the hood
(because in one of my failed attempts when I passed in the wrong
argument, I got actually the same error message that I got earlier
with sched_setaffinity), but they must do something additionally
underneath.
Anyway, I just wanted to report the result, and that there is
obviously a difference, even if can't explain it right now in
details.
Thanks
Edgar
Jeff Squyres wrote:
On Dec 2, 2008, at 11:27 AM, Edgar Gabriel wrote:
so I ran a couple of tests today and I can not confirm your
statement. I wrote simple a simple test code where a process
first sets an affinity mask and than spawns a number of threads.
The threads modify the affinity mask and every thread
( including the master thread) print out there affinity mask at
the end.
With sched_getaffinity() and sched_setaffinity() it was indeed
such that the master thread had the same affinity mask as the
thread that it spawned. This means, that the modification of the
affinity mask by a new thread in fact did affect the master
thread.
Executing the same codesquence however using the libnuma calls,
the master thread however was not affected by the new affinity
mask of the children. So clearly, libnuma must be doing
something differently.
What distro/version of Linux are you using, and what version of
libnuma?
Libnuma v2.0.x very definitely is just a wrapper around the
syscall for sched_setaffinity(). I downloaded it from:
ftp://oss.sgi.com/www/projects/libnuma/download
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab http://pstl.cs.uh.edu
Department of Computer Science University of Houston
Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab http://pstl.cs.uh.edu
Department of Computer Science University of Houston
Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems