ok, so I digged a little deeper, and have some good news. Let me start
with a set of routines, that we didn't even discuss yet, but which works
for setting thread affinity, and discuss then libnuma and
sched_setaffinity() again.
-----------
On linux systems, the pthread library has a set of routines to modify
and determine thread-affinity related information:
#define __USE_GNU
int pthread_setaffinity_np (pthread_t __th, size_t __cpusetsize,
const cpu_set_t *__cpuset);
int pthread_getaffinity_np (pthread_t __th, size_t __cpusetsize,
cpu_set_t *__cpuset)
These two routines can be used to modify the affinity of an existing
thread. If you would like to modify the affinity of a thread *before*
creating it, you can use a similar routines.
int pthread_attr_setaffinity_np (pthread_attr_t *__attr,
size_t __cpusetsize,
const cpu_set_t *__cpuset)
I tested the first two routines, and they did work for me.
-----------
Now to libnuma vs. sched_setaffinity: after digging a little deeper in
the libnuma sources, I realized that one of the differences on what they
do vs. what I did in my testcases was, that libnuma uses the
sched_setaffinity() calls with a pid of 0, instead of determining the
pid using the getpid() function. According to the sched_setaffinity()
manpages, pid of zero means 'apply the new rules to the current
process', but it does in fact mean 'to the current task/thread'. I wrote
a set of tests, where I used sched_setaffinity() with the pid zero, and
I was in fact able to modify the affinity of an individual thread using
sched_setaffinity(). If you pass in the pid of the process, it will
affect the affinity of all threads of that process.
Bottom line is, you can modify the affinity of a thread using both
libnuma on a per socket basis and the sched_setaffinity() calls on a per
core basis. Alternatively, you can use the pthread_setaffinity_np()
function to modify the affinity of a thread using a cpu_set_t similar to
sched_setaffinity.
Thanks
Edgar
Jeff Squyres wrote:
Fair enough; let me know what you find. It would be good to understand
exactly why you're seeing what you're seeing...
On Dec 2, 2008, at 5:47 PM, Edgar Gabriel wrote:
its on OpenSuSE 11 with kernel 2.6.25.11. I don't know the libnuma
library version, but I suspect that its fairly new.
I will try to investigate that in the next days a little more. I do
think that they use sched_setaffinity() underneath the hood (because
in one of my failed attempts when I passed in the wrong argument, I
got actually the same error message that I got earlier with
sched_setaffinity), but they must do something additionally underneath.
Anyway, I just wanted to report the result, and that there is
obviously a difference, even if can't explain it right now in details.
Thanks
Edgar
Jeff Squyres wrote:
On Dec 2, 2008, at 11:27 AM, Edgar Gabriel wrote:
so I ran a couple of tests today and I can not confirm your
statement. I wrote simple a simple test code where a process first
sets an affinity mask and than spawns a number of threads. The
threads modify the affinity mask and every thread ( including the
master thread) print out there affinity mask at the end.
With sched_getaffinity() and sched_setaffinity() it was indeed such
that the master thread had the same affinity mask as the thread that
it spawned. This means, that the modification of the affinity mask
by a new thread in fact did affect the master thread.
Executing the same codesquence however using the libnuma calls, the
master thread however was not affected by the new affinity mask of
the children. So clearly, libnuma must be doing something differently.
What distro/version of Linux are you using, and what version of libnuma?
Libnuma v2.0.x very definitely is just a wrapper around the syscall
for sched_setaffinity(). I downloaded it from:
ftp://oss.sgi.com/www/projects/libnuma/download
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab http://pstl.cs.uh.edu
Department of Computer Science University of Houston
Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab http://pstl.cs.uh.edu
Department of Computer Science University of Houston
Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
