Re: [O-MPI devel] Linux processor affinity
On Nov 29, 2005, at 3:04 PM, Bogdan Costescu wrote:
Here's the problem: there are 3 different APIs for processor affinity
in Linux.
Could you please list them (at least the ones that you know about) ?
Check out http://svn.open-mpi.org/svn/ompi/trunk/opal/mca/paffinity/
linux/paffinity_linux.h -- there's a big comment in that file about
the problem, to include descriptions of the 3 APIs.
In the kernel source, in kernel/sched.c, the sys_sched_setaffinity
function appears only in 2.6.0 (talking about stable kernels only). I
can also see it back-ported by Red Hat in their RHEL3 (2.4.21-based)
kernels, so I would like to know if others have back-ported it as
well and if their functions differ.
Both the official 2.6.x and the Red Hat back-ported definition of this
function is:
asmlinkage long sys_sched_setaffinity(pid_t pid, unsigned int len,
unsigned long __user
*user_mask_ptr)
(the back-ported RHEL3 doesn't have the __user attribute to the last
parameter, but that's cosmetic)
The glibc definitions of sched_setaffinity seem to change, I already
found 2 of them in RHEL3 and RHEL4, but they both call the same
underlying kernel function. So Open MPI could just bypass glibc and
call the kernel function directly, for example:
The problem is that there are some 2-parameter variants out there. :-(
Check out Paul Hargrove's solution: http://www.open-mpi.org/community/
lists/devel/2005/11/0562.php
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
Re: [O-MPI devel] Linux processor affinity
On Nov 29, 2005, at 2:51 PM, Paul H. Hargrove wrote:
The result is the following, which I've tried in limited testing:
Holy Crimminey, Batman -- this message slipped by me in my INBOX.
This is friggen' amazing.
Many thanks, Paul!
enum {
SCHED_SETAFFINITY_TAKES_2_ARGS,
SCHED_SETAFFINITY_TAKES_3_ARGS_THIRD_IS_LONG,
SCHED_SETAFFINITY_TAKES_3_ARGS_THIRD_IS_CPU_SET,
SCHED_SETAFFINITY_UNKNOWN
};
/* We want to call by this prototype, even if it is not the real
one */
extern sched_setaffinity(int pid, unsigned int len, void *mask);
int probe_setaffinity(void) {
unsigned long mask[511];
int rc;
memset(mask, 0, sizeof(mask));
mask[0] = 1;
rc = sched_setaffinity(0, sizeof(mask), mask);
if (rc >= 0) {
/* Kernel truncates over-length masks -> successful call */
return SCHED_SETAFFINITY_TAKES_3_ARGS_THIRD_IS_CPU_SET;
} else if (errno == EINVAL) {
/* Kernel returns EINVAL when len != sizeof(long) */
return SCHED_SETAFFINITY_TAKES_3_ARGS_THIRD_IS_LONG;
} else if (errno == EFAULT) {
/* Kernel returns EFAULT having rejected len as an address */
return SCHED_SETAFFINITY_TAKES_2_ARGS;
}
return SCHED_SETAFFINITY_UNKNOWN;
};
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
Re: [O-MPI devel] [PATH] ompi_info doesn't show use_mem_hooks flag
Thanks for catching that. I'll take a look and commit a fix real
soon now.
Brian
On Dec 8, 2005, at 11:03 AM, Gleb Natapov wrote:
On Thu, Dec 08, 2005 at 09:59:46AM -0500, Brian Barrett wrote:
On Dec 8, 2005, at 9:27 AM, Gleb Natapov wrote:
On Wed, Dec 07, 2005 at 10:40:51AM -0500, Brian Barrett wrote:
Hopefully this made some sense. If not, on to the next round of e-
mails :).
This made allot of sense. What is compiled by default now is
malloc_hooks
I'll compile ptmalloc and play with it and may be then will be the
next
round :)
Further clarification... All this work was done on the trunk and
probably won't come over to the v1.0 branch in the short term
(definitely not for v1.0.1). The v1.0 series still defaults to using
the malloc_hooks and all that.
I work with trunk here, and I have a problem to run openib with
ptmalloc2. It seems ptmalloc2 doesn't define posix_memalign() function
and openib uses it, so libc version is called and this doens't work.
Included patch solves the problem but perhaps there is other solution.
Index: opal/mca/memory/ptmalloc2/malloc.c
===
--- opal/mca/memory/ptmalloc2/malloc.c (revision 8398)
+++ opal/mca/memory/ptmalloc2/malloc.c (working copy)
@@ -5431,12 +5431,11 @@
*/
-#ifdef _LIBC
-# include
+#include
/* We need a wrapper function for one of the additions of POSIX. */
int
-__posix_memalign (void **memptr, size_t alignment, size_t size)
+posix_memalign (void **memptr, size_t alignment, size_t size)
{
void *mem;
__malloc_ptr_t (*hook) __MALLOC_PMT ((size_t, size_t,
@@ -5464,6 +5463,7 @@
return ENOMEM;
}
+#ifdef _LIBC
weak_alias (__posix_memalign, posix_memalign)
strong_alias (__libc_calloc, __calloc) weak_alias (__libc_calloc,
calloc)
--
Gleb.
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [O-MPI devel] [PATH] ompi_info doesn't show use_mem_hooks flag
On Thu, 8 Dec 2005, Gleb Natapov wrote: On Wed, Dec 07, 2005 at 10:40:51AM -0500, Brian Barrett wrote: Hopefully this made some sense. If not, on to the next round of e- mails :). This made allot of sense. What is compiled by default now is malloc_hooks I'll compile ptmalloc and play with it and may be then will be the next round :) Hey Gleb, Just FYI, we are still doing a bit of work with the memory hooks and as of yesterday we were seeing a bug when using the sbrk hooks. I am working on this now and should have something more later today. - Galen Thanks, -- Gleb. ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [O-MPI devel] [PATH] ompi_info doesn't show use_mem_hooks flag
On Thu, Dec 08, 2005 at 09:59:46AM -0500, Brian Barrett wrote:
> On Dec 8, 2005, at 9:27 AM, Gleb Natapov wrote:
>
> > On Wed, Dec 07, 2005 at 10:40:51AM -0500, Brian Barrett wrote:
> >> Hopefully this made some sense. If not, on to the next round of e-
> >> mails :).
> >>
> > This made allot of sense. What is compiled by default now is
> > malloc_hooks
> > I'll compile ptmalloc and play with it and may be then will be the
> > next
> > round :)
>
> Further clarification... All this work was done on the trunk and
> probably won't come over to the v1.0 branch in the short term
> (definitely not for v1.0.1). The v1.0 series still defaults to using
> the malloc_hooks and all that.
>
I work with trunk here, and I have a problem to run openib with
ptmalloc2. It seems ptmalloc2 doesn't define posix_memalign() function
and openib uses it, so libc version is called and this doens't work.
Included patch solves the problem but perhaps there is other solution.
Index: opal/mca/memory/ptmalloc2/malloc.c
===
--- opal/mca/memory/ptmalloc2/malloc.c (revision 8398)
+++ opal/mca/memory/ptmalloc2/malloc.c (working copy)
@@ -5431,12 +5431,11 @@
*/
-#ifdef _LIBC
-# include
+#include
/* We need a wrapper function for one of the additions of POSIX. */
int
-__posix_memalign (void **memptr, size_t alignment, size_t size)
+posix_memalign (void **memptr, size_t alignment, size_t size)
{
void *mem;
__malloc_ptr_t (*hook) __MALLOC_PMT ((size_t, size_t,
@@ -5464,6 +5463,7 @@
return ENOMEM;
}
+#ifdef _LIBC
weak_alias (__posix_memalign, posix_memalign)
strong_alias (__libc_calloc, __calloc) weak_alias (__libc_calloc, calloc)
--
Gleb.
Re: [O-MPI devel] [PATH] ompi_info doesn't show use_mem_hooks flag
On Dec 8, 2005, at 9:27 AM, Gleb Natapov wrote: On Wed, Dec 07, 2005 at 10:40:51AM -0500, Brian Barrett wrote: Hopefully this made some sense. If not, on to the next round of e- mails :). This made allot of sense. What is compiled by default now is malloc_hooks I'll compile ptmalloc and play with it and may be then will be the next round :) Further clarification... All this work was done on the trunk and probably won't come over to the v1.0 branch in the short term (definitely not for v1.0.1). The v1.0 series still defaults to using the malloc_hooks and all that. Brian -- Brian Barrett Open MPI developer http://www.open-mpi.org/
Re: [O-MPI devel] [PATH] ompi_info doesn't show use_mem_hooks flag
On Wed, Dec 07, 2005 at 10:40:51AM -0500, Brian Barrett wrote: > Hopefully this made some sense. If not, on to the next round of e- > mails :). > This made allot of sense. What is compiled by default now is malloc_hooks I'll compile ptmalloc and play with it and may be then will be the next round :) Thanks, -- Gleb.
