Re: [OMPI users] www.open-mpi.org certificate error?

[Gilles Gouaillardet]

Jeff,

if my understanding is correct, https requires open-mpi.org is the only
(httpd) domain served on port 443 for a given IP (e.g. no shared hosting)
a certificate is based on host name (e.g. www.open-mpi.org)  and can
contains wildcards (e.g. *.open-mpi.org)
so if the first condition is met, then you should be able to reuse the
certificate that was previously used at UI.

makes sense ?

Cheers,

Gilles

On Sunday, July 31, 2016, Jeff Squyres (jsquyres) 
wrote:

> I knew about letsencrypt (it's sponsored by my own company, Cisco --
> huzzah!).  But I (apparently foolishly) didn't think SSL was important, and
> didn't want to bother with figuring out how to do all the SSL-sysadmin-ish
> things.  :-)
>
> I just poked around with letsencrypt.org; it looks actually pretty simple
> (even on a hosted site where we have limited ssh access to the web server
> itself -- I used https://github.com/Neilpang/acme.sh and it worked like a
> champ).
>
> PSA: If you have an http web site, you should go look at letsencrypt.org.
>
> I'll look at getting www.open-mpi.org back to https shortly.
>
>
>
>
> > On Jul 30, 2016, at 12:51 PM, Craig Inches  > wrote:
> >
> > There is a free service for certificates, two that I know of infact.
> >
> > https://www.startssl.com/ and https://letsencrypt.org/
> >
> > Startssl is more your tradition cert request process and lets encrypt is
> a project for automated free certificates but if sysadmin'ing is not your
> primary thing then I would say go with Start! I use them for all my sites.
> >
> > Also Durga, the SSL is at a preceding step to the redirect, it is
> confirmed before establishing the http connection.
> >
> > Cheers, Craig
> >
> > On Sat, Jul 30, 2016 at 12:39:23PM -0400, dpchoudh . wrote:
> >
> > Hi Jeff and all Disclaimer: I know next to nothing about how the web
> works. Having said that, would it not be possible to redirect an https
> request to a http request? I believe apache mod-rewrite can do it. Or does
> this certificate check happens even before the rewrite? Regards Durga
> >
> > The woods are lovely, dark and deep; but I have promises to keep. And
> kilometers to go before I sleep; and kilometers to go before I sleep. On
> Sat, Jul 30, 2016 at 12:31 PM, Jeff Squyres (jsquyres) <[1]
> jsquy...@cisco.com > wrote:
> >
> > Meh.  That's a good point.  We might have to pony up the cost for
> > the certificates, then.  :-(
> > (Indiana University provided all this stuff to us for free; now that
> > the community has to pay for our own hosting, the funding has to
> > come from some where).
> > Please bear with us -- all this sysadmin/infrastructure stuff is
> > completely unrelated to do with our real jobs (i.e., software
> > development of Open MPI); we're doing all this migration work on
> > nights, weekends, and sometimes while waiting for lengthy
> > compiles.  We didn't think of the Google-will-have-https-links
> > issue.  :-\
> > > On Jul 30, 2016, at 12:27 PM, Bennet Fauber <[2]ben...@umich.edu
> >
> > wrote:
> > >
> > > Thanks, Jeff,
> > >
> > > Just to note, though, many, many links in Google searches will
> > have
> > > the https address.
> > >
> > > -- bennet
> > >
> > >
> > > On Sat, Jul 30, 2016 at 12:21 PM, Jeff Squyres (jsquyres)
> > > <[3]jsquy...@cisco.com > wrote:
> > >> Hmm.  Sorry about this; we just moved the web site from Indiana
> > University to Host Gator (per
> > [4]http://www.open-mpi.org/community/lists/devel/2016/06/19139.php).
> > >>
> > >> I thought I had disabled https for the web site last night when I
> > did the move -- I'll have to check into this.
> > >>
> > >> For the meantime, please just use [5]http://www.open-mpi.org/.
> > >>
> > >>
> > >>
> > >>> On Jul 30, 2016, at 11:25 AM, Bennet Fauber
> > <[6]ben...@umich.edu > wrote:
> > >>>
> > >>> I am getting a certificate error from
> > [7]https://www.open-mpi.org/
> > >>>
> > >>> The owner of [8]www.open-mpi.org has configured their website
> > improperly.
> > >>> To protect your information from being stolen, Firefox has not
> > >>> connected to this website.
> > >>>
> > >>> and if I go to advanced and ask about the certificate, it says
> > >>>
> > >>> The certificate is only valid for the following names:
> > >>> *.[9]hostgator.com, [10]hostgator.com
> > >>>
> > >>>
> > >>> Is this something I have done to myself?
> > >>> ___
> > >>> users mailing list
> > >>> [11]users@lists.open-mpi.org 
> > >>> [12]https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> > >>
> > >>
> > >> --
> > >> Jeff Squyres
> > >> [13]jsquy...@cisco.com 
> > >> For corporate legal information go to:
> > [14]http://www.cisco.com/web/about/doing_business/legal/cri/
> > >>
> > >> ___
> > >> users mailing list
> > >> [15]users@lists.open-mpi.org 
> > >> [16]https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> > > ___
> > > users mailing list
> > > [17]users@lists.ope

Re: [OMPI users] www.open-mpi.org certificate error?

[Gilles Gouaillardet]

a CSR is the signature of the server public key by the certificate
authority.

unless the same private key was used for https servers of open-mpi.org and
non open-mpi.org domains, I do not think IU providing the server key pair
is an issue.

Cheers,

Gilles

On Sunday, July 31, 2016, Bennet Fauber  wrote:

> Is the web server's private key, used to generate the CSR, also
> needed?  If so, perhaps IU cannot share that.
>
>
>
> On Sat, Jul 30, 2016 at 11:09 PM, Gilles Gouaillardet
> > wrote:
> > Jeff,
> >
> > if my understanding is correct, https requires open-mpi.org is the only
> > (httpd) domain served on port 443 for a given IP (e.g. no shared hosting)
> > a certificate is based on host name (e.g. www.open-mpi.org)  and can
> > contains wildcards (e.g. *.open-mpi.org)
> > so if the first condition is met, then you should be able to reuse the
> > certificate that was previously used at UI.
> >
> > makes sense ?
> >
> > Cheers,
> >
> > Gilles
> >
> > On Sunday, July 31, 2016, Jeff Squyres (jsquyres)  >
> > wrote:
> >>
> >> I knew about letsencrypt (it's sponsored by my own company, Cisco --
> >> huzzah!).  But I (apparently foolishly) didn't think SSL was important,
> and
> >> didn't want to bother with figuring out how to do all the
> SSL-sysadmin-ish
> >> things.  :-)
> >>
> >> I just poked around with letsencrypt.org; it looks actually pretty
> simple
> >> (even on a hosted site where we have limited ssh access to the web
> server
> >> itself -- I used https://github.com/Neilpang/acme.sh and it worked
> like a
> >> champ).
> >>
> >> PSA: If you have an http web site, you should go look at
> letsencrypt.org.
> >>
> >> I'll look at getting www.open-mpi.org back to https shortly.
> >>
> >>
> >>
> >>
> >> > On Jul 30, 2016, at 12:51 PM, Craig Inches  > wrote:
> >> >
> >> > There is a free service for certificates, two that I know of infact.
> >> >
> >> > https://www.startssl.com/ and https://letsencrypt.org/
> >> >
> >> > Startssl is more your tradition cert request process and lets encrypt
> is
> >> > a project for automated free certificates but if sysadmin'ing is not
> your
> >> > primary thing then I would say go with Start! I use them for all my
> sites.
> >> >
> >> > Also Durga, the SSL is at a preceding step to the redirect, it is
> >> > confirmed before establishing the http connection.
> >> >
> >> > Cheers, Craig
> >> >
> >> > On Sat, Jul 30, 2016 at 12:39:23PM -0400, dpchoudh . wrote:
> >> >
> >> > Hi Jeff and all Disclaimer: I know next to nothing about how the web
> >> > works. Having said that, would it not be possible to redirect an https
> >> > request to a http request? I believe apache mod-rewrite can do it. Or
> does
> >> > this certificate check happens even before the rewrite? Regards Durga
> >> >
> >> > The woods are lovely, dark and deep; but I have promises to keep. And
> >> > kilometers to go before I sleep; and kilometers to go before I sleep.
> On
> >> > Sat, Jul 30, 2016 at 12:31 PM, Jeff Squyres (jsquyres)
> >> > <[1]jsquy...@cisco.com > wrote:
> >> >
> >> > Meh.  That's a good point.  We might have to pony up the cost for
> >> > the certificates, then.  :-(
> >> > (Indiana University provided all this stuff to us for free; now that
> >> > the community has to pay for our own hosting, the funding has to
> >> > come from some where).
> >> > Please bear with us -- all this sysadmin/infrastructure stuff is
> >> > completely unrelated to do with our real jobs (i.e., software
> >> > development of Open MPI); we're doing all this migration work on
> >> > nights, weekends, and sometimes while waiting for lengthy
> >> > compiles.  We didn't think of the Google-will-have-https-links
> >> > issue.  :-\
> >> > > On Jul 30, 2016, at 12:27 PM, Bennet Fauber <[2]ben...@umich.edu
> >
> >> > wrote:
> >> > >
> >> > > Thanks, Jeff,
> >> > >
> >> > > Just to note, though, many, many links in Google searches will
> >> > have
> >> > > the https address.
> >> > >
> >> > > -- bennet
> >> > >
> >> > &g

Re: [OMPI users] open-mpi: all-recursive error when compiling

[Gilles Gouaillardet]

The error message is related to a permission issue (which is very 
puzzling in itself ...)


can you manually check the permissions ?

cd /home/pi/Downloads/openmpi-2.0.0/opal/asm

ls -l .deps/atomic-asm.Tpo atomic-asm.S


then you can

make clean

make V=1 atomic-asm.lo


and post the output


meanwhile, you can check your umask is correct (so files are created 
with the right permissions).


note unless you plan to install openmpi in /usr/local (e.g. sudo make 
install, after make)


you need to ./configure --prefix=

i also recommend you also --enable-mpirun-prefix-by-default


Cheers,


Gilles


On 8/5/2016 9:09 AM, Christiano SA wrote:

Hi,
I've downloaded the openmpi-2.0.0.tar.gz and done this:
$ tar -zxvf openmpi-2.0.0.tar.gz
$ cd openmpi-2.0.0
$ ./configure
(...)
$ make
but an error is happening:
(...)
Making all in include
make[2]: Entering directory 
'/home/pi/Downloads/openmpi-2.0.0/opal/include'

make  all-am
make[3]: Entering directory 
'/home/pi/Downloads/openmpi-2.0.0/opal/include'

make[3]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal/include'
make[2]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal/include'
Making all in asm
make[2]: Entering directory '/home/pi/Downloads/openmpi-2.0.0/opal/asm'
  CPPASatomic-asm.lo
atomic-asm.S:1:0: fatal error: opening dependency file 
.deps/atomic-asm.Tpo: Permission denied

.text
 ^
compilation terminated.
make[2]: *** [Makefile:1735: atomic-asm.lo] Error 1
make[2]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal/asm'
make[1]: *** [Makefile:2301: all-recursive] Error 1
make[1]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal'
make: *** [Makefile:1800: all-recursive] Error 1
I use raspberry pi 2 (raspbian) as my desktop computer, however, I've 
done a test in my Solaris 13.1 x86 with Oracle Developer Studio 12.5 
and the result is similar: an error around "all-recursive". I alread 
tried install the gnu-make lastest version and didn't work, same thing 
with dmake and gmake: error around "all-recursive".
More information: The .deb (raspbian) packages are working and 
packages to solaris too, however I would like to compile the lastest 
version.

What did I do wrong?


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] open-mpi: all-recursive error when compiling

[Gilles Gouaillardet]

b'
atomic-asm.S:86: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:130: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:136: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'

make[2]: *** [Makefile:1735: atomic-asm.lo] Error 1
make[2]: Leaving directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/opal/asm'

make[1]: *** [Makefile:2301: all-recursive] Error 1
make[1]: Leaving directory '/home/pi/Downloads/TEST/openmpi-2.0.0/opal'
make: *** [Makefile:1800: all-recursive] Error 1
* 3 - Hypotheses ==*
*Solaris*
I'm not expert but searching "__curbrk" I discover it belongs to glibc
http://stackoverflow.com/questions/6210685/explanation-for-this-assembly-code
And solaris doesn't use glibc, according with:
http://unix.stackexchange.com/questions/34650/is-the-solaris-libc-based-on-the-gnu-libc
*Raspbian*
Apparently my processor armv7 does not support some instructions (?).
---
Dpchoudh,
Raspbian don't have selinux in default installation (At least there is 
no explicit intuitive command indicating this.)

*Sent:* Thursday, August 04, 2016 at 9:33 PM
*From:* "Gilles Gouaillardet" 
*To:* "Open MPI Users" 
*Subject:* Re: [OMPI users] open-mpi: all-recursive error when compiling

The error message is related to a permission issue (which is very 
puzzling in itself ...)


can you manually check the permissions ?

cd /home/pi/Downloads/openmpi-2.0.0/opal/asm

ls -l .deps/atomic-asm.Tpo atomic-asm.S

then you can

make clean

make V=1 atomic-asm.lo

and post the output

meanwhile, you can check your umask is correct (so files are created 
with the right permissions).


note unless you plan to install openmpi in /usr/local (e.g. sudo make 
install, after make)


you need to ./configure --prefix=

i also recommend you also --enable-mpirun-prefix-by-default

Cheers,

Gilles

On 8/5/2016 9:09 AM, Christiano SA wrote:

Hi,
I've downloaded the openmpi-2.0.0.tar.gz and done this:
$ tar -zxvf openmpi-2.0.0.tar.gz
$ cd openmpi-2.0.0
$ ./configure
(...)
$ make
but an error is happening:
(...)
Making all in include
make[2]: Entering directory
'/home/pi/Downloads/openmpi-2.0.0/opal/include'
make  all-am
make[3]: Entering directory
'/home/pi/Downloads/openmpi-2.0.0/opal/include'
make[3]: Leaving directory
'/home/pi/Downloads/openmpi-2.0.0/opal/include'
make[2]: Leaving directory
'/home/pi/Downloads/openmpi-2.0.0/opal/include'
Making all in asm
make[2]: Entering directory
'/home/pi/Downloads/openmpi-2.0.0/opal/asm'
  CPPASatomic-asm.lo
atomic-asm.S:1:0: fatal error: opening dependency file
.deps/atomic-asm.Tpo: Permission denied
.text
 ^
compilation terminated.
make[2]: *** [Makefile:1735: atomic-asm.lo] Error 1
make[2]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal/asm'
make[1]: *** [Makefile:2301: all-recursive] Error 1
make[1]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal'
make: *** [Makefile:1800: all-recursive] Error 1
I use raspberry pi 2 (raspbian) as my desktop computer, however,
I've done a test in my Solaris 13.1 x86 with Oracle Developer
Studio 12.5 and the result is similar: an error around
"all-recursive". I alread tried install the gnu-make lastest
version and didn't work, same thing with dmake and gmake: error
around "all-recursive".
More information: The .deb (raspbian) packages are working and
packages to solaris too, however I would like to compile the
lastest version.
What did I do wrong?

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___ users mailing list 
users@lists.open-mpi.org 
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] open-mpi: all-recursive error when compiling

[Gilles Gouaillardet]

deps/&|;s|\.lo$||'`;\
/bin/sh ../../libtool --mode=compile gcc -std=gnu99 
-DHAVE_CONFIG_H  -I. -I../../opal/include -I../../ompi/include 
-I../../oshmem/include 
-I../../opal/mca/hwloc/hwloc1112/hwloc/include/private/autogen 
-I../../opal/mca/hwloc/hwloc1112/hwloc/include/hwloc/autogen 
-I../../ompi/mpiext/cuda/c   -I../.. -I../../orte/include  
-D_REENTRANT 
-I/export/home/solaris/Downloads/Test_03/openmpi-2.0.0/opal/mca/hwloc/hwloc1112/hwloc/include 
-I/export/home/solaris/Downloads/Test_03/openmpi-2.0.0/opal/mca/event/libevent2022/libevent 
-I/export/home/solaris/Downloads/Test_03/openmpi-2.0.0/opal/mca/event/libevent2022/libevent/include 
-O3 -DNDEBUG -D_REENTRANT -finline-functions -fno-strict-aliasing -MT 
atomic-asm.lo -MD -MP -MF $depbase.Tpo -c -o atomic-asm.lo 
atomic-asm.S &&\

mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. 
-I../../opal/include -I../../ompi/include -I../../oshmem/include 
-I../../opal/mca/hwloc/hwloc1112/hwloc/include/private/autogen 
-I../../opal/mca/hwloc/hwloc1112/hwloc/include/hwloc/autogen 
-I../../ompi/mpiext/cuda/c -I../.. -I../../orte/include -D_REENTRANT 
-I/export/home/solaris/Downloads/Test_03/openmpi-2.0.0/opal/mca/hwloc/hwloc1112/hwloc/include 
-I/export/home/solaris/Downloads/Test_03/openmpi-2.0.0/opal/mca/event/libevent2022/libevent 
-I/export/home/solaris/Downloads/Test_03/openmpi-2.0.0/opal/mca/event/libevent2022/libevent/include 
-O3 -DNDEBUG -D_REENTRANT -finline-functions -fno-strict-aliasing -MT 
atomic-asm.lo -MD -MP -MF .deps/atomic-asm.Tpo -c atomic-asm.S  -fPIC 
-DPIC -o .libs/atomic-asm.o

* 2 - Raspbian armv7 =*
$ ./configure
$ make
Making all in config
make[1]: Entering directory '/home/pi/Downloads/TEST/openmpi-2.0.0/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/pi/Downloads/TEST/openmpi-2.0.0/config'
Making all in contrib
make[1]: Entering directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/contrib'

make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/pi/Downloads/TEST/openmpi-2.0.0/contrib'
Making all in opal
make[1]: Entering directory '/home/pi/Downloads/TEST/openmpi-2.0.0/opal'
Making all in include
make[2]: Entering directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/opal/include'

make  all-am
make[3]: Entering directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/opal/include'
make[3]: Leaving directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/opal/include'
make[2]: Leaving directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/opal/include'

Making all in asm
make[2]: Entering directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/opal/asm'

  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-local.s" atomic-asm.S
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:70: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:86: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:130: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:136: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'

make[2]: *** [Makefile:1735: atomic-asm.lo] Error 1
make[2]: Leaving directory 
'/home/pi/Downloads/TEST/openmpi-2.0.0/opal/asm'

make[1]: *** [Makefile:2301: all-recursive] Error 1
make[1]: Leaving directory '/home/pi/Downloads/TEST/openmpi-2.0.0/opal'
make: *** [Makefile:1800: all-recursive] Error 1
* 3 - Hypotheses ==*
*Solaris*
I'm not expert but searching "__curbrk" I discover it belongs to glibc
http://stackoverflow.com/questions/6210685/explanation-for-this-assembly-code
And solaris doesn't use glibc, according with:
http://unix.stackexchange.com/questions/34650/is-the-solaris-libc-based-on-the-gnu-libc
*Raspbian*
Apparently my processor armv7 does not support some instructions (?).
---
Dpchoudh,
Raspbian don't have selinux in default installation (At least there is 
no explicit intuitive command indicating this.)

*Sent:* Thursday, Augu

Re: [OMPI users] Multiple connections to MySQL vi MPI

[Gilles Gouaillardet]

What if you run

mpirun -np 1 mysqlconnect

on your frontend (aka compilation and/or submission) host ?

does it work as expected ?


if yes, then this likely indicates a MySQL permission/configuration issue.

for example, it accepts connections from 'someUser' only from one node,

or maybe mysqld is running on your frontend and only accepts connections 
from the unix socket


(and you need to enable TCP (at least) if you want to allow connections 
from remote hosts)



Cheers,


Gilles


On 8/9/2016 8:43 AM, alev mutlu wrote:

Hello,

I am trying to connect to MySQL with the following

if (!mysql_real_connect(connection,server,user,password,database, 0, 
NULL, 0)) {

  printf("Conection error : %s\n", mysql_error(connection));
  exit(1);
}

The code compiles with g++ and works fine.

It compiles with mpicxx as well but when run through pbs script

mpirun -np1-hostfile$PBS_NODEFILEmysqlconnect


I get a run time error

Conection error : Access denied for user 'someUser'@'node2' (using 
password: YES)


I guess mysql itself adds the @ "node2" and my connection fails.

How can I prevent this, any suggestions?

thanks in advance

_alev


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] which info is needed for SIGSEGV in Java foropenmpi-dev-124-g91e9686 on Solaris

[Gilles Gouaillardet]

Siegmar,

how did you configure openmpi ? which java version did you use ?

i just found a regression and you currently have to explicitly add
CFLAGS=-D_REENTRANT CPPFLAGS=-D_REENTRANT
to your configure command line

if you want to debug this issue (i cannot reproduce it on a solaris 11
x86 virtual machine)
you can apply the attached patch, and make sure you configure with
--enable-debug and run

OMPI_ATTACH=1 mpiexec -n 1 java InitFinalizeMain

then you will need to attach the *java* process with gdb, set the _dbg
local variable to zero and continue
you should get a clean stack trace and hopefully we will be able to help

Cheers,

Gilles

On 2014/10/24 0:03, Siegmar Gross wrote:
> Hello Oscar,
>
> do you have time to look into my problem? Probably Takahiro has a
> point and gdb behaves differently on Solaris and Linux, so that
> the differing outputs have no meaning. I tried to debug my Java
> program, but without success so far, because I wasn't able to get
> into the Java program to set a breakpoint or to see the code. Have
> you succeeded to debug a mpiJava program? If so, how must I call
> gdb (I normally use "gdb mipexec" and then "run -np 1 java ...")?
> What can I do to get helpful information to track the error down?
> I have attached the error log file. Perhaps you can see if something
> is going wrong with the Java interface. Thank you very much for your
> help and any hints for the usage of gdb with mpiJava in advance.
> Please let me know if I can provide anything else.
>
>
> Kind regards
>
> Siegmar
>
>
>>> I think that it must have to do with MPI, because everything
>>> works fine on Linux and my Java program works fine with an older
>>> MPI version (openmpi-1.8.2a1r31804) as well.
>> Yes. I also think it must have to do with MPI.
>> But java process side, not mpiexec process side.
>>
>> When you run Java MPI program via mpiexec, a mpiexec process
>> process launch a java process. When the java process (your
>> Java program) calls a MPI method, native part (written in C/C++)
>> of the MPI library is called. It runs in java process, not in
>> mpiexec process. I suspect that part.
>>
>>> On Solaris things are different.
>> Are you saying the following difference?
>> After this line,
>>> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT);
>> Linux shows
>>> orte_job_state_to_str (state=1)
>>> at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217
>>> 217 switch(state) {
>> but Solaris shows
>>> orte_util_print_name_args (name=0x100118380 )
>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122
>>> 122 if (NULL == name) {
>> Each macro is defined as:
>>
>> #define ORTE_ACTIVATE_JOB_STATE(j, s)   \
>> do {\
>> orte_job_t *shadow=(j); \
>> opal_output_verbose(1, orte_state_base_framework.framework_output, \
>> "%s ACTIVATE JOB %s STATE %s AT %s:%d",  \
>> ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), \
>> (NULL == shadow) ? "NULL" : \
>> ORTE_JOBID_PRINT(shadow->jobid), \
>> orte_job_state_to_str((s)), \
>> __FILE__, __LINE__); \
>> orte_state.activate_job_state(shadow, (s)); \
>> } while(0);
>>
>> #define ORTE_NAME_PRINT(n) \
>> orte_util_print_name_args(n)
>>
>> #define ORTE_JOBID_PRINT(n) \
>> orte_util_print_jobids(n)
>>
>> I'm not sure, but I think the gdb on Solaris steps into
>> orte_util_print_name_args, but gdb on Linux doesn't step into
>> orte_util_print_name_args and orte_util_print_jobids for some
>> reason, or orte_job_state_to_str is evaluated before them.
>>
>> So I think it's not an important difference.
>>
>> You showed the following lines.
> orterun (argc=5, argv=0x7fffe0d8)
> at 
> ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084
> 1084while (orte_event_base_active) {
> (gdb) 
> 1085opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
> (gdb) 
>> I'm not familiar with this code but I think this part (in mpiexec
>> process) is only waiting the java process to terminate (normally
>> or abnormally). So I think the problem is not in a mpiexec process
>> but in a java process.
>>
>> Regards,
>> Takahiro
>>
>>> Hi Takahiro,
>>>
 mpiexec and java run as distinct processes. Your JRE message
 says java process raises SEGV. So you should trace the java
 process, not the mpiexec process. And more, your JRE message
 says the crash happened outside the Java Virtual Machine in
 native code. So usual Java program debugger is useless.
 You should trace native code part of the java process.
 Unfortunately I don't know ho

Re: [OMPI users] OMPI users] low CPU utilization with OpenMPI

[Gilles Gouaillardet]

Can you also check there is no cpu binding issue (several mpi tasks and/or 
OpenMP threads if any, bound to the same core and doing time sharing ?
A simple way to check that is to log into a compute node, run top and then 
press 1 f j
If some cores have higher usage than others, you are likely doing time sharing.
An other option is to disable cpu binding (ompi and openmp if any) and see if 
things get better
(This is suboptimal but still better than time sharing)

"Jeff Squyres (jsquyres)"  wrote:
>- Is /tmp on that machine on NFS or local?
>
>- Have you looked at the text of the help message that came out before the "9 
>more processes have sent help message help-opal-shmem-mmap.txt / mmap on nfs" 
>message?  It should contain details about what the problematic NFS directory 
>is.
>
>- Do you know that it's MPI that is causing this low CPU utilization?
>
>- You mentioned other MPI implementations; have you tested with them to see if 
>they get better CPU utilization?
>
>- What happens if you run this application on a single machine, with no 
>network messaging?
>
>- Do you know what specifically in your application is slow?  I.e., have you 
>done any instrumentation to see what steps / API calls are running slowly, and 
>then tried to figure out why?
>
>- Do you have blocking message patterns that might operate well in shared 
>memory, but expose the inefficiencies of its algorithms/design when it moves 
>to higher-latency transports?
>
>- How long does your application run for?
>
>I ask these questions because MPI applications tend to be quite complicated. 
>Sometimes it's the application itself that is the cause of slowdown / 
>inefficiencies.
>
>
>
>On Oct 23, 2014, at 9:29 PM, Vinson Leung  wrote:
>
>> Later I change another machine and set the TMPDIR to default /tmp, but the 
>> problem (low CPU utilization under 20%) still occur :<
>> 
>> Vincent
>> 
>> On Thu, Oct 23, 2014 at 10:38 PM, Jeff Squyres (jsquyres) 
>>  wrote:
>> If normal users can't write to /tmp (or if /tmp is an NFS-mounted 
>> filesystem), that's the underlying problem.
>> 
>> @Vinson -- you should probably try to get that fixed.
>> 
>> 
>> 
>> On Oct 23, 2014, at 10:35 AM, Joshua Ladd  wrote:
>> 
>> > It's not coming from OSHMEM but from the OPAL "shmem" framework. You are 
>> > going to get terrible performance - possibly slowing to a crawl having all 
>> > processes open their backing files for mmap on NSF. I think that's the 
>> > error that he's getting.
>> >
>> >
>> > Josh
>> >
>> > On Thu, Oct 23, 2014 at 6:06 AM, Vinson Leung  
>> > wrote:
>> > HI, Thanks for your reply:)
>> > I really run an MPI program (compile with OpenMPI and run with "mpirun -n 
>> > 8 .."). My OpenMPI version is 1.8.3 and my program is Gromacs. BTW, 
>> > what is OSHMEM ?
>> >
>> > Best
>> > Vincent
>> >
>> > On Thu, Oct 23, 2014 at 12:21 PM, Ralph Castain  wrote:
>> > From your error message, I gather you are not running an MPI program, but 
>> > rather an OSHMEM one? Otherwise, I find the message strange as it only 
>> > would be emitted from an OSHMEM program.
>> >
>> > What version of OMPI are you trying to use?
>> >
>> >> On Oct 22, 2014, at 7:12 PM, Vinson Leung  wrote:
>> >>
>> >> Thanks for your reply:)
>> >> Follow your advice I tried to set the TMPDIR to /var/tmp and /dev/shm and 
>> >> even reset to /tmp (I get the system permission), the problem still occur 
>> >> (CPU utilization still lower than 20%). I have no idea why and ready to 
>> >> give up OpenMPI instead of using other MPI library.
>> >>
>> >> Old Message-
>> >>
>> >> Date: Tue, 21 Oct 2014 22:21:31 -0400
>> >> From: Brock Palen 
>> >> To: Open MPI Users 
>> >> Subject: Re: [OMPI users] low CPU utilization with OpenMPI
>> >> Message-ID: 
>> >> Content-Type: text/plain; charset=us-ascii
>> >>
>> >> Doing special files on NFS can be weird,  try the other /tmp/ locations:
>> >>
>> >> /var/tmp/
>> >> /dev/shm  (ram disk careful!)
>> >>
>> >> Brock Palen
>> >> www.umich.edu/~brockp
>> >> CAEN Advanced Computing
>> >> XSEDE Campus Champion
>> >> bro...@umich.edu
>> >> (734)936-1985
>> >>
>> >>
>> >>
>> >> > On Oct 21, 2014, at 10:18 PM, Vinson Leung  
>> >> > wrote:
>> >> >
>> >> > Because of permission reason (OpenMPI can not write temporary file to 
>> >> > the default /tmp directory), I change the TMPDIR to my local directory 
>> >> > (export TMPDIR=/home/user/tmp ) and then the MPI program can run. But 
>> >> > the CPU utilization is very low under 20% (8 MPI rank running in Intel 
>> >> > Xeon 8-core CPU).
>> >> >
>> >> > And I also got some message when I run with OpenMPI:
>> >> > [cn3:28072] 9 more processes have sent help message 
>> >> > help-opal-shmem-mmap.txt / mmap on nfs
>> >> > [cn3:28072] Set MCA parameter "orte_base_help_aggregate" to 0 to see 
>> >> > all help / error messages
>> >> >
>> >> > Any idea?
>> >> > Thanks
>> >> >
>> >> > VIncent
>
>
>-- 
>Jeff Squyres
>jsquy...@cisco.com
>For corporate legal information go to: 
>http://www.cisco.com

Re: [OMPI users] OMPI users] which info is needed for SIGSEGV in Java foropenmpi-dev-124-g91e9686on Solaris

[Gilles Gouaillardet]

Hi Siegmar,

You might need to configure with --enable-debug and add -g -O0 to your CFLAGS 
and LDFLAGS

Then once you attach with gdb, you have to find the thread that is polling :
thread 1
bt
thread 2
bt
and so on until you find the good thread
If _dbg is a local variable, you need to select the right frame before you can 
change the value :
get the frame number from bt (generally 1 under linux)
f 
set _dbg=0

I hope this helps

Gilles


Siegmar Gross  wrote:
>Hi Gilles,
>
>I changed _dbg to a static variable, so that it is visible in the
>library, but unfortunately still not in the symbol table.
>
>
>tyr java 419 nm /usr/local/openmpi-1.9.0_64_gcc/lib64/libmpi_java.so | grep -i 
>_dbg
>[271]   |  1249644| 4|OBJT |LOCL |0|18 |_dbg.14258
>tyr java 420 /usr/local/gdb-7.6.1_64_gcc/bin/gdb
>GNU gdb (GDB) 7.6.1
>...
>(gdb) attach 13019
>Attaching to process 13019
>[New process 13019]
>Retry #1:
>Retry #2:
>Retry #3:
>Retry #4:
>0x7eadcb04 in ?? ()
>(gdb) symbol-file /usr/local/openmpi-1.9.0_64_gcc/lib64/libmpi_java.so
>Reading symbols from 
>/export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libmpi_java.so.0.0.0...done.
>(gdb) set var _dbg.14258=0
>No symbol "_dbg" in current context.
>(gdb) 
>
>
>Kind regards
>
>Siegmar
>
>
>
>
>> unfortunately I didn't get anything useful. It's probably my fault,
>> because I'm still not very familiar with gdb or any other debugger.
>> I did the following things.
>> 
>> 
>> 1st window:
>> ---
>> 
>> tyr java 174 setenv OMPI_ATTACH 1
>> tyr java 175 mpijavac InitFinalizeMain.java 
>> warning: [path] bad path element
>>   "/usr/local/openmpi-1.9.0_64_gcc/lib64/shmem.jar":
>>   no such file or directory
>> 1 warning
>> tyr java 176 mpiexec -np 1 java InitFinalizeMain
>> 
>> 
>> 
>> 2nd window:
>> ---
>> 
>> tyr java 379 ps -aef | grep java
>> noaccess  1345 1   0   May 22 ? 113:23 /usr/java/bin/java 
>> -server -Xmx128m -XX:+UseParallelGC 
>-XX:ParallelGCThreads=4 
>>   fd1026  3661 10753   0 14:09:12 pts/14  0:00 mpiexec -np 1 java 
>> InitFinalizeMain
>>   fd1026  3677 13371   0 14:16:55 pts/2   0:00 grep java
>>   fd1026  3663  3661   0 14:09:12 pts/14  0:01 java -cp 
>/home/fd1026/work/skripte/master/parallel/prog/mpi/java:/usr/local/jun
>> tyr java 380 /usr/local/gdb-7.6.1_64_gcc/bin/gdb
>> GNU gdb (GDB) 7.6.1
>> ...
>> (gdb) attach 3663
>> Attaching to process 3663
>> [New process 3663]
>> Retry #1:
>> Retry #2:
>> Retry #3:
>> Retry #4:
>> 0x7eadcb04 in ?? ()
>> (gdb) symbol-file /usr/local/openmpi-1.9.0_64_gcc/lib64/libmpi_java.so
>> Reading symbols from 
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libmpi_java.so.0.0.0...done.
>> (gdb) set var _dbg=0
>> No symbol "_dbg" in current context.
>> (gdb) set var JNI_OnLoad::_dbg=0
>> No symbol "_dbg" in specified context.
>> (gdb) set JNI_OnLoad::_dbg=0
>> No symbol "_dbg" in specified context.
>> (gdb) info threads
>> [New LWP 12]
>> [New LWP 11]
>> [New LWP 10]
>> [New LWP 9]
>> [New LWP 8]
>> [New LWP 7]
>> [New LWP 6]
>> [New LWP 5]
>> [New LWP 4]
>> [New LWP 3]
>> [New LWP 2]
>>   Id   Target Id Frame 
>>   12   LWP 2 0x7eadc6b0 in ?? ()
>>   11   LWP 3 0x7eadcbb8 in ?? ()
>>   10   LWP 4 0x7eadcbb8 in ?? ()
>>   9LWP 5 0x7eadcbb8 in ?? ()
>>   8LWP 6 0x7eadcbb8 in ?? ()
>>   7LWP 7 0x7eadcbb8 in ?? ()
>>   6LWP 8 0x7ead8b0c in ?? ()
>>   5LWP 9 0x7eadcbb8 in ?? ()
>>   4LWP 100x7eadcbb8 in ?? ()
>>   3LWP 110x7eadcbb8 in ?? ()
>>   2LWP 120x7eadcbb8 in ?? ()
>> * 1LWP 1 0x7eadcb04 in ?? ()
>> (gdb) 
>> 
>> 
>> 
>> It seems that "_dbg" is unknown and unavailable.
>> 
>> tyr java 399 grep _dbg 
>> /export2/src/openmpi-1.9/openmpi-dev-124-g91e9686/ompi/mpi/java/c/*
>> /export2/src/openmpi-1.9/openmpi-dev-124-g91e9686/ompi/mpi/java/c/mpi_MPI.c: 
>>volatile int _dbg = 1;
>> /export2/src/openmpi-1.9/openmpi-dev-124-g91e9686/ompi/mpi/java/c/mpi_MPI.c: 
>>while (_dbg) poll(NULL, 0, 1);
>> tyr java 400 nm /usr/local/openmpi-1.9.0_64_gcc/lib64/*.so | grep -i _dbg
>> tyr java 401 nm /usr/local/openmpi-1.9.0_64_gcc/lib64/*.so | grep -i 
>> JNI_OnLoad
>> [1057]  |  139688| 444|FUNC |GLOB |0|11 
>> |JNI_OnLoad
>> tyr java 402 
>> 
>> 
>> 
>> How can I set _dbg to zero to continue mpiexec? I also tried to
>> set a breakpoint for function JNI_OnLoad, but it seems, that the
>> function isn't called before SIGSEGV.
>> 
>> 
>> tyr java 177 unsetenv OMPI_ATTACH 
>> tyr java 178 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
>> GNU gdb (GDB) 7.6.1
>> ...
>> (gdb) b mpi_MPI.c:JNI_OnLoad
>> No source file named mpi_MPI.c.
>> Make breakpoint pending on future shared library load? (y or [n]) y
>> 
>> Breakpoint 1 (mpi_MPI.c:JNI_OnLoad) pending.
>> (gdb) run -np 1 java InitFinalizeMain 
>> Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpie

Re: [OMPI users] OMPI users] OMPI users] which info is needed for SIGSEGV inJava foropenmpi-dev-124-g91e9686on Solaris

[Gilles Gouaillardet]

It looks like we faced a similar issue :
opal_process_name_t is 64 bits aligned wheteas orte_process_name_t is 32 bits 
aligned. If you run an alignment sensitive cpu such as sparc and you are not 
lucky (so to speak) you can run into this issue.
i will make a patch for this shortly

Ralph Castain  wrote:
>Afraid this must be something about the Sparc - just ran on a Solaris 11 x86 
>box and everything works fine.
>
>
>> On Oct 26, 2014, at 8:22 AM, Siegmar Gross 
>>  wrote:
>> 
>> Hi Gilles,
>> 
>> I wanted to explore which function is called, when I call MPI_Init
>> in a C program, because this function should be called from a Java
>> program as well. Unfortunately C programs break with a Bus Error
>> once more for openmpi-dev-124-g91e9686 on Solaris. I assume that's
>> the reason why I get no useful backtrace for my Java program.
>> 
>> tyr small_prog 117 mpicc -o init_finalize init_finalize.c
>> tyr small_prog 118 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
>> ...
>> (gdb) run -np 1 init_finalize
>> Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec -np 1 
>> init_finalize
>> [Thread debugging using libthread_db enabled]
>> [New Thread 1 (LWP 1)]
>> [New LWP2]
>> [tyr:19240] *** Process received signal ***
>> [tyr:19240] Signal: Bus Error (10)
>> [tyr:19240] Signal code: Invalid address alignment (1)
>> [tyr:19240] Failing at address: 7bd1c10c
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:opal_backtrace_print+0x2c
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:0xdcc04
>> /lib/sparcv9/libc.so.1:0xd8b98
>> /lib/sparcv9/libc.so.1:0xcc70c
>> /lib/sparcv9/libc.so.1:0xcc918
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:opal_proc_set_name+0x1c
>>  [ Signal 10 (BUS)]
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/openmpi/mca_pmix_native.so:0x103e8
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/openmpi/mca_ess_pmi.so:0x33dc
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-rte.so.0.0.0:orte_init+0x67c
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libmpi.so.0.0.0:ompi_mpi_init+0x374
>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libmpi.so.0.0.0:PMPI_Init+0x2a8
>> /home/fd1026/work/skripte/master/parallel/prog/mpi/small_prog/init_finalize:main+0x20
>> /home/fd1026/work/skripte/master/parallel/prog/mpi/small_prog/init_finalize:_start+0x7c
>> [tyr:19240] *** End of error message ***
>> --
>> mpiexec noticed that process rank 0 with PID 0 on node tyr exited on signal 
>> 10 (Bus Error).
>> --
>> [LWP2 exited]
>> [New Thread 2]
>> [Switching to Thread 1 (LWP 1)]
>> sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be found to 
>> satisfy query
>> (gdb) bt
>> #0  0x7f6173d0 in rtld_db_dlactivity () from /usr/lib/sparcv9/ld.so.1
>> #1  0x7f6175a8 in rd_event () from /usr/lib/sparcv9/ld.so.1
>> #2  0x7f618950 in lm_delete () from /usr/lib/sparcv9/ld.so.1
>> #3  0x7f6226bc in remove_so () from /usr/lib/sparcv9/ld.so.1
>> #4  0x7f624574 in remove_hdl () from /usr/lib/sparcv9/ld.so.1
>> #5  0x7f61d97c in dlclose_core () from /usr/lib/sparcv9/ld.so.1
>> #6  0x7f61d9d4 in dlclose_intn () from /usr/lib/sparcv9/ld.so.1
>> #7  0x7f61db0c in dlclose () from /usr/lib/sparcv9/ld.so.1
>> #8  0x7ec87f60 in vm_close (loader_data=0x0, 
>> module=0x7c901fe0)
>>at ../../../openmpi-dev-124-g91e9686/opal/libltdl/loaders/dlopen.c:212
>> #9  0x7ec85534 in lt_dlclose (handle=0x100189b50)
>>at ../../../openmpi-dev-124-g91e9686/opal/libltdl/ltdl.c:1982
>> #10 0x7ecaabd4 in ri_destructor (obj=0x1001893a0)
>>at 
>> ../../../../openmpi-dev-124-g91e9686/opal/mca/base/mca_base_component_repository.c:382
>> #11 0x7eca9504 in opal_obj_run_destructors (object=0x1001893a0)
>>at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:446
>> #12 0x7ecaa474 in mca_base_component_repository_release (
>>component=0x7b1236f0 )
>>at 
>> ../../../../openmpi-dev-124-g91e9686/opal/mca/base/mca_base_component_repository.c:240
>> #13 0x7ecac774 in mca_base_component_unload (
>>component=0x7b1236f0 , output_id=-1)
>>at 
>> ../../../../openmpi-dev-124-g91e9686/opal/mca/base/mca_base_components_close.c:47
>> #14 0x7ecac808 in mca_base_component_close (
>>component=0x7b1236f0 , output_id=-1)
>>at 
>> ../../../../openmpi-dev-124-g91e9686/opal/mca/base/mca_base_components_close.c:60
>> #15 0x7ecac8dc in mca_base_components_close (output_id=-1, 
>>components=0x7f14ba58 , skip=0x0)
>>at 
>> ../../../../openmpi-dev-124-g91e9686/opal/mca/base/mca_base_components_close.c:86
>> #16 0x7ecac844 in mca_base_fram

Re: [OMPI users] OMPI users] OMPI users] OMPI users] which info is needed for SIGSEGV inJava foropenmpi-dev-124-g91e9686on Solaris

[Gilles Gouaillardet]

No :-(
I need some extra work to stop declaring orte_process_name_t and 
ompi_process_name_t variables.
#249 will make things much easier.
One option is to use opal_process_name_t everywhere or typedef orte and ompi 
types to the opal one.
An other (lightweight but error prone imho) is to change variable declaration 
only.
Any thought ?

Ralph Castain  wrote:
>Will PR#249 solve it? If so, we should just go with it as I suspect that is 
>the long-term solution.
>
>> On Oct 26, 2014, at 4:25 PM, Gilles Gouaillardet 
>>  wrote:
>> 
>> It looks like we faced a similar issue :
>> opal_process_name_t is 64 bits aligned wheteas orte_process_name_t is 32 
>> bits aligned. If you run an alignment sensitive cpu such as sparc and you 
>> are not lucky (so to speak) you can run into this issue.
>> i will make a patch for this shortly
>> 
>> Ralph Castain  wrote:
>>> Afraid this must be something about the Sparc - just ran on a Solaris 11 
>>> x86 box and everything works fine.
>>> 
>>> 
>>>> On Oct 26, 2014, at 8:22 AM, Siegmar Gross 
>>>>  wrote:
>>>> 
>>>> Hi Gilles,
>>>> 
>>>> I wanted to explore which function is called, when I call MPI_Init
>>>> in a C program, because this function should be called from a Java
>>>> program as well. Unfortunately C programs break with a Bus Error
>>>> once more for openmpi-dev-124-g91e9686 on Solaris. I assume that's
>>>> the reason why I get no useful backtrace for my Java program.
>>>> 
>>>> tyr small_prog 117 mpicc -o init_finalize init_finalize.c
>>>> tyr small_prog 118 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
>>>> ...
>>>> (gdb) run -np 1 init_finalize
>>>> Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec -np 1 
>>>> init_finalize
>>>> [Thread debugging using libthread_db enabled]
>>>> [New Thread 1 (LWP 1)]
>>>> [New LWP2]
>>>> [tyr:19240] *** Process received signal ***
>>>> [tyr:19240] Signal: Bus Error (10)
>>>> [tyr:19240] Signal code: Invalid address alignment (1)
>>>> [tyr:19240] Failing at address: 7bd1c10c
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:opal_backtrace_print+0x2c
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:0xdcc04
>>>> /lib/sparcv9/libc.so.1:0xd8b98
>>>> /lib/sparcv9/libc.so.1:0xcc70c
>>>> /lib/sparcv9/libc.so.1:0xcc918
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:opal_proc_set_name+0x1c
>>>>  [ Signal 10 (BUS)]
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/openmpi/mca_pmix_native.so:0x103e8
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/openmpi/mca_ess_pmi.so:0x33dc
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-rte.so.0.0.0:orte_init+0x67c
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libmpi.so.0.0.0:ompi_mpi_init+0x374
>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libmpi.so.0.0.0:PMPI_Init+0x2a8
>>>> /home/fd1026/work/skripte/master/parallel/prog/mpi/small_prog/init_finalize:main+0x20
>>>> /home/fd1026/work/skripte/master/parallel/prog/mpi/small_prog/init_finalize:_start+0x7c
>>>> [tyr:19240] *** End of error message ***
>>>> --
>>>> mpiexec noticed that process rank 0 with PID 0 on node tyr exited on 
>>>> signal 10 (Bus Error).
>>>> --
>>>> [LWP2 exited]
>>>> [New Thread 2]
>>>> [Switching to Thread 1 (LWP 1)]
>>>> sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be found to 
>>>> satisfy query
>>>> (gdb) bt
>>>> #0  0x7f6173d0 in rtld_db_dlactivity () from 
>>>> /usr/lib/sparcv9/ld.so.1
>>>> #1  0x7f6175a8 in rd_event () from /usr/lib/sparcv9/ld.so.1
>>>> #2  0x7f618950 in lm_delete () from /usr/lib/sparcv9/ld.so.1
>>>> #3  0x7f6226bc in remove_so () from /usr/lib/sparcv9/ld.so.1
>>>> #4  0x7f624574 in remove_hdl () from /usr/lib/sparcv9/ld.so.1
>>>> #5  0x7f61d97c in dlclose_core () from /usr/lib/sparcv9/ld.so.1
>>>> #6  0x7f61d9d4 in dlclose_intn () from /usr/lib/sparcv9/ld.so.1
>>>> #7  0x7f61db0c in dlclose () from 

Re: [OMPI users] OMPI users] OMPI users] OMPI users] which info is needed for SIGSEGV inJava foropenmpi-dev-124-g91e9686on Solaris

[Gilles Gouaillardet]

Ralph,

this is also a solution.
the pro is it seems more lightweight than PR #249
the two cons i can see are :
- opal_process_name_t alignment goes from 64 to 32 bits
- some functions (opal_hash_table_*) takes an uint64_t as argument so we
still need to use memcpy in order to
  * guarantee 64 bits alignment on some archs (such as sparc)
  * avoid ugly cast such as uint64_t id = *(uint64_t *)&process_name;

as far as i am concerned, i am fine with your proposed suggestion to
dump opal_identifier_t.

about the patch, did you mean you have something ready i can apply to my
PR ?
or do you expect me to do the changes (i am ok to do it if needed)

Cheers,

Gilles

On 2014/10/27 11:04, Ralph Castain wrote:
> Just took a glance thru 249 and have a few suggestions on it - will pass them 
> along tomorrow. I think the right solution is to (a) dump opal_identifier_t 
> in favor of using opal_process_name_t everywhere in the opal layer, (b) 
> typedef orte_process_name_t to opal_process_name_t, and (c) leave 
> ompi_process_name_t as typedef’d to the RTE component in the MPI layer. This 
> lets other RTEs decide for themselves how they want to handle it.
>
> If you add changes to your branch, I can pass you a patch with my suggested 
> alterations.
>
>> On Oct 26, 2014, at 5:55 PM, Gilles Gouaillardet 
>>  wrote:
>>
>> No :-(
>> I need some extra work to stop declaring orte_process_name_t and 
>> ompi_process_name_t variables.
>> #249 will make things much easier.
>> One option is to use opal_process_name_t everywhere or typedef orte and ompi 
>> types to the opal one.
>> An other (lightweight but error prone imho) is to change variable 
>> declaration only.
>> Any thought ?
>>
>> Ralph Castain  wrote:
>>> Will PR#249 solve it? If so, we should just go with it as I suspect that is 
>>> the long-term solution.
>>>
>>>> On Oct 26, 2014, at 4:25 PM, Gilles Gouaillardet 
>>>>  wrote:
>>>>
>>>> It looks like we faced a similar issue :
>>>> opal_process_name_t is 64 bits aligned wheteas orte_process_name_t is 32 
>>>> bits aligned. If you run an alignment sensitive cpu such as sparc and you 
>>>> are not lucky (so to speak) you can run into this issue.
>>>> i will make a patch for this shortly
>>>>
>>>> Ralph Castain  wrote:
>>>>> Afraid this must be something about the Sparc - just ran on a Solaris 11 
>>>>> x86 box and everything works fine.
>>>>>
>>>>>
>>>>>> On Oct 26, 2014, at 8:22 AM, Siegmar Gross 
>>>>>>  wrote:
>>>>>>
>>>>>> Hi Gilles,
>>>>>>
>>>>>> I wanted to explore which function is called, when I call MPI_Init
>>>>>> in a C program, because this function should be called from a Java
>>>>>> program as well. Unfortunately C programs break with a Bus Error
>>>>>> once more for openmpi-dev-124-g91e9686 on Solaris. I assume that's
>>>>>> the reason why I get no useful backtrace for my Java program.
>>>>>>
>>>>>> tyr small_prog 117 mpicc -o init_finalize init_finalize.c
>>>>>> tyr small_prog 118 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
>>>>>> ...
>>>>>> (gdb) run -np 1 init_finalize
>>>>>> Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec -np 1 
>>>>>> init_finalize
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> [New Thread 1 (LWP 1)]
>>>>>> [New LWP2]
>>>>>> [tyr:19240] *** Process received signal ***
>>>>>> [tyr:19240] Signal: Bus Error (10)
>>>>>> [tyr:19240] Signal code: Invalid address alignment (1)
>>>>>> [tyr:19240] Failing at address: 7bd1c10c
>>>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:opal_backtrace_print+0x2c
>>>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:0xdcc04
>>>>>> /lib/sparcv9/libc.so.1:0xd8b98
>>>>>> /lib/sparcv9/libc.so.1:0xcc70c
>>>>>> /lib/sparcv9/libc.so.1:0xcc918
>>>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:opal_proc_set_name+0x1c
>>>>>>  [ Signal 10 (BUS)]
>>>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/openmpi/mca_pmix_native.so:0x103e8
>>>>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib

Re: [OMPI users] which info is needed for SIGSEGV in Java foropenmpi-dev-124-g91e9686on Solaris

[Gilles Gouaillardet]

allel/prog/mpi/java:/usr/local/jun
>>>>>> tyr java 380 /usr/local/gdb-7.6.1_64_gcc/bin/gdb
>>>>>> GNU gdb (GDB) 7.6.1
>>>>>> ...
>>>>>> (gdb) attach 3663
>>>>>> Attaching to process 3663
>>>>>> [New process 3663]
>>>>>> Retry #1:
>>>>>> Retry #2:
>>>>>> Retry #3:
>>>>>> Retry #4:
>>>>>> 0x7eadcb04 in ?? ()
>>>>>> (gdb) symbol-file /usr/local/openmpi-1.9.0_64_gcc/lib64/libmpi_java.so
>>>>>> Reading symbols from 
>>> /export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libmpi_java.so.0.0.0...done.
>>>>>> (gdb) set var _dbg=0
>>>>>> No symbol "_dbg" in current context.
>>>>>> (gdb) set var JNI_OnLoad::_dbg=0
>>>>>> No symbol "_dbg" in specified context.
>>>>>> (gdb) set JNI_OnLoad::_dbg=0
>>>>>> No symbol "_dbg" in specified context.
>>>>>> (gdb) info threads
>>>>>> [New LWP 12]
>>>>>> [New LWP 11]
>>>>>> [New LWP 10]
>>>>>> [New LWP 9]
>>>>>> [New LWP 8]
>>>>>> [New LWP 7]
>>>>>> [New LWP 6]
>>>>>> [New LWP 5]
>>>>>> [New LWP 4]
>>>>>> [New LWP 3]
>>>>>> [New LWP 2]
>>>>>>  Id   Target Id Frame 
>>>>>>  12   LWP 2 0x7eadc6b0 in ?? ()
>>>>>>  11   LWP 3 0x7eadcbb8 in ?? ()
>>>>>>  10   LWP 4 0x7eadcbb8 in ?? ()
>>>>>>  9LWP 5 0x7eadcbb8 in ?? ()
>>>>>>  8LWP 6 0x7eadcbb8 in ?? ()
>>>>>>  7LWP 7 0x7eadcbb8 in ?? ()
>>>>>>  6LWP 8 0x7ead8b0c in ?? ()
>>>>>>  5LWP 9 0x7eadcbb8 in ?? ()
>>>>>>  4LWP 100x7eadcbb8 in ?? ()
>>>>>>  3LWP 110x7eadcbb8 in ?? ()
>>>>>>  2LWP 120x7eadcbb8 in ?? ()
>>>>>> * 1LWP 1 0x7eadcb04 in ?? ()
>>>>>> (gdb) 
>>>>>>
>>>>>>
>>>>>>
>>>>>> It seems that "_dbg" is unknown and unavailable.
>>>>>>
>>>>>> tyr java 399 grep _dbg 
>>>>>> /export2/src/openmpi-1.9/openmpi-dev-124-g91e9686/ompi/mpi/java/c/*
>>>>>> /export2/src/openmpi-1.9/openmpi-dev-124-g91e9686/ompi/mpi/java/c/mpi_MPI.c:
>>>>>> volatile 
>>> int _dbg = 1;
>>>>>> /export2/src/openmpi-1.9/openmpi-dev-124-g91e9686/ompi/mpi/java/c/mpi_MPI.c:
>>>>>> while 
>>> (_dbg) poll(NULL, 0, 1);
>>>>>> tyr java 400 nm /usr/local/openmpi-1.9.0_64_gcc/lib64/*.so | grep -i _dbg
>>>>>> tyr java 401 nm /usr/local/openmpi-1.9.0_64_gcc/lib64/*.so | grep -i 
>>>>>> JNI_OnLoad
>>>>>> [1057]  |  139688| 444|FUNC |GLOB |0|11  
>>>>>>|JNI_OnLoad
>>>>>> tyr java 402 
>>>>>>
>>>>>>
>>>>>>
>>>>>> How can I set _dbg to zero to continue mpiexec? I also tried to
>>>>>> set a breakpoint for function JNI_OnLoad, but it seems, that the
>>>>>> function isn't called before SIGSEGV.
>>>>>>
>>>>>>
>>>>>> tyr java 177 unsetenv OMPI_ATTACH 
>>>>>> tyr java 178 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
>>>>>> GNU gdb (GDB) 7.6.1
>>>>>> ...
>>>>>> (gdb) b mpi_MPI.c:JNI_OnLoad
>>>>>> No source file named mpi_MPI.c.
>>>>>> Make breakpoint pending on future shared library load? (y or [n]) y
>>>>>>
>>>>>> Breakpoint 1 (mpi_MPI.c:JNI_OnLoad) pending.
>>>>>> (gdb) run -np 1 java InitFinalizeMain 
>>>>>> Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec -np 1 java 
>>>>>> InitFinalizeMain
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> [New Thread 1 (LWP 1)]
>>>>>> [New LWP2]
>>>>>> #
>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>> #
>>>>>> #  SIGSEGV (0xb) at pc=0x7ea3c7f0, pid=3518, tid=2
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> tyr java 381 cat InitFinalizeMain.java 
>>>>>> import mpi.*;
>>>>>>
>>>>>> public class InitFinalizeMain
>>>>>> {
>>>>>>  public static void main (String args[]) throws MPIException
>>>>>>  {
>>>>>>MPI.Init (args);
>>>>>>System.out.print ("Hello!\n");
>>>>>>MPI.Finalize ();
>>>>>>  }
>>>>>> }
>>>>>>
>>>>>>
>>>>>> SIGSEGV happens in MPI.Init(args), because I can print a message
>>>>>> before I call the method.
>>>>>>
>>>>>> tyr java 192 unsetenv OMPI_ATTACH
>>>>>> tyr java 193 mpijavac InitFinalizeMain.java
>>>>>> tyr java 194 mpiexec -np 1 java InitFinalizeMain
>>>>>> Before MPI.Init()
>>>>>> #
>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>> #
>>>>>> #  SIGSEGV (0xb) at pc=0x7ea3c7f0, pid=3697, tid=2
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any ideas, how I can continue? I couldn't find a C function for
>>>>>> MPI.Init() in a C file. Do you know, which function is called first,
>>>>>> so that I can set a breakpoint? By the way, I get the same error
>>>>>> for Solaris 10 x86_64.
>>>>>>
>>>>>> tyr java 388 ssh sunpc1
>>>>>> ...
>>>>>> sunpc1 java 106 mpijavac InitFinalizeMain.java
>>>>>> sunpc1 java 107 uname -a
>>>>>> SunOS sunpc1 5.10 Generic_147441-21 i86pc i386 i86pc Solaris
>>>>>> sunpc1 java 108 isainfo -k
>>>>>> amd64
>>>>>> sunpc1 java 109 mpiexec -np 1 java InitFinalizeMain
>>>>>> #
>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>> #
>>>>>> #  SIGSEGV (0xb) at pc=0xfd7fff1d77f0, pid=20256, tid=2
>>>>>>
>>>>>>
>>>>>> Thank you very much for any help in advance.
>>>>>>
>>>>>> Kind regards
>>>>>>
>>>>>> Siegmar
>>>>>>
>>>>>>
>>>>>>
>>>>>>> thank you very much for your help.
>>>>>>>
>>>>>>>> how did you configure openmpi ? which java version did you use ?
>>>>>>>>
>>>>>>>> i just found a regression and you currently have to explicitly add
>>>>>>>> CFLAGS=-D_REENTRANT CPPFLAGS=-D_REENTRANT
>>>>>>>> to your configure command line
>>>>>>> I added "-D_REENTRANT" to my command.
>>>>>>>
>>>>>>> ../openmpi-dev-124-g91e9686/configure 
>>>>>>> --prefix=/usr/local/openmpi-1.9.0_64_gcc \
>>>>>>>  --libdir=/usr/local/openmpi-1.9.0_64_gcc/lib64 \
>>>>>>>  --with-jdk-bindir=/usr/local/jdk1.8.0/bin \
>>>>>>>  --with-jdk-headers=/usr/local/jdk1.8.0/include \
>>>>>>>  JAVA_HOME=/usr/local/jdk1.8.0 \
>>>>>>>  LDFLAGS="-m64" CC="gcc" CXX="g++" FC="gfortran" \
>>>>>>>  CFLAGS="-m64 -D_REENTRANT" CXXFLAGS="-m64" FCFLAGS="-m64" \
>>>>>>>  CPP="cpp" CXXCPP="cpp" \
>>>>>>>  CPPFLAGS="-D_REENTRANT" CXXCPPFLAGS="" \
>>>>>>>  --enable-mpi-cxx \
>>>>>>>  --enable-cxx-exceptions \
>>>>>>>  --enable-mpi-java \
>>>>>>>  --enable-heterogeneous \
>>>>>>>  --enable-mpi-thread-multiple \
>>>>>>>  --with-threads=posix \
>>>>>>>  --with-hwloc=internal \
>>>>>>>  --without-verbs \
>>>>>>>  --with-wrapper-cflags="-std=c11 -m64" \
>>>>>>>  --enable-debug \
>>>>>>>  |& tee log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_gcc
>>>>>>>
>>>>>>> I use Java 8.
>>>>>>>
>>>>>>> tyr openmpi-1.9 112 java -version
>>>>>>> java version "1.8.0"
>>>>>>> Java(TM) SE Runtime Environment (build 1.8.0-b132)
>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
>>>>>>> tyr openmpi-1.9 113 
>>>>>>>
>>>>>>> Unfortunately I still get a SIGSEGV with openmpi-dev-124-g91e9686.
>>>>>>> I have applied your patch and will try to debug my small Java
>>>>>>> program tomorrow or next week and then let you know the result.
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/10/25590.php 
>> <http://www.open-mpi.org/community/lists/users/2014/10/25590.php>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25591.php

commit c9c5d4011bf6ea1ade1a5bd9b6a77f02157dc774
Author: Gilles Gouaillardet 
List-Post: users@lists.open-mpi.org
Date:   Wed Oct 15 17:19:13 2014 +0900

Fix heterogeneous support

* correctly send OPAL_DSTORE_ARCH key

diff --git a/ompi/proc/proc.c b/ompi/proc/proc.c
index d30182f..12b781e 100644
--- a/ompi/proc/proc.c
+++ b/ompi/proc/proc.c
@@ -107,6 +107,7 @@ int ompi_proc_init(void)
 OMPI_CAST_RTE_NAME(&proc->super.proc_name)->vpid = i;

 if (i == OMPI_PROC_MY_NAME->vpid) {
+opal_value_t kv;
 ompi_proc_local_proc = proc;
 proc->super.proc_flags = OPAL_PROC_ALL_LOCAL;
 proc->super.proc_hostname = strdup(ompi_process_info.nodename);
@@ -115,8 +116,13 @@ int ompi_proc_init(void)
 opal_proc_local_set(&proc->super);
 #if OPAL_ENABLE_HETEROGENEOUS_SUPPORT
 /* add our arch to the modex */
-OPAL_MODEX_SEND_STRING(ret, PMIX_SYNC_REQD, PMIX_REMOTE, 
OPAL_DSTORE_ARCH,
-   &proc->super.proc_arch, OPAL_UINT32);
+OBJ_CONSTRUCT(&kv, opal_value_t);
+kv.key = strdup(OPAL_DSTORE_ARCH);
+kv.type = OPAL_UINT32;
+kv.data.uint32 = opal_local_arch;
+ret = opal_pmix.put(PMIX_REMOTE, &kv);
+OBJ_DESTRUCT(&kv);
+
 if (OPAL_SUCCESS != ret) {
 return ret;
 }

Re: [OMPI users] Possible Memory Leak in simple PingPong-Routine with OpenMPI 1.8.3?

[Gilles Gouaillardet]

Hi,

i tested on a RedHat 6 like linux server and could not observe any
memory leak.

BTW, are you running 32 or 64 bits cygwin ? and what is your configure
command line ?

Thanks,

Gilles

On 2014/10/27 18:26, Marco Atzeri wrote:
> On 10/27/2014 8:30 AM, maxinator333 wrote:
>> Hello,
>>
>> I noticed this weird behavior, because after a certain time of more than
>> one minute the transfer rates of MPI_Send and MPI_Recv dropped by a
>> factor of 100+. By chance I saw, that my program did allocate more and
>> more memory. I have the following minimal working example:
>>
>> #include 
>> #include 
>>
>> const uint32_t MSG_LENGTH = 256;
>>
>> int main(int argc, char* argv[]) {
>>  MPI_Init(NULL, NULL);
>>  int rank;
>>  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>
>>  volatile char * msg  = (char*) malloc( sizeof(char) *
>> MSG_LENGTH );
>>
>>  for (uint64_t i = 0; i < 1e9; i++) {
>>  if ( rank == 1 ) {
>>  MPI_Recv( const_cast(msg), MSG_LENGTH, MPI_CHAR,
>>rank-1, 0, MPI_COMM_WORLD,
>> MPI_STATUS_IGNORE);
>>  MPI_Send( const_cast(msg), MSG_LENGTH, MPI_CHAR,
>>rank-1, 0, MPI_COMM_WORLD);
>>  } else if ( rank == 0 ) {
>>  MPI_Send( const_cast(msg), MSG_LENGTH, MPI_CHAR,
>>rank+1, 0, MPI_COMM_WORLD);
>>  MPI_Recv( const_cast(msg), MSG_LENGTH, MPI_CHAR,
>>rank+1, 0, MPI_COMM_WORLD,
>> MPI_STATUS_IGNORE);
>>  }
>>  MPI_Barrier( MPI_COMM_WORLD );
>>  for (uint32_t k = 0; k < MSG_LENGTH; k++)
>>  msg[k]++;
>>  }
>>
>>  MPI_Finalize();
>>  return 0;
>> }
>>
>>
>> I run this with mpirun -n 2 ./pingpong_memleak.exe
>>
>> The program does nothing more than send a message from rank 0 to rank 1,
>> then from rank 1 to rank 0 and so on in standard blocking mode, not even
>> asynchronous.
>>
>> Running the program will allocate roughly 30mb/s (Windows Task Manager)
>> until it stops at around 1.313.180kb. This is when the transfer rates
>> (not being measured in above snippet) drop significantly to maybe a
>> second per send instead of roughly 1µs.
>>
>> I use Cygwin with Windows 7 and 16Gb RAM. I haven't tested this minimal
>> working example on other setups.
>
> Can someone test on other platforms and confirm me that is a cygwin
> specific issue ?
>
> Regards
> Marco
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25602.php

Re: [OMPI users] OMPI users] Possible Memory Leak in simple PingPong-Routine with OpenMPI 1.8.3?

[Gilles Gouaillardet]

Thanks Marco,

I could reproduce the issue even with one node sending/receiving to itself.

I will investigate this tomorrow

Cheers,

Gilles

Marco Atzeri  wrote:
>
>
>On 10/27/2014 10:30 AM, Gilles Gouaillardet wrote:
>> Hi,
>>
>> i tested on a RedHat 6 like linux server and could not observe any
>> memory leak.
>>
>> BTW, are you running 32 or 64 bits cygwin ? and what is your configure
>> command line ?
>>
>> Thanks,
>>
>> Gilles
>>
>
>the problem is present in both versions.
>
>cygwin 1.8.3-1 packages  are built with configure:
>
>  --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin 
>--libexecdir=/usr/libexec --datadir=/usr/share --localstatedir=/var 
>--sysconfdir=/etc --libdir=/usr/lib --datarootdir=/usr/share 
>--docdir=/usr/share/doc/openmpi --htmldir=/usr/share/doc/openmpi/html -C 
>LDFLAGS=-Wl,--export-all-symbols --disable-mca-dso --disable-sysv-shmem 
>--enable-cxx-exceptions --with-threads=posix --without-cs-fs 
>--with-mpi-param_check=always --enable-contrib-no-build=vt,libompitrace 
>--enable-mca-no-build=paffinity,installdirs-windows,timer-windows,shmem-sysv
>
>Regards
>Marco
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/10/25604.php

Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code

[Gilles Gouaillardet]

Michael,

Could you please run
mpirun -np 1 df -h
mpirun -np 1 df -hi
on both compute and login nodes

Thanks

Gilles

michael.rach...@dlr.de wrote:
>Dear developers of OPENMPI,
>
>We have now installed and tested the bugfixed OPENMPI Nightly Tarball  of 
>2014-10-24  (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 .
>As before (with OPENMPI-1.8.3 release version) the small Ftn-testprogram runs 
>correctly on the login-node.
>As before the program aborts on the compute node, but now with a different 
>error message: 
>
>The following message appears when launching the program with 2 processes: 
>mpiexec -np 2 -bind-to core -tag-output ./a.out
>
>[1,0]: on nodemaster: iwin= 685 :
>[1,0]:  total storage [MByte] alloc. in shared windows so far:   
>137.
>[ [1,0]: === allocation of shared window no. iwin= 686
>[1,0]:  starting now with idim_1=   5
>-
>It appears as if there is not enough space for 
>/tmp/openmpi-sessions-rachner@r5i5n13_0/48127/1/shared_window_688.r5i5n13 (the 
>shared-memory backing
>file). It is likely that your MPI job will now either abort or experience
>performance degradation.
>
>  Local host:  r5i5n13
>  Space Requested: 204256 B
>  Space Available: 208896 B
>--
>[r5i5n13:26917] *** An error occurred in MPI_Win_allocate_shared
>[r5i5n13:26917] *** reported by process [3154051073,140733193388032]
>[r5i5n13:26917] *** on communicator MPI_COMM_WORLD
>[r5i5n13:26917] *** MPI_ERR_INTERN: internal error
>[r5i5n13:26917] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will 
>now abort,
>[r5i5n13:26917] ***and potentially your MPI job)
>rachner@r5i5n13:~/dat>
>
>
>
>When I repeat the run using 24 processes (on same compute node) the same kind 
>of abort message occurs, but earlier:
>
>[1,0]: on nodemaster: iwin= 231 :
>[1,0]:  total storage [MByte] alloc. in shared windows so far:   
>46.2
> [1,0]: === allocation of shared window no. iwin= 232
>[1,0]:  starting now with idim_1=   5
>-
>It appears as if there is not enough space for 
>/tmp/openmpi-sessions-rachner@r5i5n13_0/48029/1/shared_window_234.r5i5n13 (the 
>shared-memory backing
>file). It is likely that your MPI job will now either abort or experience
>performance degradation.
>
>  Local host:  r5i5n13
>  Space Requested: 204784 B
>  Space Available: 131072 B
>--
>[r5i5n13:26947] *** An error occurred in MPI_Win_allocate_shared
>[r5i5n13:26947] *** reported by process [3147628545,140733193388032]
>[r5i5n13:26947] *** on communicator MPI_COMM_WORLD
>[r5i5n13:26947] *** MPI_ERR_INTERN: internal error
>[r5i5n13:26947] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will 
>now abort,
>[r5i5n13:26947] ***and potentially your MPI job)
>rachner@r5i5n13:~/dat>
>
>
>So the problem is not yet resolved.
>
>Greetings
> Michael Rachner
>
>
>
>
>
>
>-Ursprüngliche Nachricht-
>Von: Rachner, Michael 
>Gesendet: Montag, 27. Oktober 2014 11:49
>An: 'Open MPI Users'
>Betreff: AW: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared 
>memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
>
>Dear Mr. Squyres.
>
>We will try to install your bug-fixed nigthly tarball of 2014-10-24 on 
>Cluster5 to see whether it works or not.
>The installation however will take some time. I get back to you, if I know 
>more.
>
>Let me add the information that on the Laki each nodes has 16 GB of shared 
>memory (there it worked), the login-node on Cluster 5 has 64 GB (there it 
>worked too), whereas the compute nodes on Cluster5 have 128 GB (there it did 
>not work).
>So possibly the bug might have something to do with the size of the physical 
>shared memory available on the node.
>
>Greetings
>Michael Rachner
>
>-Ursprüngliche Nachricht-
>Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Jeff Squyres 
>(jsquyres)
>Gesendet: Freitag, 24. Oktober 2014 22:45
>An: Open MPI User's List
>Betreff: Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared 
>memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
>
>Nathan tells me that this may well be related to a fix that was literally just 
>pulled into the v1.8 branch today:
>
>https://github.com/open-mpi/ompi-release/pull/56
>
>Would you mind testing any nightly tarball after tonight?  (i.e

Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code

[Gilles Gouaillardet]

Michael,

The available space must be greater than the requested size + 5%

From the logs, the error message makes sense to me : there is not enough space 
in /tmp
Since the compute nodes have a lot of memory, you might want to try using 
/dev/shm instead of /tmp for the backing files

Cheers,

Gilles

michael.rach...@dlr.de wrote:
>Dear developers of OPENMPI,
>
>We have now installed and tested the bugfixed OPENMPI Nightly Tarball  of 
>2014-10-24  (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 .
>As before (with OPENMPI-1.8.3 release version) the small Ftn-testprogram runs 
>correctly on the login-node.
>As before the program aborts on the compute node, but now with a different 
>error message: 
>
>The following message appears when launching the program with 2 processes: 
>mpiexec -np 2 -bind-to core -tag-output ./a.out
>
>[1,0]: on nodemaster: iwin= 685 :
>[1,0]:  total storage [MByte] alloc. in shared windows so far:   
>137.
>[ [1,0]: === allocation of shared window no. iwin= 686
>[1,0]:  starting now with idim_1=   5
>-
>It appears as if there is not enough space for 
>/tmp/openmpi-sessions-rachner@r5i5n13_0/48127/1/shared_window_688.r5i5n13 (the 
>shared-memory backing
>file). It is likely that your MPI job will now either abort or experience
>performance degradation.
>
>  Local host:  r5i5n13
>  Space Requested: 204256 B
>  Space Available: 208896 B
>--
>[r5i5n13:26917] *** An error occurred in MPI_Win_allocate_shared
>[r5i5n13:26917] *** reported by process [3154051073,140733193388032]
>[r5i5n13:26917] *** on communicator MPI_COMM_WORLD
>[r5i5n13:26917] *** MPI_ERR_INTERN: internal error
>[r5i5n13:26917] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will 
>now abort,
>[r5i5n13:26917] ***and potentially your MPI job)
>rachner@r5i5n13:~/dat>
>
>
>
>When I repeat the run using 24 processes (on same compute node) the same kind 
>of abort message occurs, but earlier:
>
>[1,0]: on nodemaster: iwin= 231 :
>[1,0]:  total storage [MByte] alloc. in shared windows so far:   
>46.2
> [1,0]: === allocation of shared window no. iwin= 232
>[1,0]:  starting now with idim_1=   5
>-
>It appears as if there is not enough space for 
>/tmp/openmpi-sessions-rachner@r5i5n13_0/48029/1/shared_window_234.r5i5n13 (the 
>shared-memory backing
>file). It is likely that your MPI job will now either abort or experience
>performance degradation.
>
>  Local host:  r5i5n13
>  Space Requested: 204784 B
>  Space Available: 131072 B
>--
>[r5i5n13:26947] *** An error occurred in MPI_Win_allocate_shared
>[r5i5n13:26947] *** reported by process [3147628545,140733193388032]
>[r5i5n13:26947] *** on communicator MPI_COMM_WORLD
>[r5i5n13:26947] *** MPI_ERR_INTERN: internal error
>[r5i5n13:26947] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will 
>now abort,
>[r5i5n13:26947] ***and potentially your MPI job)
>rachner@r5i5n13:~/dat>
>
>
>So the problem is not yet resolved.
>
>Greetings
> Michael Rachner
>
>
>
>
>
>
>-Ursprüngliche Nachricht-
>Von: Rachner, Michael 
>Gesendet: Montag, 27. Oktober 2014 11:49
>An: 'Open MPI Users'
>Betreff: AW: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared 
>memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
>
>Dear Mr. Squyres.
>
>We will try to install your bug-fixed nigthly tarball of 2014-10-24 on 
>Cluster5 to see whether it works or not.
>The installation however will take some time. I get back to you, if I know 
>more.
>
>Let me add the information that on the Laki each nodes has 16 GB of shared 
>memory (there it worked), the login-node on Cluster 5 has 64 GB (there it 
>worked too), whereas the compute nodes on Cluster5 have 128 GB (there it did 
>not work).
>So possibly the bug might have something to do with the size of the physical 
>shared memory available on the node.
>
>Greetings
>Michael Rachner
>
>-Ursprüngliche Nachricht-
>Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Jeff Squyres 
>(jsquyres)
>Gesendet: Freitag, 24. Oktober 2014 22:45
>An: Open MPI User's List
>Betreff: Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared 
>memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
>
>Nathan tells me that this may well be related to a fi

Re: [OMPI users] OMPI users] OMPI users] OMPI users] which info is needed for SIGSEGV inJava foropenmpi-dev-124-g91e9686on Solaris

[Gilles Gouaillardet]

Ralph,

On 2014/10/28 0:46, Ralph Castain wrote:
> Actually, I propose to also remove that issue. Simple enough to use a
> hash_table_32 to handle the jobids, and let that point to a
> hash_table_32 of vpids. Since we rarely have more than one jobid
> anyway, the memory overhead actually decreases with this model, and we
> get rid of that annoying need to memcpy everything. 
sounds good to me.
from an implementation/performance point of view, should we put treat
the local jobid differently ?
(e.g. use a special variable for the hash_table_32 of the vpids of the
current jobid)
>> as far as i am concerned, i am fine with your proposed suggestion to
>> dump opal_identifier_t.
>>
>> about the patch, did you mean you have something ready i can apply to my
>> PR ?
>> or do you expect me to do the changes (i am ok to do it if needed)
> Why don’t I grab your branch, create a separate repo based on it (just to 
> keep things clean), push it to my area and give you write access? We can then 
> collaborate on the changes and create a PR from there. This way, you don’t 
> need to give me write access to your entire repo.
>
> Make sense?
ok to work on an other "somehow shared" repo for that issue.
i am not convinced you should grab my branch since all the changes i
made are will be no more valid.
anyway, feel free to fork a repo from my branch or the master and i will
work from here.

Cheers,

Gilles

Re: [OMPI users] Possible Memory Leak in simple PingPong-Routine with OpenMPI 1.8.3?

[Gilles Gouaillardet]

Marco,

here is attached a patch that fixes the issue
/* i could not find yet why this does not occurs on Linux ... */

could you please give it a try ?

Cheers,

Gilles

On 2014/10/27 18:45, Marco Atzeri wrote:
>
>
> On 10/27/2014 10:30 AM, Gilles Gouaillardet wrote:
>> Hi,
>>
>> i tested on a RedHat 6 like linux server and could not observe any
>> memory leak.
>>
>> BTW, are you running 32 or 64 bits cygwin ? and what is your configure
>> command line ?
>>
>> Thanks,
>>
>> Gilles
>>
>
> the problem is present in both versions.
>
> cygwin 1.8.3-1 packages  are built with configure:
>
>  --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin
> --sbindir=/usr/sbin --libexecdir=/usr/libexec --datadir=/usr/share
> --localstatedir=/var --sysconfdir=/etc --libdir=/usr/lib
> --datarootdir=/usr/share --docdir=/usr/share/doc/openmpi
> --htmldir=/usr/share/doc/openmpi/html -C
> LDFLAGS=-Wl,--export-all-symbols --disable-mca-dso
> --disable-sysv-shmem --enable-cxx-exceptions --with-threads=posix
> --without-cs-fs --with-mpi-param_check=always
> --enable-contrib-no-build=vt,libompitrace
> --enable-mca-no-build=paffinity,installdirs-windows,timer-windows,shmem-sysv
>
> Regards
> Marco
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/10/25604.php

diff --git a/ompi/mca/pml/ob1/pml_ob1_recvreq.c 
b/ompi/mca/pml/ob1/pml_ob1_recvreq.c
index 7c8853f..c4a 100644
--- a/ompi/mca/pml/ob1/pml_ob1_recvreq.c
+++ b/ompi/mca/pml/ob1/pml_ob1_recvreq.c
@@ -16,6 +16,8 @@
  * Copyright (c) 2011-2012 Los Alamos National Security, LLC. All rights
  * reserved.
  * Copyright (c) 2012  FUJITSU LIMITED.  All rights reserved.
+ * Copyright (c) 2014  Research Organization for Information Science
+ * and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
  * 
  * Additional copyrights may follow
@@ -152,11 +154,16 @@ static void 
mca_pml_ob1_recv_request_construct(mca_pml_ob1_recv_request_t* reque
 OBJ_CONSTRUCT(&request->lock, opal_mutex_t);
 }

+static void mca_pml_ob1_recv_request_destruct(mca_pml_ob1_recv_request_t* 
request)
+{
+OBJ_DESTRUCT(&request->lock);
+}
+
 OBJ_CLASS_INSTANCE(
 mca_pml_ob1_recv_request_t,
 mca_pml_base_recv_request_t,
 mca_pml_ob1_recv_request_construct,
-NULL);
+mca_pml_ob1_recv_request_destruct);


 /*

Re: [OMPI users] SIGBUS in openmpi-dev-178-ga16c1e4 on Solaris 10 Sparc

[Gilles Gouaillardet]

Hi Siegmar,

From the jvm logs, there is an alignment error in native_get_attr but i could 
not find it by reading the source code.

Could you please do
ulimit -c unlimited
mpiexec ...
and then
gdb /bin/java core
And run bt on all threads until you get a line number in native_get_attr

Thanks

Gilles

Siegmar Gross  wrote:
>Hi,
>
>today I installed openmpi-dev-178-ga16c1e4 on Solaris 10 Sparc
>with gcc-4.9.1 and Java 8. Now a very simple Java program works
>as expected, but other Java programs still break. I removed the
>warnings about "shmem.jar" and used the following configure
>command.
>
>tyr openmpi-dev-178-ga16c1e4-SunOS.sparc.64_gcc 406 head config.log \
>  | grep openmpi
>$ ../openmpi-dev-178-ga16c1e4/configure
>  --prefix=/usr/local/openmpi-1.9.0_64_gcc
>  --libdir=/usr/local/openmpi-1.9.0_64_gcc/lib64
>  --with-jdk-bindir=/usr/local/jdk1.8.0/bin
>  --with-jdk-headers=/usr/local/jdk1.8.0/include
>  JAVA_HOME=/usr/local/jdk1.8.0
>  LDFLAGS=-m64 CC=gcc CXX=g++ FC=gfortran CFLAGS=-m64 -D_REENTRANT
>  CXXFLAGS=-m64 FCFLAGS=-m64 CPP=cpp CXXCPP=cpp
>  CPPFLAGS= -D_REENTRANT CXXCPPFLAGS=
>  --enable-mpi-cxx --enable-cxx-exceptions --enable-mpi-java
>  --enable-mpi-thread-multiple --with-threads=posix
>  --with-hwloc=internal
>  --without-verbs --with-wrapper-cflags=-std=c11 -m64
>  --with-wrapper-cxxflags=-m64 --enable-debug
>
>
>tyr java 290 ompi_info | grep -e "Open MPI repo revision:" -e "C compiler 
>version:"
>  Open MPI repo revision: dev-178-ga16c1e4
>  C compiler version: 4.9.1
>
>
>
>> > regarding the BUS error reported by Siegmar, i also commited
>> > 62bde1fcb554079143030bb305512c236672386f
>> > in order to fix it (this is based on code review only, i have no sparc64
>> > hardware to test it is enough)
>> 
>> I'll test it, when a new nightly snapshot is available for the trunk.
>
>
>tyr java 291 mpijavac InitFinalizeMain.java 
>tyr java 292 mpiexec -np 1 java InitFinalizeMain
>Hello!
>
>tyr java 293 mpijavac BcastIntMain.java 
>tyr java 294 mpiexec -np 2 java BcastIntMain
>#
># A fatal error has been detected by the Java Runtime Environment:
>#
>#  SIGBUS (0xa) at pc=0xfffee3210bfc, pid=24792, tid=2
>...
>
>
>
>tyr java 296 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
>...
>(gdb) run -np 2 java BcastIntMain
>Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec -np 2 java 
>BcastIntMain
>[Thread debugging using libthread_db enabled]
>[New Thread 1 (LWP 1)]
>[New LWP2]
>#
># A fatal error has been detected by the Java Runtime Environment:
>#
>#  SIGBUS (0xa) at pc=0xfffee3210bfc, pid=24814, tid=2
>#
># JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build 1.8.0-b132)
># Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode 
>solaris-sparc compressed oops)
># Problematic frame:
># C  [mca_pmix_native.so+0x10bfc]  native_get_attr+0x3000
>#
># Failed to write core dump. Core dumps have been disabled. To enable core 
>dumping, try "ulimit -c unlimited" before starting Java again
>#
># An error report file with more information is saved as:
># /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid24814.log
>#
># A fatal error has been detected by the Java Runtime Environment:
>#
>#  SIGBUS (0xa) at pc=0xfffee3210bfc, pid=24812, tid=2
>#
># JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build 1.8.0-b132)
># Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode 
>solaris-sparc compressed oops)
># Problematic frame:
># C  [mca_pmix_native.so+0x10bfc]  native_get_attr+0x3000
>#
># Failed to write core dump. Core dumps have been disabled. To enable core 
>dumping, try "ulimit -c unlimited" before starting Java again
>#
># An error report file with more information is saved as:
># /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid24812.log
>#
># If you would like to submit a bug report, please visit:
>#   http://bugreport.sun.com/bugreport/crash.jsp
># The crash happened outside the Java Virtual Machine in native code.
># See problematic frame for where to report the bug.
>#
>[tyr:24814] *** Process received signal ***
>[tyr:24814] Signal: Abort (6)
>[tyr:24814] Signal code:  (-1)
>/export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:opal_backtrace_print+0x2c
>/export2/prog/SunOS_sparc/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0.0.0:0xdc2d4
>/lib/sparcv9/libc.so.1:0xd8b98
>/lib/sparcv9/libc.so.1:0xcc70c
>/lib/sparcv9/libc.so.1:0xcc918
>/lib/sparcv9/libc.so.1:0xdd2d0 [ Signal 6 (ABRT)]
>/lib/sparcv9/libc.so.1:_thr_sigsetmask+0x1c4
>/lib/sparcv9/libc.so.1:sigprocmask+0x28
>/lib/sparcv9/libc.so.1:_sigrelse+0x5c
>/lib/sparcv9/libc.so.1:abort+0xc0
>/export2/prog/SunOS_sparc/jdk1.8.0/jre/lib/sparcv9/server/libjvm.so:0xb3cb90
>/export2/prog/SunOS_sparc/jdk1.8.0/jre/lib/sparcv9/server/libjvm.so:0xd97a04
>/export2/prog/SunOS_sparc/jdk1.8.0/jre/lib/sparcv9/server/libjvm.so:JVM_handle_solaris_signal+0xc0c
>/export2/prog/SunOS_sparc/jdk1.8.0/jre/lib/sparcv9/server/libjvm.so:0xb44e84
>/lib/sparcv9

Re: [OMPI users] OMPI users] Possible Memory Leak in simple PingPong-Routine with OpenMPI 1.8.3?

[Gilles Gouaillardet]

Thanks Marco,

pthread_mutex_init calls calloc under cygwin but does not allocate memory under 
linux, so not invoking pthread_mutex_destroy causes a memory leak only under 
cygwin.

Gilles

Marco Atzeri  wrote:
>On 10/28/2014 12:04 PM, Gilles Gouaillardet wrote:
>> Marco,
>>
>> here is attached a patch that fixes the issue
>> /* i could not find yet why this does not occurs on Linux ... */
>>
>> could you please give it a try ?
>>
>> Cheers,
>>
>> Gilles
>>
>
>It solves the issue on 64 bit.
>I see no growing memory usage anymore
>
>I will build 32 bit and then upload both as 1.8.3-2
>
>Thanks
>Marco
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/10/25630.php

Re: [OMPI users] OMPI users] OMPI users] Possible Memory Leak in simple PingPong-Routine with OpenMPI 1.8.3?

[Gilles Gouaillardet]

Yep, will do today

Ralph Castain  wrote:
>Gilles: will you be committing this to trunk and PR to 1.8?
>
>
>> On Oct 28, 2014, at 11:05 AM, Marco Atzeri  wrote:
>> 
>> On 10/28/2014 4:41 PM, Gilles Gouaillardet wrote:
>>> Thanks Marco,
>>> 
>>> pthread_mutex_init calls calloc under cygwin but does not allocate memory 
>>> under linux, so not invoking pthread_mutex_destroy causes a memory leak 
>>> only under cygwin.
>>> 
>>> Gilles
>> 
>> thanks for the work .
>> 
>> uploading 1.8.3-2 on www.cygwin.com
>> 
>> Regards
>> Marco
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/10/25634.php
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/10/25636.php

Re: [OMPI users] SIGBUS in openmpi-dev-178-ga16c1e4 on Solaris 10 Sparc

[Gilles Gouaillardet]

fff7f6173d0 in rtld_db_dlactivity () from 
>>> /usr/lib/sparcv9/ld.so.1
>>> #1  0xffff7f6175a8 in rd_event () from /usr/lib/sparcv9/ld.so.1
>>> #2  0x7f618950 in lm_delete () from /usr/lib/sparcv9/ld.so.1
>>> #3  0x7f6226bc in remove_so () from /usr/lib/sparcv9/ld.so.1
>>> #4  0x7f624574 in remove_hdl () from /usr/lib/sparcv9/ld.so.1
>>> #5  0x7f61d97c in dlclose_core () from /usr/lib/sparcv9/ld.so.1
>>> #6  0x7f61d9d4 in dlclose_intn () from /usr/lib/sparcv9/ld.so.1
>>> #7  0x7f61db0c in dlclose () from /usr/lib/sparcv9/ld.so.1
>>> #8  0x7ec87ca0 in vm_close ()
>>>   from /usr/local/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0
>>> #9  0x7ec85274 in lt_dlclose ()
>>>   from /usr/local/openmpi-1.9.0_64_gcc/lib64/libopen-pal.so.0
>>> #10 0x7ecaa5dc in ri_destructor (obj=0x100187ae0)
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/opal/mca/base/mca_base_component_repository.c:382
>>> #11 0x7eca8fd8 in opal_obj_run_destructors (object=0x100187ae0)
>>>at ../../../../openmpi-dev-178-ga16c1e4/opal/class/opal_object.h:446
>>> #12 0x7eca9eac in mca_base_component_repository_release (
>>>component=0x7b0236f0 )
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/opal/mca/base/mca_base_component_repository.c:240
>>> #13 0x7ecac17c in mca_base_component_unload (
>>>component=0x7b0236f0 , output_id=-1)
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/opal/mca/base/mca_base_components_close.c:47
>>> #14 0x7ecac210 in mca_base_component_close (
>>>component=0x7b0236f0 , output_id=-1)
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/opal/mca/base/mca_base_components_close.c:60
>>> #15 0x7ecac2e4 in mca_base_components_close (output_id=-1, 
>>>components=0x7f14bc58 , skip=0x0)
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/opal/mca/base/mca_base_components_close.c:86
>>> #16 0x7ecac24c in mca_base_framework_components_close (
>>>framework=0x7f14bc08 , skip=0x0)
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/opal/mca/base/mca_base_components_close.c:66
>>> #17 0x7efcaf80 in orte_oob_base_close ()
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/orte/mca/oob/base/oob_base_frame.c:112
>>> #18 0x7ecc0d74 in mca_base_framework_close (
>>>framework=0x7f14bc08 )
>>>at 
>>> ../../../../openmpi-dev-178-ga16c1e4/opal/mca/base/mca_base_framework.c:187
>>> #19 0x7bd07858 in rte_finalize ()
>>>at 
>>> ../../../../../openmpi-dev-178-ga16c1e4/orte/mca/ess/hnp/ess_hnp_module.c:857
>>> #20 0x7ef338bc in orte_finalize ()
>>>at ../../openmpi-dev-178-ga16c1e4/orte/runtime/orte_finalize.c:66
>>> #21 0x0001723c in orterun (argc=4, argv=0x7fffe0e8)
>>>at ../../../../openmpi-dev-178-ga16c1e4/orte/tools/orterun/orterun.c:1103
>>> #22 0x00013e80 in main (argc=4, argv=0x7fffe0e8)
>>>at ../../../../openmpi-dev-178-ga16c1e4/orte/tools/orterun/main.c:13
>>> (gdb) 
>>>
>>>
>>>
>>> Do you need any other information?
>>>
>>>
>>> Kind regards
>>>
>>> Siegmar
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25635.php

commit 8c556bbc66c06fb19c6e46c67624bac1d6719b12
Author: Gilles Gouaillardet 
List-Post: users@lists.open-mpi.org
Date:   Wed Oct 29 13:19:23 2014 +0900

pmix: fix alignment issue

diff --git a/opal/mca/pmix/native/pmix_native.c 
b/opal/mca/pmix/native/pmix_native.c
index 6e771ea..b3c03da 100644
--- a/opal/mca/pmix/native/pmix_native.c
+++ b/opal/mca/pmix/native/pmix_native.c
@@ -1097,6 +1097,7 @@ static bool native_get_attr(const char *attr, 
opal_value_t **kv)
 continue;
 }
 native_pname.vid = vid;
+memcpy(&id, &native_pname, sizeof(opal_identifier_t));
 #if OPAL_HAVE_HWLOC
 OBJ_CONSTRUCT(&vals, opal_list_t);
 if (OPAL_SUCCESS != (rc = opal_dstore.fetch(opal_dstore_internal, 
(opal_identifier_t*)&native_pname,
@@ -1104,7 +1105,7 @@ static bool native_get_attr(const char *attr, 
opal_value_t **kv)
 opal_output_verbose(2, opal_pmix_base_framework.framework_output,
 "%s cpuset for local proc %s not found",
 OPAL_NAME_PRINT(OPAL_PROC_MY_NAME),
-
OPAL_NAME_PRINT(*(opal_identifier_t*)&native_pname));
+OPAL_NAME_PRINT(id));
 OPAL_LIST_DESTRUCT(&vals);
 /* even though the cpuset wasn't found, we at least know it is
  * on the same node with us */
@@ -1131,7 +1132,7 @@ static bool native_get_attr(const char *attr, 
opal_value_t **kv)
 OPAL_OUTPUT_VERBOSE((1, opal_pmix_base_framework.framework_output,
  "%s pmix:native proc %s locality %s",
  OPAL_NAME_PRINT(OPAL_PROC_MY_NAME),
- 
OPAL_NAME_PRINT(*(opal_identifier_t*)&native_pname),
+ OPAL_NAME_PRINT(id),
  opal_hwloc_base_print_locality(locality)));

 OBJ_CONSTRUCT(&kvn, opal_value_t);

Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code

[Gilles Gouaillardet]

Michael,

could you please share your test program so we can investigate it ?

Cheers,

Gilles

On 2014/10/31 18:53, michael.rach...@dlr.de wrote:
> Dear developers of OPENMPI,
>
> There remains a hanging observed in MPI_WIN_ALLOCATE_SHARED.
>
> But first: 
> Thank you for your advices to employ shmem_mmap_relocate_backing_file = 1
> It indeed turned out, that the bad (but silent) allocations  by 
> MPI_WIN_ALLOCATE_SHARED, which I observed in the past after ~140 MB of 
> allocated shared memory, 
> were indeed caused by  a too small available storage for the sharedmem 
> backing files. Applying the MCA parameter resolved the problem.
>
> Now the allocation of shared data windows by  MPI_WIN_ALLOCATE_SHARED in the 
> OPENMPI-1.8.3 release version works on both clusters!
> I tested it both with my small sharedmem-Ftn-testprogram  as well as with our 
> Ftn-CFD-code.
> It worked  even when allocating 1000 shared data windows containing a total 
> of 40 GB.  Very well.
>
> But now I come to the problem remaining:
> According to the attached email of Jeff (see below) of 2014-10-24, 
> we have alternatively installed and tested the bugfixed OPENMPI Nightly 
> Tarball  of 2014-10-24  (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 .
> That version worked well, when our CFD-code was running on only 1 node.
> But I observe now, that when running the CFD-code on 2 node with  2 processes 
> per node,
> after having allocated a total of 200 MB of data in 20 shared windows, the 
> allocation of the 21-th window fails, 
> because all 4 processes enter MPI_WIN_ALLOCATE_SHARED but never leave it. The 
> code hangs in that routine, without any message.
>
> In contrast, that bug does NOT occur with the  OPENMPI-1.8.3 release version  
>  with same program on same machine.
>
> That means for you:  
>In openmpi-dev-176-g9334abc.tar.gz   the new-introduced  bugfix concerning 
> the shared memory allocation may be not yet correctly coded ,
>or that version contains another new bug in sharedmemory allocation  
> compared to the working(!) 1.8.3-release version.
>
> Greetings to you all
>   Michael Rachner
> 
>
>
>
> -Ursprüngliche Nachricht-
> Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Jeff Squyres 
> (jsquyres)
> Gesendet: Freitag, 24. Oktober 2014 22:45
> An: Open MPI User's List
> Betreff: Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared 
> memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
>
> Nathan tells me that this may well be related to a fix that was literally 
> just pulled into the v1.8 branch today:
>
> https://github.com/open-mpi/ompi-release/pull/56
>
> Would you mind testing any nightly tarball after tonight?  (i.e., the v1.8 
> tarballs generated tonight will be the first ones to contain this fix)
>
> http://www.open-mpi.org/nightly/master/
>
>
>
> On Oct 24, 2014, at 11:46 AM,  
>  wrote:
>
>> Dear developers of OPENMPI,
>>  
>> I am running a small downsized Fortran-testprogram for shared memory 
>> allocation (using MPI_WIN_ALLOCATE_SHARED and  MPI_WIN_SHARED_QUERY) )
>> on only 1 node   of 2 different Linux-clusters with OPENMPI-1.8.3 and 
>> Intel-14.0.4 /Intel-13.0.1, respectively.
>>  
>> The program simply allocates a sequence of shared data windows, each 
>> consisting of 1 integer*4-array.
>> None of the windows is freed, so the amount of allocated data  in shared 
>> windows raises during the course of the execution.
>>  
>> That worked well on the 1st cluster (Laki, having 8 procs per node))  
>> when allocating even 1000 shared windows each having 5 integer*4 array 
>> elements, i.e. a total of  200 MBytes.
>> On the 2nd cluster (Cluster5, having 24 procs per node) it also worked on 
>> the login node, but it did NOT work on a compute node.
>> In that error case, there occurs something like an internal storage limit of 
>> ~ 140 MB for the total storage allocated in all shared windows.
>> When that limit is reached, all later shared memory allocations fail (but 
>> silently).
>> So the first attempt to use such a bad shared data window results in a bus 
>> error due to the bad storage address encountered.
>>  
>> That strange behavior could be observed in the small testprogram but also 
>> with my large Fortran CFD-code.
>> If the error occurs, then it occurs with both codes, and both at a storage 
>> limit of  ~140 MB.
>> I found that this storage limit depends only weakly on  the number of 
>> processes (for np=2,4,8,16,24  it is: 144.4 , 144.0, 141.0, 137.0, 
>> 132.2 MB)
>>  
>> Note that the shared memory storage available on both clusters was very 
>> large (many GB of free memory).
>>  
>> Here is the error message when running with np=2 and an  array 
>> dimension of idim_1=5  for the integer*4 array allocated per shared 
>> window on the compute node of Cluster5:
>> In that case, the error occurred at the 723-th shared window, which is the 
>> 1st badly allocated window in tha

Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code

[Gilles Gouaillardet]

Hi Michael,

bigger the program, bigger the fun ;-)

i will have a look at it.

Cheers,

Gilles

On 2014/11/05 19:08, michael.rach...@dlr.de wrote:
> Dear Gilles,
>
> My small downsized Ftn-testprogram for testing the shared memory  feature 
> (MPI_WIN_ALLOCATE_SHARED,  MPI_WIN_SHARED_QUERY, C_F_POINTER)
>  presumes for simplicity that all processes are running on the same node 
> (i.e. the communicator containing the procs on the same node  is just 
> MPI_COMM_WORLD).
> So the hanging of MPI_WIN_ALLOCATE_SHARED when running on 2 nodes could only 
> be observed with our large CFD-code. 
>
> Are OPENMPI-developers nevertheless interested in that testprogram?
>
> Greetings
> Michael
>
>
>
>
>
>
> -Ursprüngliche Nachricht-
> Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Gilles 
> Gouaillardet
> Gesendet: Mittwoch, 5. November 2014 10:46
> An: Open MPI Users
> Betreff: Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared 
> memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
>
> Michael,
>
> could you please share your test program so we can investigate it ?
>
> Cheers,
>
> Gilles
>
> On 2014/10/31 18:53, michael.rach...@dlr.de wrote:
>> Dear developers of OPENMPI,
>>
>> There remains a hanging observed in MPI_WIN_ALLOCATE_SHARED.
>>
>> But first: 
>> Thank you for your advices to employ shmem_mmap_relocate_backing_file = 1
>> It indeed turned out, that the bad (but silent) allocations  by 
>> MPI_WIN_ALLOCATE_SHARED, which I observed in the past after ~140 MB of 
>> allocated shared memory, were indeed caused by  a too small available 
>> storage for the sharedmem backing files. Applying the MCA parameter resolved 
>> the problem.
>>
>> Now the allocation of shared data windows by  MPI_WIN_ALLOCATE_SHARED in the 
>> OPENMPI-1.8.3 release version works on both clusters!
>> I tested it both with my small sharedmem-Ftn-testprogram  as well as with 
>> our Ftn-CFD-code.
>> It worked  even when allocating 1000 shared data windows containing a total 
>> of 40 GB.  Very well.
>>
>> But now I come to the problem remaining:
>> According to the attached email of Jeff (see below) of 2014-10-24, we 
>> have alternatively installed and tested the bugfixed OPENMPI Nightly Tarball 
>>  of 2014-10-24  (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 .
>> That version worked well, when our CFD-code was running on only 1 node.
>> But I observe now, that when running the CFD-code on 2 node with  2 
>> processes per node, after having allocated a total of 200 MB of data 
>> in 20 shared windows, the allocation of the 21-th window fails, because all 
>> 4 processes enter MPI_WIN_ALLOCATE_SHARED but never leave it. The code hangs 
>> in that routine, without any message.
>>
>> In contrast, that bug does NOT occur with the  OPENMPI-1.8.3 release version 
>>   with same program on same machine.
>>
>> That means for you:  
>>In openmpi-dev-176-g9334abc.tar.gz   the new-introduced  bugfix 
>> concerning the shared memory allocation may be not yet correctly coded ,
>>or that version contains another new bug in sharedmemory allocation  
>> compared to the working(!) 1.8.3-release version.
>>
>> Greetings to you all
>>   Michael Rachner
>> 
>>
>>
>>
>> -Ursprüngliche Nachricht-
>> Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Jeff 
>> Squyres (jsquyres)
>> Gesendet: Freitag, 24. Oktober 2014 22:45
>> An: Open MPI User's List
>> Betreff: Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in 
>> shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
>>
>> Nathan tells me that this may well be related to a fix that was literally 
>> just pulled into the v1.8 branch today:
>>
>> https://github.com/open-mpi/ompi-release/pull/56
>>
>> Would you mind testing any nightly tarball after tonight?  (i.e., the 
>> v1.8 tarballs generated tonight will be the first ones to contain this 
>> fix)
>>
>> http://www.open-mpi.org/nightly/master/
>>
>>
>>
>> On Oct 24, 2014, at 11:46 AM,  
>>  wrote:
>>
>>> Dear developers of OPENMPI,
>>>  
>>> I am running a small downsized Fortran-testprogram for shared memory 
>>> allocation (using MPI_WIN_ALLOCATE_SHARED and  MPI_WIN_SHARED_QUERY) )
>>> on only 1 node   of 2 different Linux-clusters with OPENMPI-1.8.3 and 
>>> Intel-14.0.4 /Intel-13.0.1, respectively.
>>>  
>>> The program sim

Re: [OMPI users] OPENMPI-1.8.3: missing fortran bindings for MPI_SIZEOF

[Gilles Gouaillardet]

Michael,

the root cause is openmpi was not compiled with the intel compilers but
the gnu compiler.
fortran modules are not binary compatible so openmpi and your
application must be compiled
with the same compiler.

Cheers,

Gilles

On 2014/11/05 18:25, michael.rach...@dlr.de wrote:
> Dear OPENMPI developers,
>
> In OPENMPI-1.8.3 the Ftn-bindings for  MPI_SIZEOF  are missing, when using 
> the mpi-module and when using mpif.h .
> (I have not controlled, whether they are present in the mpi_08 module.)
>
> I get this message from the linker (Intel-14.0.2):
>  /home/vat/src/KERNEL/mpi_ini.f90:534: undefined reference to 
> `mpi_sizeof0di4_'
>
> So can you add  the Ftn-bindings for MPI_SIZEOF?
>
> Once again I feel, that Fortran-bindings are unloved step-children for 
> C-programmers. 
>
> Greetings to you all
>  Michael Rachner
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/11/25676.php

Re: [OMPI users] OMPI users] OPENMPI-1.8.3: missing fortran bindings for MPI_SIZEOF

[Gilles Gouaillardet]

Michael,

Did you recompile with gfortran compiler or relink only ?
You need to recompile and relink
Can you attach your program so i can have a look ?

You really need one mpi install per compiler, and more if compilers versions 
from the same vendor are not compatible.
modules are useful to make this easy for end users, and this is out of the 
scope of openmpi.

Cheers,

Gilles

michael.rach...@dlr.de wrote:
>Sorry, Gilles, you might be wrong:
>
>The error occurs also with gfortran-4.9.1, when running my small shared memory 
>testprogram:
>
>This is the answer of the linker with gfortran-4.9.1 :  
> sharedmemtest.f90:(.text+0x1145): undefined reference to `mpi_sizeof0di4_'
>
>and this is the answer with intel-14.0.4:
>sharedmemtest.f90:(.text+0x11c3): undefined reference to `mpi_sizeof0di4_'
>
>
>If openmpi  actually provides a module file   mpi.mod,  that was  precompiled 
>by openmpi for a certain Fortran compiler,
>then the whole installation of openmpi on a User machine from the 
>openmpi-sourcecode for a User chosen Ftn-compiler would be a farce.
>The module file  mpi.mod  must be either generated during the installation 
>process of openmpi on the User-machine for the User chosen Ftn-compiler,
>or alternatively Openmpi must provide the module not by a  mpi.mod file,  but 
>a mpi.f90 file.  MS-MPI does it that way.
>In my opinion, providing a  mpi.f90  file is indeed  better than providing an  
>mpi.mod file, because the User can look inside the module
>and can directly see, if something is missing or possibly wrongly coded. 
>
>Greetings 
>  Michael Rachner
>
>
>-Ursprüngliche Nachricht-
>Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Gilles 
>Gouaillardet
>Gesendet: Mittwoch, 5. November 2014 11:33
>An: Open MPI Users
>Betreff: Re: [OMPI users] OPENMPI-1.8.3: missing fortran bindings for 
>MPI_SIZEOF
>
>Michael,
>
>the root cause is openmpi was not compiled with the intel compilers but the 
>gnu compiler.
>fortran modules are not binary compatible so openmpi and your application must 
>be compiled with the same compiler.
>
>Cheers,
>
>Gilles
>
>On 2014/11/05 18:25, michael.rach...@dlr.de wrote:
>> Dear OPENMPI developers,
>>
>> In OPENMPI-1.8.3 the Ftn-bindings for  MPI_SIZEOF  are missing, when using 
>> the mpi-module and when using mpif.h .
>> (I have not controlled, whether they are present in the mpi_08 
>> module.)
>>
>> I get this message from the linker (Intel-14.0.2):
>>  /home/vat/src/KERNEL/mpi_ini.f90:534: undefined reference to 
>> `mpi_sizeof0di4_'
>>
>> So can you add  the Ftn-bindings for MPI_SIZEOF?
>>
>> Once again I feel, that Fortran-bindings are unloved step-children for 
>> C-programmers. 
>>
>> Greetings to you all
>>  Michael Rachner
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/11/25676.php
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25682.php
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25683.php

Re: [OMPI users] OMPI users] How OMPI picks ethernet interfaces

[Gilles Gouaillardet]

Brock,

Is your post related to ib0/eoib0 being used at all, or being used with load 
balancing ?

let me clarify this :
--mca btl ^openib
disables the openib btl aka *native* infiniband.
This does not disable ib0 and eoib0 that are handled by the tcp btl.
As you already figured out, btl_tcp_if_include (or btl_tcp_if_exclude) can be 
used for that purpose.

Cheers,

Gilles




Ralph Castain  wrote:
>OMPI discovers all active interfaces and automatically considers them 
>available for its use unless instructed otherwise via the params. I’d have to 
>look at the TCP BTL code to see the loadbalancing algo - I thought we didn’t 
>have that “on” by default across BTLs, but I don’t know if the TCP one 
>automatically uses all available Ethernet interfaces by default. Sounds like 
>it must.
>
>
>> On Nov 7, 2014, at 11:53 AM, Brock Palen  wrote:
>> 
>> I was doing a test on our IB based cluster, where I was diabling IB
>> 
>> --mca btl ^openib --mca mtl ^mxm
>> 
>> I was sending very large messages >1GB  and I was surppised by the speed.
>> 
>> I noticed then that of all our ethernet interfaces
>> 
>> eth0  (1gig-e)
>> ib0  (ip over ib, for lustre configuration at vendor request)
>> eoib0  (ethernet over IB interface for IB -> Ethernet gateway for some 
>> extrnal storage support at >1Gig speed
>> 
>> I saw all three were getting traffic.
>> 
>> We use torque for our Resource Manager and use TM support, the hostnames 
>> given by torque match the eth0 interfaces.
>> 
>> How does OMPI figure out that it can also talk over the others?  How does it 
>> chose to load balance?
>> 
>> BTW that is fine, but we will use if_exclude on one of the IB ones as ib0 
>> and eoib0  are the same physical device and may screw with load balancing if 
>> anyone ver falls back to TCP.
>> 
>> Brock Palen
>> www.umich.edu/~brockp
>> CAEN Advanced Computing
>> XSEDE Campus Champion
>> bro...@umich.edu
>> (734)936-1985
>> 
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/11/25709.php
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25710.php

Re: [OMPI users] OMPI users] How OMPI picks ethernet interfaces

[Gilles Gouaillardet]

Ralph,

IIRC there is load balancing accros all the btl, for example
between vader and scif.
So load balancing between ib0 and eoib0 is just a particular case that might 
not necessarily be handled by the btl tcp.

Cheers,

Gilles

Ralph Castain  wrote:
>OMPI discovers all active interfaces and automatically considers them 
>available for its use unless instructed otherwise via the params. I’d have to 
>look at the TCP BTL code to see the loadbalancing algo - I thought we didn’t 
>have that “on” by default across BTLs, but I don’t know if the TCP one 
>automatically uses all available Ethernet interfaces by default. Sounds like 
>it must.
>
>
>> On Nov 7, 2014, at 11:53 AM, Brock Palen  wrote:
>> 
>> I was doing a test on our IB based cluster, where I was diabling IB
>> 
>> --mca btl ^openib --mca mtl ^mxm
>> 
>> I was sending very large messages >1GB  and I was surppised by the speed.
>> 
>> I noticed then that of all our ethernet interfaces
>> 
>> eth0  (1gig-e)
>> ib0  (ip over ib, for lustre configuration at vendor request)
>> eoib0  (ethernet over IB interface for IB -> Ethernet gateway for some 
>> extrnal storage support at >1Gig speed
>> 
>> I saw all three were getting traffic.
>> 
>> We use torque for our Resource Manager and use TM support, the hostnames 
>> given by torque match the eth0 interfaces.
>> 
>> How does OMPI figure out that it can also talk over the others?  How does it 
>> chose to load balance?
>> 
>> BTW that is fine, but we will use if_exclude on one of the IB ones as ib0 
>> and eoib0  are the same physical device and may screw with load balancing if 
>> anyone ver falls back to TCP.
>> 
>> Brock Palen
>> www.umich.edu/~brockp
>> CAEN Advanced Computing
>> XSEDE Campus Champion
>> bro...@umich.edu
>> (734)936-1985
>> 
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/11/25709.php
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25710.php

Re: [OMPI users] How OMPI picks ethernet interfaces

[Gilles Gouaillardet]

Hi,

IIRC there were some bug fixes between 1.8.1 and 1.8.2 in order to really
use all the published interfaces.

by any change, are you running a firewall on your head node ?
one possible explanation is the compute node tries to access the public
interface of the head node, and packets get dropped by the firewall.

if you are running a firewall, can you make a test without it ?
/* if you do need NAT, then just remove the DROP and REJECT rules "/

an other possible explanation is the compute node is doing (reverse) dns
requests with the public name and/or ip of the head node and that takes
some time to complete (success or failure, this does not really matter here)

/* a simple test is to make sure all the hosts/ip of the head node are in
the /etc/hosts of the compute node */

could you check your network config (firewall and dns) ?

can you reproduce the delay when running mpirun on the head node and with
one mpi task on the compute node ?

if yes, then the hard way to trace the delay issue would be to strace -ttt
both orted and mpi task that are launched on the compute node and see where
the time is lost.
/* at this stage, i would suspect orted ... */

Cheers,

Gilles

On Mon, Nov 10, 2014 at 5:56 PM, Reuti  wrote:

> Hi,
>
> Am 10.11.2014 um 16:39 schrieb Ralph Castain:
>
> > That is indeed bizarre - we haven’t heard of anything similar from other
> users. What is your network configuration? If you use oob_tcp_if_include or
> exclude, can you resolve the problem?
>
> Thx - this option helped to get it working.
>
> These tests were made for sake of simplicity between the headnode of the
> cluster and one (idle) compute node. I tried then between the (identical)
> compute nodes and this worked fine. The headnode of the cluster and the
> compute node are slightly different though (i.e. number of cores), and
> using eth1 resp. eth0 for the internal network of the cluster.
>
> I tried --hetero-nodes with no change.
>
> Then I turned to:
>
> reuti@annemarie:~> date; mpiexec -mca btl self,tcp --mca
> oob_tcp_if_include 192.168.154.0/26 -n 4 --hetero-nodes --hostfile
> machines ./mpihello; date
>
> and the application started instantly. On another cluster, where the
> headnode is identical to the compute nodes but with the same network setup
> as above, I observed a delay of "only" 30 seconds. Nevertheless, also on
> this cluster the working addition was the correct "oob_tcp_if_include" to
> solve the issue.
>
> The questions which remain: a) is this a targeted behavior, b) what
> changed in this scope between 1.8.1 and 1.8.2?
>
> -- Reuti
>
>
> >
> >> On Nov 10, 2014, at 4:50 AM, Reuti  wrote:
> >>
> >> Am 10.11.2014 um 12:50 schrieb Jeff Squyres (jsquyres):
> >>
> >>> Wow, that's pretty terrible!  :(
> >>>
> >>> Is the behavior BTL-specific, perchance?  E.G., if you only use
> certain BTLs, does the delay disappear?
> >>
> >> You mean something like:
> >>
> >> reuti@annemarie:~> date; mpiexec -mca btl self,tcp -n 4 --hostfile
> machines ./mpihello; date
> >> Mon Nov 10 13:44:34 CET 2014
> >> Hello World from Node 1.
> >> Total: 4
> >> Universe: 4
> >> Hello World from Node 0.
> >> Hello World from Node 3.
> >> Hello World from Node 2.
> >> Mon Nov 10 13:46:42 CET 2014
> >>
> >> (the above was even the latest v1.8.3-186-g978f61d)
> >>
> >> Falling back to 1.8.1 gives (as expected):
> >>
> >> reuti@annemarie:~> date; mpiexec -mca btl self,tcp -n 4 --hostfile
> machines ./mpihello; date
> >> Mon Nov 10 13:49:51 CET 2014
> >> Hello World from Node 1.
> >> Total: 4
> >> Universe: 4
> >> Hello World from Node 0.
> >> Hello World from Node 2.
> >> Hello World from Node 3.
> >> Mon Nov 10 13:49:53 CET 2014
> >>
> >>
> >> -- Reuti
> >>
> >>> FWIW: the use-all-IP interfaces approach has been in OMPI forever.
> >>>
> >>> Sent from my phone. No type good.
> >>>
>  On Nov 10, 2014, at 6:42 AM, Reuti 
> wrote:
> 
> > Am 10.11.2014 um 12:24 schrieb Reuti:
> >
> > Hi,
> >
> >> Am 09.11.2014 um 05:38 schrieb Ralph Castain:
> >>
> >> FWIW: during MPI_Init, each process “publishes” all of its
> interfaces. Each process receives a complete map of that info for every
> process in the job. So when the TCP btl sets itself up, it attempts to
> connect across -all- the interfaces published by the other end.
> >>
> >> So it doesn’t matter what hostname is provided by the RM. We
> discover and “share” all of the interface info for every node, and then use
> them for loadbalancing.
> >
> > does this lead to any time delay when starting up? I stayed with
> Open MPI 1.6.5 for some time and tried to use Open MPI 1.8.3 now. As there
> is a delay when the applications starts in my first compilation of 1.8.3 I
> disregarded even all my extra options and run it outside of any
> queuingsystem - the delay remains - on two different clusters.
> 
>  I forgot to mention: the delay is more or less exactly 2 minutes from
> the time I issued `mpiexec` until the `mpihello` starts up (there is 

Re: [OMPI users] OMPI users] How OMPI picks ethernet interfaces

[Gilles Gouaillardet]

Could you please send the output of netstat -nr on both head and compute node ?
no problem obfuscating the ip of the head node, i am only interested in 
netmasks and routes.

Ralph Castain  wrote:
>
>> On Nov 12, 2014, at 2:45 PM, Reuti  wrote:
>> 
>> Am 12.11.2014 um 17:27 schrieb Reuti:
>> 
>>> Am 11.11.2014 um 02:25 schrieb Ralph Castain:
>>> 
 Another thing you can do is (a) ensure you built with —enable-debug, and 
 then (b) run it with -mca oob_base_verbose 100  (without the 
 tcp_if_include option) so we can watch the connection handshake and see 
 what it is doing. The —hetero-nodes will have not affect here and can be 
 ignored.
>>> 
>>> Done. It really tries to connect to the outside interface of the headnode. 
>>> But being there a firewall or not: the nodes have no clue how to reach 
>>> 137.248.0.0 - they have no gateway to this network at all.
>> 
>> I have to revert this. They think that there is a gateway although it isn't. 
>> When I remove the entry by hand for the gateway in the routing table it 
>> starts up instantly too.
>> 
>> While I can do this on my own cluster I still have the 30 seconds delay on a 
>> cluster where I'm not root, while this can be because of the firewall there. 
>> The gateway on this cluster is indeed going to the outside world.
>> 
>> Personally I find this behavior a little bit too aggressive to use all 
>> interfaces. If you don't check this carefully beforehand and start a long 
>> running application one might even not notice the delay during the startup.
>
>Agreed - do you have any suggestions on how we should choose the order in 
>which to try them? I haven’t been able to come up with anything yet. Jeff has 
>some fancy algo in his usnic BTL that we are going to discuss after SC that 
>I’m hoping will help, but I’d be open to doing something better in the interim 
>for 1.8.4
>
>> 
>> -- Reuti
>> 
>> 
>>> It tries so independent from the internal or external name of the headnode 
>>> given in the machinefile - I hit ^C then. I attached the output of Open MPI 
>>> 1.8.1 for this setup too.
>>> 
>>> -- Reuti
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2014/11/25777.php
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/11/25781.php
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25782.php

Re: [OMPI users] mpirun fails across nodes

[Gilles Gouaillardet]

Hi,

it seems you messed up the command line

could you try

$ mpirun --mca btl ^openib --host compute-01-01,compute-01-06 ring_c


can you also try to run mpirun from a compute node instead of the head
node ?

Cheers,

Gilles

On 2014/11/13 16:07, Syed Ahsan Ali wrote:
> Here is what I see when disabling openib support.\
>
>
> [pmdtest@pmd ~]$ mpirun --host --mca btl ^openib
> compute-01-01,compute-01-06 ring_c
> ssh:  orted: Temporary failure in name resolution
> ssh:  orted: Temporary failure in name resolution
> --
> A daemon (pid 7608) died unexpectedly with status 255 while attempting
> to launch so we are aborting.
>
> While nodes can still ssh each other
>
> [pmdtest@compute-01-01 ~]$ ssh compute-01-06
> Last login: Thu Nov 13 12:05:58 2014 from compute-01-01.private.dns.zone
> [pmdtest@compute-01-06 ~]$
>
>
>
>
> On Thu, Nov 13, 2014 at 12:03 PM, Syed Ahsan Ali  
> wrote:
>>  Hi Jefff
>>
>> No firewall is enabled. Running the diagnostics I found that non
>> communication mpi job is running . While ring_c remains stuck. There
>> are of course warnings for open fabrics but in my case I an running
>> application by disabling openib., Please see below
>>
>>  [pmdtest@pmd ~]$ mpirun --host compute-01-01,compute-01-06 hello_c.out
>> --
>> WARNING: There is at least one OpenFabrics device found but there are
>> no active ports detected (or Open MPI was unable to use them).  This
>> is most certainly not what you wanted.  Check your cables, subnet
>> manager configuration, etc.  The openib BTL will be ignored for this
>> job.
>>   Local host: compute-01-01.private.dns.zone
>> --
>> Hello, world, I am 0 of 2
>> Hello, world, I am 1 of 2
>> [pmd.pakmet.com:06386] 1 more process has sent help message
>> help-mpi-btl-openib.txt / no active ports found
>> [pmd.pakmet.com:06386] Set MCA parameter "orte_base_help_aggregate" to
>> 0 to see all help / error messages
>>
>> [pmdtest@pmd ~]$ mpirun --host compute-01-01,compute-01-06 ring_c
>> --
>> WARNING: There is at least one OpenFabrics device found but there are
>> no active ports detected (or Open MPI was unable to use them).  This
>> is most certainly not what you wanted.  Check your cables, subnet
>> manager configuration, etc.  The openib BTL will be ignored for this
>> job.
>>   Local host: compute-01-01.private.dns.zone
>> --
>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>> Process 0 sent to 1
>> Process 0 decremented value: 9
>> [compute-01-01.private.dns.zone][[54687,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
>> connect() to 192.168.108.10 failed: No route to host (113)
>> [pmd.pakmet.com:15965] 1 more process has sent help message
>> help-mpi-btl-openib.txt / no active ports found
>> [pmd.pakmet.com:15965] Set MCA parameter "orte_base_help_aggregate" to
>> 0 to see all help / error messages
>> 
>>
>>
>>
>>
>>
>> On Wed, Nov 12, 2014 at 7:32 PM, Jeff Squyres (jsquyres)
>>  wrote:
>>> Do you have firewalling enabled on either server?
>>>
>>> See this FAQ item:
>>>
>>> 
>>> http://www.open-mpi.org/faq/?category=running#diagnose-multi-host-problems
>>>
>>>
>>>
>>> On Nov 12, 2014, at 4:57 AM, Syed Ahsan Ali  wrote:
>>>
 Dear All

 I need your advice. While trying to run mpirun job across nodes I get
 following error. It seems that the two nodes i.e, compute-01-01 and
 compute-01-06 are not able to communicate with each other. While nodes
 see each other on ping.

 [pmdtest@pmd ERA_CLM45]$ mpirun -np 16 -hostfile hostlist --mca btl
 ^openib ../bin/regcmMPICLM45 regcm.in

 [compute-01-06.private.dns.zone][[48897,1],7][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 192.168.108.14 failed: No route to host (113)
 [compute-01-06.private.dns.zone][[48897,1],4][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 192.168.108.14 failed: No route to host (113)
 [compute-01-06.private.dns.zone][[48897,1],5][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 192.168.108.14 failed: No route to host (113)
 [compute-01-01.private.dns.zone][[48897,1],10][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 [compute-01-01.private.dns.zone][[48897,1],12][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 192.168.108.10 failed: No route to host (113)
 [compute-01-01.private.dns.zone][[48897,1],14][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 192.168.108.10 failed: No route to host (113)
 connect() to 192.168.108.10 failed: No route to host (113)

>>>

Re: [OMPI users] mpirun fails across nodes

[Gilles Gouaillardet]

mpirun complains about the 192.168.108.10 ip address, but ping reports a
10.0.0.8 address

is the 192.168.* network a point to point network (for example between a
host and a mic) so two nodes
cannot ping each other via this address ?
/* e.g. from compute-01-01 can you ping the 192.168.108.* ip address of
compute-01-06 ? */

could you also run

mpirun --mca btl ^openib --host compute-01-01,compute-01-06 --mca
btl_tcp_if_include 10.0.0.0/8 ring_c

and see whether it helps ?


On 2014/11/13 16:24, Syed Ahsan Ali wrote:
> Same result in both cases
>
> [pmdtest@pmd ~]$ mpirun --mca btl ^openib --host
> compute-01-01,compute-01-06 ring_c
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> [compute-01-01.private.dns.zone][[47139,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
> connect() to 192.168.108.10 failed: No route to host (113)
>
>
> [pmdtest@compute-01-01 ~]$ mpirun --mca btl ^openib --host
> compute-01-01,compute-01-06 ring_c
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> [compute-01-01.private.dns.zone][[11064,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
> connect() to 192.168.108.10 failed: No route to host (113)
>
>
> On Thu, Nov 13, 2014 at 12:11 PM, Gilles Gouaillardet
>  wrote:
>> Hi,
>>
>> it seems you messed up the command line
>>
>> could you try
>>
>> $ mpirun --mca btl ^openib --host compute-01-01,compute-01-06 ring_c
>>
>>
>> can you also try to run mpirun from a compute node instead of the head
>> node ?
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/11/13 16:07, Syed Ahsan Ali wrote:
>>> Here is what I see when disabling openib support.\
>>>
>>>
>>> [pmdtest@pmd ~]$ mpirun --host --mca btl ^openib
>>> compute-01-01,compute-01-06 ring_c
>>> ssh:  orted: Temporary failure in name resolution
>>> ssh:  orted: Temporary failure in name resolution
>>> --
>>> A daemon (pid 7608) died unexpectedly with status 255 while attempting
>>> to launch so we are aborting.
>>>
>>> While nodes can still ssh each other
>>>
>>> [pmdtest@compute-01-01 ~]$ ssh compute-01-06
>>> Last login: Thu Nov 13 12:05:58 2014 from compute-01-01.private.dns.zone
>>> [pmdtest@compute-01-06 ~]$
>>>
>>>
>>>
>>>
>>> On Thu, Nov 13, 2014 at 12:03 PM, Syed Ahsan Ali  
>>> wrote:
>>>>  Hi Jefff
>>>>
>>>> No firewall is enabled. Running the diagnostics I found that non
>>>> communication mpi job is running . While ring_c remains stuck. There
>>>> are of course warnings for open fabrics but in my case I an running
>>>> application by disabling openib., Please see below
>>>>
>>>>  [pmdtest@pmd ~]$ mpirun --host compute-01-01,compute-01-06 hello_c.out
>>>> --
>>>> WARNING: There is at least one OpenFabrics device found but there are
>>>> no active ports detected (or Open MPI was unable to use them).  This
>>>> is most certainly not what you wanted.  Check your cables, subnet
>>>> manager configuration, etc.  The openib BTL will be ignored for this
>>>> job.
>>>>   Local host: compute-01-01.private.dns.zone
>>>> --
>>>> Hello, world, I am 0 of 2
>>>> Hello, world, I am 1 of 2
>>>> [pmd.pakmet.com:06386] 1 more process has sent help message
>>>> help-mpi-btl-openib.txt / no active ports found
>>>> [pmd.pakmet.com:06386] Set MCA parameter "orte_base_help_aggregate" to
>>>> 0 to see all help / error messages
>>>>
>>>> [pmdtest@pmd ~]$ mpirun --host compute-01-01,compute-01-06 ring_c
>>>> --
>>>> WARNING: There is at least one OpenFabrics device found but there are
>>>> no active ports detected (or Open MPI was unable to use them).  This
>>>> is most certainly not what you wanted.  Check your cables, subnet
>>>> manager configuration, etc.  The openib BTL will be ignored for this
>>>> job.
>>>>   Local host: compute-01-01.private.dns.zone
>>>> --
>>>> Process 0 

Re: [OMPI users] mpirun fails across nodes

[Gilles Gouaillardet]

--mca btl ^openib
disables the openib btl, which is native infiniband only.

ib0 is treated as any TCP interface and then handled by the tcp btl

an other option is you to use
--mca btl_tcp_if_exclude ib0

On 2014/11/13 16:43, Syed Ahsan Ali wrote:
> You are right it is running on 10.0.0.0 interface [pmdtest@pmd ~]$
> mpirun --mca btl ^openib --host compute-01-01,compute-01-06 --mca
> btl_tcp_if_include 10.0.0.0/8 ring_c
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> Process 0 decremented value: 8
> Process 0 decremented value: 7
> Process 0 decremented value: 6
> Process 1 exiting
> Process 0 decremented value: 5
> Process 0 decremented value: 4
> Process 0 decremented value: 3
> Process 0 decremented value: 2
> Process 0 decremented value: 1
> Process 0 decremented value: 0
> Process 0 exiting
> [pmdtest@pmd ~]$
>
> While the ip addresses 192.168.108* are for ib interface.
>
>  [root@compute-01-01 ~]# ifconfig
> eth0  Link encap:Ethernet  HWaddr 00:24:E8:59:4C:2A
>   inet addr:10.0.0.3  Bcast:10.255.255.255  Mask:255.0.0.0
>   inet6 addr: fe80::224:e8ff:fe59:4c2a/64 Scope:Link
>   UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>   RX packets:65588 errors:0 dropped:0 overruns:0 frame:0
>   TX packets:14184 errors:0 dropped:0 overruns:0 carrier:0
>   collisions:0 txqueuelen:1000
>   RX bytes:18692977 (17.8 MiB)  TX bytes:1834122 (1.7 MiB)
>   Interrupt:169 Memory:dc00-dc012100
> ib0   Link encap:InfiniBand  HWaddr
> 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
>   inet addr:192.168.108.14  Bcast:192.168.108.255  Mask:255.255.255.0
>   UP BROADCAST MULTICAST  MTU:65520  Metric:1
>   RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>   TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>   collisions:0 txqueuelen:256
>   RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>
>
>
> So the point is why mpirun is following the ib  path while I it has
> been disabled. Possible solutions?
>
> On Thu, Nov 13, 2014 at 12:32 PM, Gilles Gouaillardet
>  wrote:
>> mpirun complains about the 192.168.108.10 ip address, but ping reports a
>> 10.0.0.8 address
>>
>> is the 192.168.* network a point to point network (for example between a
>> host and a mic) so two nodes
>> cannot ping each other via this address ?
>> /* e.g. from compute-01-01 can you ping the 192.168.108.* ip address of
>> compute-01-06 ? */
>>
>> could you also run
>>
>> mpirun --mca btl ^openib --host compute-01-01,compute-01-06 --mca
>> btl_tcp_if_include 10.0.0.0/8 ring_c
>>
>> and see whether it helps ?
>>
>>
>> On 2014/11/13 16:24, Syed Ahsan Ali wrote:
>>> Same result in both cases
>>>
>>> [pmdtest@pmd ~]$ mpirun --mca btl ^openib --host
>>> compute-01-01,compute-01-06 ring_c
>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>> Process 0 sent to 1
>>> Process 0 decremented value: 9
>>> [compute-01-01.private.dns.zone][[47139,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
>>> connect() to 192.168.108.10 failed: No route to host (113)
>>>
>>>
>>> [pmdtest@compute-01-01 ~]$ mpirun --mca btl ^openib --host
>>> compute-01-01,compute-01-06 ring_c
>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>> Process 0 sent to 1
>>> Process 0 decremented value: 9
>>> [compute-01-01.private.dns.zone][[11064,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
>>> connect() to 192.168.108.10 failed: No route to host (113)
>>>
>>>
>>> On Thu, Nov 13, 2014 at 12:11 PM, Gilles Gouaillardet
>>>  wrote:
>>>> Hi,
>>>>
>>>> it seems you messed up the command line
>>>>
>>>> could you try
>>>>
>>>> $ mpirun --mca btl ^openib --host compute-01-01,compute-01-06 ring_c
>>>>
>>>>
>>>> can you also try to run mpirun from a compute node instead of the head
>>>> node ?
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>> On 2014/11/13 16:07, Syed Ahsan Ali wrote:
>>>>> Here is what I see when disabling openib support.\
>>>>>
>>>>>
>>>>> [pmdtest@pmd ~]$ mpirun --host --mca btl ^openib
>>>>> compute-01-01,compute-01-06 ring_c
>>>>> ssh:  orted: Temporary failure in name resolution
>>>>> ssh: 

Re: [OMPI users] mpirun fails across nodes

[Gilles Gouaillardet]

This is really weird ?

is the loopback interface up and running on both nodes and with the same
ip ?

can you run on both compute nodes ?
netstat -nr


On 2014/11/13 16:50, Syed Ahsan Ali wrote:
> Now it looks through the loopback address
>
> [pmdtest@pmd ~]$ mpirun --host compute-01-01,compute-01-06 --mca
> btl_tcp_if_exclude ib0 ring_c
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> [compute-01-01.private.dns.zone][[37713,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
> connect() to 127.0.0.1 failed: Connection refused (111)
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> [pmd.pakmet.com:30867] 1 more process has sent help message
> help-mpi-btl-openib.txt / no active ports found
> [pmd.pakmet.com:30867] Set MCA parameter "orte_base_help_aggregate" to
> 0 to see all help / error messages
>
>
>
> On Thu, Nov 13, 2014 at 12:46 PM, Gilles Gouaillardet
>  wrote:
>> --mca btl ^openib
>> disables the openib btl, which is native infiniband only.
>>
>> ib0 is treated as any TCP interface and then handled by the tcp btl
>>
>> an other option is you to use
>> --mca btl_tcp_if_exclude ib0
>>
>> On 2014/11/13 16:43, Syed Ahsan Ali wrote:
>>> You are right it is running on 10.0.0.0 interface [pmdtest@pmd ~]$
>>> mpirun --mca btl ^openib --host compute-01-01,compute-01-06 --mca
>>> btl_tcp_if_include 10.0.0.0/8 ring_c
>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>> Process 0 sent to 1
>>> Process 0 decremented value: 9
>>> Process 0 decremented value: 8
>>> Process 0 decremented value: 7
>>> Process 0 decremented value: 6
>>> Process 1 exiting
>>> Process 0 decremented value: 5
>>> Process 0 decremented value: 4
>>> Process 0 decremented value: 3
>>> Process 0 decremented value: 2
>>> Process 0 decremented value: 1
>>> Process 0 decremented value: 0
>>> Process 0 exiting
>>> [pmdtest@pmd ~]$
>>>
>>> While the ip addresses 192.168.108* are for ib interface.
>>>
>>>  [root@compute-01-01 ~]# ifconfig
>>> eth0  Link encap:Ethernet  HWaddr 00:24:E8:59:4C:2A
>>>   inet addr:10.0.0.3  Bcast:10.255.255.255  Mask:255.0.0.0
>>>   inet6 addr: fe80::224:e8ff:fe59:4c2a/64 Scope:Link
>>>   UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>   RX packets:65588 errors:0 dropped:0 overruns:0 frame:0
>>>   TX packets:14184 errors:0 dropped:0 overruns:0 carrier:0
>>>   collisions:0 txqueuelen:1000
>>>   RX bytes:18692977 (17.8 MiB)  TX bytes:1834122 (1.7 MiB)
>>>   Interrupt:169 Memory:dc00-dc012100
>>> ib0   Link encap:InfiniBand  HWaddr
>>> 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
>>>   inet addr:192.168.108.14  Bcast:192.168.108.255  
>>> Mask:255.255.255.0
>>>   UP BROADCAST MULTICAST  MTU:65520  Metric:1
>>>   RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>>   TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>>   collisions:0 txqueuelen:256
>>>   RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>>>
>>>
>>>
>>> So the point is why mpirun is following the ib  path while I it has
>>> been disabled. Possible solutions?
>>>
>>> On Thu, Nov 13, 2014 at 12:32 PM, Gilles Gouaillardet
>>>  wrote:
>>>> mpirun complains about the 192.168.108.10 ip address, but ping reports a
>>>> 10.0.0.8 address
>>>>
>>>> is the 192.168.* network a point to point network (for example between a
>>>> host and a mic) so two nodes
>>>> cannot ping each other via this address ?
>>>> /* e.g. from compute-01-01 can you ping the 192.168.108.* ip address of
>>>> compute-01-06 ? */
>>>>
>>>> could you also run
>>>>
>>>> mpirun --mca btl ^openib --host compute-01-01,compute-01-06 --mca
>>>> btl_tcp_if_include 10.0.0.0/8 ring_c
>>>>
>>>> and see whether it helps ?
>>>>
>>>>
>>>> On 2014/11/13 16:24, Syed Ahsan Ali wrote:
>>>>> Same result in both cases
>>>>>
>>>>> [pmdtest@pmd ~]$ mpirun --mca btl ^openib --host
>>>>> compute-01-01,compute-01-06 ring_c
>>>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>>>> Process 0 sent to 1
>>>>> Process 0 decremented value: 9
>>>>> [compute-01-

Re: [OMPI users] mpirun fails across nodes

[Gilles Gouaillardet]

but it is running on your head node isnt't it ?

you might want to double check why there is no loopback interface on
your compute nodes.
in the mean time, you can disable lo and ib0 interfaces

Cheers,

Gilles

On 2014/11/13 16:59, Syed Ahsan Ali wrote:
>  I don't see it running
>
> [pmdtest@compute-01-01 ~]$ netstat -nr
> Kernel IP routing table
> Destination Gateway Genmask Flags   MSS Window  irtt Iface
> 192.168.108.0   0.0.0.0 255.255.255.0   U 0 0  0 ib0
> 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0  0 ib0
> 239.0.0.0   0.0.0.0 255.0.0.0   U 0 0  0 eth0
> 10.0.0.00.0.0.0 255.0.0.0   U 0 0  0 eth0
> 0.0.0.0 10.0.0.10.0.0.0 UG0 0  0 eth0
> [pmdtest@compute-01-01 ~]$ exit
> logout
> Connection to compute-01-01 closed.
> [pmdtest@pmd ~]$ ssh compute-01-06
> Last login: Thu Nov 13 12:06:14 2014 from compute-01-01.private.dns.zone
> [pmdtest@compute-01-06 ~]$ netstat -nr
> Kernel IP routing table
> Destination Gateway Genmask Flags   MSS Window  irtt Iface
> 192.168.108.0   0.0.0.0 255.255.255.0   U 0 0  0 ib0
> 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0  0 ib0
> 239.0.0.0   0.0.0.0 255.0.0.0   U 0 0  0 eth0
> 10.0.0.00.0.0.0 255.0.0.0   U 0 0  0 eth0
> 0.0.0.0 10.0.0.10.0.0.0 UG0 0  0 eth0
> [pmdtest@compute-01-06 ~]$
> 
>
> On Thu, Nov 13, 2014 at 12:56 PM, Gilles Gouaillardet
>  wrote:
>> This is really weird ?
>>
>> is the loopback interface up and running on both nodes and with the same
>> ip ?
>>
>> can you run on both compute nodes ?
>> netstat -nr
>>
>>
>> On 2014/11/13 16:50, Syed Ahsan Ali wrote:
>>> Now it looks through the loopback address
>>>
>>> [pmdtest@pmd ~]$ mpirun --host compute-01-01,compute-01-06 --mca
>>> btl_tcp_if_exclude ib0 ring_c
>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>> [compute-01-01.private.dns.zone][[37713,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
>>> connect() to 127.0.0.1 failed: Connection refused (111)
>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>> [pmd.pakmet.com:30867] 1 more process has sent help message
>>> help-mpi-btl-openib.txt / no active ports found
>>> [pmd.pakmet.com:30867] Set MCA parameter "orte_base_help_aggregate" to
>>> 0 to see all help / error messages
>>>
>>>
>>>
>>> On Thu, Nov 13, 2014 at 12:46 PM, Gilles Gouaillardet
>>>  wrote:
>>>> --mca btl ^openib
>>>> disables the openib btl, which is native infiniband only.
>>>>
>>>> ib0 is treated as any TCP interface and then handled by the tcp btl
>>>>
>>>> an other option is you to use
>>>> --mca btl_tcp_if_exclude ib0
>>>>
>>>> On 2014/11/13 16:43, Syed Ahsan Ali wrote:
>>>>> You are right it is running on 10.0.0.0 interface [pmdtest@pmd ~]$
>>>>> mpirun --mca btl ^openib --host compute-01-01,compute-01-06 --mca
>>>>> btl_tcp_if_include 10.0.0.0/8 ring_c
>>>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>>>> Process 0 sent to 1
>>>>> Process 0 decremented value: 9
>>>>> Process 0 decremented value: 8
>>>>> Process 0 decremented value: 7
>>>>> Process 0 decremented value: 6
>>>>> Process 1 exiting
>>>>> Process 0 decremented value: 5
>>>>> Process 0 decremented value: 4
>>>>> Process 0 decremented value: 3
>>>>> Process 0 decremented value: 2
>>>>> Process 0 decremented value: 1
>>>>> Process 0 decremented value: 0
>>>>> Process 0 exiting
>>>>> [pmdtest@pmd ~]$
>>>>>
>>>>> While the ip addresses 192.168.108* are for ib interface.
>>>>>
>>>>>  [root@compute-01-01 ~]# ifconfig
>>>>> eth0  Link encap:Ethernet  HWaddr 00:24:E8:59:4C:2A
>>>>>   inet addr:10.0.0.3  Bcast:10.255.255.255  Mask:255.0.0.0
>>>>>   inet6 addr: fe80::224:e8ff:fe59:4c2a/64 Scope:Link
>>>>>   UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>>>   RX packets:65588 errors:0 dropped:0 overruns:0 frame:0
>>>>>   TX packets:141

Re: [OMPI users] How OMPI picks ethernet interfaces

[Gilles Gouaillardet]

My 0.02 US$

first, the root cause of the problem was a default gateway was
configured on the node,
but this gateway was unreachable.
imho, this is incorrect system setting that can lead to unpredictable
results :
- openmpi 1.8.1 works (you are lucky, good for you)
- openmpi 1.8.3 fails (no luck this time, too bad)
so i believe it is incorrect to blame openmpi for this.

that being said, you raise some good points of how to improve user
friendliness for end users
that have limited skills and/or interest in OpenMPI and system
administration.

basically, i agree with Gus. HPC is complex, not every clusters are the same
and imho some minimal config/tuning might not be avoided to get OpenMPI
working,
or operating at full speed.


let me give a few examples :

you recommend OpenMPI uses only the interfaces that matches the
hostnames in the machinefile.
what if you submit from the head node ? should you use the interface
that matches the hostname ?
what if this interface is the public interface, there is a firewall
and/or compute nodes have no default gateway ?
that will simply not work ...
so mpirun needs to pass orted all its interfaces.
which one should be picked by orted ?
- the first one ? it might be the unreachable public interface ...
- the one on the same subnet ? what if none is on the same subnet ?
  on the cluster i am working, eth0 are in different subnets, ib0 is on
a single subnet
  and i do *not* want to use ib0. but on some other clusters, the
ethernet network is so cheap
  they *want* to use ib0.

on your cluster, you want to use eth0 for oob and mpi, and eth1 for NFS.
that is legitimate.
in my case, i want to use eth0 (gigE) for oob and eth2 (10gigE) for MPI.
that is legitimate too.

we both want OpenMPI works *and* with best performance out of the box.
it is a good thing to have high expectations, but they might not all be met.

i'd rather implement some pre-defined policies that rules how ethernet
interfaces should be picked up,
and add a FAQ that mentions : if it does not work (or does not work as
fast as expected) out of the box, you should
at first try an other policy.

then the next legitimate question will be "what is the default policy" ?
regardless the answer, it will be good for some and bad for others.


imho, posting a mail to the OMPI users mailing list was the right thing
to do :
- you got help on how to troubleshot and fix the issue
- we got some valuable feedback on endusers expectations.

Cheers,

Gilles

On 2014/11/14 3:36, Gus Correa wrote:
> On 11/13/2014 11:14 AM, Ralph Castain wrote:
>> Hmmm…I’m beginning to grok the issue. It is a tad unusual for people to
>> assign different hostnames to their interfaces - I’ve seen it in the
>> Hadoop world, but not in HPC. Still, no law against it.
>
> No, not so unusual.
> I have clusters from respectable vendors that come with
> /etc/hosts for name resolution of the various interfaces.
> If I remember right, Rocks clusters also does that (or actually
> allow the sys admin to setup additional networks and at that point
> will append /etc/hosts with the additional names, or perhaps put those
> names in DHCP).
> I am not so familiar to xcat, but I think it has similar DHCP
> functionality, or maybe DNS on the head node.
>
> Having said that, I don't think this is an obstacle to setting up the
> right "if_include/if_exlculde" choices (along with the btl, oob, etc),
> for each particular cluster in the mca parameter configuration file.
> That is what my parallel conversation with Reuti was about.
>
> I believe the current approach w.r.t. interfaces:
> "use everythint, let the sysadmin/user restrict as
> (s)he sees fit" is both a wise and flexible way to do it.
> Guessing the "right interface to use" sounds risky to me (wrong
> choices may happen), and a bit of a cast.
>
>>
>> This will take a little thought to figure out a solution. One problem
>> that immediately occurs is if someone includes a hostfile that has lines
>> which refer to the same physical server, but using different interface
>> names. We’ll think those are completely distinct servers, and so the
>> process placement will be totally messed up.
>>
>
> Sure, and besides this, there will be machines with
> inconsistent/wrong/conflicting name resolution schemes
> that the current OMPI approach simply (and wisely) ignores.
>
>
>> We’ll also encounter issues with the daemon when it reports back, as the
>> hostname it gets will almost certainly differ from the hostname we were
>> expecting. Not as critical, but need to check to see where that will
>> impact the code base
>>
>
> I'm sure that will happen.
> Torque uses hostname by default for several things, and it can be a
> configuration nightmare to workaround that when what hostname reports
> is not what you want.
>
> IMHO, you may face a daunting guesswork task to get this right,
> to pick the
> interfaces that are best for a particular computer or cluster.
> It is so much easier to let the sysadmin/user, who presumably know

Re: [OMPI users] OMPI users] error building openmpi-dev-274-g2177f9e withgcc-4.9.2

[Gilles Gouaillardet]

Siegmar,

This is correct, --enable-heterogenous is now fixed in the trunk.

Please also note that -D_REENTRANT is now automatically set on solaris

Cheers

Gilles

Siegmar Gross  wrote:
>Hi Jeff, hi Ralph,
>
>> This issue should now be fixed, too.
>
>Yes, it is. Thank you very much for your help. It seems that even
>--enable-heterogeneous is working once more (Gilles told me last
>month that it was broken in the trunk), because I could successfully
>multiply a small matrix in my heterogeneous environment.
>
>
>Kind regards and thank you very much once more
>
>Siegmar
>
>
>
>> On Nov 14, 2014, at 12:04 PM, Siegmar Gross 
>>  wrote:
>> 
>> > Hi,
>> > 
>> > today I tried to install openmpi-dev-274-g2177f9e on my machines
>> > (Solaris 10 Sparc, Solaris 10 x86_64, and openSUSE Linux 12.1
>> > x86_64) with gcc-4.9.2 and got the following error on all three
>> > platforms.
>> > 
>> > tyr openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc 117 tail -25 
>> > log.make.Linux.x86_64.64_gcc
>> >  SED  mpi/man/man3/MPI_Wtime.3
>> >  SED  mpi/man/man3/OpenMPI.3
>> > make[2]: Leaving directory 
>`/export2/src/openmpi-1.9/openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc/ompi'
>> > Making all in mpi/cxx
>> > make[2]: Entering directory 
>`/export2/src/openmpi-1.9/openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc/ompi/mpi/cxx'
>> >  CXX  mpicxx.lo
>> > In file included from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/mca/rte/orte/rte_orte.h:33:0,
>> > from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/mca/rte/rte.h:195,
>> > from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/errhandler/errhandler.h:34,
>> > from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/mpi/cxx/mpicxx.cc:37:
>> > ../../../../openmpi-dev-274-g2177f9e/orte/mca/routed/routed.h:52:8: error: 
>> > using typedef-name 
>'orte_process_name_t' after 'struct'
>> > struct orte_process_name_t;
>> >^
>> > In file included from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/mca/rte/orte/rte_orte.h:29:0,
>> > from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/mca/rte/rte.h:195,
>> > from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/errhandler/errhandler.h:34,
>> > from 
>> > ../../../../openmpi-dev-274-g2177f9e/ompi/mpi/cxx/mpicxx.cc:37:
>> > ../../../../openmpi-dev-274-g2177f9e/orte/include/orte/types.h:102:29: 
>> > note: 'orte_process_name_t' 
>has a previous declaration here
>> > typedef opal_process_name_t orte_process_name_t;
>> > ^
>> > make[2]: *** [mpicxx.lo] Error 1
>> > make[2]: Leaving directory 
>`/export2/src/openmpi-1.9/openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc/ompi/mpi/cxx'
>> > make[1]: *** [all-recursive] Error 1
>> > make[1]: Leaving directory 
>`/export2/src/openmpi-1.9/openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc/ompi'
>> > make: *** [all-recursive] Error 1
>> > tyr openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc 118 
>> > 
>> > 
>> > I used the following configure command.
>> > 
>> > tyr openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc 118 head config.log | 
>> > grep openmpi
>> >  $ ../openmpi-dev-274-g2177f9e/configure 
>> > --prefix=/usr/local/openmpi-1.9.0_64_gcc 
>--libdir=/usr/local/openmpi-1.9.0_64_gcc/lib64 
>> > --with-jdk-bindir=/usr/local/jdk1.8.0/bin 
>> > --with-jdk-headers=/usr/local/jdk1.8.0/include 
>JAVA_HOME=/usr/local/jdk1.8.0 LDFLAGS=-m64 
>> > CC=gcc CXX=g++ FC=gfortran CFLAGS=-m64 -D_REENTRANT CXXFLAGS=-m64 
>> > FCFLAGS=-m64 CPP=cpp CXXCPP=cpp 
>CPPFLAGS= -D_REENTRANT 
>> > CXXCPPFLAGS= --enable-mpi-cxx --enable-cxx-exceptions --enable-mpi-java 
>--enable-mpi-thread-multiple --with-threads=posix 
>> > --with-hwloc=internal --without-verbs --with-wrapper-cflags=-std=c11 -m64 
>--with-wrapper-cxxflags=-m64 --enable-debug
>> > tyr openmpi-dev-274-g2177f9e-Linux.x86_64.64_gcc 119 
>> > 
>> > 
>> > I would be grateful. if somebody can fix the problem. Thank
>> > you very much for any help in advance.
>> > 
>> > 
>> > Kind regards
>> > 
>> > Siegmar
>> > 
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post: 
>> > http://www.open-mpi.org/community/lists/users/2014/11/25815.php
>> 
>> 
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> 
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25819.php

Re: [OMPI users] Fortran and OpenMPI 1.8.3 compiled with Intel-15 does nothing silently

[Gilles Gouaillardet]

Hi John,

do you MPI_Init() or do you MPI_Init_thread(MPI_THREAD_MULTIPLE) ?

does your program calls MPI anywhere from an OpenMP region ?
does your program calls MPI only within an !$OMP MASTER section ?
does your program does not invoke MPI at all from any OpenMP region ?

can you reproduce this issue with a simple fortran program ? or can you
publish all your files ?

Cheers,

Gilles

On 2014/11/18 1:41, John Bray wrote:
> I have succesfully been using OpenMPI 1.8.3 compiled with Intel-14, using
>
> ./configure --prefix=/usr/local/mpi/$(basename $PWD) --with-threads=posix
> --enable-mpi-thread-multiple --disable-vt --with-scif=no
>
> I have now switched to Intel 15.0.1, and configuring with the same options,
> I get minor changes in config.log about warning spotting, but it makes all
> the binaries, and I can compile my own fortran code with mpif90/mpicc
>
> but a command 'mpiexec --verbose -n 12 ./fortran_binary' does nothing
>
> I checked the FAQ and started using
>
> ./configure --prefix=/usr/local/mpi/$(basename $PWD) --with-threads=posix
> --enable-mpi-thread-multiple --disable-vt --with-scif=no CC=icc CXX=icpc
> F77=ifort FC=ifort
>
> but that makes no difference.
>
> Only with -d do I get any more information
>
> mpirun -d --verbose -n 12
> /home/jbray/5.0/mic2/one/intel-15_openmpi-1.8.3/one_f_debug.exe
> [mic2:21851] procdir: /tmp/openmpi-sessions-jbray@mic2_0/27642/0/0
> [mic2:21851] jobdir: /tmp/openmpi-sessions-jbray@mic2_0/27642/0
> [mic2:21851] top: openmpi-sessions-jbray@mic2_0
> [mic2:21851] tmp: /tmp
> [mic2:21851] sess_dir_cleanup: job session dir does not exist
> [mic2:21851] procdir: /tmp/openmpi-sessions-jbray@mic2_0/27642/0/0
> [mic2:21851] jobdir: /tmp/openmpi-sessions-jbray@mic2_0/27642/0
> [mic2:21851] top: openmpi-sessions-jbray@mic2_0
> [mic2:21851] tmp: /tmp
> [mic2:21851] sess_dir_finalize: proc session dir does not exist
> <12 times>
>
>
> [mic2:21851] sess_dir_cleanup: job session dir does not exist
> exiting with status 139
>
> My C codes do not have this problem
>
> Compiler options are
>
> mpicxx -g -O0 -fno-inline-functions -openmp -o one_c_debug.exe async.c
> collective.c compute.c memory.c one.c openmp.c p2p.c variables.c
> auditmpi.c   control.c inout.c perfio.c ring.c wave.c io.c   leak.c mpiio.c
> pthreads.c -openmp -lpthread
>
> mpif90 -g -O0  -fno-inline-functions -openmp -o one_f_debug.exe control.o
> io.f90 leak.f90 memory.f90 one.f90 ring.f90 slow.f90 swapbounds.f90
> variables.f90 wave.f90 *.F90 -openmp
>
> Any suggestions as to what is upsetting Fortran with Intel-15
>
> John
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/11/25823.php

Re: [OMPI users] collective algorithms

[Gilles Gouaillardet]

Daniel,

you can run
$ ompi_info --parseable --all | grep _algorithm: | grep enumerator

that will give you the list of supported algo for the collectives,
here is a sample output :

mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:0:ignore
mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:1:basic_linear
mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:2:nonoverlapping
mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:3:recursive_doubling
mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:4:ring
mca:coll:tuned:param:coll_tuned_allreduce_algorithm:enumerator:value:5:segmented_ring


the decision (which algo is used based on communicator size/message
size/...) is made in
ompi/mca/coll/tuned/coll_tuned_decision_fixed.c
and can be overriden via config file or environment variable

i cannot point you to a paper, and hopefully someone else will

Cheers,

Gilles


On 2014/11/18 12:53, Faraj, Daniel A wrote:
> I am trying to survey the collective algorithms in Open MPI.
> I looked at the src code but could not make out the guts of the communication 
> algorithms.
> There are some open mpi papers but not detailed, where they talk about what 
> algorithms are using in certain collectives.
> Has anybody done this sort of work, or point me to a paper?
>
> Basically, for a given collective operation, what are:
>
> a)  Communication algorithm being used for a given criteria (i.e. message 
> size or np)
>
> b)  What is theoretical algorithm cost
>
> Thanx
>
>
> ---
> Daniel Faraj
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/11/25831.php

Re: [OMPI users] MPI_Neighbor_alltoallw fails with mpi-1.8.3

[Gilles Gouaillardet]

Hi Ghislain,

that sound like a but in MPI_Dist_graph_create :-(

you can use MPI_Dist_graph_create_adjacent instead :

MPI_Dist_graph_create_adjacent(MPI_COMM_WORLD, degrees, &targets[0],
&weights[0],
degrees, &targets[0], &weights[0], info,
rankReordering, &commGraph);

it does not crash and as far as i understand, it produces correct results,

according the the mpi standard (example 7.3) that should do the same
thing, that's why
i think there is a bug in MPI_Dist_graph_create

Cheers,

Gilles



On 2014/11/21 2:21, Howard Pritchard wrote:
> Hi Ghislain,
>
> I tried to run your test with mvapich 1.9 and get a "message truncated"
> failure at three ranks.
>
> Howard
>
>
> 2014-11-20 8:51 GMT-07:00 Ghislain Viguier :
>
>> Dear support,
>>
>> I'm encountering an issue with the MPI_Neighbor_alltoallw request of
>> mpi-1.8.3.
>> I have enclosed a test case with information of my workstation.
>>
>> In this test, I define a weighted topology for 5 processes, where the
>> weight represent the number of buffers to send/receive :
>> rank
>>   0 : | x |
>>   1 : | 2 | x |
>>   2 : | 1 | 1 | x |
>>   3 : | 3 | 2 | 3 | x |
>>   4 : | 5 | 2 | 2 | 2 | x |
>>
>> In this topology, the rank 1 will send/receive :
>>2 buffers to/from the rank 0,
>>1 buffer to/from the rank 2,
>>2 buffers to/from the rank 3,
>>2 buffers to/from the rank 4,
>>
>> The send buffer are defined with the MPI_Type_create_hindexed_block. This
>> allows to use a same buffer for several communications without duplicating
>> it (read only).
>> Here the rank 1 will have 2 send buffers (the max of 2, 1, 2, 2).
>> The receiver buffer is a contiguous buffer defined with
>> MPI_Type_contiguous request.
>> Here, the receiver buffer of the rank 1 is of size : 7 (2+1+2+2)
>>
>> This test case succesful for 2 or 3 processes. For 4 processes, the test
>> fails 1 times for 3 successes. For 5 processes, the test fails all the time.
>>
>> The error code is : *** MPI_ERR_IN_STATUS: error code in status
>>
>> I don't understand what I am doing wrong.
>>
>> Could you please have a look on it?
>>
>> Thank you very much.
>>
>> Best regards,
>> Ghislain Viguier
>>
>> --
>> Ghislain Viguier
>> Tél. 06 31 95 03 17
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/11/25850.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/11/25852.php

diff --git a/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c 
b/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c
index 28ecf04..4069212 100644
--- a/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c
+++ b/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c
@@ -181,7 +181,7 @@ mca_coll_basic_neighbor_alltoallw_dist_graph(const void 
*sbuf, const int scounts
 /* post all receives first */
 for (neighbor = 0, reqs = basic_module->mccb_reqs ; neighbor < indegree ; 
++neighbor) {
 rc = MCA_PML_CALL(irecv((char *) rbuf + rdisps[neighbor], 
rcounts[neighbor], rdtypes[neighbor],
-inedges[neighbor], MCA_COLL_BASE_TAG_ALLTOALL, 
comm, reqs++));
+outedges[neighbor], 
MCA_COLL_BASE_TAG_ALLTOALL, comm, reqs++));
 if (OMPI_SUCCESS != rc) break;
 }

//
// Name: 027_MPI_Neighbor_alltoallw_synthetic.cpp
// Author  : 
// Version :
// Copyright   : Your copyright notice
// Description : Hello World in C++, Ansi-style
//

#include 
#include 
#include 
#include 
#include 
#include 
#include 

using namespace std;

int main(int argc, char *argv[]) {

	const int sendBufferSize = 1;

	///   MPI initialization   ///

	int ierr;
	int nbProc;
	int rank;
	ierr = MPI_Init(&argc, &argv);
	assert(!ierr);
	ierr = MPI_Comm_size(MPI_COMM_WORLD, &nbProc);
	assert(!ierr);
	ierr = MPI_Comm_rank(MPI_COMM_WORLD, &rank);
	assert(!ierr);

	assert(nbProc <= 5);

	///   weighted topology   ///
	//   0  | x |
	//   1  | 2 | x |
	//   2  | 1 | 1 | x |
	//   3  | 3 | 2 | 3 | x |
	//   4  | 5 | 2 | 2 | 2 | x |
	// rank   0   1   2   3   4

	int degrees = nbProc - 1;
	vector targets(4);
	vector weights(4);
	switch (rank) {

		case 0:
		targets[0] = 1; targets[1] = 2; targets[2] = 3; targets[3] = 4;
		weights[0] = 2; weights[1] = 1; weights[2] = 3; weights[3] = 5;
		break;

		case 1:
		targets[0] = 0; targets[1] = 2; targets[2] = 3; targets[3] = 4;
		weights[0] = 2; weights[1] = 1; weights[2] = 2; weights[3] = 2;
		break;

		case 2:
		ta

Re: [OMPI users] MPI_Neighbor_alltoallw fails with mpi-1.8.3

[Gilles Gouaillardet]

Ghislain,

i can confirm there is a bug in mca_topo_base_dist_graph_distribute

FYI a proof of concept is available at
https://github.com/open-mpi/ompi/pull/283
and i recommend you use MPI_Dist_graph_create_adjacent if this meets
your needs.

as a side note, the right way to set the info is
MPI_Info info = MPI_INFO_NULL;

/* mpich is more picky and crashes with info = NULL */

Cheers,

Gilles

On 2014/11/21 18:21, Ghislain Viguier wrote:
> Hi Gilles and Howard,
>
> The use of MPI_Dist_graph_create_adjacent solves the issue :)
>
> Thanks for your help!
>
> Best reagrds,
> Ghislain
>
> 2014-11-21 7:23 GMT+01:00 Gilles Gouaillardet > :
>>  Hi Ghislain,
>>
>> that sound like a but in MPI_Dist_graph_create :-(
>>
>> you can use MPI_Dist_graph_create_adjacent instead :
>>
>> MPI_Dist_graph_create_adjacent(MPI_COMM_WORLD, degrees, &targets[0],
>> &weights[0],
>> degrees, &targets[0], &weights[0], info,
>> rankReordering, &commGraph);
>>
>> it does not crash and as far as i understand, it produces correct results,
>>
>> according the the mpi standard (example 7.3) that should do the same
>> thing, that's why
>> i think there is a bug in MPI_Dist_graph_create
>>
>> Cheers,
>>
>> Gilles
>>
>>
>>
>>
>> On 2014/11/21 2:21, Howard Pritchard wrote:
>>
>> Hi Ghislain,
>>
>> I tried to run your test with mvapich 1.9 and get a "message truncated"
>> failure at three ranks.
>>
>> Howard
>>
>>
>> 2014-11-20 8:51 GMT-07:00 Ghislain Viguier  
>> :
>>
>>
>>  Dear support,
>>
>> I'm encountering an issue with the MPI_Neighbor_alltoallw request of
>> mpi-1.8.3.
>> I have enclosed a test case with information of my workstation.
>>
>> In this test, I define a weighted topology for 5 processes, where the
>> weight represent the number of buffers to send/receive :
>> rank
>>   0 : | x |
>>   1 : | 2 | x |
>>   2 : | 1 | 1 | x |
>>   3 : | 3 | 2 | 3 | x |
>>   4 : | 5 | 2 | 2 | 2 | x |
>>
>> In this topology, the rank 1 will send/receive :
>>2 buffers to/from the rank 0,
>>1 buffer to/from the rank 2,
>>2 buffers to/from the rank 3,
>>2 buffers to/from the rank 4,
>>
>> The send buffer are defined with the MPI_Type_create_hindexed_block. This
>> allows to use a same buffer for several communications without duplicating
>> it (read only).
>> Here the rank 1 will have 2 send buffers (the max of 2, 1, 2, 2).
>> The receiver buffer is a contiguous buffer defined with
>> MPI_Type_contiguous request.
>> Here, the receiver buffer of the rank 1 is of size : 7 (2+1+2+2)
>>
>> This test case succesful for 2 or 3 processes. For 4 processes, the test
>> fails 1 times for 3 successes. For 5 processes, the test fails all the time.
>>
>> The error code is : *** MPI_ERR_IN_STATUS: error code in status
>>
>> I don't understand what I am doing wrong.
>>
>> Could you please have a look on it?
>>
>> Thank you very much.
>>
>> Best regards,
>> Ghislain Viguier
>>
>> --
>> Ghislain Viguier
>> Tél. 06 31 95 03 17
>>
>> ___
>> users mailing listus...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this 
>> post:http://www.open-mpi.org/community/lists/users/2014/11/25850.php
>>
>>
>>
>> ___
>> users mailing listus...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/11/25852.php
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/11/25853.php
>>
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/11/25855.php

Re: [OMPI users] MPI_Neighbor_alltoallw fails with mpi-1.8.3

[Gilles Gouaillardet]

George,

imho, you are right !

here is attached a new version of Ghislain's program and that uses
MPI_Dist_graph_neighbors_count and MPI_Dist_graph_neighbors
as you suggested.

it produces correct results

/* note that in this case, realDestinations is similar to targets,
so i might have left some silent bugs in the program */

Bottom line, though Open MPI implementation of MPI_Dist_graph_create is not
deterministic, it is compliant with the MPI standard.
/* not to mention this is not the right place to argue what the standard
could or should have been ... */

Cheers,

Gilles


On 2014/11/24 12:47, George Bosilca wrote:
> I would argue this is a typical user level bug.
>
> The major difference between the dist_create and dist_create_adjacent is
> that in the later each process provides its neighbors in an order that is
> expected (and that match the info provided to the MPI_Neighbor_alltoallw
> call. When the topology is created with dist_create, every process will
> end-up having the correct partial topology, but in an order that doesn't
> match what the user expected (not in the rank-order of the neighbors).
> However, I can't find anything in the standard that would require from the
> MPI library to sort the neighbors. I would assume is the user
> responsibility, to make sure that they are using the topology in the right
> order, where the right order is what the communicator really contains and
> not what the user expect based on prior knowledge.
>
>   George.
>
>
> On Fri, Nov 21, 2014 at 3:48 AM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
>>  Ghislain,
>>
>> i can confirm there is a bug in mca_topo_base_dist_graph_distribute
>>
>> FYI a proof of concept is available at
>> https://github.com/open-mpi/ompi/pull/283
>> and i recommend you use MPI_Dist_graph_create_adjacent if this meets your
>> needs.
>>
>> as a side note, the right way to set the info is
>> MPI_Info info = MPI_INFO_NULL;
>>
>> /* mpich is more picky and crashes with info = NULL */
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/11/21 18:21, Ghislain Viguier wrote:
>>
>> Hi Gilles and Howard,
>>
>> The use of MPI_Dist_graph_create_adjacent solves the issue :)
>>
>> Thanks for your help!
>>
>> Best reagrds,
>> Ghislain
>>
>> 2014-11-21 7:23 GMT+01:00 Gilles Gouaillardet >
>>  :
>>
>>Hi Ghislain,
>>
>> that sound like a but in MPI_Dist_graph_create :-(
>>
>> you can use MPI_Dist_graph_create_adjacent instead :
>>
>> MPI_Dist_graph_create_
>> adjacent(MPI_COMM_WORLD, degrees, &targets[0],
>> &weights[0],
>> degrees, &targets[0], &weights[0], info,
>> rankReordering, &commGraph);
>>
>> it does not crash and as far as i understand, it produces correct results,
>>
>> according the the mpi standard (example 7.3) that should do the same
>> thing, that's why
>> i think there is a bug in MPI_Dist_graph_create
>>
>> Cheers,
>>
>> Gilles
>>
>>
>>
>>
>> On 2014/11/21 2:21, Howard Pritchard wrote:
>>
>> Hi Ghislain,
>>
>> I tried to run your test with mvapich 1.9 and get a "message truncated"
>> failure at three ranks.
>>
>> Howard
>>
>>
>> 2014-11-20 8:51 GMT-07:00 Ghislain Viguier  
>>   
>> :
>>
>>
>>  Dear support,
>>
>> I'm encountering an issue with the MPI_Neighbor_alltoallw request of
>> mpi-1.8.3.
>> I have enclosed a test case with information of my workstation.
>>
>> In this test, I define a weighted topology for 5 processes, where the
>> weight represent the number of buffers to send/receive :
>> rank
>>   0 : | x |
>>   1 : | 2 | x |
>>   2 : | 1 | 1 | x |
>>   3 : | 3 | 2 | 3 | x |
>>   4 : | 5 | 2 | 2 | 2 | x |
>>
>> In this topology, the rank 1 will send/receive :
>>2 buffers to/from the rank 0,
>>1 buffer to/from the rank 2,
>>2 buffers to/from the rank 3,
>>2 buffers to/from the rank 4,
>>
>> The send buffer are defined with the MPI_Type_create_hindexed_block. This
>> allows to use a same buffer for several communications without duplicating
>> it (read only).
>> Here the rank 1 will have 2 send buffers (the max of 2, 1, 2, 2).
>> The receiver buffer is a contiguous buffer defined with
>> MPI_Type_contiguous request.
>> Here, the receiver buffer of the rank 1 is of size : 7 (2+1+2+2)
>>
>> This test case succe

Re: [OMPI users] mpi_wtime implementation

[Gilles Gouaillardet]

Folks,

one drawback of retrieving time with rdtsc is that this value is core
specific :
if a task is not bound to a core, then the value returned by MPI_Wtime()
might go backward.

if i run the following program with
taskset -c 1 ./time

and then move it accross between cores
(taskset -cp 0  ; taskset -cp 2 ; ...)
then the program can abort. in my environment, i can measure up to 150ms
difference.

/* some mtt tests will abort if this condition is met */


i was unable to observe this behavior with gettimeofday()

/* though it could occur when ntpd synchronizes the clock */

is there any plan to make the timer function selectable via a mca param ?
or to automatically fallback to gettimeofday if a task is not bound on a
core ?

Cheers,

Gilles

$ cat time.c
#include 
#include 

int main (int argc, char *argv[]) {
int i;
double t = 0;
MPI_Init(&argc, &argv);
for (;;) {
double _t = MPI_Wtime();
if (_t < t) {
fprintf(stderr, "going back in time %lf < %lf\n", _t, t);
MPI_Abort(MPI_COMM_WORLD, 1);
}
t = _t;
}
MPI_Finalize();
return 0;
}

On 2014/11/25 1:59, Dave Goodell (dgoodell) wrote:
> On Nov 24, 2014, at 12:06 AM, George Bosilca  wrote:
>
>> https://github.com/open-mpi/ompi/pull/285 is a potential answer. I would 
>> like to hear Dave Goodell comment on this before pushing it upstream.
>>
>>   George.
> I'll take a look at it today.  My notification settings were messed up when 
> you originally CCed me on the PR, so I didn't see this until now.
>
> -Dave
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/11/25863.php

Re: [OMPI users] "default-only MCA variable"?

[Gilles Gouaillardet]

It could be because configure did not find the knem headers and hence knem is 
not supported and hence this mca parameter is read-only

My 0.2 us$ ...

Dave Love さんのメール:
>Why can't I set parameters like this (not the only one) with 1.8.3?
>
>  WARNING: A user-supplied value attempted to override the default-only MCA
>  variable named "btl_sm_use_knem".
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25882.php

Re: [OMPI users] Warning about not enough registerable memory on SL6.6

[Gilles Gouaillardet]

Folks,

FWIW, i observe a similar behaviour on my system.

imho, the root cause is OFED has been upgraded from a (quite) older
version to latest 3.12 version

here is the relevant part of code (btl_openib.c from the master) :


static uint64_t calculate_max_reg (void)
{
if (0 == stat("/sys/module/mlx4_core/parameters/log_num_mtt",
&statinfo)) {
} else if (0 == stat("/sys/module/ib_mthca/parameters/num_mtt",
&statinfo)) {
mtts_per_seg = 1 <<
read_module_param("/sys/module/ib_mthca/parameters/log_mtts_per_seg", 1);
num_mtt =
read_module_param("/sys/module/ib_mthca/parameters/num_mtt", 1 << 20);
reserved_mtt =
read_module_param("/sys/module/ib_mthca/parameters/fmr_reserved_mtts", 0);

max_reg = (num_mtt - reserved_mtt) * opal_getpagesize () *
mtts_per_seg;
} else if (
(0 == stat("/sys/module/mlx5_core", &statinfo)) ||
(0 == stat("/sys/module/mlx4_core/parameters", &statinfo)) ||
(0 == stat("/sys/module/ib_mthca/parameters", &statinfo))
) {
/* mlx5 means that we have ofed 2.0 and it can always register
2xmem_total for any mlx hca */
max_reg = 2 * mem_total;
} else {
}

/* Print a warning if we can't register more than 75% of physical
   memory.  Abort if the abort_not_enough_reg_mem MCA param was
   set. */
if (max_reg < mem_total * 3 / 4) {
}
return (max_reg * 7) >> 3;
}

with OFED 3.12, the /sys/module/mlx4_core/parameters/log_num_mtt pseudo
file does *not* exist any more
/sys/module/ib_mthca/parameters/num_mtt exists so the second path is taken
and mtts_per_seg is read from
/sys/module/ib_mthca/parameters/log_mtts_per_seg

i noted that log_mtts_per_seg is also a parameter of mlx4_core :
/sys/module/mlx4_core/parameters/log_mtts_per_seg

the value is 3 in ib_mthca (and leads to a warning) but 5 in mlx4_core
(big enough, and does not lead to a warning if this value is read)


i had no time to read the latest ofed doc, so i cannot answer :
- should log_mtts_per_seg be read from mlx4_core instead ?
- is the warning a false positive ?


my only point is this warning *might* be a false positive and the root
cause *might* be calculate_max_reg logic
*could* be wrong with the latest OFED stack.

Could the Mellanox folks comment on this ?

Cheers,

Gilles




On 2014/12/09 3:18, Götz Waschk wrote:
> Hi,
>
> here's another test with openmpi 1.8.3. With 1.8.1, 32GB was detected, now
> it is just 16:
> % mpirun -np 2 /usr/lib64/openmpi-intel/bin/mpitests-osu_get_bw
> --
> WARNING: It appears that your OpenFabrics subsystem is configured to only
> allow registering part of your physical memory.  This can cause MPI jobs to
> run with erratic performance, hang, and/or crash.
>
> This may be caused by your OpenFabrics vendor limiting the amount of
> physical memory that can be registered.  You should investigate the
> relevant Linux kernel module parameters that control how much physical
> memory can be registered, and increase them to allow registering all
> physical memory on your machine.
>
> See this Open MPI FAQ item for more information on these Linux kernel module
> parameters:
>
> http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
>
>   Local host:  pax95
>   Registerable memory: 16384 MiB
>   Total memory:49106 MiB
>
> Your MPI job will continue, but may be behave poorly and/or hang.
> --
> # OSU MPI_Get Bandwidth Test v4.3
> # Window creation: MPI_Win_allocate
> # Synchronization: MPI_Win_flush
> # Size  Bandwidth (MB/s)
> 1  28.56
> 2  58.74
>
>
> So it wasn't fixed for RHEL 6.6.
>
> Regards, Götz
>
> On Mon, Dec 8, 2014 at 4:00 PM, Götz Waschk  wrote:
>
>> Hi,
>>
>> I had tested 1.8.4rc1 and it wasn't fixed. I can try again though,
>> maybe I had made an error.
>>
>> Regards, Götz Waschk
>>
>> On Mon, Dec 8, 2014 at 3:17 PM, Joshua Ladd  wrote:
>>> Hi,
>>>
>>> This should be fixed in OMPI 1.8.3. Is it possible for you to give 1.8.3
>> a
>>> shot?
>>>
>>> Best,
>>>
>>> Josh
>>>
>>> On Mon, Dec 8, 2014 at 8:43 AM, Götz Waschk 
>> wrote:
 Dear Open-MPI experts,

 I have updated my little cluster from Scientific Linux 6.5 to 6.6,
 this included extensive changes in the Infiniband drivers and a newer
 openmpi version (1.8.1). Now I'm getting this message on all nodes
 with more than 32 GB of RAM:


 WARNING: It appears that your OpenFabrics subsystem is configured to
>> only
 allow registering part of your physical memory.  This can cause MPI jobs
 to
 run with erratic performance, hang, and/or crash.

 This may be caused by your OpenFabrics vendor limiting the amount of
 physical memory that can be registered.  You should investigate the
 relevant Linux kernel module parameters that co

Re: [OMPI users] Open mpi based program runs as root and gives SIGSEGV under unprivileged user

[Gilles Gouaillardet]

Luca,

your email mentions openmpi 1.6.5
but gdb output points to openmpi 1.8.1.

could the root cause be a mix of versions that does not occur with root
account ?

which openmpi version are you expecting ?

you can run
pmap 
when your binary is running and/or under gdb to confirm the openmpi library
that is really used

Cheers,

Gilles

On Wed, Dec 10, 2014 at 7:21 PM, Luca Fini  wrote:

> I've a problem running a well tested MPI based application.
>
> The program has been used for years with no problems. Suddenly the
> executable which was run many times with no problems crashed with
> SIGSEGV. The very same executable if run with root privileges works
> OK. The same happens with other executables and across various
> recompilation attempts.
>
> We could not find any relevant difference in the O.S. since a few days
> ago when the program worked also under unprivileged user ID. Actually
> about in the same span of time we changed the GID of the user
> experiencing the fault, but we think this is not relevant because the
> same SIGSEGV happens to another user which was not modified. Moreover
> we cannot see how that change can affect the running executabe (we
> checked all file permissions in the directory tree where the program
> is used).
>
> Running the program under GDB we get the trace reported below. The
> segfault happens at the very beginning during MPI initialization.
>
> We can use the program with sudo, but I'd like to find out what
> happened to go back to "normal" usage.
>
> I'd appreciate any hint on the issue.
>
> Many thanks,
>
>Luca Fini
>
> ==
> Here follows a few environment details:
>
> Program started with: mpirun -debug -debugger gdb  -np 1
>
> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD
>
> OPEN-MPI 1.6.5
>
> Linux 2.6.32-431.29.2.2.6.32-431.29.2.el6.x86_64
>
> Intel fortran Compiler: 2011.7.256
>
> =
> Here follows the stack trace:
>
> Starting program:
>
> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD
>
> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD
> [Thread debugging using libthread_db enabled]
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x2af652c7 in mca_base_component_find (directory=0x0,
> type=0x3b914a7fb5 "rte", static_components=0x3b916cb040,
> requested_component_names=0x0, include_mode=128, found_components=0x1,
> open_dso_components=16)
> at mca_base_component_find.c:162
> 162OBJ_CONSTRUCT(found_components, opal_list_t);
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.12-1.149.el6.x86_64 libgcc-4.4.7-11.el6.x86_64
> libgfortran-4.4.7-11.el6.x86_64 libtool-ltdl-2.2.6-15.5.el6.x86_64
> openmpi-1.8.1-1.el6.x86_64
> (gdb) where
> #0  0x2af652c7 in mca_base_component_find (directory=0x0,
> type=0x3b914a7fb5 "rte", static_components=0x3b916cb040,
> requested_component_names=0x0, include_mode=128, found_components=0x1,
> open_dso_components=16)
> at mca_base_component_find.c:162
> #1  0x003b90c4870a in mca_base_framework_components_register ()
> from /usr/lib64/openmpi/lib/libopen-pal.so.6
> #2  0x003b90c48c06 in mca_base_framework_register () from
> /usr/lib64/openmpi/lib/libopen-pal.so.6
> #3  0x003b90c48def in mca_base_framework_open () from
> /usr/lib64/openmpi/lib/libopen-pal.so.6
> #4  0x003b914407e7 in ompi_mpi_init () from
> /usr/lib64/openmpi/lib/libmpi.so.1
> #5  0x003b91463200 in PMPI_Init () from
> /usr/lib64/openmpi/lib/libmpi.so.1
> #6  0x2acd9295 in mpi_init_f (ierr=0x7fffd268) at pinit_f.c:75
> #7  0x005bb159 in MODE_MNH_WORLD::init_nmnh_comm_world
> (kinfo_ll=Cannot access memory at address 0x0
> ) at
> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/MASTER/spll_mode_mnh_world.f90:45
> #8  0x005939d3 in MODE_IO_LL::initio_ll () at
>
> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/MASTER/spll_mode_io_ll.f90:107
> #9  0x0049d02f in prep_pgd () at
>
> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/MASTER/spll_prep_pgd.f90:130
> #10 0x0049cf8c in main ()
>
> --
> Luca Fini.  INAF - Oss. Astrofisico di Arcetri
> L.go E.Fermi, 5. 50125 Firenze. Italy
> Tel: +39 055 2752 307 Fax: +39 055 2752 292
> Skype: l.fini
> Web: http://www.arcetri.inaf.it/~lfini
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/25945.php
>

Re: [OMPI users] Open mpi based program runs as root and gives SIGSEGV under unprivileged user

[Gilles Gouaillardet]

Luca,

you might want to double check the environment :
env | grep ^OMPI
and the per user config
ls $HOME/.openmpi

Cheers,

Gilles

On 2014/12/11 17:40, Luca Fini wrote:
> Many thanks for the replies.
>
> The mismatch in OpeMPI version is my fault: while writing the request
> for help I looked at the name of the directory where OpenMPI was built
> (I did not build it myself) and did not notice that the name of the
> directory did not reflect the version actually compiled.
>
> I had already checked the ulimits defined for the account where the
> SIGSEGV happens and they seems OK.
>
> Moreover I have a further result: I created a brand new account with
> default privileges and tried to run the program under that one, and it
> works!
>
> I'm still trying to spot out the differences between the two
> unprivileged accounts.
>
> Cheers,
>l.
>
> On Wed, Dec 10, 2014 at 6:12 PM, Gus Correa  wrote:
>> Hi Luca
>>
>> Another possibility that comes to mind,
>> besides mixed versions mentioned by Gilles,
>> is the OS limits.
>> Limits may vary according to the user and user privileges.
>>
>> Large programs tend to require big stacksize (even unlimited),
>> and typically segfault when the stack is not large enough.
>> Max number of open files is yet another hurdle.
>> And if you're using Infinband, the max locked memory size should be
>> unlimited.
>> Check /etc/security/limits.conf and "ulimit -a".
>>
>> I hope this helps,
>> Gus Correa
>>
>> On 12/10/2014 08:28 AM, Gilles Gouaillardet wrote:
>>> Luca,
>>>
>>> your email mentions openmpi 1.6.5
>>> but gdb output points to openmpi 1.8.1.
>>>
>>> could the root cause be a mix of versions that does not occur with root
>>> account ?
>>>
>>> which openmpi version are you expecting ?
>>>
>>> you can run
>>> pmap 
>>> when your binary is running and/or under gdb to confirm the openmpi
>>> library that is really used
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On Wed, Dec 10, 2014 at 7:21 PM, Luca Fini >> <mailto:lf...@arcetri.astro.it>> wrote:
>>>
>>> I've a problem running a well tested MPI based application.
>>>
>>> The program has been used for years with no problems. Suddenly the
>>> executable which was run many times with no problems crashed with
>>> SIGSEGV. The very same executable if run with root privileges works
>>> OK. The same happens with other executables and across various
>>> recompilation attempts.
>>>
>>> We could not find any relevant difference in the O.S. since a few days
>>> ago when the program worked also under unprivileged user ID. Actually
>>> about in the same span of time we changed the GID of the user
>>> experiencing the fault, but we think this is not relevant because the
>>> same SIGSEGV happens to another user which was not modified. Moreover
>>> we cannot see how that change can affect the running executabe (we
>>> checked all file permissions in the directory tree where the program
>>> is used).
>>>
>>> Running the program under GDB we get the trace reported below. The
>>> segfault happens at the very beginning during MPI initialization.
>>>
>>> We can use the program with sudo, but I'd like to find out what
>>> happened to go back to "normal" usage.
>>>
>>> I'd appreciate any hint on the issue.
>>>
>>> Many thanks,
>>>
>>> Luca Fini
>>>
>>> ==
>>> Here follows a few environment details:
>>>
>>> Program started with: mpirun -debug -debugger gdb  -np 1
>>>
>>> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD
>>>
>>> OPEN-MPI 1.6.5
>>>
>>> Linux 2.6.32-431.29.2.2.6.32-431.29.2.el6.x86_64
>>>
>>> Intel fortran Compiler: 2011.7.256
>>>
>>> =
>>> Here follows the stack trace:
>>>
>>> Starting program:
>>>
>>> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD
>>>
>>> /home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_

Re: [OMPI users] MPI inside MPI (still)

[Gilles Gouaillardet]

Alex,

can you try something like
call system(sh -c 'env -i /.../mpirun -np 2 /.../app_name')

-i start with an empty environment
that being said, you might need to set a few environment variables
manually :
env -i PATH=/bin ...

and that being also said, this "trick" could be just a bad idea :
you might be using a scheduler, and if you empty the environment, the
scheduler
will not be aware of the "inside" run.

on top of that, invoking system might fail depending on the interconnect
you use.

Bottom line, i believe Ralph's reply is still valid, even if five years
have passed :
changing your workflow, or using MPI_Comm_spawn is a much better approach.

Cheers,

Gilles

On 2014/12/12 11:22, Alex A. Schmidt wrote:
> Dear OpenMPI users,
>
> Regarding to this previous post
>  from 2009,
> I wonder if the reply
> from Ralph Castain is still valid. My need is similar but quite simpler:
> to make a system call from an openmpi fortran application to run a
> third party openmpi application. I don't need to exchange mpi messages
> with the application. I just need to read the resulting output file
> generated
> by it. I have tried to do the following system call from my fortran openmpi
> code:
>
> call system("sh -c 'mpirun -n 2 app_name")
>
> but I get
>
> **
>
> Open MPI does not support recursive calls of mpirun
>
> **
>
> Is there a way to make this work?
>
> Best regards,
>
> Alex
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/25966.php

Re: [OMPI users] MPI inside MPI (still)

[Gilles Gouaillardet]

Alex,

just ask MPI_Comm_spawn to start (up to) 5 tasks via the maxprocs
parameter :

   int MPI_Comm_spawn(char *command, char *argv[], int maxprocs,
MPI_Info info,
 int root, MPI_Comm comm, MPI_Comm *intercomm,
 int array_of_errcodes[])

INPUT PARAMETERS
   maxprocs
  - maximum number of processes to start (integer,
significant only at root)

Cheers,

Gilles

On 2014/12/12 12:23, Alex A. Schmidt wrote:
> Hello Gilles,
>
> Thanks for your reply. The "env -i PATH=..." stuff seems to work!!!
>
> call system("sh -c 'env -i PATH=/usr/lib64/openmpi/bin:/bin mpirun -n 2
> hello_world' ")
>
> did produce the expected result with a simple openmi "hello_world" code I
> wrote.
>
> I might be harder though with the real third party app I have in mind. And
> I realize
> getting passed over a job scheduler with this approach might not work at
> all...
>
> I have looked at the MPI_Comm_spawn call but I failed to understand how it
> could help here. For instance, can I use it to launch an mpi app with the
> option "-n 5" ?
>
> Alex
>
> 2014-12-12 0:36 GMT-02:00 Gilles Gouaillardet > :
>>
>>  Alex,
>>
>> can you try something like
>> call system(sh -c 'env -i /.../mpirun -np 2 /.../app_name')
>>
>> -i start with an empty environment
>> that being said, you might need to set a few environment variables
>> manually :
>> env -i PATH=/bin ...
>>
>> and that being also said, this "trick" could be just a bad idea :
>> you might be using a scheduler, and if you empty the environment, the
>> scheduler
>> will not be aware of the "inside" run.
>>
>> on top of that, invoking system might fail depending on the interconnect
>> you use.
>>
>> Bottom line, i believe Ralph's reply is still valid, even if five years
>> have passed :
>> changing your workflow, or using MPI_Comm_spawn is a much better approach.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/12 11:22, Alex A. Schmidt wrote:
>>
>> Dear OpenMPI users,
>>
>> Regarding to this previous 
>> post<http://www.open-mpi.org/community/lists/users/2009/06/9560.php> 
>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php> from 2009,
>> I wonder if the reply
>> from Ralph Castain is still valid. My need is similar but quite simpler:
>> to make a system call from an openmpi fortran application to run a
>> third party openmpi application. I don't need to exchange mpi messages
>> with the application. I just need to read the resulting output file
>> generated
>> by it. I have tried to do the following system call from my fortran openmpi
>> code:
>>
>> call system("sh -c 'mpirun -n 2 app_name")
>>
>> but I get
>>
>> **
>>
>> Open MPI does not support recursive calls of mpirun
>>
>> **
>>
>> Is there a way to make this work?
>>
>> Best regards,
>>
>> Alex
>>
>>
>>
>>
>> ___
>> users mailing listus...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/12/25966.php
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/12/25967.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/25968.php

Re: [OMPI users] MPI inside MPI (still)

[Gilles Gouaillardet]

Alex,

just to make sure ...
this is the behavior you expected, right ?

Cheers,

Gilles

On 2014/12/12 13:27, Alex A. Schmidt wrote:
> Gilles,
>
> Ok, very nice!
>
> When I excute
>
> do rank=1,3
> call  MPI_Comm_spawn('hello_world','
> ',5,MPI_INFO_NULL,rank,MPI_COMM_WORLD,my_intercomm,MPI_ERRCODES_IGNORE,status)
> enddo
>
> I do get 15 instances of the 'hello_world' app running: 5 for each parent
> rank 1, 2 and 3.
>
> Thanks a lot, Gilles.
>
> Best regargs,
>
> Alex
>
>
>
>
> 2014-12-12 1:32 GMT-02:00 Gilles Gouaillardet > :
>>
>>  Alex,
>>
>> just ask MPI_Comm_spawn to start (up to) 5 tasks via the maxprocs
>> parameter :
>>
>>int MPI_Comm_spawn(char *command, char *argv[], int maxprocs,
>> MPI_Info info,
>>  int root, MPI_Comm comm, MPI_Comm *intercomm,
>>  int array_of_errcodes[])
>>
>> INPUT PARAMETERS
>>maxprocs
>>   - maximum number of processes to start (integer, significant
>> only at root)
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/12 12:23, Alex A. Schmidt wrote:
>>
>> Hello Gilles,
>>
>> Thanks for your reply. The "env -i PATH=..." stuff seems to work!!!
>>
>> call system("sh -c 'env -i PATH=/usr/lib64/openmpi/bin:/bin mpirun -n 2
>> hello_world' ")
>>
>> did produce the expected result with a simple openmi "hello_world" code I
>> wrote.
>>
>> I might be harder though with the real third party app I have in mind. And
>> I realize
>> getting passed over a job scheduler with this approach might not work at
>> all...
>>
>> I have looked at the MPI_Comm_spawn call but I failed to understand how it
>> could help here. For instance, can I use it to launch an mpi app with the
>> option "-n 5" ?
>>
>> Alex
>>
>> 2014-12-12 0:36 GMT-02:00 Gilles Gouaillardet >
>>  :
>>
>>  Alex,
>>
>> can you try something like
>> call system(sh -c 'env -i /.../mpirun -np 2 /.../app_name')
>>
>> -i start with an empty environment
>> that being said, you might need to set a few environment variables
>> manually :
>> env -i PATH=/bin ...
>>
>> and that being also said, this "trick" could be just a bad idea :
>> you might be using a scheduler, and if you empty the environment, the
>> scheduler
>> will not be aware of the "inside" run.
>>
>> on top of that, invoking system might fail depending on the interconnect
>> you use.
>>
>> Bottom line, i believe Ralph's reply is still valid, even if five years
>> have passed :
>> changing your workflow, or using MPI_Comm_spawn is a much better approach.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/12 11:22, Alex A. Schmidt wrote:
>>
>> Dear OpenMPI users,
>>
>> Regarding to this previous 
>> post<http://www.open-mpi.org/community/lists/users/2009/06/9560.php> 
>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php> 
>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php> 
>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php> from 2009,
>> I wonder if the reply
>> from Ralph Castain is still valid. My need is similar but quite simpler:
>> to make a system call from an openmpi fortran application to run a
>> third party openmpi application. I don't need to exchange mpi messages
>> with the application. I just need to read the resulting output file
>> generated
>> by it. I have tried to do the following system call from my fortran openmpi
>> code:
>>
>> call system("sh -c 'mpirun -n 2 app_name")
>>
>> but I get
>>
>> **
>>
>> Open MPI does not support recursive calls of mpirun
>>
>> **
>>
>> Is there a way to make this work?
>>
>> Best regards,
>>
>> Alex
>>
>>
>>
>>
>> ___
>> users mailing listus...@open-mpi.org
>>
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/12/25966.php
>>
>>
>>
>> ___
>> users mailing listus...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this 
>> post:http://www.open-mpi.org/community/lists/users/2014/12/25967.php
>>
>>
>>
>> ___
>> users mailing listus...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/12/25968.php
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/12/25969.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/25970.php

Re: [OMPI users] OMPI users] MPI inside MPI (still)

[Gilles Gouaillardet]

Alex,

You need MPI_Comm_disconnect at least.
I am not sure if this is 100% correct nor working.

If you are using third party apps, why dont you do something like
system("env -i qsub ...")
with the right options to make qsub blocking or you manually wait for the end 
of the job ?

That looks like a much cleaner and simpler approach to me.

Cheers,

Gilles

"Alex A. Schmidt"  wrote:
>Hello Gilles,
>
>Ok, I believe I have a simple toy app running as I think it should:
>'n' parent processes running under mpi_comm_world, each one
>
>spawning its own 'm' child processes (each child group work 
>together nicely, returning the expected result for an mpi_allreduce call).
>
>Now, as I mentioned before, the apps I want to run in the spawned 
>
>processes are third party mpi apps and I don't think it will be possible 
>to exchange messages with them from my app. So, I do I tell 
>when the spawned processes have finnished running? All I have to work
>
>with is the intercommunicator returned from the mpi_comm_spawn call...
>
>
>Alex
>
>
>
>
>
>2014-12-12 2:42 GMT-02:00 Alex A. Schmidt :
>
>Gilles,
>
>Well, yes, I guess
>
>I'll do tests with the real third party apps and let you know.
>
>These are huge quantum chemistry codes (dftb+, siesta and Gaussian)
>
>which greatly benefits from a parallel environment. My code is just
>a front end to use those, but since we have a lot of data to process
>
>it also benefits from a parallel environment. 
>
>
>Alex
>
> 
>
>
>2014-12-12 2:30 GMT-02:00 Gilles Gouaillardet :
>
>Alex,
>
>just to make sure ...
>this is the behavior you expected, right ?
>
>Cheers,
>
>Gilles
>
>
>
>On 2014/12/12 13:27, Alex A. Schmidt wrote:
>
>Gilles, Ok, very nice! When I excute do rank=1,3 call 
>MPI_Comm_spawn('hello_world',' 
>',5,MPI_INFO_NULL,rank,MPI_COMM_WORLD,my_intercomm,MPI_ERRCODES_IGNORE,status) 
>enddo I do get 15 instances of the 'hello_world' app running: 5 for each 
>parent rank 1, 2 and 3. Thanks a lot, Gilles. Best regargs, Alex 2014-12-12 
>1:32 GMT-02:00 Gilles Gouaillardet 
>: Alex, just ask MPI_Comm_spawn to start (up to) 5 tasks via the maxprocs 
>parameter : int MPI_Comm_spawn(char *command, char *argv[], int maxprocs, 
>MPI_Info info, int root, MPI_Comm comm, MPI_Comm *intercomm, int 
>array_of_errcodes[]) INPUT PARAMETERS maxprocs - maximum number of processes 
>to start (integer, significant only at root) Cheers, Gilles On 2014/12/12 
>12:23, Alex A. Schmidt wrote: Hello Gilles, Thanks for your reply. The "env -i 
>PATH=..." stuff seems to work!!! call system("sh -c 'env -i 
>PATH=/usr/lib64/openmpi/bin:/bin mpirun -n 2 hello_world' ") did produce the 
>expected result with a simple openmi "hello_world" code I wrote. I might be 
>harder though with the real third party app I have in mind. And I realize 
>getting passed over a job scheduler with this approach might not work at 
>all... I have looked at the MPI_Comm_spawn call but I failed to understand how 
>it could help here. For instance, can I use it to launch an mpi app with the 
>option "-n 5" ? Alex 2014-12-12 0:36 GMT-02:00 Gilles Gouaillardet 
>
>: Alex, can you try something like call system(sh -c 'env -i /.../mpirun -np 2 
>/.../app_name') -i start with an empty environment that being said, you might 
>need to set a few environment variables manually : env -i PATH=/bin ... and 
>that being also said, this "trick" could be just a bad idea : you might be 
>using a scheduler, and if you empty the environment, the scheduler will not be 
>aware of the "inside" run. on top of that, invoking system might fail 
>depending on the interconnect you use. Bottom line, i believe Ralph's reply is 
>still valid, even if five years have passed : changing your workflow, or using 
>MPI_Comm_spawn is a much better approach. Cheers, Gilles On 2014/12/12 11:22, 
>Alex A. Schmidt wrote: Dear OpenMPI users, Regarding to this previous 
>post<http://www.open-mpi.org/community/lists/users/2009/06/9560.php> 
><http://www.open-mpi.org/community/lists/users/2009/06/9560.php> 
><http://www.open-mpi.org/community/lists/users/2009/06/9560.php> 
><http://www.open-mpi.org/community/lists/users/2009/06/9560.php> from 2009, I 
>wonder if the reply from Ralph Castain is still valid. My need is similar but 
>quite simpler: to make a system call from an openmpi fortran application to 
>run a third party openmpi application. I don't need to exchange mpi messages 
>with the application. I just need to read the resulting output file generated 
>by it. I have tried to do 

Re: [OMPI users] OMPI users] OMPI users] MPI inside MPI (still)

[Gilles Gouaillardet]

George is right about the semantic

However i am surprised it returns immediatly...
That should either work or hang imho

The second point is no more mpi related, and is batch manager specific.

You will likely find a submit parameter to make the command block until the job 
completes. Or you can write your own wrapper.
Or you can retrieve the jobid and qstat periodically to get the job state.
If an api is available, this is also an option.

Cheers,

Gilles

George Bosilca  wrote:
>You have to call MPI_Comm_disconnect on both sides of the intercommunicator. 
>On the spawner processes you should call it on the intercom, while on the 
>spawnees you should call it on the MPI_Comm_get_parent.
>
>
>  George.
>
>
>On Dec 12, 2014, at 20:43 , Alex A. Schmidt  wrote:
>
>
>Gilles,
>
>MPI_comm_disconnect seem to work but not quite.
>
>The call to it returns almost immediatly while
>
>the spawn processes keep piling up in the background
>
>until they are all done...
>
>I think system('env -i qsub...') to launch the third party apps
>
>would take the execution of every call back to the scheduler 
>queue. How would I track each one for their completion?
>
>Alex
>
>
>2014-12-12 22:35 GMT-02:00 Gilles Gouaillardet :
>
>Alex,
>
>You need MPI_Comm_disconnect at least.
>I am not sure if this is 100% correct nor working.
>
>If you are using third party apps, why dont you do something like
>system("env -i qsub ...")
>with the right options to make qsub blocking or you manually wait for the end 
>of the job ?
>
>That looks like a much cleaner and simpler approach to me.
>
>Cheers,
>
>Gilles
>
>"Alex A. Schmidt"  wrote:
>
>Hello Gilles,
>
>Ok, I believe I have a simple toy app running as I think it should:
>'n' parent processes running under mpi_comm_world, each one
>
>spawning its own 'm' child processes (each child group work 
>together nicely, returning the expected result for an mpi_allreduce call).
>
>Now, as I mentioned before, the apps I want to run in the spawned 
>
>processes are third party mpi apps and I don't think it will be possible 
>to exchange messages with them from my app. So, I do I tell 
>when the spawned processes have finnished running? All I have to work
>
>with is the intercommunicator returned from the mpi_comm_spawn call...
>
>
>Alex
>
>
>
>
>
>2014-12-12 2:42 GMT-02:00 Alex A. Schmidt :
>
>Gilles,
>
>Well, yes, I guess
>
>I'll do tests with the real third party apps and let you know.
>
>These are huge quantum chemistry codes (dftb+, siesta and Gaussian)
>
>which greatly benefits from a parallel environment. My code is just
>a front end to use those, but since we have a lot of data to process
>
>it also benefits from a parallel environment. 
>
>
>Alex
>
> 
>
>
>2014-12-12 2:30 GMT-02:00 Gilles Gouaillardet :
>
>Alex,
>
>just to make sure ...
>this is the behavior you expected, right ?
>
>Cheers,
>
>Gilles
>
>
>
>On 2014/12/12 13:27, Alex A. Schmidt wrote:
>
>Gilles, Ok, very nice! When I excute do rank=1,3 call 
>MPI_Comm_spawn('hello_world',' 
>',5,MPI_INFO_NULL,rank,MPI_COMM_WORLD,my_intercomm,MPI_ERRCODES_IGNORE,status) 
>enddo I do get 15 instances of the 'hello_world' app running: 5 for each 
>parent rank 1, 2 and 3. Thanks a lot, Gilles. Best regargs, Alex 2014-12-12 
>1:32 GMT-02:00 Gilles Gouaillardet 
>: Alex, just ask MPI_Comm_spawn to start (up to) 5 tasks via the maxprocs 
>parameter : int MPI_Comm_spawn(char *command, char *argv[], int maxprocs, 
>MPI_Info info, int root, MPI_Comm comm, MPI_Comm *intercomm, int 
>array_of_errcodes[]) INPUT PARAMETERS maxprocs - maximum number of processes 
>to start (integer, significant only at root) Cheers, Gilles On 2014/12/12 
>12:23, Alex A. Schmidt wrote: Hello Gilles, Thanks for your reply. The "env -i 
>PATH=..." stuff seems to work!!! call system("sh -c 'env -i 
>PATH=/usr/lib64/openmpi/bin:/bin mpirun -n 2 hello_world' ") did produce the 
>expected result with a simple openmi "hello_world" code I wrote. I might be 
>harder though with the real third party app I have in mind. And I realize 
>getting passed over a job scheduler with this approach might not work at 
>all... I have looked at the MPI_Comm_spawn call but I failed to understand how 
>it could help here. For instance, can I use it to launch an mpi app with the 
>option "-n 5" ? Alex 2014-12-12 0:36 GMT-02:00 Gilles Gouaillardet 
>
>: Alex, can you try something like call system(sh -c 'env -i /.../mpirun -np 2 
>/.../app_name') -i start with an empty

Re: [OMPI users] OMPI users] OMPI users] OMPI users] MPI inside MPI (still)

[Gilles Gouaillardet]

Alex,

Are you calling MPI_Comm_disconnect in the 3 "master" tasks and with the same 
remote communicator ?

I also read the man page again, and MPI_Comm_disconnect does not ensure the 
remote processes have finished or called MPI_Comm_disconnect, so that might not 
be the thing you need.
George, can you please comment on that ?

Cheers,

Gilles

George Bosilca  wrote:
>MPI_Comm_disconnect should be a local operation, there is no reason for it to 
>deadlock. I looked at the code and everything is local with the exception of a 
>call to PMIX.FENCE. Can you attach to your deadlocked processes and confirm 
>that they are stopped in the pmix.fence?
>
>
>  George.
>
>
>
>On Sat, Dec 13, 2014 at 8:47 AM, Alex A. Schmidt  wrote:
>
>Hi
>
>Sorry, I was calling mpi_comm_disconnect on the group comm handler, not
>on the intercomm handler returned from the spawn call as it should be.
>
>Well, calling the disconnect on the intercomm handler does halt the spwaner
>side but the wait is never completed since, as George points out, there is no
>disconnect call being made on the spawnee side and that brings me back
>to the beginning of the problem since, being a third party app, that call would
>never be there. I guess an mpi wrapper to deal with that could be made for
>the app, but I fell the wrapper itself, at the end, would face the same problem
>we face right now.
>
>My application is a genetic algorithm code that search optimal configuration
>(minimum or maximum energy) of cluster of atoms. The work flow bottleneck
>is the calculation of the cluster energy. For the cases which an analytical
>potential is available the calculation can be made internally and the workload
>is distributed among slaves nodes from a master node. This is also done
>when an analytical potential is not available and the energy calculation must
>be done externally by a quantum chemistry code like dftb+, siesta and Gaussian.
>So far, we have been running these codes in serial mode. No need to say that
>we could do a lot better if they could be executed in parallel.
>
>I am not familiar with DMRAA but it seems to be the right choice to deal with
>job schedulers as it covers the ones I am interested in (pbs/torque and 
>loadlever).
>
>Alex
>
>
>2014-12-13 7:49 GMT-02:00 Gilles Gouaillardet :
>
>George is right about the semantic
>
>However i am surprised it returns immediatly...
>That should either work or hang imho
>
>The second point is no more mpi related, and is batch manager specific.
>
>You will likely find a submit parameter to make the command block until the 
>job completes. Or you can write your own wrapper.
>Or you can retrieve the jobid and qstat periodically to get the job state.
>If an api is available, this is also an option.
>
>Cheers,
>
>Gilles
>
>George Bosilca  wrote:
>You have to call MPI_Comm_disconnect on both sides of the intercommunicator. 
>On the spawner processes you should call it on the intercom, while on the 
>spawnees you should call it on the MPI_Comm_get_parent.
>
>
>  George.
>
>
>On Dec 12, 2014, at 20:43 , Alex A. Schmidt  wrote:
>
>
>Gilles,
>
>MPI_comm_disconnect seem to work but not quite.
>
>The call to it returns almost immediatly while
>
>the spawn processes keep piling up in the background
>
>until they are all done...
>
>I think system('env -i qsub...') to launch the third party apps
>
>would take the execution of every call back to the scheduler 
>queue. How would I track each one for their completion?
>
>Alex
>
>
>2014-12-12 22:35 GMT-02:00 Gilles Gouaillardet :
>
>Alex,
>
>You need MPI_Comm_disconnect at least.
>I am not sure if this is 100% correct nor working.
>
>If you are using third party apps, why dont you do something like
>system("env -i qsub ...")
>with the right options to make qsub blocking or you manually wait for the end 
>of the job ?
>
>That looks like a much cleaner and simpler approach to me.
>
>Cheers,
>
>Gilles
>
>"Alex A. Schmidt"  wrote:
>
>Hello Gilles,
>
>Ok, I believe I have a simple toy app running as I think it should:
>'n' parent processes running under mpi_comm_world, each one
>
>spawning its own 'm' child processes (each child group work 
>together nicely, returning the expected result for an mpi_allreduce call).
>
>Now, as I mentioned before, the apps I want to run in the spawned 
>
>processes are third party mpi apps and I don't think it will be possible 
>to exchange messages with them from my app. So, I do I tell 
>when the spawned processes have finnished running? All I have to work
>
>with is the intercommunicator returned from th

Re: [OMPI users] OpenMPI 1.8.4rc3, 1.6.5 and 1.6.3: segmentation violation in mca_io_romio_dist_MPI_File_close

[Gilles Gouaillardet]

Eric,

can you make your test case (source + input file + howto) available so i
can try to reproduce and fix this ?

Based on the stack trace, i assume this is a complete end user application.
have you tried/been able to reproduce the same kind of crash with a
trimmed test program ?

BTW, what kind of filesystem is hosting Resultats.Eta1 ? (e.g. ext4 /
nfs / lustre / other)

Cheers,

Gilles

On 2014/12/15 4:06, Eric Chamberland wrote:
> Hi,
>
> I finally (thanks for fixing oversubscribing) tested with 1.8.4rc3 for
> my problem with collective MPI I/O.
>
> A problem still there.  In this 2 processes example, process rank 1
> dies with segfault while process rank 0 wait indefinitely...
>
> Running with valgrind, I found these errors which may gives hints:
>
> *
> Rank 1:
> *
> On process rank 1, without valgrind it ends with either a segmentation
> violation or memory corruption or invalide free without valgrind).
>
> But running with valgrind, it tells:
>
> ==16715== Invalid write of size 2
> ==16715==at 0x4C2E793: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
> ==16715==by 0x1F60AA91: opal_convertor_unpack (opal_convertor.c:321)
> ==16715==by 0x25AA8DD3: mca_pml_ob1_recv_frag_callback_match
> (pml_ob1_recvfrag.c:225)
> ==16715==by 0x2544110C: mca_btl_vader_check_fboxes
> (btl_vader_fbox.h:220)
> ==16715==by 0x25443577: mca_btl_vader_component_progress
> (btl_vader_component.c:695)
> ==16715==by 0x1F5F0F27: opal_progress (opal_progress.c:207)
> ==16715==by 0x1ACB40B3: opal_condition_wait (condition.h:93)
> ==16715==by 0x1ACB4201: ompi_request_wait_completion (request.h:381)
> ==16715==by 0x1ACB4305: ompi_request_default_wait (req_wait.c:39)
> ==16715==by 0x26BA2FFB: ompi_coll_tuned_bcast_intra_generic
> (coll_tuned_bcast.c:254)
> ==16715==by 0x26BA36F7: ompi_coll_tuned_bcast_intra_binomial
> (coll_tuned_bcast.c:385)
> ==16715==by 0x26B94289: ompi_coll_tuned_bcast_intra_dec_fixed
> (coll_tuned_decision_fixed.c:258)
> ==16715==by 0x1ACD55F2: PMPI_Bcast (pbcast.c:110)
> ==16715==by 0x2FE1CC48: ADIOI_Shfp_fname (shfp_fname.c:67)
> ==16715==by 0x2FDEB493: mca_io_romio_dist_MPI_File_open (open.c:177)
> ==16715==by 0x2FDE3B0D: mca_io_romio_file_open
> (io_romio_file_open.c:40)
> ==16715==by 0x1AD52344: module_init (io_base_file_select.c:455)
> ==16715==by 0x1AD51DFA: mca_io_base_file_select
> (io_base_file_select.c:238)
> ==16715==by 0x1ACA582F: ompi_file_open (file.c:130)
> ==16715==by 0x1AD30DA3: PMPI_File_open (pfile_open.c:94)
> ==16715==by 0x13F9B36F:
> PAIO::ouvreFichierMPIIO(PAGroupeProcessus&, std::string const&, int,
> ompi_file_t*&, bool) (PAIO.cc:290)
> ==16715==by 0xCA44252:
> GISLectureEcriture::litGISMPI(std::string,
> GroupeInfoSur&, std::string&) (GISLectureEcriture.icc:411)
> ==16715==by 0xCA23F0D: Champ::importeParallele(std::string const&)
> (Champ.cc:951)
> ==16715==by 0x4D0DEE: main (Test.NormesEtProjectionChamp.cc:789)
> ==16715==  Address 0x32ef3e50 is 0 bytes after a block of size 256
> alloc'd
> ==16715==at 0x4C2C5A4: malloc (vg_replace_malloc.c:296)
> ==16715==by 0x2FE1C78E: ADIOI_Malloc_fn (malloc.c:50)
> ==16715==by 0x2FE1C951: ADIOI_Shfp_fname (shfp_fname.c:25)
> ==16715==by 0x2FDEB493: mca_io_romio_dist_MPI_File_open (open.c:177)
> ==16715==by 0x2FDE3B0D: mca_io_romio_file_open
> (io_romio_file_open.c:40)
> ==16715==by 0x1AD52344: module_init (io_base_file_select.c:455)
> ==16715==by 0x1AD51DFA: mca_io_base_file_select
> (io_base_file_select.c:238)
> ==16715==by 0x1ACA582F: ompi_file_open (file.c:130)
> ==16715==by 0x1AD30DA3: PMPI_File_open (pfile_open.c:94)
> ==16715==by 0x13F9B36F:
> PAIO::ouvreFichierMPIIO(PAGroupeProcessus&, std::string const&, int,
> ompi_file_t*&, bool) (PAIO.cc:290)
> ==16715==by 0xCA44252:
> GISLectureEcriture::litGISMPI(std::string,
> GroupeInfoSur&, std::string&) (GISLectureEcriture.icc:411)
> ==16715==by 0xCA23F0D: Champ::importeParallele(std::string const&)
> (Champ.cc:951)
> ==16715==by 0x4D0DEE: main (Test.NormesEtProjectionChamp.cc:789)
> ...
> ...
> ==16715== Invalid write of size 1
> ==16715==at 0x4C2E7BB: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
> ==16715==by 0x1F60AA91: opal_convertor_unpack (opal_convertor.c:321)
> ==16715==by 0x25AA8DD3: mca_pml_ob1_recv_frag_callback_match
> (pml_ob1_recvfrag.c:225)
> ==16715==by 0x2544110C: mca_btl_vader_check_fboxes
> (btl_vader_fbox.h:220)
> ==16715==by 0x25443577: mca_btl_vader_component_progress
> (btl_vader_component.c:695)
> ==16715==by 0x1F5F0F27: opal_progress (opal_progress.c:207)
> ==16715==by 0x1ACB40B3: opal_condition_wait (condition.h:93)
> ==16715==by 0x1ACB4201: ompi_request_wait_completion (request.h:381)
> ==16715==by 0x1ACB4305: ompi_request_default_wait (req_wait.c:39)
> ==16715==by 0x26BA2FFB: ompi_coll_tuned_

Re: [OMPI users] OpenMPI 1.8.4rc3, 1.6.5 and 1.6.3: segmentation violation in mca_io_romio_dist_MPI_File_close

[Gilles Gouaillardet]

Eric,

i checked the source code (v1.8) and the limit for the shared_fp_fname
is 256 (hard coded).

i am now checking if the overflow is correctly detected (that could
explain the one byte overflow reported by valgrind)

Cheers,

Gilles

On 2014/12/15 11:52, Eric Chamberland wrote:
> Hi again,
>
> some new hints that might help:
>
> 1- With valgrind : If I run the same test case, same data, but
> moved to a shorter path+filename, then *valgrind* does *not*
> complains!!
> 2- Without valgrind: *Sometimes*, the test case with long
> path+filename passes without "segfaulting"!
> 3- It seems to happen at the fourth file I try to open using the
> following described procedure:
>
> Also, I was wondering about this: In this 2 processes test case
> (running in the same node), I :
>
> 1- open the file collectively (which resides on the same ssd drive on
> my computer)
> 2-  MPI_File_read_at_all a long int and 3 chars (11 bytes)
> 3- stop (because I detect I am not reading my MPIIO file format)
> 4- close the file
>
> A guess (FWIW): Can process rank 0, for example close the file too
> quickly, which destroys the string reserved for the filename that is
> used by process rank 1 which could be using shared memory on the same
> node?
>
> Thanks,
>
> Eric
>
> On 12/14/2014 02:06 PM, Eric Chamberland wrote:
>> Hi,
>>
>> I finally (thanks for fixing oversubscribing) tested with 1.8.4rc3 for
>> my problem with collective MPI I/O.
>>
>> A problem still there.  In this 2 processes example, process rank 1
>> dies with segfault while process rank 0 wait indefinitely...
>>
>> Running with valgrind, I found these errors which may gives hints:
>>
>> *
>> Rank 1:
>> *
>> On process rank 1, without valgrind it ends with either a segmentation
>> violation or memory corruption or invalide free without valgrind).
>>
>> But running with valgrind, it tells:
>>
>> ==16715== Invalid write of size 2
>> ==16715==at 0x4C2E793: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
>> ==16715==by 0x1F60AA91: opal_convertor_unpack (opal_convertor.c:321)
>> ==16715==by 0x25AA8DD3: mca_pml_ob1_recv_frag_callback_match
>> (pml_ob1_recvfrag.c:225)
>> ==16715==by 0x2544110C: mca_btl_vader_check_fboxes
>> (btl_vader_fbox.h:220)
>> ==16715==by 0x25443577: mca_btl_vader_component_progress
>> (btl_vader_component.c:695)
>> ==16715==by 0x1F5F0F27: opal_progress (opal_progress.c:207)
>> ==16715==by 0x1ACB40B3: opal_condition_wait (condition.h:93)
>> ==16715==by 0x1ACB4201: ompi_request_wait_completion (request.h:381)
>> ==16715==by 0x1ACB4305: ompi_request_default_wait (req_wait.c:39)
>> ==16715==by 0x26BA2FFB: ompi_coll_tuned_bcast_intra_generic
>> (coll_tuned_bcast.c:254)
>> ==16715==by 0x26BA36F7: ompi_coll_tuned_bcast_intra_binomial
>> (coll_tuned_bcast.c:385)
>> ==16715==by 0x26B94289: ompi_coll_tuned_bcast_intra_dec_fixed
>> (coll_tuned_decision_fixed.c:258)
>> ==16715==by 0x1ACD55F2: PMPI_Bcast (pbcast.c:110)
>> ==16715==by 0x2FE1CC48: ADIOI_Shfp_fname (shfp_fname.c:67)
>> ==16715==by 0x2FDEB493: mca_io_romio_dist_MPI_File_open (open.c:177)
>> ==16715==by 0x2FDE3B0D: mca_io_romio_file_open
>> (io_romio_file_open.c:40)
>> ==16715==by 0x1AD52344: module_init (io_base_file_select.c:455)
>> ==16715==by 0x1AD51DFA: mca_io_base_file_select
>> (io_base_file_select.c:238)
>> ==16715==by 0x1ACA582F: ompi_file_open (file.c:130)
>> ==16715==by 0x1AD30DA3: PMPI_File_open (pfile_open.c:94)
>> ==16715==by 0x13F9B36F:
>> PAIO::ouvreFichierMPIIO(PAGroupeProcessus&, std::string const&, int,
>> ompi_file_t*&, bool) (PAIO.cc:290)
>> ==16715==by 0xCA44252:
>> GISLectureEcriture::litGISMPI(std::string,
>> GroupeInfoSur&, std::string&) (GISLectureEcriture.icc:411)
>> ==16715==by 0xCA23F0D: Champ::importeParallele(std::string const&)
>> (Champ.cc:951)
>> ==16715==by 0x4D0DEE: main (Test.NormesEtProjectionChamp.cc:789)
>> ==16715==  Address 0x32ef3e50 is 0 bytes after a block of size 256
>> alloc'd
>> ==16715==at 0x4C2C5A4: malloc (vg_replace_malloc.c:296)
>> ==16715==by 0x2FE1C78E: ADIOI_Malloc_fn (malloc.c:50)
>> ==16715==by 0x2FE1C951: ADIOI_Shfp_fname (shfp_fname.c:25)
>> ==16715==by 0x2FDEB493: mca_io_romio_dist_MPI_File_open (open.c:177)
>> ==16715==by 0x2FDE3B0D: mca_io_romio_file_open
>> (io_romio_file_open.c:40)
>> ==16715==by 0x1AD52344: module_init (io_base_file_select.c:455)
>> ==16715==by 0x1AD51DFA: mca_io_base_file_select
>> (io_base_file_select.c:238)
>> ==16715==by 0x1ACA582F: ompi_file_open (file.c:130)
>> ==16715==by 0x1AD30DA3: PMPI_File_open (pfile_open.c:94)
>> ==16715==by 0x13F9B36F:
>> PAIO::ouvreFichierMPIIO(PAGroupeProcessus&, std::string const&, int,
>> ompi_file_t*&, bool) (PAIO.cc:290)
>> ==16715==by 0xCA44252:
>> GISLectureEcriture::litGISMPI(std::string,
>> GroupeInfoSur&, std::string&) (GISLectureEcriture.icc

Re: [OMPI users] OpenMPI 1.8.4rc3, 1.6.5 and 1.6.3: segmentation violation in mca_io_romio_dist_MPI_File_close

[Gilles Gouaillardet]

Eric,

here is a patch for the v1.8 series, it fixes a one byte overflow.

valgrind should stop complaining, and assuming this is the root cause of
the memory corruption,
that could also fix your program.

that being said, shared_fp_fname is limited to 255 characters (this is
hard coded) so even if
it gets truncated to 255 characters (instead of 256), the behavior could
be kind of random.

/* from ADIOI_Shfp_fname :
  If the real file is /tmp/thakur/testfile, the shared-file-pointer
   file will be /tmp/thakur/.testfile.shfp., where  is

FWIW,  is a random number that takes between 1 and 10 characters

could you please give this patch a try and let us know the results ?

Cheers,

Gilles

On 2014/12/15 11:52, Eric Chamberland wrote:
> Hi again,
>
> some new hints that might help:
>
> 1- With valgrind : If I run the same test case, same data, but
> moved to a shorter path+filename, then *valgrind* does *not*
> complains!!
> 2- Without valgrind: *Sometimes*, the test case with long
> path+filename passes without "segfaulting"!
> 3- It seems to happen at the fourth file I try to open using the
> following described procedure:
>
> Also, I was wondering about this: In this 2 processes test case
> (running in the same node), I :
>
> 1- open the file collectively (which resides on the same ssd drive on
> my computer)
> 2-  MPI_File_read_at_all a long int and 3 chars (11 bytes)
> 3- stop (because I detect I am not reading my MPIIO file format)
> 4- close the file
>
> A guess (FWIW): Can process rank 0, for example close the file too
> quickly, which destroys the string reserved for the filename that is
> used by process rank 1 which could be using shared memory on the same
> node?
>
> Thanks,
>
> Eric
>
> On 12/14/2014 02:06 PM, Eric Chamberland wrote:
>> Hi,
>>
>> I finally (thanks for fixing oversubscribing) tested with 1.8.4rc3 for
>> my problem with collective MPI I/O.
>>
>> A problem still there.  In this 2 processes example, process rank 1
>> dies with segfault while process rank 0 wait indefinitely...
>>
>> Running with valgrind, I found these errors which may gives hints:
>>
>> *
>> Rank 1:
>> *
>> On process rank 1, without valgrind it ends with either a segmentation
>> violation or memory corruption or invalide free without valgrind).
>>
>> But running with valgrind, it tells:
>>
>> ==16715== Invalid write of size 2
>> ==16715==at 0x4C2E793: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:915)
>> ==16715==by 0x1F60AA91: opal_convertor_unpack (opal_convertor.c:321)
>> ==16715==by 0x25AA8DD3: mca_pml_ob1_recv_frag_callback_match
>> (pml_ob1_recvfrag.c:225)
>> ==16715==by 0x2544110C: mca_btl_vader_check_fboxes
>> (btl_vader_fbox.h:220)
>> ==16715==by 0x25443577: mca_btl_vader_component_progress
>> (btl_vader_component.c:695)
>> ==16715==by 0x1F5F0F27: opal_progress (opal_progress.c:207)
>> ==16715==by 0x1ACB40B3: opal_condition_wait (condition.h:93)
>> ==16715==by 0x1ACB4201: ompi_request_wait_completion (request.h:381)
>> ==16715==by 0x1ACB4305: ompi_request_default_wait (req_wait.c:39)
>> ==16715==by 0x26BA2FFB: ompi_coll_tuned_bcast_intra_generic
>> (coll_tuned_bcast.c:254)
>> ==16715==by 0x26BA36F7: ompi_coll_tuned_bcast_intra_binomial
>> (coll_tuned_bcast.c:385)
>> ==16715==by 0x26B94289: ompi_coll_tuned_bcast_intra_dec_fixed
>> (coll_tuned_decision_fixed.c:258)
>> ==16715==by 0x1ACD55F2: PMPI_Bcast (pbcast.c:110)
>> ==16715==by 0x2FE1CC48: ADIOI_Shfp_fname (shfp_fname.c:67)
>> ==16715==by 0x2FDEB493: mca_io_romio_dist_MPI_File_open (open.c:177)
>> ==16715==by 0x2FDE3B0D: mca_io_romio_file_open
>> (io_romio_file_open.c:40)
>> ==16715==by 0x1AD52344: module_init (io_base_file_select.c:455)
>> ==16715==by 0x1AD51DFA: mca_io_base_file_select
>> (io_base_file_select.c:238)
>> ==16715==by 0x1ACA582F: ompi_file_open (file.c:130)
>> ==16715==by 0x1AD30DA3: PMPI_File_open (pfile_open.c:94)
>> ==16715==by 0x13F9B36F:
>> PAIO::ouvreFichierMPIIO(PAGroupeProcessus&, std::string const&, int,
>> ompi_file_t*&, bool) (PAIO.cc:290)
>> ==16715==by 0xCA44252:
>> GISLectureEcriture::litGISMPI(std::string,
>> GroupeInfoSur&, std::string&) (GISLectureEcriture.icc:411)
>> ==16715==by 0xCA23F0D: Champ::importeParallele(std::string const&)
>> (Champ.cc:951)
>> ==16715==by 0x4D0DEE: main (Test.NormesEtProjectionChamp.cc:789)
>> ==16715==  Address 0x32ef3e50 is 0 bytes after a block of size 256
>> alloc'd
>> ==16715==at 0x4C2C5A4: malloc (vg_replace_malloc.c:296)
>> ==16715==by 0x2FE1C78E: ADIOI_Malloc_fn (malloc.c:50)
>> ==16715==by 0x2FE1C951: ADIOI_Shfp_fname (shfp_fname.c:25)
>> ==16715==by 0x2FDEB493: mca_io_romio_dist_MPI_File_open (open.c:177)
>> ==16715==by 0x2FDE3B0D: mca_io_romio_file_open
>> (io_romio_file_open.c:40)
>> ==16715==by 0x1AD52344: module_init (io_base_file_select.c:455)
>> ==16715=

Re: [OMPI users] ERROR: C_FUNLOC function

[Gilles Gouaillardet]

Hi Siegmar,

a similar issue was reported in mpich with xlf compilers :
http://trac.mpich.org/projects/mpich/ticket/2144

They concluded this is a compiler issue (e.g. the compiler does not
implement TS 29113 subclause 8.1)


Jeff,
i made PR 315 https://github.com/open-mpi/ompi/pull/315
f08 binding support is disabled if TS29113 subclause 8.1 is not implemented
could you please review/comment on this ?


Cheers,

Gilles


On 2014/12/12 2:28, Siegmar Gross wrote:
> Hi Jeff,
>
> will you have the time to fix the Fortran problem for the new Oracle
> Solaris Studio 12.4 compiler suite in openmpi-1.8.4?
>
> tyr openmpi-1.8.4rc2-SunOS.sparc.64_cc 103 tail -15 
> log.make.SunOS.sparc.64_cc 
>   PPFC comm_compare_f08.lo
>   PPFC comm_connect_f08.lo
>   PPFC comm_create_errhandler_f08.lo
>
>fn = c_funloc(comm_errhandler_fn)
>  ^   
> "../../../../../openmpi-1.8.4rc2/ompi/mpi/fortran/use-mpi-f08/comm_create_errhan
> dler_f08.F90", Line = 22, Column = 18: ERROR: C_FUNLOC function argument must 
> be 
> a procedure that is interoperable or a procedure pointer associated with an 
> interoperable procedure.
> ...
>
>
> Kind regards
>
> Siegmar
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/25963.php

Re: [OMPI users] OpenMPI 1.8.4rc3, 1.6.5 and 1.6.3: segmentation violation in mca_io_romio_dist_MPI_File_close

[Gilles Gouaillardet]

Eric,

thanks for the simple test program.

i think i see what is going wrong and i will make some changes to avoid
the memory overflow.

that being said, there is a hard coded limit of 256 characters, and your
path is bigger than 300 characters.
bottom line, and even if there is no more memory overflow, that cannot
work as expected.

i will report this to the mpich folks, since romio is currently imported
from mpich.

Cheers,

Gilles

On 2014/12/16 0:16, Eric Chamberland wrote:
> Hi Gilles,
>
> just created a very simple test case!
>
> with this setup, you will see the bug with valgrind:
>
> export
> too_long=./this/is/a_very/long/path/that/contains/a/not/so/long/filename/but/trying/to/collectively/mpi_file_open/it/you/will/have/a/memory/corruption/resulting/of/invalide/writing/or/reading/past/the/end/of/one/or/some/hidden/strings/in/mpio/Simple/user/would/like/to/have/the/parameter/checked/and/an/error/returned/or/this/limit/removed
>
> mpicc -o bug_MPI_File_open_path_too_long
> bug_MPI_File_open_path_too_long.c
>
> mkdir -p $too_long
> echo "header of a text file" > $too_long/toto.txt
>
> mpirun -np 2 valgrind ./bug_MPI_File_open_path_too_long 
> $too_long/toto.txt
>
> and watch the errors!
>
> unfortunately, the memory corruptions here doesn't seem to segfault
> this simple test case, but in my case, it is fatal and with valgrind,
> it is reported...
>
> OpenMPI 1.6.5, 1.8.3rc3 are affected
>
> MPICH-3.1.3 also have the error!
>
> thanks,
>
> Eric
>

Re: [OMPI users] OpenMPI 1.8.4rc3, 1.6.5 and 1.6.3: segmentation violation in mca_io_romio_dist_MPI_File_close

[Gilles Gouaillardet]

Eric and all,

That is clearly a limitation in romio, and this is being tracked at
https://trac.mpich.org/projects/mpich/ticket/2212

in the mean time, what we can do in OpenMPI is update
mca_io_romio_file_open() and fails with a user friendly error message
if strlen(filename) is larger that 225.

Cheers,

Gilles

On 2014/12/16 12:43, Gilles Gouaillardet wrote:
> Eric,
>
> thanks for the simple test program.
>
> i think i see what is going wrong and i will make some changes to avoid
> the memory overflow.
>
> that being said, there is a hard coded limit of 256 characters, and your
> path is bigger than 300 characters.
> bottom line, and even if there is no more memory overflow, that cannot
> work as expected.
>
> i will report this to the mpich folks, since romio is currently imported
> from mpich.
>
> Cheers,
>
> Gilles
>
> On 2014/12/16 0:16, Eric Chamberland wrote:
>> Hi Gilles,
>>
>> just created a very simple test case!
>>
>> with this setup, you will see the bug with valgrind:
>>
>> export
>> too_long=./this/is/a_very/long/path/that/contains/a/not/so/long/filename/but/trying/to/collectively/mpi_file_open/it/you/will/have/a/memory/corruption/resulting/of/invalide/writing/or/reading/past/the/end/of/one/or/some/hidden/strings/in/mpio/Simple/user/would/like/to/have/the/parameter/checked/and/an/error/returned/or/this/limit/removed
>>
>> mpicc -o bug_MPI_File_open_path_too_long
>> bug_MPI_File_open_path_too_long.c
>>
>> mkdir -p $too_long
>> echo "header of a text file" > $too_long/toto.txt
>>
>> mpirun -np 2 valgrind ./bug_MPI_File_open_path_too_long 
>> $too_long/toto.txt
>>
>> and watch the errors!
>>
>> unfortunately, the memory corruptions here doesn't seem to segfault
>> this simple test case, but in my case, it is fatal and with valgrind,
>> it is reported...
>>
>> OpenMPI 1.6.5, 1.8.3rc3 are affected
>>
>> MPICH-3.1.3 also have the error!
>>
>> thanks,
>>
>> Eric
>>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/26005.php

Re: [OMPI users] OMPI users] OMPI users] OMPI users] OMPI users] MPI inside MPI (still)

[Gilles Gouaillardet]

#x27;< infile' , 2 ,...)"
>
>to execute 'mpirun -n 2 siesta < infile' on the spawnee side. That was 
>my first choice. Well, siesta behaves as if no stdin file was present...
>
>Alex
>
>
>2014-12-15 17:07 GMT-02:00 Ralph Castain :
>
>You should be able to just include that in your argv that you pass to the 
>Comm_spawn API.
>
>
>
>On Mon, Dec 15, 2014 at 9:27 AM, Alex A. Schmidt  wrote:
>
>George,
>
>Thanks for the tip. In fact, calling mpi_comm_spawn right away with 
>MPI_COMM_SELF
>
>has worked for me just as well -- no subgroups needed at all.
>
>I am testing this openmpi app named "siesta" in parallel. The source code is 
>available,
>so making it "spawn ready" by adding the pair mpi_comm_get_parent + 
>mpi_comm_disconnect 
>into the main code can be done.  If it works, maybe the siesta's developers 
>can be convinced 
>to add this feature in a future release.
>
>
>However, siesta is launched only by specifying input/output files with i/o 
>redirection like 
>
>mpirun -n   siesta < infile > outfile
>
>
>So far, I could not find anything about how to set an stdin file for an 
>spawnee process. 
>Specifiyng it in a app context file doesn't seem to work. Can it be done? 
>Maybe through 
>an MCA parameter?
>
>
>Alex
>
>
>
>
>
>
>2014-12-15 2:43 GMT-02:00 George Bosilca :
>
>Alex,
>
>
>The code looks good, and is 100% MPI standard accurate.
>
>
>I would change the way you create the subcoms in the parent. You do a lot of 
>useless operations, as you can achieve exactly the same outcome (one 
>communicator per node), either by duplicating MPI_COMM_SELF or doing an 
>MPI_Comm_split with the color equal to your rank.
>
>
>  George.
>
>
>
>On Sun, Dec 14, 2014 at 2:20 AM, Alex A. Schmidt  wrote:
>
>Hi,
>
>Sorry, guys. I don't think the newbie here can follow any discussion 
>beyond basic mpi... 
>
>Anyway, if I add the pair     
>
>call MPI_COMM_GET_PARENT(mpi_comm_parent,ierror)
>call MPI_COMM_DISCONNECT(mpi_comm_parent,ierror)
>
>on the spawnee side I get the proper response in the spawning processes.
>
>Please, take a look at the attached toy codes parent.F and child.F 
>I've been playing with. 'mpirun -n 2 parent' seems to work as expected.
>
>Alex
>
>
>2014-12-13 23:46 GMT-02:00 Gilles Gouaillardet :
>
>Alex,
>
>Are you calling MPI_Comm_disconnect in the 3 "master" tasks and with the same 
>remote communicator ?
>
>I also read the man page again, and MPI_Comm_disconnect does not ensure the 
>remote processes have finished or called MPI_Comm_disconnect, so that might 
>not be the thing you need.
>George, can you please comment on that ?
>
>Cheers,
>
>Gilles
>
>George Bosilca  wrote:
>
>MPI_Comm_disconnect should be a local operation, there is no reason for it to 
>deadlock. I looked at the code and everything is local with the exception of a 
>call to PMIX.FENCE. Can you attach to your deadlocked processes and confirm 
>that they are stopped in the pmix.fence?
>
>
>  George.
>
>
>
>On Sat, Dec 13, 2014 at 8:47 AM, Alex A. Schmidt  wrote:
>
>Hi
>
>Sorry, I was calling mpi_comm_disconnect on the group comm handler, not
>on the intercomm handler returned from the spawn call as it should be.
>
>Well, calling the disconnect on the intercomm handler does halt the spwaner
>side but the wait is never completed since, as George points out, there is no
>disconnect call being made on the spawnee side and that brings me back
>to the beginning of the problem since, being a third party app, that call would
>never be there. I guess an mpi wrapper to deal with that could be made for
>the app, but I fell the wrapper itself, at the end, would face the same problem
>we face right now.
>
>My application is a genetic algorithm code that search optimal configuration
>(minimum or maximum energy) of cluster of atoms. The work flow bottleneck
>is the calculation of the cluster energy. For the cases which an analytical
>potential is available the calculation can be made internally and the workload
>is distributed among slaves nodes from a master node. This is also done
>when an analytical potential is not available and the energy calculation must
>be done externally by a quantum chemistry code like dftb+, siesta and Gaussian.
>So far, we have been running these codes in serial mode. No need to say that
>we could do a lot better if they could be executed in parallel.
>
>I am not familiar with DMRAA but it seems to be the right choice to deal with
>job schedulers as it covers the ones I am interested in (pbs/torque a

Re: [OMPI users] OMPI users] OpenMPI 1.8.4rc3, 1.6.5 and 1.6.3: segmentation violation in mca_io_romio_dist_MPI_File_close

[Gilles Gouaillardet]

Eric,

As long as lFileNameWithoutTooLongPath length is less than 226 characters and 
you do not run into some threads related race conditions, that should be just 
fine, and that roughly covers 99% cases.

Thanks for sharing this workaround !

Cheers,

Gilles

Eric Chamberland  wrote:
>Hi!
>
>Here is a "poor man's fix" that works for me (the idea is not from me, 
>thanks to Thomas H.):
>
>#1- char* lCwd = getcwd(0,0);
>#2- chdir(lPathToFile);
>#3- MPI_File_open(...,lFileNameWithoutTooLongPath,...);
>#4- chdir(lCwd);
>#5- ...
>
>I think there are some limitations but it works very well for our 
>uses... and until a "real" fix is proposed...
>
>Thanks for helping!
>
>Eric
>
>
>On 12/15/2014 11:42 PM, Gilles Gouaillardet wrote:
>> Eric and all,
>>
>> That is clearly a limitation in romio, and this is being tracked at
>> https://trac.mpich.org/projects/mpich/ticket/2212
>>
>> in the mean time, what we can do in OpenMPI is update
>> mca_io_romio_file_open() and fails with a user friendly error message
>> if strlen(filename) is larger that 225.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/16 12:43, Gilles Gouaillardet wrote:
>>> Eric,
>>>
>>> thanks for the simple test program.
>>>
>>> i think i see what is going wrong and i will make some changes to avoid
>>> the memory overflow.
>>>
>>> that being said, there is a hard coded limit of 256 characters, and your
>>> path is bigger than 300 characters.
>>> bottom line, and even if there is no more memory overflow, that cannot
>>> work as expected.
>>>
>>> i will report this to the mpich folks, since romio is currently imported
>>> from mpich.
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On 2014/12/16 0:16, Eric Chamberland wrote:
>>>> Hi Gilles,
>>>>
>>>> just created a very simple test case!
>>>>
>>>> with this setup, you will see the bug with valgrind:
>>>>
>>>> export
>>>> too_long=./this/is/a_very/long/path/that/contains/a/not/so/long/filename/but/trying/to/collectively/mpi_file_open/it/you/will/have/a/memory/corruption/resulting/of/invalide/writing/or/reading/past/the/end/of/one/or/some/hidden/strings/in/mpio/Simple/user/would/like/to/have/the/parameter/checked/and/an/error/returned/or/this/limit/removed
>>>>
>>>> mpicc -o bug_MPI_File_open_path_too_long
>>>> bug_MPI_File_open_path_too_long.c
>>>>
>>>> mkdir -p $too_long
>>>> echo "header of a text file" > $too_long/toto.txt
>>>>
>>>> mpirun -np 2 valgrind ./bug_MPI_File_open_path_too_long
>>>> $too_long/toto.txt
>>>>
>>>> and watch the errors!
>>>>
>>>> unfortunately, the memory corruptions here doesn't seem to segfault
>>>> this simple test case, but in my case, it is fatal and with valgrind,
>>>> it is reported...
>>>>
>>>> OpenMPI 1.6.5, 1.8.3rc3 are affected
>>>>
>>>> MPICH-3.1.3 also have the error!
>>>>
>>>> thanks,
>>>>
>>>> Eric
>>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2014/12/26005.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/12/26006.php
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/12/26022.php

Re: [OMPI users] OMPI users] ERROR: C_FUNLOC function

[Gilles Gouaillardet]

FWIW

I faced a simlar issue on my linux virtualbox.
My shared folder is a vboxfs filesystem, but statfs returns the nfs magic id.
That causes some mess and the test fails.
At this stage i cannot tell whether i should blame the glibc, the kernel, a 
virtualbox driver or myself

Cheer,

Gilles

Mike Dubman さんのメール:
>Hi Siegmar,
>
>Could you please check the /etc/mtab file for real FS type for the following 
>mount points:
>
>
>get_mounts: dirs[16]:/misc fs:autofs nfs:No
>get_mounts: dirs[17]:/net fs:autofs nfs:No
>get_mounts: dirs[18]:/home fs:autofs nfs:No
>
>
>could you please check if mntent.h and paths.h were detected by "configure" in 
>config.log?
>
>
>Thanks
>
>
>
>On Thu, Dec 18, 2014 at 12:39 AM, Jeff Squyres (jsquyres)  
>wrote:
>
>Siegmar --
>
>I filed https://github.com/open-mpi/ompi/issues/317 and 
>https://github.com/open-mpi/ompi/issues/318.
>
>
>
>
>On Dec 17, 2014, at 3:33 PM, Siegmar Gross 
> wrote:
>
>> Hi Jeff,
>>
>>> This fix was just pushed to the OMPI master.  A new master tarball
>>> should be available shortly (probably within an hour or so -- look
>>> for a tarball dated Dec 17 at http://www.open-mpi.org/nightly/master/).
>>
>> Yes, I could build it now. Thank you very much to everybody who helped
>> to fix the problem. I get an error for "make check" on Solaris 10 Sparc,
>> Solaris 10 x86_64, and OpenSUSE Linux with both gcc-4.9.2 and Sun C 5.13.
>> Hopefully I have some time tomorrow to to test this version with some
>> simple programs.
>>
>> Linux, Sun C 5.13:
>> ==
>> ...
>> PASS: opal_bit_ops
>> Failure : Mismatch: input "/home", expected:0 got:1
>>
>> Failure : Mismatch: input "/net", expected:0 got:1
>>
>> Failure : Mismatch: input "/misc", expected:0 got:1
>>
>> SUPPORT: OMPI Test failed: opal_path_nfs() (3 of 20 failed)
>> Test usage: ./opal_path_nfs [DIR]
>> On Linux interprets output from mount(8) to check for nfs and verify 
>> opal_path_nfs()
>> Additionally, you may specify multiple DIR on the cmd-line, of which you the 
>> output
>> get_mounts: dirs[0]:/dev fs:devtmpfs nfs:No
>> get_mounts: dirs[1]:/dev/shm fs:tmpfs nfs:No
>> get_mounts: dirs[2]:/run fs:tmpfs nfs:No
>> get_mounts: dirs[3]:/dev/pts fs:devpts nfs:No
>> get_mounts: dirs[4]:/ fs:ext4 nfs:No
>> get_mounts: dirs[5]:/proc fs:proc nfs:No
>> get_mounts: dirs[6]:/sys fs:sysfs nfs:No
>> get_mounts: dirs[7]:/sys/kernel/debug fs:debugfs nfs:No
>> get_mounts: dirs[8]:/sys/kernel/security fs:securityfs nfs:No
>> get_mounts: dirs[9]:/local fs:ext4 nfs:No
>> get_mounts: dirs[10]:/var/lock fs:tmpfs nfs:No
>> get_mounts: dirs[11]:/var/run fs:tmpfs nfs:No
>> get_mounts: dirs[12]:/media fs:tmpfs nfs:No
>> get_mounts: dirs[13]:/usr/local fs:nfs nfs:Yes
>> get_mounts: dirs[14]:/opt/global fs:nfs nfs:Yes
>> get_mounts: already know dir[13]:/usr/local
>> get_mounts: dirs[13]:/usr/local fs:nfs nfs:Yes
>> get_mounts: dirs[15]:/export2 fs:nfs nfs:Yes
>> get_mounts: already know dir[14]:/opt/global
>> get_mounts: dirs[14]:/opt/global fs:nfs nfs:Yes
>> get_mounts: dirs[16]:/misc fs:autofs nfs:No
>> get_mounts: dirs[17]:/net fs:autofs nfs:No
>> get_mounts: dirs[18]:/home fs:autofs nfs:No
>> get_mounts: dirs[19]:/home/fd1026 fs:nfs nfs:Yes
>> test(): file:/home/fd1026 bool:1
>> test(): file:/home bool:0
>> test(): file:/net bool:0
>> test(): file:/misc bool:0
>> test(): file:/export2 bool:1
>> test(): file:/opt/global bool:1
>> test(): file:/usr/local bool:1
>> test(): file:/media bool:0
>> test(): file:/var/run bool:0
>> test(): file:/var/lock bool:0
>> test(): file:/local bool:0
>> test(): file:/sys/kernel/security bool:0
>> test(): file:/sys/kernel/debug bool:0
>> test(): file:/sys bool:0
>> test(): file:/proc bool:0
>> test(): file:/ bool:0
>> test(): file:/dev/pts bool:0
>> test(): file:/run bool:0
>> test(): file:/dev/shm bool:0
>> test(): file:/dev bool:0
>> FAIL: opal_path_nfs
>> 
>> 1 of 2 tests failed
>> Please report to http://www.open-mpi.org/community/help/
>> 
>> make[3]: *** [check-TESTS] Error 1
>> make[3]: Leaving directory
>> `/export2/src/openmpi-1.9/openmpi-dev-557-g01a24c4-Linux.x86_64.64_cc/test/util'
>> make[2]: *** [check-am] Error 2
>> make[2]: Leaving directory
>> `/export2/src/openmpi-1.9/openmpi-dev-557-g01a24c4-Linux.x86_64.64_cc/test/util'
>> make[1]: *** [check-recursive] Error 1
>> make[1]: Leaving directory 
>> `/export2/src/openmpi-1.9/openmpi-dev-557-g01a24c4-Linux.x86_64.64_cc/test'
>> make: *** [check-recursive] Error 1
>> tyr openmpi-dev-557-g01a24c4-Linux.x86_64.64_cc 133 dtmail_ssh &
>> [1] 17531
>> tyr openmpi-dev-557-g01a24c4-Linux.x86_64.64_cc 134 libSDtMail: Warning: Xt 
>> Warning: Missing charsets in
>> String to FontSet conversion
>> libSDtMail: Warning: Xt Warning: Cannot convert string "-dt-interface
>> user-medium-r-normal-s*-*-*-*-*-*-*-*-*" to type FontSet
>>
>> tyr openmpi-dev-557-g01a24c4-Linux.x86_64.64_cc 134
>>
>>
>>
>> Linux, gcc-4.9.2:
>> ==

Re: [OMPI users] processes hang with openmpi-dev-602-g82c02b4

[Gilles Gouaillardet]

Siegmar,

could you please give a try to the attached patch ?
/* and keep in mind this is just a workaround that happen to work */

Cheers,

Gilles

On 2014/12/22 22:48, Siegmar Gross wrote:
> Hi,
>
> today I installed openmpi-dev-602-g82c02b4 on my machines (Solaris 10 Sparc,
> Solaris 10 x86_64, and openSUSE Linux 12.1 x86_64) with gcc-4.9.2 and the
> new Solaris Studio 12.4 compilers. All build processes finished without
> errors, but I have a problem running a very small program. It works for
> three processes but hangs for six processes. I have the same behaviour
> for both compilers.
>
> tyr small_prog 139 time; mpiexec -np 3 --host sunpc1,linpc1,tyr 
> init_finalize; time
> 827.161u 210.126s 30:51.08 56.0%0+0k 4151+20io 2898pf+0w
> Hello!
> Hello!
> Hello!
> 827.886u 210.335s 30:54.68 55.9%0+0k 4151+20io 2898pf+0w
> tyr small_prog 140 time; mpiexec -np 6 --host sunpc1,linpc1,tyr 
> init_finalize; time
> 827.946u 210.370s 31:15.02 55.3%0+0k 4151+20io 2898pf+0w
> ^CKilled by signal 2.
> Killed by signal 2.
> 869.242u 221.644s 33:40.54 53.9%0+0k 4151+20io 2898pf+0w
> tyr small_prog 141 
>
> tyr small_prog 145 ompi_info | grep -e "Open MPI repo revision:" -e "C 
> compiler:"
>   Open MPI repo revision: dev-602-g82c02b4
>   C compiler: cc
> tyr small_prog 146 
>
>
> tyr small_prog 146 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
> GNU gdb (GDB) 7.6.1
> ...
> (gdb) run -np 3 --host sunpc1,linpc1,tyr init_finalize
> Starting program: /usr/local/openmpi-1.9.0_64_cc/bin/mpiexec -np 3 --host 
> sunpc1,linpc1,tyr 
> init_finalize
> [Thread debugging using libthread_db enabled]
> [New Thread 1 (LWP 1)]
> [New LWP2]
> Hello!
> Hello!
> Hello!
> [LWP2 exited]
> [New Thread 2]
> [Switching to Thread 1 (LWP 1)]
> sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be found to 
> satisfy query
> (gdb) run -np 6 --host sunpc1,linpc1,tyr init_finalize
> The program being debugged has been started already.
> Start it from the beginning? (y or n) y
>
> Starting program: /usr/local/openmpi-1.9.0_64_cc/bin/mpiexec -np 6 --host 
> sunpc1,linpc1,tyr 
> init_finalize
> [Thread debugging using libthread_db enabled]
> [New Thread 1 (LWP 1)]
> [New LWP2]
> ^CKilled by signal 2.
> Killed by signal 2.
>
> Program received signal SIGINT, Interrupt.
> [Switching to Thread 1 (LWP 1)]
> 0x7d1dc6b0 in __pollsys () from /lib/sparcv9/libc.so.1
> (gdb) bt
> #0  0x7d1dc6b0 in __pollsys () from /lib/sparcv9/libc.so.1
> #1  0x7d1cb468 in _pollsys () from /lib/sparcv9/libc.so.1
> #2  0x7d170ed8 in poll () from /lib/sparcv9/libc.so.1
> #3  0x7e69a630 in poll_dispatch ()
>from /usr/local/openmpi-1.9.0_64_cc/lib64/libopen-pal.so.0
> #4  0x7e6894ec in opal_libevent2021_event_base_loop ()
>from /usr/local/openmpi-1.9.0_64_cc/lib64/libopen-pal.so.0
> #5  0x0001eb14 in orterun (argc=1757447168, argv=0xff7ed8550cff)
> at ../../../../openmpi-dev-602-g82c02b4/orte/tools/orterun/orterun.c:1090
> #6  0x00014e2c in main (argc=256, argv=0xff7ed8af5c00)
> at ../../../../openmpi-dev-602-g82c02b4/orte/tools/orterun/main.c:13
> (gdb) 
>
> Any ideas? Unfortunately I'm leaving for vaccation so that I cannot test
> any patches until the end of the year. Neverthess I wanted to report the
> problem. At the moment I cannot test if I have the same behaviour in a
> homogeneous environment with three machines because the new version isn't
> available before tomorrow on the other machines. I used the following
> configure command.
>
> ../openmpi-dev-602-g82c02b4/configure --prefix=/usr/local/openmpi-1.9.0_64_cc 
> \
>   --libdir=/usr/local/openmpi-1.9.0_64_cc/lib64 \
>   --with-jdk-bindir=/usr/local/jdk1.8.0/bin \
>   --with-jdk-headers=/usr/local/jdk1.8.0/include \
>   JAVA_HOME=/usr/local/jdk1.8.0 \
>   LDFLAGS="-m64 -mt" \
>   CC="cc" CXX="CC" FC="f95" \
>   CFLAGS="-m64 -mt" CXXFLAGS="-m64 -library=stlport4" FCFLAGS="-m64" \
>   CPP="cpp" CXXCPP="cpp" \
>   CPPFLAGS="" CXXCPPFLAGS="" \
>   --enable-mpi-cxx \
>   --enable-cxx-exceptions \
>   --enable-mpi-java \
>   --enable-heterogeneous \
>   --enable-mpi-thread-multiple \
>   --with-threads=posix \
>   --with-hwloc=internal \
>   --without-verbs \
>   --with-wrapper-cflags="-m64 -mt" \
>   --with-wrapper-cxxflags="-m64 -library=stlport4" \
>   --with-wrapper-ldflags="-mt" \
>   --enable-debug \
>   |& tee log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_cc
>
> Furthermore I used the following test program.
>
> #include 
> #include 
> #include "mpi.h"
>
> int main (int argc, char *argv[])
> {
>   MPI_Init (&argc, &argv);
>   printf ("Hello!\n");
>   MPI_Finalize ();
>   return EXIT_SUCCESS;
> }
>
>
>
> Kind regards
>
> Siegmar
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community

Re: [OMPI users] processes hang with openmpi-dev-602-g82c02b4

[Gilles Gouaillardet]

Kawashima-san,

i'd rather consider this as a bug in the README (!)


heterogenous support has been broken for some time, but it was
eventually fixed.

truth is there are *very* limited resources (both human and hardware)
maintaining heterogeneous
support, but that does not mean heterogeneous support should not be
used, nor that bug report
will be ignored.

Cheers,

Gilles

On 2014/12/24 9:26, Kawashima, Takahiro wrote:
> Hi Siegmar,
>
> Heterogeneous environment is not supported officially.
>
> README of Open MPI master says:
>
> --enable-heterogeneous
>   Enable support for running on heterogeneous clusters (e.g., machines
>   with different endian representations).  Heterogeneous support is
>   disabled by default because it imposes a minor performance penalty.
>
>   *** THIS FUNCTIONALITY IS CURRENTLY BROKEN - DO NOT USE ***
>
>> Hi,
>>
>> today I installed openmpi-dev-602-g82c02b4 on my machines (Solaris 10 Sparc,
>> Solaris 10 x86_64, and openSUSE Linux 12.1 x86_64) with gcc-4.9.2 and the
>> new Solaris Studio 12.4 compilers. All build processes finished without
>> errors, but I have a problem running a very small program. It works for
>> three processes but hangs for six processes. I have the same behaviour
>> for both compilers.
>>
>> tyr small_prog 139 time; mpiexec -np 3 --host sunpc1,linpc1,tyr 
>> init_finalize; time
>> 827.161u 210.126s 30:51.08 56.0%0+0k 4151+20io 2898pf+0w
>> Hello!
>> Hello!
>> Hello!
>> 827.886u 210.335s 30:54.68 55.9%0+0k 4151+20io 2898pf+0w
>> tyr small_prog 140 time; mpiexec -np 6 --host sunpc1,linpc1,tyr 
>> init_finalize; time
>> 827.946u 210.370s 31:15.02 55.3%0+0k 4151+20io 2898pf+0w
>> ^CKilled by signal 2.
>> Killed by signal 2.
>> 869.242u 221.644s 33:40.54 53.9%0+0k 4151+20io 2898pf+0w
>> tyr small_prog 141 
>>
>> tyr small_prog 145 ompi_info | grep -e "Open MPI repo revision:" -e "C 
>> compiler:"
>>   Open MPI repo revision: dev-602-g82c02b4
>>   C compiler: cc
>> tyr small_prog 146 
>>
>>
>> tyr small_prog 146 /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec
>> GNU gdb (GDB) 7.6.1
>> ...
>> (gdb) run -np 3 --host sunpc1,linpc1,tyr init_finalize
>> Starting program: /usr/local/openmpi-1.9.0_64_cc/bin/mpiexec -np 3 --host 
>> sunpc1,linpc1,tyr 
>> init_finalize
>> [Thread debugging using libthread_db enabled]
>> [New Thread 1 (LWP 1)]
>> [New LWP2]
>> Hello!
>> Hello!
>> Hello!
>> [LWP2 exited]
>> [New Thread 2]
>> [Switching to Thread 1 (LWP 1)]
>> sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be found to 
>> satisfy query
>> (gdb) run -np 6 --host sunpc1,linpc1,tyr init_finalize
>> The program being debugged has been started already.
>> Start it from the beginning? (y or n) y
>>
>> Starting program: /usr/local/openmpi-1.9.0_64_cc/bin/mpiexec -np 6 --host 
>> sunpc1,linpc1,tyr 
>> init_finalize
>> [Thread debugging using libthread_db enabled]
>> [New Thread 1 (LWP 1)]
>> [New LWP2]
>> ^CKilled by signal 2.
>> Killed by signal 2.
>>
>> Program received signal SIGINT, Interrupt.
>> [Switching to Thread 1 (LWP 1)]
>> 0x7d1dc6b0 in __pollsys () from /lib/sparcv9/libc.so.1
>> (gdb) bt
>> #0  0x7d1dc6b0 in __pollsys () from /lib/sparcv9/libc.so.1
>> #1  0x7d1cb468 in _pollsys () from /lib/sparcv9/libc.so.1
>> #2  0x7d170ed8 in poll () from /lib/sparcv9/libc.so.1
>> #3  0x7e69a630 in poll_dispatch ()
>>from /usr/local/openmpi-1.9.0_64_cc/lib64/libopen-pal.so.0
>> #4  0x7e6894ec in opal_libevent2021_event_base_loop ()
>>from /usr/local/openmpi-1.9.0_64_cc/lib64/libopen-pal.so.0
>> #5  0x0001eb14 in orterun (argc=1757447168, argv=0xff7ed8550cff)
>> at ../../../../openmpi-dev-602-g82c02b4/orte/tools/orterun/orterun.c:1090
>> #6  0x00014e2c in main (argc=256, argv=0xff7ed8af5c00)
>> at ../../../../openmpi-dev-602-g82c02b4/orte/tools/orterun/main.c:13
>> (gdb) 
>>
>> Any ideas? Unfortunately I'm leaving for vaccation so that I cannot test
>> any patches until the end of the year. Neverthess I wanted to report the
>> problem. At the moment I cannot test if I have the same behaviour in a
>> homogeneous environment with three machines because the new version isn't
>> available before tomorrow on the other machines. I used the following
>> configure command.
>>
>> ../openmpi-dev-602-g82c02b4/configure 
>> --prefix=/usr/local/openmpi-1.9.0_64_cc \
>>   --libdir=/usr/local/openmpi-1.9.0_64_cc/lib64 \
>>   --with-jdk-bindir=/usr/local/jdk1.8.0/bin \
>>   --with-jdk-headers=/usr/local/jdk1.8.0/include \
>>   JAVA_HOME=/usr/local/jdk1.8.0 \
>>   LDFLAGS="-m64 -mt" \
>>   CC="cc" CXX="CC" FC="f95" \
>>   CFLAGS="-m64 -mt" CXXFLAGS="-m64 -library=stlport4" FCFLAGS="-m64" \
>>   CPP="cpp" CXXCPP="cpp" \
>>   CPPFLAGS="" CXXCPPFLAGS="" \
>>   --enable-mpi-cxx \
>>   --enable-cxx-exceptions \
>>   --enable-mpi-java \
>>   --enable-heterogeneous \
>>   --enable-mpi-thread-multiple \
>>   --with-threads=po

Re: [OMPI users] OMPI users] What could cause a segfault in OpenMPI?

[Gilles Gouaillardet]

Where does the error occurs ?
MPI_Init ?
MPI_Finalize ?
In between ?

In the first case, the bug is likely a mishandled error case,
which means OpenMPI is unlikely the root cause of the crash.

Did you check infniband is up and running on your cluster ?

Cheers,

Gilles 

Saliya Ekanayake さんのメール:
>It's been a while on this, but we are still having trouble getting OpenMPI to 
>work with Infiniband on this cluster. We tried with latest 1.8.4 as well, but 
>it's still the same.
>
>
>To recap, we get the following error when MPI initializes (in the simple Hello 
>world C example) with Infiniband. Everything works fine if we explicitly turn 
>off openib with --mca btl ^openib
>
>
>This is the error I got after debugging with gdb as you suggested.
>
>
>hello_c: connect/btl_openib_connect_udcm.c:736: udcm_module_finalize: 
>Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == ((opal_object_t *) 
>(&m->cm_recv_msg_queue))->obj_magic_id' failed.
>
>
>Thank you,
>
>Saliya
>
>
>On Mon, Nov 10, 2014 at 10:01 AM, Saliya Ekanayake  wrote:
>
>Thank you Jeff, I'll try this and  let you know. 
>
>Saliya 
>
>On Nov 10, 2014 6:42 AM, "Jeff Squyres (jsquyres)"  wrote:
>
>I am sorry for the delay; I've been caught up in SC deadlines.  :-(
>
>I don't see anything blatantly wrong in this output.
>
>Two things:
>
>1. Can you try a nightly v1.8.4 snapshot tarball?  This will check to see if 
>whatever the bug is has been fixed for the upcoming release:
>
>    http://www.open-mpi.org/nightly/v1.8/
>
>2. Build Open MPI with the --enable-debug option (note that this adds a 
>slight-but-noticeable performance penalty).  When you run, it should dump a 
>core file.  Load that core file in a debugger and see where it is failing 
>(i.e., file and line in the OMPI source).
>
>We don't usually have to resort to asking users to perform #2, but there's no 
>additional information to give a clue as to what is happening.  :-(
>
>
>
>On Nov 9, 2014, at 11:43 AM, Saliya Ekanayake  wrote:
>
>> Hi Jeff,
>>
>> You are probably busy, but just checking if you had a chance to look at this.
>>
>> Thanks,
>> Saliya
>>
>> On Thu, Nov 6, 2014 at 9:19 AM, Saliya Ekanayake  wrote:
>> Hi Jeff,
>>
>> I've attached a tar file with information.
>>
>> Thank you,
>> Saliya
>>
>> On Tue, Nov 4, 2014 at 4:18 PM, Jeff Squyres (jsquyres)  
>> wrote:
>> Looks like it's failing in the openib BTL setup.
>>
>> Can you send the info listed here?
>>
>>     http://www.open-mpi.org/community/help/
>>
>>
>>
>> On Nov 4, 2014, at 1:10 PM, Saliya Ekanayake  wrote:
>>
>> > Hi,
>> >
>> > I am using OpenMPI 1.8.1 in a Linux cluster that we recently setup. It 
>> > builds fine, but when I try to run even the simplest hello.c program it'll 
>> > cause a segfault. Any suggestions on how to correct this?
>> >
>> > The steps I did and error message are below.
>> >
>> > 1. Built OpenMPI 1.8.1 on the cluster. The ompi_info is attached.
>> > 2. cd to examples directory and mpicc hello_c.c
>> > 3. mpirun -np 2 ./a.out
>> > 4. Error text is attached.
>> >
>> > Please let me know if you need more info.
>> >
>> > Thank you,
>> > Saliya
>> >
>> >
>> > --
>> > Saliya Ekanayake esal...@gmail.com
>> > Cell 812-391-4914 Home 812-961-6383
>> > http://saliya.org
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post: 
>> > http://www.open-mpi.org/community/lists/users/2014/11/25668.php
>>
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/11/25672.php
>>
>>
>>
>> --
>> Saliya Ekanayake esal...@gmail.com
>> Cell 812-391-4914 Home 812-961-6383
>> http://saliya.org
>>
>>
>>
>> --
>> Saliya Ekanayake esal...@gmail.com
>> Cell 812-391-4914 Home 812-961-6383
>> http://saliya.org
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/11/25717.php
>
>
>--
>Jeff Squyres
>jsquy...@cisco.com
>For corporate legal information go to: 
>http://www.cisco.com/web/about/doing_business/legal/cri/
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/11/25723.php
>
>
>
>
>-- 
>
>Saliya Ekanayake
>
>Ph.D. Candidate | Research Assistant
>
>School of Informatics and Computing | Digital Science Center
>
>Indiana University, Bloomington
>Cell 812-391-4914
>http://saliya.org
>

Re: [OMPI users] OMPI users] OMPI users] What could cause a segfault in OpenMPI?

[Gilles Gouaillardet]

Do you mean OMPI did not issue such a warning ?

If there was no such warning, could you please detail
How much ram on your system
The value returned by ulimit -l
The ofed stack you are running
The kernel modules you loaded
The model of your ib card

Cheers

Gilles

Saliya Ekanayake さんのメール:
>I meant it works now, sorry for the confusion.
>
>
>Running the test revealed a warning on memory registration, which we fixed by 
>setting unlimited in ulimit -l. Then running OMPI sample worked too.
>
>
>Thank you,
>
>saliya
>
>
>
>
>On Sun, Dec 28, 2014 at 11:18 PM, Ralph Castain  wrote:
>
>So you are saying the test worked, but you are still encountering an error 
>when executing an MPI job? Or are you saying things now work?
>
>
>
>On Dec 28, 2014, at 5:58 PM, Saliya Ekanayake  wrote:
>
>
>Thank you Ralph. This produced the warning on memory limits similar to [1] and 
>setting ulimit -l unlimited worked.
>
>
>[1] http://lists.openfabrics.org/pipermail/general/2007-June/036941.html
>
>
>Saliya
>
>
>On Sun, Dec 28, 2014 at 5:57 PM, Ralph Castain  wrote:
>
>Have the admin try running the ibv_ud_pingpong test - that will exercise the 
>portion of the system under discussion.
>
>
>
>On Dec 28, 2014, at 2:31 PM, Saliya Ekanayake  wrote:
>
>
>What I heard from the administrator is that, 
>
>"The tests that work are the simple utilities ib_read_lat and ib_read_bw
>that measures latency and bandwith between two nodes. They are part of
>the "perftest" repo package."
>
>On Dec 28, 2014 10:20 AM, "Saliya Ekanayake"  wrote:
>
>This happens at MPI_Init. I've attached the full error message.
>
>
>The sys admin mentioned Infiniband utility tests ran OK. I'll contact him for 
>more details and let you know.
>
>
>Thank you,
>Saliya
>
>
>On Sun, Dec 28, 2014 at 3:18 AM, Gilles Gouaillardet 
> wrote:
>
>Where does the error occurs ?
>MPI_Init ?
>MPI_Finalize ?
>In between ?
>
>In the first case, the bug is likely a mishandled error case,
>which means OpenMPI is unlikely the root cause of the crash.
>
>Did you check infniband is up and running on your cluster ?
>
>Cheers,
>
>Gilles 
>
>Saliya Ekanayake さんのメール:
>
>It's been a while on this, but we are still having trouble getting OpenMPI to 
>work with Infiniband on this cluster. We tried with latest 1.8.4 as well, but 
>it's still the same.
>
>
>To recap, we get the following error when MPI initializes (in the simple Hello 
>world C example) with Infiniband. Everything works fine if we explicitly turn 
>off openib with --mca btl ^openib
>
>
>This is the error I got after debugging with gdb as you suggested.
>
>
>hello_c: connect/btl_openib_connect_udcm.c:736: udcm_module_finalize: 
>Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == ((opal_object_t *) 
>(&m->cm_recv_msg_queue))->obj_magic_id' failed.
>
>
>Thank you,
>
>Saliya
>
>
>On Mon, Nov 10, 2014 at 10:01 AM, Saliya Ekanayake  wrote:
>
>Thank you Jeff, I'll try this and  let you know. 
>
>Saliya 
>
>On Nov 10, 2014 6:42 AM, "Jeff Squyres (jsquyres)"  wrote:
>
>I am sorry for the delay; I've been caught up in SC deadlines.  :-(
>
>I don't see anything blatantly wrong in this output.
>
>Two things:
>
>1. Can you try a nightly v1.8.4 snapshot tarball?  This will check to see if 
>whatever the bug is has been fixed for the upcoming release:
>
>    http://www.open-mpi.org/nightly/v1.8/
>
>2. Build Open MPI with the --enable-debug option (note that this adds a 
>slight-but-noticeable performance penalty).  When you run, it should dump a 
>core file.  Load that core file in a debugger and see where it is failing 
>(i.e., file and line in the OMPI source).
>
>We don't usually have to resort to asking users to perform #2, but there's no 
>additional information to give a clue as to what is happening.  :-(
>
>
>
>On Nov 9, 2014, at 11:43 AM, Saliya Ekanayake  wrote:
>
>> Hi Jeff,
>>
>> You are probably busy, but just checking if you had a chance to look at this.
>>
>> Thanks,
>> Saliya
>>
>> On Thu, Nov 6, 2014 at 9:19 AM, Saliya Ekanayake  wrote:
>> Hi Jeff,
>>
>> I've attached a tar file with information.
>>
>> Thank you,
>> Saliya
>>
>> On Tue, Nov 4, 2014 at 4:18 PM, Jeff Squyres (jsquyres)  
>> wrote:
>> Looks like it's failing in the openib BTL setup.
>>
>> Can you send the info listed here?
>>
>>     http://www.open-mpi.org/community/help/
>>
>>
>>
>> On Nov 4

Re: [OMPI users] OMPI users] Icreasing OFED registerable memory

[Gilles Gouaillardet]

FWIW ompi does not yet support XRC with OFED 3.12.

Cheers,

Gilles

Deva さんのメール:
>Hi Waleed,
>
>
>It is highly recommended to upgrade to latest OFED.  Meanwhile, Can you try 
>latest OMPI release (v1.8.4), where this warning is ignored on older OFEDs
>
>
>-Devendar 
>
>
>On Sun, Dec 28, 2014 at 6:03 AM, Waleed Lotfy  wrote:
>
>I have a bunch of 8 GB memory nodes in a cluster who were lately
>upgraded to 16 GB. When I run any jobs I get the following warning:
>--
>WARNING: It appears that your OpenFabrics subsystem is configured to
>only
>allow registering part of your physical memory.  This can cause MPI jobs
>to
>run with erratic performance, hang, and/or crash.
>
>This may be caused by your OpenFabrics vendor limiting the amount of
>physical memory that can be registered.  You should investigate the
>relevant Linux kernel module parameters that control how much physical
>memory can be registered, and increase them to allow registering all
>physical memory on your machine.
>
>See this Open MPI FAQ item for more information on these Linux kernel
>module
>parameters:
>
>    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
>
>  Local host:              comp022.local
>  Registerable memory:     8192 MiB
>  Total memory:            16036 MiB
>
>Your MPI job will continue, but may be behave poorly and/or hang.
>--
>
>Searching for a fix to this issue, I found that I have to set
>log_num_mtt within the kernel module, so I added this line to
>modprobe.conf:
>
>options mlx4_core log_num_mtt=21
>
>But then ib0 interface fails to start showing this error:
>ib_ipoib device ib0 does not seem to be present, delaying
>initialization.
>
>Reducing the value of log_num_mtt to 20, allows ib0 to start but shows
>the registerable memory of 8 GB warning.
>
>I am using OFED 1.3.1, I know it is pretty old and we are planning to
>upgrade soon.
>
>Output on all nodes for 'ompi_info  -v ompi full --parsable':
>
>ompi:version:full:1.2.7
>ompi:version:svn:r19401
>orte:version:full:1.2.7
>orte:version:svn:r19401
>opal:version:full:1.2.7
>opal:version:svn:r19401
>
>Any help would be appreciated.
>
>Waleed Lotfy
>Bibliotheca Alexandrina
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2014/12/26076.php
>
>
>
>
>-- 
>
>
>
>-Devendar
>

Re: [OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Diego,

First, i recommend you redefine tParticle and add a padding integer so 
everything is aligned.


Before invoking MPI_Type_create_struct, you need to 
call MPI_Get_address(dummy, base, MPI%err)
displacements = displacements - base

MPI_Type_create_resized might be unnecessary if tParticle is aligned 
And the lower bound should be zero.

BTW, which compiler are you using ?
Is tParticle object a common ?
iirc, intel compiler aligns types automatically, but not commons, and that 
means MPI_Type_create_struct is not aligned as it should most of the time.

Cheers,

Gilles 

Diego Avesani さんのメール:
>dear all,
>
>
>I have a problem with MPI_Type_Create_Struct and MPI_TYPE_CREATE_RESIZED.
>
>
>I have this variable type:
>
>
>  TYPE tParticle
>
>     INTEGER  :: ip
>
>     REAL     :: RP(2)
>
>     REAL     :: QQ(2)
>
>  ENDTYPE tParticle
>
>
>Then I define:
>
>
>Nstruct=3
>
>ALLOCATE(TYPES(Nstruct))
>
>ALLOCATE(LENGTHS(Nstruct))
>
>ALLOCATE(DISPLACEMENTS(Nstruct))
>
>!set the types
>
>TYPES(1) = MPI_INTEGER
>
>TYPES(2) = MPI_DOUBLE_PRECISION
>
>TYPES(3) = MPI_DOUBLE_PRECISION
>
>!set the lengths
>
>LENGTHS(1) = 1
>
>LENGTHS(2) = 2
>
>LENGTHS(3) = 2
>
>
>As gently suggested by Nick Papior Andersen and George Bosilca some months 
>ago, I checked the variable adress to resize my struct variable to avoid empty 
>space and
>
>to have a more general definition.
>
>
> !
>
> CALL MPI_GET_ADDRESS(dummy%ip,    DISPLACEMENTS(1), MPI%iErr)
>
> CALL MPI_GET_ADDRESS(dummy%RP(1), DISPLACEMENTS(2), MPI%iErr)
>
> CALL MPI_GET_ADDRESS(dummy%QQ(1), DISPLACEMENTS(3), MPI%iErr)
>
> !
>
> CALL 
>MPI_Type_Create_Struct(Nstruct,LENGTHS,DISPLACEMENTS,TYPES,MPI_PARTICLE_TYPE_OLD,MPI%iErr)
>
> CALL MPI_Type_Commit(MPI_PARTICLE_TYPE_OLD,MPI%iErr)
>
> !
>
> CALL MPI_TYPE_CREATE_RESIZED(MPI_PARTICLE_TYPE_OLD, 
>DISPLACEMENTS(1),DISPLACEMENTS(2) - DISPLACEMENTS(1), MPI_PARTICLE_TYPE)
>
>
>
>This does not work. When my program run, I get an error:
>
>
>forrtl: severe (174): SIGSEGV, segmentation fault occurred.
>
>
>I have read the manual but probably I am not able to understand  
>MPI_TYPE_CREATE_RESIZED. 
>
>
>Someone could help me?
>
>                                                                               
>                                                              
>
>Thanks a lot
>
>Diego
>
>
>
>Diego
>

Re: [OMPI users] OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Diego,

George gave you the solution,

The snippet you posted has two mistakes
You did not remove mpi_get_address(dummy) from all displacements
(See my previous reply)
You pass incorrect values to mpi_type_create_resized

Can you post a trimmed version of your program instead of a snippet ?

Gus is right about using double precision vs real and -r8

Cheers,

Gilles

Diego Avesani さんのメール:
>Dear Gilles Dear all,
>
>
>I have done all that to avoid to pedding an integer, as suggested by George.
>
>I define tParticle as a common object. 
>
>I am using Intel fortran compiler. 
>
>
>George suggests:
>
>
>"" The displacements are relative to the benign of your particle type. Thus 
>the first one is not 0 but the displacement of “integer :: ip” due to the fact 
>that the compiler is allowed to introduce gaps in order to better align.
>
>  DISPLACEMENTS(1)=MPI_GET_ADDRESS(dummy%ip)
>
>  DISPLACEMENTS(2)=MPI_GET_ADDRESS(dummy%RP[1])
>
>  DISPLACEMENTS(3)=MPI_GET_ADDRESS(dummy%QQ[1])
>
>and then remove the MPI_GET_ADDRESS(dummy) from all of them.
>
>
>3. After creating the structure type you need to resize it in order to 
>correctly determine the span of the entire structure, and how an array of such 
>structures lays in memory. Something like:
>
>MPI_TYPE_CREATE_RESIZED(old type, DISPLACEMENT(1),
>
>   MPI_GET_ADDRESS(dummy[2]) - MPI_GET_ADDRESS(dummy[1]), newt) ""
>
>
>What do you think?
>
>George, Did i miss something?
>
>
>Thanks a lot
>
>
>
>
>Diego
>
>
>On 2 January 2015 at 12:51, Gilles Gouaillardet 
> wrote:
>
>Diego,
>
>First, i recommend you redefine tParticle and add a padding integer so 
>everything is aligned.
>
>
>Before invoking MPI_Type_create_struct, you need to 
>call MPI_Get_address(dummy, base, MPI%err)
>displacements = displacements - base
>
>MPI_Type_create_resized might be unnecessary if tParticle is aligned 
>And the lower bound should be zero.
>
>BTW, which compiler are you using ?
>Is tParticle object a common ?
>iirc, intel compiler aligns types automatically, but not commons, and that 
>means MPI_Type_create_struct is not aligned as it should most of the time.
>
>Cheers,
>
>Gilles 
>
>Diego Avesani さんのメール:
>
>
>dear all,
>
>
>I have a problem with MPI_Type_Create_Struct and MPI_TYPE_CREATE_RESIZED.
>
>
>I have this variable type:
>
>
>  TYPE tParticle
>
>     INTEGER  :: ip
>
>     REAL     :: RP(2)
>
>     REAL     :: QQ(2)
>
>  ENDTYPE tParticle
>
>
>Then I define:
>
>
>Nstruct=3
>
>ALLOCATE(TYPES(Nstruct))
>
>ALLOCATE(LENGTHS(Nstruct))
>
>ALLOCATE(DISPLACEMENTS(Nstruct))
>
>!set the types
>
>TYPES(1) = MPI_INTEGER
>
>TYPES(2) = MPI_DOUBLE_PRECISION
>
>TYPES(3) = MPI_DOUBLE_PRECISION
>
>!set the lengths
>
>LENGTHS(1) = 1
>
>LENGTHS(2) = 2
>
>LENGTHS(3) = 2
>
>
>As gently suggested by Nick Papior Andersen and George Bosilca some months 
>ago, I checked the variable adress to resize my struct variable to avoid empty 
>space and
>
>to have a more general definition.
>
>
> !
>
> CALL MPI_GET_ADDRESS(dummy%ip,    DISPLACEMENTS(1), MPI%iErr)
>
> CALL MPI_GET_ADDRESS(dummy%RP(1), DISPLACEMENTS(2), MPI%iErr)
>
> CALL MPI_GET_ADDRESS(dummy%QQ(1), DISPLACEMENTS(3), MPI%iErr)
>
> !
>
> CALL 
>MPI_Type_Create_Struct(Nstruct,LENGTHS,DISPLACEMENTS,TYPES,MPI_PARTICLE_TYPE_OLD,MPI%iErr)
>
> CALL MPI_Type_Commit(MPI_PARTICLE_TYPE_OLD,MPI%iErr)
>
> !
>
> CALL MPI_TYPE_CREATE_RESIZED(MPI_PARTICLE_TYPE_OLD, 
>DISPLACEMENTS(1),DISPLACEMENTS(2) - DISPLACEMENTS(1), MPI_PARTICLE_TYPE)
>
>
>
>This does not work. When my program run, I get an error:
>
>
>forrtl: severe (174): SIGSEGV, segmentation fault occurred.
>
>
>I have read the manual but probably I am not able to understand  
>MPI_TYPE_CREATE_RESIZED. 
>
>
>Someone could help me?
>
>                                                                               
>                                                              
>
>Thanks a lot
>
>Diego
>
>
>
>Diego
>
>
>___
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2015/01/26092.php
>
>

Re: [OMPI users] OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Diego,

here is an updated revision i will double check tomorrow
/* i dit not test it yet, so forgive me it it does not compile/work */

Cheers,

Gilles

On Sun, Jan 4, 2015 at 6:48 PM, Diego Avesani 
wrote:

> Dear Gilles, Dear all,
>
> in the attachment you can find the program.
>
> What do you meam "remove mpi_get_address(dummy) from all displacements".
>
> Thanks for all your help
>
> Diego
>
>
>
> Diego
>
>
> On 3 January 2015 at 00:45, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
>> Diego,
>>
>> George gave you the solution,
>>
>> The snippet you posted has two mistakes
>> You did not remove mpi_get_address(dummy) from all displacements
>> (See my previous reply)
>> You pass incorrect values to mpi_type_create_resized
>>
>> Can you post a trimmed version of your program instead of a snippet ?
>>
>> Gus is right about using double precision vs real and -r8
>>
>> Cheers,
>>
>> Gilles
>>
>> Diego Avesani さんのメール:
>> Dear Gilles Dear all,
>>
>> I have done all that to avoid to pedding an integer, as suggested by
>> George.
>> I define tParticle as a common object.
>> I am using Intel fortran compiler.
>>
>> George suggests:
>>
>> *"" The displacements are relative to the benign of your particle type.
>> Thus the first one is not 0 but the displacement of “integer :: ip” due to
>> the fact that the compiler is allowed to introduce gaps in order to better
>> align.*
>>
>> *  DISPLACEMENTS(1)=MPI_GET_ADDRESS(dummy%ip)*
>> *  DISPLACEMENTS(2)=**MPI_GET_ADDRESS(dummy%RP[1])*
>>
>> *  DISPLACEMENTS(3)=**MPI_GET_ADDRESS(dummy%QQ[1])*
>>
>> *and then remove the MPI_GET_ADDRESS(dummy) from all of them.*
>>
>> *3. After creating the structure type you need to resize it in order to
>> correctly determine the span of the entire structure, and how an array of
>> such structures lays in memory. Something like:*
>> *MPI_TYPE_CREATE_RESIZED(old type, DISPLACEMENT(1),*
>> *   MPI_GET_ADDRESS(dummy[2]) - MPI_GET_ADDRESS(dummy[1]), newt) ""*
>>
>> What do you think?
>> George, Did i miss something?
>>
>> Thanks a lot
>>
>>
>>
>> Diego
>>
>>
>> On 2 January 2015 at 12:51, Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com> wrote:
>>
>>> Diego,
>>>
>>> First, i recommend you redefine tParticle and add a padding integer so
>>> everything is aligned.
>>>
>>>
>>> Before invoking MPI_Type_create_struct, you need to
>>> call MPI_Get_address(dummy, base, MPI%err)
>>> displacements = displacements - base
>>>
>>> MPI_Type_create_resized might be unnecessary if tParticle is aligned
>>> And the lower bound should be zero.
>>>
>>> BTW, which compiler are you using ?
>>> Is tParticle object a common ?
>>> iirc, intel compiler aligns types automatically, but not commons, and
>>> that means MPI_Type_create_struct is not aligned as it should most of the
>>> time.
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> Diego Avesani さんのメール:
>>>
>>> dear all,
>>>
>>> I have a problem with MPI_Type_Create_Struct and MPI_TYPE_CREATE_RESIZED.
>>>
>>> I have this variable type:
>>>
>>> *  TYPE tParticle*
>>> * INTEGER  :: ip*
>>> * REAL :: RP(2)*
>>> * REAL :: QQ(2)*
>>> *  ENDTYPE tParticle*
>>>
>>> Then I define:
>>>
>>> Nstruct=3
>>> *ALLOCATE(TYPES(Nstruct))*
>>> *ALLOCATE(LENGTHS(Nstruct))*
>>> *ALLOCATE(DISPLACEMENTS(Nstruct))*
>>> *!set the types*
>>> *TYPES(1) = MPI_INTEGER*
>>> *TYPES(2) = MPI_DOUBLE_PRECISION*
>>> *TYPES(3) = MPI_DOUBLE_PRECISION*
>>> *!set the lengths*
>>> *LENGTHS(1) = 1*
>>> *LENGTHS(2) = 2*
>>> *LENGTHS(3) = 2*
>>>
>>> As gently suggested by Nick Papior Andersen and George Bosilca some
>>> months ago, I checked the variable adress to resize my struct variable to
>>> avoid empty space and
>>> to have a more general definition.
>>>
>>> * !*
>>> * CALL MPI_GET_ADDRESS(dummy%ip,DISPLACEMENTS(1), MPI%iErr)*
>>> * CALL MPI_GET_ADDRESS(dummy%RP(1), DISPLACEMENTS(2), MPI%iErr)*
>>> * CALL MPI_GET_ADDRESS(dummy%QQ(1), DISPLACEMENTS(3), MPI%iErr)*
>>> * !*
>>> * CALL
>>> MPI_Type_Create_Struct(

Re: [OMPI users] OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Diego,

MPI_Get_address was invoked with parameters in the wrong order

here is attached a fixed version

Cheers,

Gilles

On 2015/01/05 2:32, Diego Avesani wrote:
> Dear Gilles, Dear all,
>
> It works. The only thing that is missed is:
>
> *CALL MPI_Finalize(MPI%iErr)*
>
> at the end of the program.
>
> Now, I have to test it sending some data from a processor to another.
> I would like to ask you if you could explain me what you have done.
> I wrote in the program:
>
> *   IF(MPI%myrank==1)THEN*
> *  WRITE(*,*) DISPLACEMENTS*
> *   ENDIF*
>
> and the results is:
>
>*139835891001320  -139835852218120  -139835852213832*
> *  -139835852195016   8030673735967299609*
>
> I am not able to understand it.
>
> Thanks a lot.
>
> In the attachment you can find the program
>
>
>
>
>
>
>
>
> Diego
>
>
> On 4 January 2015 at 12:10, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
>> Diego,
>>
>> here is an updated revision i will double check tomorrow
>> /* i dit not test it yet, so forgive me it it does not compile/work */
>>
>> Cheers,
>>
>> Gilles
>>
>> On Sun, Jan 4, 2015 at 6:48 PM, Diego Avesani 
>> wrote:
>>
>>> Dear Gilles, Dear all,
>>>
>>> in the attachment you can find the program.
>>>
>>> What do you meam "remove mpi_get_address(dummy) from all displacements".
>>>
>>> Thanks for all your help
>>>
>>> Diego
>>>
>>>
>>>
>>> Diego
>>>
>>>
>>> On 3 January 2015 at 00:45, Gilles Gouaillardet <
>>> gilles.gouaillar...@gmail.com> wrote:
>>>
>>>> Diego,
>>>>
>>>> George gave you the solution,
>>>>
>>>> The snippet you posted has two mistakes
>>>> You did not remove mpi_get_address(dummy) from all displacements
>>>> (See my previous reply)
>>>> You pass incorrect values to mpi_type_create_resized
>>>>
>>>> Can you post a trimmed version of your program instead of a snippet ?
>>>>
>>>> Gus is right about using double precision vs real and -r8
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>> Diego Avesani ??:
>>>> Dear Gilles Dear all,
>>>>
>>>> I have done all that to avoid to pedding an integer, as suggested by
>>>> George.
>>>> I define tParticle as a common object.
>>>> I am using Intel fortran compiler.
>>>>
>>>> George suggests:
>>>>
>>>> *"" The displacements are relative to the benign of your particle type.
>>>> Thus the first one is not 0 but the displacement of "integer :: ip" due to
>>>> the fact that the compiler is allowed to introduce gaps in order to better
>>>> align.*
>>>>
>>>> *  DISPLACEMENTS(1)=MPI_GET_ADDRESS(dummy%ip)*
>>>> *  DISPLACEMENTS(2)=**MPI_GET_ADDRESS(dummy%RP[1])*
>>>>
>>>> *  DISPLACEMENTS(3)=**MPI_GET_ADDRESS(dummy%QQ[1])*
>>>>
>>>> *and then remove the MPI_GET_ADDRESS(dummy) from all of them.*
>>>>
>>>> *3. After creating the structure type you need to resize it in order to
>>>> correctly determine the span of the entire structure, and how an array of
>>>> such structures lays in memory. Something like:*
>>>> *MPI_TYPE_CREATE_RESIZED(old type, DISPLACEMENT(1),*
>>>> *   MPI_GET_ADDRESS(dummy[2]) - MPI_GET_ADDRESS(dummy[1]), newt) ""*
>>>>
>>>> What do you think?
>>>> George, Did i miss something?
>>>>
>>>> Thanks a lot
>>>>
>>>>
>>>>
>>>> Diego
>>>>
>>>>
>>>> On 2 January 2015 at 12:51, Gilles Gouaillardet <
>>>> gilles.gouaillar...@gmail.com> wrote:
>>>>
>>>>> Diego,
>>>>>
>>>>> First, i recommend you redefine tParticle and add a padding integer so
>>>>> everything is aligned.
>>>>>
>>>>>
>>>>> Before invoking MPI_Type_create_struct, you need to
>>>>> call MPI_Get_address(dummy, base, MPI%err)
>>>>> displacements = displacements - base
>>>>>
>>>>> MPI_Type_create_resized might be unnecessary if tParticle is aligned
>>>>> And the lower bound should be zero.
>>&

Re: [OMPI users] OMPI users] OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Diego,

The compiler likely added some padding after %ip to have data aligned on 128 
bits.

You need two dummies in case the compiler adds some padding at the end of the 
type.

Cheers,

Gilles

Diego Avesani さんのメール:
>Dear Gilles, Dear all,
>
>thanks, thanks a lot.
>
>
>Could you explain it to me, please? 
>
>
>I mean, when I print displacements I get:
>
>
>displacements(0)= 6922656
>
>displacements(1)= 0             
>
>displacements(2)= 16
>
>displacements(3)= 48
>
>displacements(4)= 112
>
> 
>
>Why do I have 16 spaces in displacements(2), I have only an integer in 
>dummy%ip?
>
>Why do you use dummy(1) and dummy(2)?
>
>
>Thanks a lot    
>
>
>
>Diego
>
>
>On 5 January 2015 at 02:44, Gilles Gouaillardet 
> wrote:
>
>Diego,
>
>MPI_Get_address was invoked with parameters in the wrong order
>
>here is attached a fixed version
>
>Cheers,
>
>Gilles
>
>On 2015/01/05 2:32, Diego Avesani wrote:
>
>Dear Gilles, Dear all, It works. The only thing that is missed is: *CALL 
>MPI_Finalize(MPI%iErr)* at the end of the program. Now, I have to test it 
>sending some data from a processor to another. I would like to ask you if you 
>could explain me what you have done. I wrote in the program: * 
>IF(MPI%myrank==1)THEN* * WRITE(*,*) DISPLACEMENTS* * ENDIF* and the results 
>is: *139835891001320 -139835852218120 -139835852213832* * -139835852195016 
>8030673735967299609* I am not able to understand it. Thanks a lot. In the 
>attachment you can find the program Diego On 4 January 2015 at 12:10, Gilles 
>Gouaillardet < gilles.gouaillar...@gmail.com> wrote: 
>
>Diego, here is an updated revision i will double check tomorrow /* i dit not 
>test it yet, so forgive me it it does not compile/work */ Cheers, Gilles On 
>Sun, Jan 4, 2015 at 6:48 PM, Diego Avesani  wrote: 
>
>Dear Gilles, Dear all, in the attachment you can find the program. What do you 
>meam "remove mpi_get_address(dummy) from all displacements". Thanks for all 
>your help Diego Diego On 3 January 2015 at 00:45, Gilles Gouaillardet < 
>gilles.gouaillar...@gmail.com> wrote: 
>
>Diego, George gave you the solution, The snippet you posted has two mistakes 
>You did not remove mpi_get_address(dummy) from all displacements (See my 
>previous reply) You pass incorrect values to mpi_type_create_resized Can you 
>post a trimmed version of your program instead of a snippet ? Gus is right 
>about using double precision vs real and -r8 Cheers, Gilles Diego Avesani 
>さんのメー
>
>ル: Dear Gilles Dear all, I have done all that to avoid to pedding an integer, 
>as suggested by George. I define tParticle as a common object. I am using 
>Intel fortran compiler. George suggests: *"" The displacements are relative to 
>the benign of your particle type. Thus the first one is not 0 but the 
>displacement of “integer :: ip” due to the fact that the compiler is allowed 
>to introduce gaps in order to better align.* * 
>DISPLACEMENTS(1)=MPI_GET_ADDRESS(dummy%ip)* * 
>DISPLACEMENTS(2)=**MPI_GET_ADDRESS(dummy%RP[1])* * 
>DISPLACEMENTS(3)=**MPI_GET_ADDRESS(dummy%QQ[1])* *and then remove the 
>MPI_GET_ADDRESS(dummy) from all of them.* *3. After creating the structure 
>type you need to resize it in order to correctly determine the span of the 
>entire structure, and how an array of such structures lays in memory. 
>Something like:* *MPI_TYPE_CREATE_RESIZED(old type, DISPLACEMENT(1),* * 
>MPI_GET_ADDRESS(dummy[2]) - MPI_GET_ADDRESS(dummy[1]), newt) ""* What do you 
>think? George, Did i miss something? Thanks a lot Diego On 2 January 2015 at 
>12:51, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: 
>
>Diego, First, i recommend you redefine tParticle and add a padding integer so 
>everything is aligned. Before invoking MPI_Type_create_struct, you need to 
>call MPI_Get_address(dummy, base, MPI%err) displacements = displacements - 
>base MPI_Type_create_resized might be unnecessary if tParticle is aligned And 
>the lower bound should be zero. BTW, which compiler are you using ? Is 
>tParticle object a common ? iirc, intel compiler aligns types automatically, 
>but not commons, and that means MPI_Type_create_struct is not aligned as it 
>should most of the time. Cheers, Gilles Diego Avesani 
>さんのメー
>
>ル: dear all, I have a problem with MPI_Type_Create_Struct and 
>MPI_TYPE_CREATE_RESIZED. I have this variable type: * TYPE tParticle* * 
>INTEGER :: ip* * REAL :: RP(2)* * REAL :: QQ(2)* * ENDTYPE tParticle* Then I 
>define: Nstruct=3 *ALLOCATE(TYPES(Nstruct))* *ALLOCATE(LENGTHS(Nstruct))* 
>*ALLOCATE(DISPLACEMENTS(Nstruct))* *!set the types* *TYPES(1) = MPI_INTEGER* 
>*TYPES(2) = MPI_DOUBLE_PRECISION* *TYPES(3

Re: [OMPI users] OMPI users] OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Diego,

my bad, i should have passed displacements(1) to MPI_Type_create_struct

here is an updated version

(note you have to use a REQUEST integer for MPI_Isend and MPI_Irecv,
and you also have to call MPI_Wait to ensure the requests complete)

Cheers,

Gilles

On 2015/01/08 8:23, Diego Avesani wrote:
> Dear Gilles, Dear all,
>
> I'm sorry to bother you again, but I have some problem with send and
> receive the struct_data.
>
> I tried to send a MPI_Type_Create_Struct but I get a segmentation fault
> occurred and I do not know why. The program is very simple, it is the old
> one with the isend and irecv subroutines
>
> (you can find it in the attachment)
>
> Thanks again
>
>
> Diego
>
>
> On 5 January 2015 at 15:54, Diego Avesani  wrote:
>
>> Dear Gilles,
>>
>> Thanks, Thanks a lot.
>> Now is more clear.
>>
>> Again, thanks a lot
>>
>> Diego
>>
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26116.php

MODULE MOD_PRECISION
integer, parameter :: dp = selected_real_kind(p=16)
ENDMODULE

PROGRAM PROVA_STRUCT
USE MOD_PRECISION
IMPLICIT NONE
INCLUDE 'mpif.h'
!
TYPE tMPI
INTEGER  :: myrank, nCPU, iErr, status
END TYPE tMPI
!
type particle
 sequence
 integer  :: ip
 real(dp) :: rp(2)
 real(dp) :: QQ(4)
end type particle
!
TYPE(tMPI) :: MPI
INTEGER:: COMM_CART
INTEGER:: MPI_PARTICLE_TYPE_OLD
INTEGER:: MPI_PARTICLE_TYPE

INTEGER  :: nstruct

INTEGER,ALLOCATABLE  :: TYPES(:)
INTEGER,ALLOCATABLE  :: LENGTHS(:)
INTEGER  :: REQUEST(2)
INTEGER(MPI_ADDRESS_KIND),ALLOCATABLE,DIMENSION(:)   ::DISPLACEMENTS

type(particle) :: dummy(2)  ! Used for calculation of displacement



   CALL MPI_INIT(MPI%iErr)
   CALL MPI_COMM_RANK(MPI_COMM_WORLD, MPI%myrank, MPI%iErr)
   CALL MPI_COMM_SIZE(MPI_COMM_WORLD, MPI%nCPU,   MPI%iErr)
   !
   !
   nstruct=3
   ALLOCATE(TYPES(nstruct))
   ALLOCATE(LENGTHS(nstruct))
   ALLOCATE(DISPLACEMENTS(0:nstruct+1))
   !
   TYPES(1)=MPI_INTEGER
   TYPES(2)=MPI_DOUBLE_PRECISION
   TYPES(3)=MPI_DOUBLE_PRECISION
   !
   LENGTHS(1)=1
   LENGTHS(2)=2
   LENGTHS(3)=4
   ! 
   !
   CALL MPI_GET_ADDRESS(dummy(1),DISPLACEMENTS(0),MPI%iErr)
   CALL MPI_GET_ADDRESS(dummy(1)%ip,DISPLACEMENTS(1),MPI%iErr)
   CALL MPI_GET_ADDRESS(dummy(1)%RP(1),DISPLACEMENTS(2),MPI%iErr)
   CALL MPI_GET_ADDRESS(dummy(1)%QQ(1),DISPLACEMENTS(3),MPI%iErr)
   CALL MPI_GET_ADDRESS(dummy(2),DISPLACEMENTS(4),MPI%iErr)
   !
   DISPLACEMENTS(1:nstruct+1)= DISPLACEMENTS(1:nstruct+1)-DISPLACEMENTS(0)
   !
   CALL 
MPI_TYPE_CREATE_STRUCT(nstruct,lengths,displacements(1),types,MPI_PARTICLE_TYPE_OLD,MPI%iErr)
   CALL MPI_TYPE_COMMIT(MPI_PARTICLE_TYPE_OLD,MPI%iErr)
   !
   !
   CALL MPI_TYPE_CREATE_RESIZED(MPI_PARTICLE_TYPE_OLD, DISPLACEMENTS(1), 
DISPLACEMENTS(4), MPI_PARTICLE_TYPE, MPI%iErr)
   CALL MPI_TYPE_COMMIT(MPI_PARTICLE_TYPE,MPI%iErr)
   !
   IF(MPI%myrank==0)THEN
  WRITE(*,*) DISPLACEMENTS
   ENDIF
   !
   IF(MPI%myrank==0)THEN
  CALL 
MPI_ISEND(DUMMY(1),1,MPI_PARTICLE_TYPE,1,0,MPI_COMM_WORLD,REQUEST,MPI%iErr)
   ENDIF
   !
   IF(MPI%myrank==1)THEN
  CALL 
MPI_IRECV(DUMMY(1),1,MPI_PARTICLE_TYPE,0,0,MPI_COMM_WORLD,REQUEST,MPI%iErr)
   ENDIF

   CALL MPI_WAIT(REQUEST, MPI_STATUS_IGNORE)
   ! 
   CALL MPI_Finalize(MPI%iErr)
   !

ENDPROGRAM

Re: [OMPI users] difference of behaviour for MPI_Publish_name between openmpi-1.4.5 and openmpi-1.8.4

[Gilles Gouaillardet]

Well, per the source code, this is not a bug but a feature :


from publish function from ompi/mca/pubsub/orte/pubsub_orte.c

ompi_info_get_bool(info, "ompi_unique", &unique, &flag);
if (0 == flag) {
/* uniqueness not specified - overwrite by default */
unique = false;
}

fwiw, and at first glance, i would have expected the default behaviour
is to *not* overwrite (e.g. unique = true;).

anyway, in order to get the expected result, the user program can be
modified like this :

MPI_Info info;
MPI_Info_create(&info);
MPI_Info_set(info, "ompi_unique", "true");

and then invoke MPI_Publish_name() with info instead of MPI_INFO_NULL

an updated version of the program

Cheers,

Gilles

On 2015/01/08 10:12, Ralph Castain wrote:
> Hmmm…I confess this API gets little, if any, testing as it is so seldom used, 
> so it is quite possible that a buglet has crept into it. I’ll take a look and 
> try to have something in 1.8.5.
>
> Thanks!
> Ralph
>
>> On Jan 7, 2015, at 3:14 AM, Bernard Secher  wrote:
>>
>> Hello,
>>
>> With the version openmpi-1.4.5 I got an error  when I tried to publish the 
>> same name twice with the MPI_Publish_name routine
>> With the version openmpi-1.8.4 I got no error when I published the same name 
>> twice with the MPI_Publish_name routine
>>
>> I used the attached script and source code to perform the test.
>>
>> With this test, it works well with openmpi-1.4.5, but I get a deadlock with 
>> openmpi-1.8.4. I can suppress the deadlock with openmpi-1.8.4 if I modify 
>> the shell script and add a "sleep 1" command between the 2 mpirun commands.
>>
>> Bernard
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26114.php
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26117.php

// Copyright (C) 2011-2014  CEA/DEN, EDF R&D, OPEN CASCADE
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; either
// version 2.1 of the License, or (at your option) any later version.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
//
// See http://www.salome-platform.org/ or email : 
webmaster.sal...@opencascade.com
//

#include 
#include 
#include 
#include 
#define TIMEOUT 20
#define EPSILON 0.0001

#ifndef WIN32
# include 
#endif

int main(int argc, char**argv)
{
  int *indg;
  double *vector, sum=0., norm, etalon;
  int rank, size, grank, gsize, rsize;
  int vsize=20, lvsize, rlvsize;
  int i, k1, k2, imin, imax, nb;
  int srv=0;
  MPI_Comm com, icom;
  MPI_Status status; 
  MPI_Info info;
  char   port_name [MPI_MAX_PORT_NAME]; 
  char   port_name_clt [MPI_MAX_PORT_NAME]; 
  std::string service = "SERVICE";
  bool debug=false;

#ifndef OPEN_MPI
  std::cout << "This test only works with openmpi implementation" << std::endl;
  exit(1);
#endif

  for(i=1;i

Re: [OMPI users] difference of behaviour for MPI_Publish_name between openmpi-1.4.5 and openmpi-1.8.4

[Gilles Gouaillardet]

Ralph,

honestly, i do not know if the standard says anything
(all i can do for now is blame the coin ;-) )

from now, i ll make a PR to update the man page (only master refers
ompi_unique)

Cheers,

Gilles

commit 7d2e3028d608163247975397a09f30dbe7bd192a
Author: Ralph Castain 
List-Post: users@lists.open-mpi.org
Date:   Wed Aug 14 04:24:17 2013 +

Add unique info_key to documentation

On 2015/01/08 11:51, Ralph Castain wrote:
> Does the standard say anything about the default behavior? IIRC, we set it 
> this way because (a) we had no direction, and (b) it seemed just as 
> reasonable as the alternative (I believe we flipped a coin)
>
>
>> On Jan 7, 2015, at 6:47 PM, Gilles Gouaillardet 
>>  wrote:
>>
>> Well, per the source code, this is not a bug but a feature :
>>
>>
>> from publish function from ompi/mca/pubsub/orte/pubsub_orte.c
>>
>>ompi_info_get_bool(info, "ompi_unique", &unique, &flag);
>>if (0 == flag) {
>>/* uniqueness not specified - overwrite by default */
>>unique = false;
>>}
>>
>> fwiw, and at first glance, i would have expected the default behaviour
>> is to *not* overwrite (e.g. unique = true;).
>>
>> anyway, in order to get the expected result, the user program can be
>> modified like this :
>>
>> MPI_Info info;
>> MPI_Info_create(&info);
>> MPI_Info_set(info, "ompi_unique", "true");
>>
>> and then invoke MPI_Publish_name() with info instead of MPI_INFO_NULL
>>
>> an updated version of the program
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2015/01/08 10:12, Ralph Castain wrote:
>>> Hmmm…I confess this API gets little, if any, testing as it is so seldom 
>>> used, so it is quite possible that a buglet has crept into it. I’ll take a 
>>> look and try to have something in 1.8.5.
>>>
>>> Thanks!
>>> Ralph
>>>
>>>> On Jan 7, 2015, at 3:14 AM, Bernard Secher  wrote:
>>>>
>>>> Hello,
>>>>
>>>> With the version openmpi-1.4.5 I got an error  when I tried to publish the 
>>>> same name twice with the MPI_Publish_name routine
>>>> With the version openmpi-1.8.4 I got no error when I published the same 
>>>> name twice with the MPI_Publish_name routine
>>>>
>>>> I used the attached script and source code to perform the test.
>>>>
>>>> With this test, it works well with openmpi-1.4.5, but I get a deadlock 
>>>> with openmpi-1.8.4. I can suppress the deadlock with openmpi-1.8.4 if I 
>>>> modify the shell script and add a "sleep 1" command between the 2 mpirun 
>>>> commands.
>>>>
>>>> Bernard
>>>> ___
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post: 
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26114.php
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2015/01/26117.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26119.php
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26120.php

Re: [OMPI users] OMPI users] OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Diego,

yes, it works for me (at least with the v1.8 head and gnu compilers)

Cheers,

Gilles

On 2015/01/08 17:51, Diego Avesani wrote:
> Dear Gilles,
> thanks again, however it does not work.
>
> the program says:  "SIGSEGV, segmentation fault occurred"
>
> Does the program run in your case?
>
> Thanks again
>
>
>
> Diego
>
>
> On 8 January 2015 at 03:02, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
>>  Diego,
>>
>> my bad, i should have passed displacements(1) to MPI_Type_create_struct
>>
>> here is an updated version
>>
>> (note you have to use a REQUEST integer for MPI_Isend and MPI_Irecv,
>> and you also have to call MPI_Wait to ensure the requests complete)
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2015/01/08 8:23, Diego Avesani wrote:
>>
>> Dear Gilles, Dear all,
>>
>> I'm sorry to bother you again, but I have some problem with send and
>> receive the struct_data.
>>
>> I tried to send a MPI_Type_Create_Struct but I get a segmentation fault
>> occurred and I do not know why. The program is very simple, it is the old
>> one with the isend and irecv subroutines
>>
>> (you can find it in the attachment)
>>
>> Thanks again
>>
>>
>> Diego
>>
>>
>> On 5 January 2015 at 15:54, Diego Avesani  
>>  wrote:
>>
>>
>>  Dear Gilles,
>>
>> Thanks, Thanks a lot.
>> Now is more clear.
>>
>> Again, thanks a lot
>>
>> Diego
>>
>>
>>
>>
>>
>> ___
>> users mailing listus...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26116.php
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2015/01/26118.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26124.php

Re: [OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

[Gilles Gouaillardet]

Jeff,

thanks for all the good catches.

MPI_Type_create_resized is not required in this example because
send/recv are called
with count=1.

Generally speaking, if count > 1, MPI_Type_create_resized is required
because
the compiler might add some padding at the end of the type.

Cheers,

Gilles

On 2015/01/08 20:08, Jeff Squyres (jsquyres) wrote:
> There were still some minor errors left over in the attached program.
>
> I strongly encourage you to use "use mpi" instead of "include 'mpif.h'" 
> because you will get compile time errors when you pass incorrect / forget to 
> pass parameters to MPI subroutines.  When I switched your program to "use 
> mpi", here's what the compiler showed me:
>
> 1. the name "MPI" is reserved
> 2. need to pass displacements(1:nstruct+1) to mpi_type_create_struct
> 3. need to pass request(1) to mpi_isend
> 4. need to pass request(1) to mpi_wait
> 5. need to pass ierr argument to mpi_wait
>
> 1-4 are technically not correct, but the program would likely (usually) 
> compile/run "correctly" anyway.  5 is probably what caused your segv.
>
> Attached is my copy of your program with fixes for the above-mentioned issues.
>
> BTW, I missed the beginning of this thread -- I assume that this is an 
> artificial use of mpi_type_create_resized for the purposes of a small 
> example.  The specific use of it in this program appears to be superfluous.
>
>
>
>
>
> On Jan 8, 2015, at 4:26 AM, Gilles Gouaillardet 
>  wrote:
>
>> Diego,
>>
>> yes, it works for me (at least with the v1.8 head and gnu compilers)
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2015/01/08 17:51, Diego Avesani wrote:
>>> Dear Gilles,
>>> thanks again, however it does not work.
>>>
>>> the program says:  "SIGSEGV, segmentation fault occurred"
>>>
>>> Does the program run in your case?
>>>
>>> Thanks again
>>>
>>>
>>>
>>> Diego
>>>
>>>
>>> On 8 January 2015 at 03:02, Gilles Gouaillardet <
>>>
>>> gilles.gouaillar...@iferc.org
>>>> wrote:
>>>
>>>>  Diego,
>>>>
>>>> my bad, i should have passed displacements(1) to MPI_Type_create_struct
>>>>
>>>> here is an updated version
>>>>
>>>> (note you have to use a REQUEST integer for MPI_Isend and MPI_Irecv,
>>>> and you also have to call MPI_Wait to ensure the requests complete)
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>>
>>>> On 2015/01/08 8:23, Diego Avesani wrote:
>>>>
>>>> Dear Gilles, Dear all,
>>>>
>>>> I'm sorry to bother you again, but I have some problem with send and
>>>> receive the struct_data.
>>>>
>>>> I tried to send a MPI_Type_Create_Struct but I get a segmentation fault
>>>> occurred and I do not know why. The program is very simple, it is the old
>>>> one with the isend and irecv subroutines
>>>>
>>>> (you can find it in the attachment)
>>>>
>>>> Thanks again
>>>>
>>>>
>>>> Diego
>>>>
>>>>
>>>> On 5 January 2015 at 15:54, Diego Avesani
>>>>  
>>>>  wrote:
>>>>
>>>>
>>>>  Dear Gilles,
>>>>
>>>> Thanks, Thanks a lot.
>>>> Now is more clear.
>>>>
>>>> Again, thanks a lot
>>>>
>>>> Diego
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ___
>>>> users mailing
>>>> listus...@open-mpi.org
>>>>
>>>> Subscription:
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26116.php
>>>>
>>>>
>>>>
>>>>
>>>> ___
>>>> users mailing list
>>>>
>>>> us...@open-mpi.org
>>>>
>>>> Subscription:
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> Link to this post:
>>>>
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26118.php
>>>>
>>>>
>>>>
>>>
>>> ___
>>> users mailing list
>>>
>>> us...@open-mpi.org
>>>
>>> Subscription:
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2015/01/26124.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26126.php
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26127.php

Re: [OMPI users] error building openmpi-dev-685-g881b1dc on Soalris 10

[Gilles Gouaillardet]

Hi Siegmar,

could you please try again with adding '-D_STDC_C99' to your CFLAGS ?

Thanks and regards,

Gilles

On 2015/01/12 20:54, Siegmar Gross wrote:
> Hi,
>
> today I tried to build openmpi-dev-685-g881b1dc on my machines
> (Solaris 10 Sparc, Solaris 10 x86_64, and openSUSE Linux 12.1
> x86_64) with gcc-4.9.2 and the new Solaris Studio 12.4 compilers.
> I succedded on Linux but failed on both Solaris systems for both
> compilers with the same error.
>
> ...
>   CC   adio/common/ad_prealloc.lo
>   CC   adio/common/ad_read.lo
> "/usr/include/sys/feature_tests.h", line 337: #error: "Compiler or options 
> invalid for pre-UNIX 03 
> X/Open applications and pre-2001 POSIX applications"
> cc: acomp failed for 
> ../../../../../../openmpi-dev-685-g881b1dc/ompi/mca/io/romio/romio/adio/common/ad_read.c
> make[4]: *** [adio/common/ad_read.lo] Error 1
>
>
>
> ...
>   CC   adio/common/ad_read.lo
> In file included from /usr/include/unistd.h:18:0,
>  from 
> ../../../../../../openmpi-dev-685-g881b1dc/ompi/mca/io/romio/romio/adio/common/ad_read.c:16:
> /export2/prog/SunOS_sparc/gcc-4.9.2/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/sys/feature_
> tests.h:346:2: error: #error "Compiler or options invalid for pre-UNIX 03 
> X/Open applications 
> and pre-2001 POSIX applications"
>  #error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
>   ^
> make[4]: *** [adio/common/ad_read.lo] Error 1
>
>
> I would be grateful if somebody can fix the problem. Thank you
> very much in advance
>
>
> Kind regards
>
> Siegmar
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26165.php

Re: [OMPI users] Problems compiling OpenMPI 1.8.4 with GCC 4.9.2

[Gilles Gouaillardet]

Ryan,

this issue has already been reported.

please refer to
http://www.open-mpi.org/community/lists/users/2015/01/26134.php for a
workaround

Cheers,

Gilles

On 2015/01/14 16:35, Novosielski, Ryan wrote:
> OpenMPI 1.8.4 does not appear to be buildable with GCC 4.9.2. The output, as 
> requested by the Getting Help page, is attached.
>
> I believe I tried GCC 4.9.0 too and it didn't work.
>
> I did successfully build it with Intel's compiler suite v15.0.1, so I do 
> appear to know what I'm doing.
>
> Thanks in advance for your help.
>
> --
>  *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
>  || \\UTGERS  |-*O*-
>  ||_// Biomedical | Ryan Novosielski - Senior Technologist
>  || \\ and Health | novos...@rutgers.edu - 973/972.0922 (2x0922)
>  ||  \\  Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark
>   `'
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26173.php

Re: [OMPI users] Segfault in mpi-java

[Gilles Gouaillardet]

Alexander,

i was able to reproduce this behaviour.

basically, bad things happen when the garbage collector is invoked ...
i was even able to reproduce some crashes (but that happen at random
stages) very early in the code
by manually inserting calls to the garbage collector (e.g. System.gc();)

Cheers,

Gilles

On 2015/01/19 9:03, Alexander Daryin wrote:
> Hi
>
> I am using Java MPI bindings and periodically get fatal erros. This is 
> illustrated by the following model Java program.
>
> import mpi.MPI;
> import mpi.MPIException;
> import mpi.Prequest;
> import mpi.Request;
> import mpi.Status;
>
> import java.nio.ByteBuffer;
> import java.util.Random;
>
> public class TestJavaMPI {
>
>private static final int NREQ = 16;
>private static final int BUFFSIZE = 0x2000;
>private static final int NSTEP = 10;
>
>public static void main(String... args) throws MPIException {
>MPI.Init(args);
>Random random = new Random();
>Prequest[] receiveRequests = new Prequest[NREQ];
>Request[] sendRequests = new Request[NREQ];
>ByteBuffer[] receiveBuffers = new ByteBuffer[NREQ];
>ByteBuffer[] sendBuffers = new ByteBuffer[NREQ];
>for(int i = 0; i < NREQ; i++) {
>receiveBuffers[i] = MPI.newByteBuffer(BUFFSIZE);
>sendBuffers[i] = MPI.newByteBuffer(BUFFSIZE);
>receiveRequests[i] = MPI.COMM_WORLD.recvInit(receiveBuffers[i], 
> BUFFSIZE, MPI.BYTE, MPI.ANY_SOURCE, MPI.ANY_TAG);
>receiveRequests[i].start();
>sendRequests[i] = MPI.COMM_WORLD.iSend(sendBuffers[i], 0, 
> MPI.BYTE, MPI.PROC_NULL, 0);
>}
>for(int step = 0; step < NSTEP; step++) {
>if( step % 128 == 0 ) System.out.println(step);
>int index;
>do {
>Status status = Request.testAnyStatus(receiveRequests);
>if( status != null )
>receiveRequests[status.getIndex()].start();
>index = Request.testAny(sendRequests);
>} while( index == MPI.UNDEFINED );
>sendRequests[index].free();
>sendRequests[index] = MPI.COMM_WORLD.iSend(sendBuffers[index], 
> BUFFSIZE, MPI.BYTE,
>random.nextInt(MPI.COMM_WORLD.getSize()), 0);
>}
>MPI.Finalize();
>}
> }
>
> On Linux, this produces a segfault after about a million steps. On OS X, 
> instead of segfault it prints the following error message
>
> java(64053,0x127e4d000) malloc: *** error for object 0x7f80eb828808: 
> incorrect checksum for freed object - object was probably modified after 
> being freed.
> *** set a breakpoint in malloc_error_break to debug
> [mbp:64053] *** Process received signal ***
> [mbp:64053] Signal: Abort trap: 6 (6)
> [mbp:64053] Signal code:  (0)
> [mbp:64053] [ 0] 0   libsystem_platform.dylib0x7fff86b5ff1a 
> _sigtramp + 26
> [mbp:64053] [ 1] 0   ??? 0x 
> 0x0 + 0
> [mbp:64053] [ 2] 0   libsystem_c.dylib   0x7fff80c7bb73 
> abort + 129
> [mbp:64053] [ 3] 0   libsystem_malloc.dylib  0x7fff8c26ce06 
> szone_error + 625
> [mbp:64053] [ 4] 0   libsystem_malloc.dylib  0x7fff8c2645c8 
> small_free_list_remove_ptr + 154
> [mbp:64053] [ 5] 0   libsystem_malloc.dylib  0x7fff8c2632bf 
> szone_free_definite_size + 1856
> [mbp:64053] [ 6] 0   libjvm.dylib0x00010e257d89 
> _ZN2os4freeEPvt + 63
> [mbp:64053] [ 7] 0   libjvm.dylib0x00010dea2b0a 
> _ZN9ChunkPool12free_all_butEm + 136
> [mbp:64053] [ 8] 0   libjvm.dylib0x00010e30ab33 
> _ZN12PeriodicTask14real_time_tickEi + 77
> [mbp:64053] [ 9] 0   libjvm.dylib0x00010e3372a3 
> _ZN13WatcherThread3runEv + 267
> [mbp:64053] [10] 0   libjvm.dylib0x00010e25d87e 
> _ZL10java_startP6Thread + 246
> [mbp:64053] [11] 0   libsystem_pthread.dylib 0x7fff8f1402fc 
> _pthread_body + 131
> [mbp:64053] [12] 0   libsystem_pthread.dylib 0x7fff8f140279 
> _pthread_body + 0
> [mbp:64053] [13] 0   libsystem_pthread.dylib 0x7fff8f13e4b1 
> thread_start + 13
> [mbp:64053] *** End of error message ***
>
> OpenMPI version is 1.8.4. Java version is 1.8.0_25-b17.
>
> Best regards,
> Alexander Daryin
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26215.php

Re: [OMPI users] using multiple IB connections between hosts

[Gilles Gouaillardet]

Dave,

the QDR Infiniband uses the openib btl (by default :
btl_openib_exclusivity=1024)
i assume the RoCE 10Gbps card is using the tcp btl (by default :
btl_tcp_exclusivity=100)

that means that by default, when both openib and tcp btl could be used,
the tcp btl is discarded.

could you give a try by settings the same exclusivity value on both btl ?
e.g.
OMPI_MCA_btl_tcp_exclusivity=1024 mpirun ...

assuming this is enough the get traffic on both interfaces, you migh
want *not* to use the eth0 interface
(e.g. OMPI_MCA_btl_tcp_if_exlude=eth0 ...)

you might also have to tweak the bandwidth parameters (i assume QDR
interface should get 4 times more
traffic than the 10Gbe interface)
by default :
btl_openib_bandwidth=4
btl_tcp_bandwidth=100
/* value is in Mbps, so the openib value should be 40960 (!), and in
your case, tcp bandwidth should be 10240 */
you might also want to try btl_*_bandwidth=0 (auto-detect value at run time)

i hope this helps,

Cheers,

Gilles
On 2015/01/29 9:45, Dave Turner wrote:
>  I ran some aggregate bandwidth tests between 2 hosts connected by
> both QDR InfiniBand and RoCE enabled 10 Gbps Mellanox cards.  The tests
> measured the aggregate performance for 16 cores on one host communicating
> with 16 on the second host.  I saw the same performance as with the QDR
> InfiniBand alone, so it appears that the addition of the 10 Gbps RoCE cards
> is
> not helping.
>
>  Should OpenMPI be using both in this case by default, or is there
> something
> I need to configure to allow for this?  I suppose this is the same question
> as
> how to make use of 2 identical IB connections on each node, or is the system
> simply ignoring the 10 Gbps cards because they are the slower option.
>
>  Any clarification on this would be helpful.  The only posts I've found
> are very
> old and discuss mostly channel bonding of 1 Gbps cards.
>
>  Dave Turner
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26243.php

Re: [OMPI users] cross-compiling openmpi-1.8.4 with static linking

[Gilles Gouaillardet]

Simona,

On 2015/02/08 20:45, simona bellavista wrote:
> I have two systems A (aka Host) and B (aka Target). On A a compiler suite
> is installed (intel 14.0.2), on B there is no compiler. I want to compile
> openmpi on A for running it on system B (in particular, I want to use
> mpirun and mpif90), so I want to have static linking to the intel
> libraries. First of all, I would like to know if it is possible to do this.
as Ralph already wrote, there is no need to cross compile.

you are building on a fedora box, and plan running on an ubuntu box,
all i can think is you might get some issues with system libraries versions.
also, since both systems have different paths, that might be a mess to build
(the simplest option is to run configure and make on system A and run
make install on system B)

do you really want to use mpif90 on system B ?
since system B has no compilers, that will not work.
-static-intel means the intel runtime is linked statically.
this is necessary only if system B do not have the intel runtime installed,
and you might want to configure with
--with-wrapper-{c,cxx,fc}flags=-static-intel

if you want to build ompi with static libs only, then you can configure with
--enable-static --disable-shared

about the missing libraries (liblsf and libbat), i can only assume
system A has LSF installed
but not system B. in this case, you can add the --without-lsf parameter
to your configure
line.

since system B is an ubuntu box, i think things would be much easier for
you if you built
an ubuntu virtual machine with the same os version, set accounts and
path the way they
are set on system B (you are root on your vm) , and build ompi on this VM.

Cheers,

Gilles

Re: [OMPI users] Open MPI collectives algorithm selection

[Gilles Gouaillardet]

Khalid,

i am not aware of such a mechanism.

/* there might be a way to use MPI_T_* mechanisms to force the algorithm,
and i will let other folks comment on that */

you definetly cannot directly invoke ompi_coll_tuned_bcast_intra_binomial
(abstraction violation, non portable, and you miss the some parameters)

out of curiosity, what do you have in mind for (some_condition) ?
/* since it seems implicit (some_condition) is independant of communicator
size, and message size */

Cheers,

Gilles

On Tue, Mar 10, 2015 at 10:04 PM, Khalid Hasanov  wrote:

> Hello,
>
> I would like to know if Open MPI provides some kind of mechanism to select
> collective algorithms such as MPI broadcast during run time depending on
> some logic. For example, I would like to
> use something like this:
>
> if (some_condition)  ompi_binomial_broadcast(...);
> else   ompi_pipeline_broadcast(...);
>
> I know it is possible to use some fixed algorithm by 
> coll_tuned_use_dynamic_rules
> or to define a custom selection rule using
> coll_tuned_dynamic_rules_filename. But
> I think it is not suitable in this situation as the dynamic rules mainly
> based on the message size, segment size and communicator size.
>
> Another option could be using Open MPI internal APIS like
>
> ompi_coll_tuned_bcast_intra_binomial( buf, count, dtype, root, comm,
> module, segsize);
>
> But it highly depends on Open MPI internals as it uses
> mca_coll_base_module_t .
>
> Is there any better option? (except using my own implementation of
> collectives)
>
> Any suggestion highly appreciated.
>
> Thanks
>
> Regards,
> Khalid
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/03/26446.php
>

Re: [OMPI users] open mpi on blue waters

[Gilles Gouaillardet]

Mark,

munge is an authentication mechanism based
on a secret key shared between hosts.

there are both a daemon part and a library/client part.

it its simplest form, you can run on node0 :

echo "hello" | munge | ssh node1 unmunge
(see sample output below)

if everything is correctly set (e.g. same shared secret key and munge
daemons are running)
then node1 will know for sure (well, at least as long as the secret key
stays secret ...)
the "hello" message was sent from node0 by this user, at that time and
was not altered.
there is also a time-to-live (ttl) for each message and an anti replay
mechanism

for example, munge can be used by SLURM in order to authentificate
messages between compute and head nodes.


so at first glance, it makes little sense to have munge running on some
nodes and not on others.
but since blue waters is running torque (at least this is what i found
on google), munge could be not needed
(and in this case, it is probably best to have it disabled on all nodes)
or it could be used by other services.


Ralph,

the commit log of PR #497 is :
"Let the initiator of the connection determine the method to be used -
if the receiver cannot support it, then that's an error that will cause
the connection attempt to fail."

from a security point of view, and imho, that seems wrong to me.
(it's like entering a bank and asking "i have no credit card, no id but
i ask you to trust me and give me my money").

also, that does not handle all cases :
/* if munge is not running on some nodes, and those nodes happen *not*
to be the initiator, that will break,
a trivial (and dumb) corner case is if mpirun runs on the compute node,
and the head node is part of the node list ... */

back to ompi, i d rather have the initiator send its authentication
method with its authentication key, and have the server
check it is using the same authentication method, and fail with an user
friendly error message otherwise.
(authentication type mismatch : node0 uses munge but node1 uses basic,
you should contact your sysadmin
or if you know what you are doing, you can try mpirun -mca sec basic)

on blue waters, that would mean ompi does not run out of the box, but
fails with an understandable message.
that would be less user friendly, but more secure

any thoughts ?

Cheers,

Gilles




[gouaillardet@node0 ~]$ echo coucou | munge | ssh node1 unmunge
STATUS:   Success (0)
ENCODE_HOST:  node0 (10.7.67.3)
ENCODE_TIME:  2015-03-26 09:55:16 (1427331316)
DECODE_TIME:  2015-03-26 09:55:16 (1427331316)
TTL:  300
CIPHER:   aes128 (4)
MAC:  sha1 (3)
ZIP:  none (0)
UID:  gouaillardet (1011)
GID:  gouaillardet (1011)
LENGTH:   7

coucou

On 2015/03/26 5:59, Mark Santcroos wrote:
> Hi Ralph,
>
>> On 25 Mar 2015, at 21:25 , Ralph Castain  wrote:
>> I think I have this resolved,
>> though that I still suspect their is something wrong on that system. You 
>> shouldn’t have some nodes running munge and others not running it.
> For completeness, it's not "some" nodes, its the MOM (service) nodes that run 
> it, and the compute nodes don't.
> I don't know munge well enough to judge whether it makes sense to have it 
> there only and not on the compute nodes?
>
>> I wonder if someone was experimenting and started munge on some of the 
>> nodes, and forgot to turn it off afterwards??
> If the answer to my request for clarification is along the lines of "No!", 
> then I can ask the admins whats up.
>
>> Anyway, see if this fixes the problem.
>>
>> https://github.com/open-mpi/ompi/pull/497
> Will get back to you later how that works for me.
>
> Thanks
>
> Mark
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/03/26533.php

Re: [OMPI users] open mpi on blue waters

[Gilles Gouaillardet]

On 2015/03/26 13:00, Ralph Castain wrote:
> Well, I did some digging around, and this PR looks like the right solution.
ok then :-)

following stuff is not directly related to ompi, but you might want to
comment on that anyway ...
> Second, the running of munge on the IO nodes is not only okay but required by 
> Luster.
this is the first time i hear that.
i googled "lustre munge" and could not find any relevant info about that.
is this a future feature of Lustre ?
as far as i am concerned, only Lustre MDS need a "unix" authentication
system
(ldap, nis, /etc/passwd, ...) and munge does not provide this service.
>  Future systems are increasingly going to run the user’s job script 
> (including mpirun) on the IO nodes as this (a) frees up the login node for 
> interactive editing, and (b) avoids the jitter introduced by running the job 
> script on the same node as application procs, or wasting a compute node to 
> just run the job script.
that does make sense not to run the script on a compute node.
but once again i am surprised ... as far as i am concerned, lustre IO
nodes (MDS and/or OSS) do not mount the filesystem
(i mean you cannot access the filesystem as if you were on a lustre client).
of course, you can write your script so it does not require any access
to the lustre filesystem, but that sounds like a lot of pain
for a small benefit.
/* that is specific to Lustre. GPFS for example can access the
filesystem from an IO node */

Cheers,

Gilles

Re: [OMPI users] open mpi on blue waters

[Gilles Gouaillardet]

Mark,

thanks for the link.

i tried to read between the lines, and "found" that in the case of
torque+munge,
munge might be required only on admin nodes and submission hosts (which
could be restricted
to login nodes on most systems)

on the other hand, slurm does require munge on compute nodes, even if
there is no job submission from compute nodes.

Cheers,

Gilles

On 2015/03/26 17:33, Mark Santcroos wrote:
> Hi guys,
>
> Thanks for the follow-up.
>
> It appears that you are ruling out that Munge is required because the system 
> runs TORQUE, but as far as I can see Munge is/can be used by both SLURM and 
> TORQUE.
> (http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/1-installConfig/serverConfig.htm#usingMUNGEAuth)
>
> If I misunderstood the drift, please ignore ;-)
>
> Mark
>
>
>> On 26 Mar 2015, at 5:38 , Gilles Gouaillardet 
>>  wrote:
>>
>> On 2015/03/26 13:00, Ralph Castain wrote:
>>> Well, I did some digging around, and this PR looks like the right solution.
>> ok then :-)
>>
>> following stuff is not directly related to ompi, but you might want to
>> comment on that anyway ...
>>> Second, the running of munge on the IO nodes is not only okay but required 
>>> by Luster.
>> this is the first time i hear that.
>> i googled "lustre munge" and could not find any relevant info about that.
>> is this a future feature of Lustre ?
>> as far as i am concerned, only Lustre MDS need a "unix" authentication
>> system
>> (ldap, nis, /etc/passwd, ...) and munge does not provide this service.
>>> Future systems are increasingly going to run the user’s job script 
>>> (including mpirun) on the IO nodes as this (a) frees up the login node for 
>>> interactive editing, and (b) avoids the jitter introduced by running the 
>>> job script on the same node as application procs, or wasting a compute node 
>>> to just run the job script.
>> that does make sense not to run the script on a compute node.
>> but once again i am surprised ... as far as i am concerned, lustre IO
>> nodes (MDS and/or OSS) do not mount the filesystem
>> (i mean you cannot access the filesystem as if you were on a lustre client).
>> of course, you can write your script so it does not require any access
>> to the lustre filesystem, but that sounds like a lot of pain
>> for a small benefit.
>> /* that is specific to Lustre. GPFS for example can access the
>> filesystem from an IO node */
>>
>> Cheers,
>>
>> Gilles
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/03/26539.php
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/03/26540.php

Re: [OMPI users] Isend, Recv and Test

[Gilles Gouaillardet]

per the error message, you likely misspeled vader (e.g. missed the "r")

Jeff,
the behavior was initially reported on a single node, so the tcp btl is
unlikely used

Cheers,

Gilles

On Friday, May 6, 2016, Zhen Wang  wrote:

>
>
> 2016-05-05 9:27 GMT-05:00 Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com
> >:
>
>> Out of curiosity, can you try
>> mpirun --mca btl self,sm ...
>>
> Same as before. Many MPI_Test calls.
>
>> and
>> mpirun --mca btl self,vader ...
>>
> A requested component was not found, or was unable to be opened.  This
> means that this component is either not installed or is unable to be
> used on your system (e.g., sometimes this means that shared libraries
> that the component requires are unable to be found/loaded).  Note that
> Open MPI stopped checking at the first component that it did not find.
>
> Host:  VirtualBox
> Framework: btl
> Component: vade
> --
> *** An error occurred in MPI_Init
> --
> It looks like MPI_INIT failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during MPI_INIT; some of which are due to configuration or environment
> problems.  This failure appears to be an internal failure; here's some
> additional information (which may only be relevant to an Open MPI
> developer):
>
>   mca_bml_base_open() failed
>   --> Returned "Not found" (-13) instead of "Success" (0)
> --
> *** on a NULL communicator
> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
> ***and potentially your MPI job)
> [VirtualBox:2188] Local abort before MPI_INIT completed successfully; not
> able to aggregate error messages, and not able to guarantee that all other
> processes were killed!
> ---
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code.. Per user-direction, the job has been aborted.
> ---
> --
> mpirun detected that one or more processes exited with non-zero status,
> thus causing
> the job to be terminated. The first process to do so was:
>
>   Process name: [[9235,1],0]
>   Exit code:1
> --
> [VirtualBox:02186] 1 more process has sent help message help-mca-base.txt
> / find-available:not-valid
> [VirtualBox:02186] Set MCA parameter "orte_base_help_aggregate" to 0 to
> see all help / error messages
>
>
>>
>> and see if one performs better than the other ?
>>
>> Cheers,
>>
>> Gilles
>>
>> On Thursday, May 5, 2016, Zhen Wang > > wrote:
>>
>>> Gilles,
>>>
>>> Thanks for your reply.
>>>
>>> Best regards,
>>> Zhen
>>>
>>> On Wed, May 4, 2016 at 8:43 PM, Gilles Gouaillardet <
>>> gilles.gouaillar...@gmail.com> wrote:
>>>
>>>> Note there is no progress thread in openmpi 1.10
>>>> from a pragmatic point of view, that means that for "large" messages,
>>>> no data is sent in MPI_Isend, and the data is sent when MPI "progresses"
>>>> e.g. call a MPI_Test, MPI_Probe, MPI_Recv or some similar subroutine.
>>>> in your example, the data is transferred after the first usleep
>>>> completes.
>>>>
>>> I agree.
>>>
>>>>
>>>> that being said, it takes quite a while, and there could be an issue.
>>>> what if you use MPI_Send instead () ?
>>>>
>>> Works as expected.
>>>
>>> MPI 1: Recv of 0 started at 08:37:10.
>>> MPI 1: Recv of 0 finished at 08:37:10.
>>> MPI 0: Send of 0 started at 08:37:10.
>>> MPI 0: Send of 0 finished at 08:37:10.
>>>
>>>
>>>> what if you send/Recv a large message first (to "warmup" connections),
>>>> MPI_Barrier, and then start your MPI_Isend ?
>>>>
>>> Not working. For what I want to accomplish, is my code the right way to
>>> go? Is there an altenative method? Thanks.
>>>
>>> MPI 1: Recv of 0 started at 08:38:46.
>>> MPI 0: Isend of 0 started at 08:38:46.
>>> MPI 0: Isend of 1 started at 08:38:46.
>>> MPI 0: Isend of 2 started at 08:38:4

Re: [OMPI users] SLOAVx alltoallv

[Gilles Gouaillardet]

Dave,

I briefly read the papers and it suggests the SLOAVx algorithm is
implemented by the ml collective module
this module had some issues and was judged not good for production.
it is disabled by default in the v1.10 series, and has been simply removed
from the v2.x branch.

you can either use (at your own risk ...) v1.10 or master with
mpirun --mca coll_ml_priority 100 ...

Cheers,

Gilles

On Friday, May 6, 2016, Dave Love  wrote:

> At the risk of banging on too much about collectives:
>
> I came across a writeup of the "SLOAVx" algorithm for alltoallv
> .  It was implemented
> in OMPI with apparently good results, but I can't find any code.
>
> I wonder if anyone knows the story on that.  Was it not contributed, or
> is it actually not worthwhile?  Otherwise, might it be worth investigating?
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/05/29113.php
>

Re: [OMPI users] Error building openmpi-dev-4010-g6c9d65c on Linux with Sun C

[Gilles Gouaillardet]

Siegmar,

at first glance, this looks like a crash of the compiler.
so I guess the root cause is not openmpi
(that being said, a workaround could be implemented in openmpi)

Cheers,

Gilles

On Saturday, May 7, 2016, Siegmar Gross <
siegmar.gr...@informatik.hs-fulda.de> wrote:

> Hi,
>
> today I tried to build openmpi-dev-4010-g6c9d65c on my
> machines (Solaris 10 Sparc, Solaris 10 x86_64, and openSUSE
> Linux 12.1 x86_64) with gcc-5.1.0 and Sun C 5.13. I was
> successful on most machines, but I got the following error
> on my Linux machine for the Sun C compiler.
>
> tyr openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 123 tail -7
> log.make.Linux.x86_64.64_cc
> "../../../../../openmpi-dev-4010-g6c9d65c/opal/mca/reachable/netlink/reachable_netlink_utils_common.c",
> line 322: warning: extern inline function "nl_object_priv" not defined in
> translation unit
> cc: Fatal error in /opt/sun/solarisstudio12.4/lib/compilers/acomp : Signal
> number = 11
> make[2]: *** [reachable_netlink_utils_common.lo] Error 1
> make[2]: Leaving directory
> `/export2/src/openmpi-master/openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc/opal/mca/reachable/netlink'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory
> `/export2/src/openmpi-master/openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc/opal'
> make: *** [all-recursive] Error 1
> tyr openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 124
>
>
> I would be grateful, if somebody can fix the problem.
> Thank you very much for any help in advance.
>
>
> Kind regards
>
> Siegmar
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/05/29122.php
>

Re: [OMPI users] warning message for process binding with openmpi-dev-4010-g6c9d65c

[Gilles Gouaillardet]

Siegmar,

did you upgrade your os recently ? or change hyper threading settings ?
this error message typically appears when the numactl-devel rpm is not
installed
(numactl-devel on redhat, the package name might differ on sles)

if not, would you mind retesting frI'm scratch a previous tarball that used
to work without any warning ?

Cheers,

Gilles


On Saturday, May 7, 2016, Siegmar Gross <
siegmar.gr...@informatik.hs-fulda.de> wrote:

> Hi,
>
> yesterday I installed openmpi-dev-4010-g6c9d65c on my "SUSE Linux
> Enterprise Server 12 (x86_64)" with Sun C 5.13  and gcc-5.3.0.
> Unfortunately I get the following warning message.
>
> loki hello_1 128 ompi_info | grep -e "OPAL repo revision" -e "C compiler
> absolute"
>   OPAL repo revision: dev-4010-g6c9d65c
>  C compiler absolute: /opt/solstudio12.4/bin/cc
> loki hello_1 129 mpiexec -np 3 --host loki --slot-list 0:0-5,1:0-5
> hello_1_mpi
> --
> WARNING: a request was made to bind a process. While the system
> supports binding the process itself, at least one node does NOT
> support binding memory to the process location.
>
>   Node:  loki
>
> Open MPI uses the "hwloc" library to perform process and memory
> binding. This error message means that hwloc has indicated that
> processor binding support is not available on this machine.
>
> On OS X, processor and memory binding is not available at all (i.e.,
> the OS does not expose this functionality).
>
> On Linux, lack of the functionality can mean that you are on a
> platform where processor and memory affinity is not supported in Linux
> itself, or that hwloc was built without NUMA and/or processor affinity
> support. When building hwloc (which, depending on your Open MPI
> installation, may be embedded in Open MPI itself), it is important to
> have the libnuma header and library files available. Different linux
> distributions package these files under different names; look for
> packages with the word "numa" in them. You may also need a developer
> version of the package (e.g., with "dev" or "devel" in the name) to
> obtain the relevant header files.
>
> If you are getting this message on a non-OS X, non-Linux platform,
> then hwloc does not support processor / memory affinity on this
> platform. If the OS/platform does actually support processor / memory
> affinity, then you should contact the hwloc maintainers:
> https://github.com/open-mpi/hwloc.
>
> This is a warning only; your job will continue, though performance may
> be degraded.
> --
> Process 0 of 3 running on loki
> Process 2 of 3 running on loki
> Process 1 of 3 running on loki
>
>
> Now 2 slave tasks are sending greetings.
>
> Greetings from task 1:
>   message type:3
> ...
>
>
>
> loki openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 122 ls -l
> /usr/lib64/*numa*
> -rwxr-xr-x 1 root root 48256 Nov 24 16:29 /usr/lib64/libnuma.so.1
> loki openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 123 grep numa
> log.configure.Linux.x86_64.64_cc
> checking numaif.h usability... no
> checking numaif.h presence... yes
> configure: WARNING: numaif.h: present but cannot be compiled
> configure: WARNING: numaif.h: check for missing prerequisite headers?
> configure: WARNING: numaif.h: see the Autoconf documentation
> configure: WARNING: numaif.h: section "Present But Cannot Be Compiled"
> configure: WARNING: numaif.h: proceeding with the compiler's result
> checking for numaif.h... no
> loki openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 124
>
>
>
> I didn't get the warning for openmpi-v1.10.2-176-g9d45e07 and
> openmpi-v2.x-dev-1404-g74d8ea0 as you can see in my previous emails,
> although I have the same messages in log.configure.*. I would be
> grateful, if somebody can fix the problem if it is a problem
> and not an intended message. Thank you very much for any help in
> advance.
>
>
> Kind regards
>
> Siegmar
>

Re: [OMPI users] problem with Sun C 5.14 beta

[Gilles Gouaillardet]

Siegmar,

per the config.log, you need to update your CXXFLAGS="-m64
-library=stlport4 -std=sun03"
or just CXXFLAGS="-m64"

Cheers,

Gilles

On Saturday, May 7, 2016, Siegmar Gross <
siegmar.gr...@informatik.hs-fulda.de> wrote:

> Hi,
>
> today I tried to install openmpi-v1.10.2-176-g9d45e07 on my "SUSE Linux
> Enterprise Server 12 (x86_64)" with Sun C 5.14 beta. Unfortunately
> "configure" breaks, because it thinks that C and C++ are link
> incompatible. I used the following configure command.
>
> ../openmpi-v1.10.2-176-g9d45e07/configure \
>   --prefix=/usr/local/openmpi-1.10.3_64_cc \
>   --libdir=/usr/local/openmpi-1.10.3_64_cc/lib64 \
>   --with-jdk-bindir=/usr/local/jdk1.8.0_66/bin \
>   --with-jdk-headers=/usr/local/jdk1.8.0_66/include \
>   JAVA_HOME=/usr/local/jdk1.8.0_66 \
>   LDFLAGS="-m64 -mt -Wl,-z -Wl,noexecstack" CC="cc" CXX="CC" FC="f95" \
>   CFLAGS="-m64 -mt" CXXFLAGS="-m64 -library=stlport4" FCFLAGS="-m64" \
>   CPP="cpp" CXXCPP="cpp" \
>   --enable-mpi-cxx \
>   --enable-cxx-exceptions \
>   --enable-mpi-java \
>   --enable-heterogeneous \
>   --enable-mpi-thread-multiple \
>   --with-hwloc=internal \
>   --without-verbs \
>   --with-wrapper-cflags="-m64 -mt" \
>   --with-wrapper-cxxflags="-m64 -library=stlport4" \
>   --with-wrapper-fcflags="-m64" \
>   --with-wrapper-ldflags="-mt" \
>   --enable-debug \
>   |& tee log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_cc
>
>
> I don't know if it is a compiler problem or a problem with the
> configure command. Perhaps you are nevertheless interested in
> the problem.
>
>
> Kind regards
>
> Siegmar
>

Re: [OMPI users] Incorrect function call in simple C program

[Gilles Gouaillardet]

Devon,

send() is a libc function that is used internally by Open MPI, and it uses
your user function instead of the libc ne.
simply rename your function mysend() or something else that is not used by
libc, and your issue will likely be fixed

Cheers,

Gilles

On Tuesday, May 10, 2016, Devon Hollowood  wrote:

> Hello,
>
> I am having trouble understanding why I am getting an error when running
> the program produced by the attached C file. In this file, there are three
> short functions: send(), bounce() and main(). send() and bounce() both use
> MPI_Send() and MPI_Recv(), but critically, neither one is called from
> main(), so as I understand it, neither function should ever be run. main()
> is just:
>
> int main(int argc, char *argv[]) {
>> int rank;
>>
>> MPI_Init(&argc, &argv);
>> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>
>> MPI_Finalize();
>> }
>>
>
> Despite the fact that the program should never enter send() or bounce(),
> when I compile with
>
> mpicc mpi-issue.c -o mpi-issue
>>
>
> and run with
>
> mpirun -n 2 --verbose ./mpi-issue
>>
>
> I get the following:
>
> *** An error occurred in MPI_Send
>> *** on a NULL communicator
>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>> ***and potentially your MPI job)
>> [dhcp-visitor-enr-117-111.slac.stanford.edu:99119] Local abort before
>> MPI_INIT completed successfully; not able to aggregate error messages, and
>> not able to guarantee that all other processes were killed!
>> ---
>> Primary job  terminated normally, but 1 process returned
>> a non-zero exit code.. Per user-direction, the job has been aborted.
>> ---
>> --
>> mpirun detected that one or more processes exited with non-zero status,
>> thus causing
>> the job to be terminated. The first process to do so was:
>>
>>   Process name: [[2829,1],0]
>>   Exit code:1
>> --
>>
>>
>
> How is it possible to be getting an error in MPI_Send(), if MPI_Send()
> never gets run?
>
> I am running open-mpi 1.10.2, installed via the Homebrew open-mpi package,
> and this is running on my Macbook, which is running OSX Yosemite. I have
> attached the results of ompi_info --all as ompi_info.txt.bz2
>
> Any help would be appreciated! Sorry if this is a newb question and I am
> missing something obvious--I tried my best to search for this issue but I
> couldn't find anything.
>
> -Devon
>

Re: [OMPI users] 'AINT' undeclared

[Gilles Gouaillardet]

Hi,


i was able to build openmpi 1.10.2 with the same configure command line 
(after i quoted the LDFLAGS parameters)



can you please run

grep SIZEOF_PTRDIFF_T config.status


it should be 4 or 8, but it seems different in your environment (!)


are you running 32 or 64 bit kernel ? on which processor ?


Cheers,


Gilles


On 5/10/2016 1:41 AM, Ilias Miroslav wrote:


Greetings,


I am trying to install OpenMPI 1.10.1/1.10.2 with gcc (GCC) 5.2.1 
20150902 (Red Hat 5.2.1-2) statically,



$ ./configure --prefix=/home/ilias/bin/openmpi-1.10.1_gnu_static 
CXX=g++ CC=gcc F77=gfortran FC=gfortran LDFLAGS=--static -ldl -lrt 
--disable-shared --enable-static --disable-vt


However, I am getting this error below. Any help, please ? Am I 
missing something in my GNU installation ?




make[2]: Entering directory 
`/home/ilias/bin/openmpi-1.10.1_gnu_static/openmpi-1.10.2/ompi/datatype'

  CC   ompi_datatype_args.lo
  CC   ompi_datatype_create.lo
  CC   ompi_datatype_create_contiguous.lo
  CC   ompi_datatype_create_indexed.lo
  CC   ompi_datatype_create_struct.lo
  CC   ompi_datatype_create_vector.lo
  CC   ompi_datatype_create_darray.lo
  CC   ompi_datatype_create_subarray.lo
  CC   ompi_datatype_external32.lo
  CC   ompi_datatype_match_size.lo
  CC   ompi_datatype_module.lo
  CC   ompi_datatype_sndrcv.lo
ompi_datatype_module.c:280:44: warning: implicit declaration of 
function 'OMPI_DATATYPE_INIT_UNAVAILABLE_BASIC_TYPE' 
[-Wimplicit-function-declaration]
 ompi_predefined_datatype_t ompi_mpi_aint = 
OMPI_DATATYPE_INIT_UNAVAILABLE_BASIC_TYPE(INT64_T, AINT, 
OMPI_DATATYPE_FLAG_DATA_C | OMPI_DATATYPE_FLAG_DATA_INT);

^
ompi_datatype_module.c:280:86: error: 'INT64_T' undeclared here (not 
in a function)
 ompi_predefined_datatype_t ompi_mpi_aint = 
OMPI_DATATYPE_INIT_UNAVAILABLE_BASIC_TYPE(INT64_T, AINT, 
OMPI_DATATYPE_FLAG_DATA_C | OMPI_DATATYPE_FLAG_DATA_INT);

^
ompi_datatype_module.c:280:95: error: 'AINT' undeclared here (not in a 
function)
 ompi_predefined_datatype_t ompi_mpi_aint = 
OMPI_DATATYPE_INIT_UNAVAILABLE_BASIC_TYPE(INT64_T, AINT, 
OMPI_DATATYPE_FLAG_DATA_C | OMPI_DATATYPE_FLAG_DATA_INT);

 ^
make[2]: *** [ompi_datatype_module.lo] Error 1
make[2]: *** Waiting for unfinished jobs
make[2]: Leaving directory 
`/home/ilias/bin/openmpi-1.10.1_gnu_static/openmpi-1.10.2/ompi/datatype'

make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory 
`/home/ilias/bin/openmpi-1.10.1_gnu_static/openmpi-1.10.2/ompi'

make: *** [all-recursive] Error 1
il...@grid.ui.savba.sk:~/bin/openmpi-1.10.1_gnu_static/openmpi-1.10.2/.


M.




___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/05/29142.php

Re: [OMPI users] mpirun command won't run unless the firewalld daemon is disabled

[Gilles Gouaillardet]

you can direct OpenMPI to only use a specific range of ports (that 
should be open in your firewall configuration)



mpirun --mca oob_tcp_static_ipv4_ports - ...


if you use the tcp btl, you can (also) use

mpirun --mca btl_tcp_port_min_v4  --mca btl_tcp_port_range_v4 
 ...



Cheers,

Gilles

On 5/10/2016 3:31 AM, Llolsten Kaonga wrote:


Hello all,

We’ve been running openmpi for a long time and up to version 1.8.2 and 
CentOS 6.7 with commands such as the one below:


usr/local/bin/mpirun --allow-run-as-root --mca btl openib,self,sm 
--mca pml ob1 -np 2 -np 8 -hostfile /root/mpi-hosts 
/usr/local/bin/IMB-MPI1


To be able to run the above command, we normally just disabled the 
IPv4 firewall and SELinux.


We recently made the following updates:

OS: CentOS 7.2

IMB:  4.1

OpenMPI: 1.10.2

When we tried to execute the above mpirun command, we got a TCP Broken 
Pipe error. There was no IP assignment conflict and eventually, we 
narrowed down the problem to the firewall. Disabling the firewalld 
daemon allows the command to run to completion. We would prefer not to 
disable the daemon as our servers may sometimes be connected to the 
rest of our subnet.


Are there other options such as perhaps specifying a port (I am 
guessing, so specific instructions will be greatly appreciated).


I thank you.



___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/05/29143.php

Re: [OMPI users] problem with ld for Sun C 5.14 beta and openmpi-dev-4010-g6c9d65c

[Gilles Gouaillardet]

Siegmar,


this issue was previously reported at 
http://www.open-mpi.org/community/lists/devel/2016/05/18923.php


i just pushed the patch


Cheers,


Gilles


On 5/10/2016 2:27 PM, Siegmar Gross wrote:

Hi,

I tried to install openmpi-dev-4010-g6c9d65c on my "SUSE Linux
Enterprise Server 12 (x86_64)" with Sun C 5.14 beta. Unfortunately
"make" breaks with the following error.

make[2]: Entering directory 
'/export2/src/openmpi-master/openmpi-dev-4010-g6c9d6

  GENERATE mpi-ignore-tkr-sizeof.h
  GENERATE mpi-ignore-tkr-sizeof.f90
  PPFC mpi-ignore-tkr.lo
  FCLD libmpi_usempi_ignore_tkr.la
f90: Warning: Option -shared passed to ld, if ld is invoked, ignored 
otherwise
f90: Warning: Option -path passed to ld, if ld is invoked, ignored 
otherwise
f90: Warning: Option -path passed to ld, if ld is invoked, ignored 
otherwise
f90: Warning: Option -soname passed to ld, if ld is invoked, ignored 
otherwise

/usr/bin/ld: unrecognized option '-path'
/usr/bin/ld: use the --help option for usage information
Makefile:1909: recipe for target 'libmpi_usempi_ignore_tkr.la' failed
make[2]: *** [libmpi_usempi_ignore_tkr.la] Error 2
make[2]: Leaving directory 
'/export2/src/openmpi-master/openmpi-dev-4010-g6c9d65

Makefile:3433: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory 
'/export2/src/openmpi-master/openmpi-dev-4010-g6c9d65

Makefile:1898: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
loki openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 132


loki openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 135 ld -version
GNU ld (GNU Binutils; SUSE Linux Enterprise 12) 2.25.0
Copyright (C) 2014 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later 
version.

This program has absolutely no warranty.
loki openmpi-dev-4010-g6c9d65c-Linux.x86_64.64_cc 136


I could built it with "Sun C 5.13". Can somebody fix the problem? 
Thank you

very much for any help in advance.


Kind regards

Siegmar
___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/05/29151.php

Re: [OMPI users] Incorrect function call in simple C program

[Gilles Gouaillardet]

except if you #include the libc header in your app, *and* your send
function has a different prototype, I do not see how clang can issue a
warning
(except of course if clang "knows" all the libc subroutines ...)

Cheers,

Gilles

On Tuesday, May 10, 2016, Devon Hollowood  wrote:

> That worked perfectly. Thank you. I'm surprised that clang didn't emit a
> warning about this!
>
> -Devon
>
> On Mon, May 9, 2016 at 3:42 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com
> > wrote:
>
>> Devon,
>>
>> send() is a libc function that is used internally by Open MPI, and it
>> uses your user function instead of the libc ne.
>> simply rename your function mysend() or something else that is not used
>> by libc, and your issue will likely be fixed
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On Tuesday, May 10, 2016, Devon Hollowood > > wrote:
>>
>>> Hello,
>>>
>>> I am having trouble understanding why I am getting an error when running
>>> the program produced by the attached C file. In this file, there are three
>>> short functions: send(), bounce() and main(). send() and bounce() both use
>>> MPI_Send() and MPI_Recv(), but critically, neither one is called from
>>> main(), so as I understand it, neither function should ever be run. main()
>>> is just:
>>>
>>> int main(int argc, char *argv[]) {
>>>> int rank;
>>>>
>>>> MPI_Init(&argc, &argv);
>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>>>
>>>> MPI_Finalize();
>>>> }
>>>>
>>>
>>> Despite the fact that the program should never enter send() or bounce(),
>>> when I compile with
>>>
>>> mpicc mpi-issue.c -o mpi-issue
>>>>
>>>
>>> and run with
>>>
>>> mpirun -n 2 --verbose ./mpi-issue
>>>>
>>>
>>> I get the following:
>>>
>>> *** An error occurred in MPI_Send
>>>> *** on a NULL communicator
>>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>>>> ***and potentially your MPI job)
>>>> [dhcp-visitor-enr-117-111.slac.stanford.edu:99119] Local abort before
>>>> MPI_INIT completed successfully; not able to aggregate error messages, and
>>>> not able to guarantee that all other processes were killed!
>>>> ---
>>>> Primary job  terminated normally, but 1 process returned
>>>> a non-zero exit code.. Per user-direction, the job has been aborted.
>>>> ---
>>>>
>>>> --
>>>> mpirun detected that one or more processes exited with non-zero status,
>>>> thus causing
>>>> the job to be terminated. The first process to do so was:
>>>>
>>>>   Process name: [[2829,1],0]
>>>>   Exit code:1
>>>>
>>>> --
>>>>
>>>>
>>>
>>> How is it possible to be getting an error in MPI_Send(), if MPI_Send()
>>> never gets run?
>>>
>>> I am running open-mpi 1.10.2, installed via the Homebrew open-mpi
>>> package, and this is running on my Macbook, which is running OSX Yosemite.
>>> I have attached the results of ompi_info --all as ompi_info.txt.bz2
>>>
>>> Any help would be appreciated! Sorry if this is a newb question and I am
>>> missing something obvious--I tried my best to search for this issue but I
>>> couldn't find anything.
>>>
>>> -Devon
>>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org 
>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/05/29147.php
>>
>
>

Re: [OMPI users] mpirun command won't run unless the firewalld daemon is disabled

[Gilles Gouaillardet]

I was basically suggesting you open a few ports to anyone (e.g. any IP
address), and Jeff suggests you open all ports to a few trusted IP
addresses.

btw, how many network ports do you have ?
if you have two ports (e.g. eth0 for external access and eth1 for private
network) and MPI should only use the internal network, then you can allow
all traffic on the internal port, and
mpirun --mca oob_tcp_if_include eth1 --mca btl_tcp_if_include eth1 ...

Cheers,

Gilles

On Wednesday, May 11, 2016, Llolsten Kaonga  wrote:

> Hello Jeff,
>
> I think what you suggest is likely exactly what we want to see happen. We
> run the interop tests with at least two servers, sometimes more. We also
> have other devices (InfiniBand or RoCE switches) between the servers.
>
> I will have to ask a stupid question here but when you suggest that we open
> the firewall to trust random TCP connections, how is that different from
> disabling it? Is there some configuration besides the suggestion by Gilles
> to specify ports or a range of ports?
>
> I thank you.
> --
> Llolsten
>
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org ] On Behalf
> Of Jeff Squyres
> (jsquyres)
> Sent: Tuesday, May 10, 2016 3:47 PM
> To: Open MPI User's List >
> Subject: Re: [OMPI users] mpirun command won't run unless the firewalld
> daemon is disabled
>
> Open MPI generally needs to be able to communicate on random TCP ports
> between machines in the MPI job (and the machine where mpirun is invoked,
> if
> that is a different machine).
>
> You could also open your firewall to trust random TCP connections just
> between the servers in your cluster.
>
>
>
> > On May 10, 2016, at 3:44 PM, Llolsten Kaonga  > wrote:
> >
> > Hello Orion,
> >
> > I actually rather like the new CentOS 7.2 system better and would like
> > to not remove firewalld. We will try Gilles' suggestion and see what
> happens.
> >
> > I thank you.
> > --
> > Llolsten
> >
> > -Original Message-
> > From: users [mailto:users-boun...@open-mpi.org ] On
> Behalf Of Orion
> > Poplawski
> > Sent: Tuesday, May 10, 2016 3:31 PM
> > To: Open MPI Users >
> > Subject: Re: [OMPI users] mpirun command won't run unless the
> > firewalld daemon is disabled
> >
> > On 05/10/2016 09:24 AM, Llolsten Kaonga wrote:
> >> Hello Durga,
> >>
> >> As I mentioned earlier, up to version 1.8.2, we would just disable
> >> SELinux and the IPv4 firewall and things run smoothly. It was only
> >> when we installed version 1.10.2 (CentOS 7.2) that we run into these
> >> troubles. CentOS 7.2 no longer seems to bother with the IPv4
> >> firewall, so
> > you can't do:
> >>
> >>
> >>
> >> # service iptables save
> >>
> >> # service iptables stop
> >>
> >> # chkconfig iptables off
> >
> > I'll just note that you can either embrace the new firewalld config
> > (and use firewall-cmd to open your needed ports) or you can remove
> > firewalld and install iptables-services and go back to the old
> > iptables method of configuring the firewall.  If you don't want a
> > firewall at all, just remove firewalld.
> >
> > --
> > Orion Poplawski
> > Technical Manager 303-415-9701 x222
> > NWRA, Boulder/CoRA Office FAX: 303-415-9702
> > 3380 Mitchell Lane   or...@nwra.com 
> > Boulder, CO 80301   http://www.nwra.com
> > ___
> > users mailing list
> > us...@open-mpi.org 
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> > http://www.open-mpi.org/community/lists/users/2016/05/29160.php
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org 
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> > http://www.open-mpi.org/community/lists/users/2016/05/29161.php
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com 
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/05/29162.php
>
>
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/05/29163.php
>

Re: [OMPI users] 'AINT' undeclared

[Gilles Gouaillardet]

Ilias,

at first glance, you are using the PGI preprocessor (!)

can you re-run configure with CPP=cpp,
or after removing all PGI related environment variables,
and see it it helps ?

Cheers,

Gilles

On Wednesday, May 11, 2016, Ilias Miroslav  wrote:

> https://www.open-mpi.org/community/lists/users/2016/05/29148.php
>
> Dear Gill,
>
> Ooops, I don't have SIZEOF_PTRDIFF_T in config.status/config.log.
>
> I am attaching config.log, config.status files.
>
> This is virtual machine :
> Linux grid.ui.savba.sk 2.6.32-573.26.1.el6.x86_64 #1 SMP Tue May 3
> 14:22:07 CDT 2016 x86_64 x86_64 x86_64 GNU/Linux ; 4 cpu Westmere
> E56xx/L56xx/X56xx (Nehalem-C).
>
> Miro
>
>
>
>

Re: [OMPI users] Question about mpirun mca_oob_tcp_recv_handler error.

[Gilles Gouaillardet]

Hi,


Where did you get the openmpi package from ?

fc20 ships openmpi 1.7.3 ...


does it work as expected if you do not use mpirun

(e.g. ./hello_c)


if yes, then you can try

ldd hello_c

which mpirun

ldd mpirun

mpirun -np 1 ldd hello_c


and confirm both mpirun and hello_c use the same mpi libraries


Cheers,


Gilles


On 5/11/2016 12:06 PM, lzfneu wrote:

Hi Ralph and all,

Thanks for your respond, but the problems is not solved.

My system is fedora20 64-bit with openmpi-1.8.4 package installed in 
my laptop.


The mpirun command just run in my single laptop which disconnect to 
the internet and  I also performe the following command to find mpirun 
path and add it to .bashcr file. However, the results with no effect.


[user@localhost ~]$ which mpirun
/usr/lib64/openmpi/bin/mpirun

Any idea and thanks in advance!


*Subject:* Re: [OMPI users] Question about mpirun 
mca_oob_tcp_recv_handler error.

*From:* Ralph Castain (/rhc_at_[hidden]/)
*Date:* 2016-05-10 14:43:39

  * *Next message:* Orion Poplawski: "Re: [OMPI users] mpirun command
won't run unless the firewalld daemon is disabled"

  * *Previous message:* Gus Correa: "Re: [OMPI users] No core dump in
some cases"

  * *In reply to:* lzfneu: "[OMPI users] Question about mpirun
mca_oob_tcp_recv_handler error."




This usually indicates that the remote process is using a different OMPI
version. You might check to ensure that the paths on the remote nodes are
correct.



From: lzf...@live.com
To: us...@open-mpi.org
CC: lzf...@live.com
Subject: Question about mpirun mca_oob_tcp_recv_handler error.
Date: Tue, 10 May 2016 15:46:45 +

Hi everyone,

I have a problem to consult you, when I cd to the /examples folder 
contained in the openmpi-1.8.4 package, and test the hello_c example 
program with mpirun command errors occured:


Here are the command and the error messages in details:
[user@localhost examples]$ mpirun -np 2 hello_c
[localhost.localdomain:06965] [[52154,0],0] mca_oob_tcp_recv_handler: 
invalid message type: 14
[localhost.localdomain:06965] [[52154,0],0] mca_oob_tcp_recv_handler: 
invalid message type: 14

---
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
---
--
mpirun detected that one or more processes exited with non-zero 
status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[52154,1],0]
  Exit code:65
--

The problem was not occured before, but in recently when i execute 
every progams by using the mpirun command, the error message is 
reproducible. I don't know why.

Could you please help me and thanks in advance !

Zhefu


___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/05/29167.php