Re: [OMPI users] error running mpirun command

2019-05-03 Thread Gilles Gouaillardet via users
Eric,

which version of Open MPI are you using ? how many hosts in your hostsfile ?

The error message suggests this could be a bug within Open MPI, and a
potential workaround for you would be to try
mpirun -np 84  - -hostfile hostsfile --mca routed direct ./openmpi_hello.c

You might also want to double check all your hosts can access each
other with TCP/IP and on all ports (e.g. no firewall should be
running)

Cheers,

Gilles


On Sat, May 4, 2019 at 9:41 AM Eric F. Alemany via users
 wrote:
>
> Hello everyone,
>
> I am new to Open MPI please forgive me for my beginner mistake. I read 
> through the FAQ of open-mpi.org website and built a small cluster (9 nodes - 
> including a master node).
> I thought i followed the instructions accordingly but i am having issue 
> running a simple mpirun.
>
> $ mpirun -np 84  - -hostfile hostsfile ./openmpi_hello.c
>
> mpirun: Forwarding signal 20 to job
> --
> ORTE does not know how to route a message to the specified daemon
> located on the indicated node:
>
>   my node:   phaser-manager
>   target node:  radonc-phaser01
>
> This is usually an internal programming error that should be
> reported to the developers. In the meantime, a workaround may
> be to set the MCA param routed=direct on the command line or
> in your environment. We apologize for the problem.
> —
>
> I dont understand the meaning of the error message. I can share more of my 
> configuration files if someone would be interested in helping me.
>
> Thank you in advance for your help.
>
>
> Best,
> Eric
>
> _
>
> Eric F.  Alemany
> System Administrator for Research
>
> IRT
> Division of Radiation & Cancer  Biology
> Department of Radiation Oncology
>
> Stanford University School of Medicine
> Stanford, California 94305
>
> Tel:1-650-498-7969  No Texting
> Fax:1-650-723-7382
>
>
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] 3.0.4, 4.0.1 build failure on OSX Mojave with LLVM

2019-04-24 Thread Gilles Gouaillardet via users
John,

what if you move some parameters to CPPFLAGS and CXXCPPFLAGS (see the
new configure command line below)

Cheers,

Gilles

'/Users/cary/projects/ulixesall-llvm/builds/openmpi-4.0.1/nodl/../configure'
\
--prefix=/Volumes/GordianStorage/opt/contrib-llvm7_appleclang/openmpi-4.0.1-nodl
\
CC='/Volumes/GordianStorage/opt/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang'
\
CXX='/Volumes/GordianStorage/opt/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++'
\
   FC='/opt/homebrew/bin/gfortran-6' \
   F77='/opt/homebrew/bin/gfortran-6' \
   CFLAGS='-fvisibility=default -mmacosx-version-min=10.10 -fPIC -pipe' \
   
CPPFLAGS='-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include
-DSTDC_HEADERS' \
  CXXFLAGS='-std=c++11 -stdlib=libc++ -fvisibility=default
-mmacosx-version-min=10.10 -fPIC -pipe' \
  
CXXCPFLAGS='-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include'
\
   FCFLAGS='-fPIC -pipe' \
   --enable-static \
   --with-pic \
   --disable-dlopen \
   --enable-mpirun-prefix-by-default

On Wed, Apr 24, 2019 at 7:01 PM Jeff Squyres (jsquyres) via users
 wrote:
>
> Can you send at least your config.log file?  It would be good to know why the 
> "STDC" test is reporting that this setup does not have STDC headers when it 
> actually does.
>
>
> > On Apr 23, 2019, at 8:14 PM, John R. Cary  wrote:
> >
> > It appears that the problem is with AC_HEADER_STDC, which is reporting
> > that this setup does not have stdc headers when in fact it does.
> >
> > Overriding with
> >
> >  CFLAGS='-fvisibility=default -mmacosx-version-min=10.10 
> > -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include
> >  -fPIC -pipe -DSTDC_HEADERS -DOPAL_STDC_HEADERS'
> >
> > in particular, the last two defines, fixes this.
> >
> > .John
> >
> >
> >
> > On 4/23/2019 4:59 PM, Jeff Squyres (jsquyres) wrote:
> >> The version of LLVM that I have installed on my Mac 10.14.4 is:
> >>
> >> $ where clang
> >> /usr/bin/clang
> >> $ clang --version
> >> Apple LLVM version 10.0.1 (clang-1001.0.46.4)
> >> Target: x86_64-apple-darwin18.5.0
> >> Thread model: posix
> >> InstalledDir: 
> >> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
> >>
> >> I don't know how that compares to upstream clang v7.x...?
> >>
> >> More below.
> >>
> >>
> >>
> >>> On Apr 23, 2019, at 6:26 PM, John R. Cary via users 
> >>>  wrote:
> >>>
> >>> The failure is
> >>>
> >>> In file included from 
> >>> /Users/cary/projects/ulixesall-llvm/builds/openmpi-4.0.1/nodl/../ompi/datatype/ompi_datatype_external.c:29:
> >>> In file included from 
> >>> /Users/cary/projects/ulixesall-llvm/builds/openmpi-4.0.1/nodl/../ompi/communicator/communicator.h:38:
> >>> In file included from 
> >>> /Users/cary/projects/ulixesall-llvm/builds/openmpi-4.0.1/nodl/../ompi/errhandler/errhandler.h:33:
> >>> ../../ompi/include/mpi.h:397:9: error: unknown type name 'ptrdiff_t'
> >>> typedef OMPI_MPI_AINT_TYPE MPI_Aint;
> >>> ^
> >>> ../../opal/include/opal_config.h:1911:28: note: expanded from macro 
> >>> 'OMPI_MPI_AINT_TYPE'
> >>> #define OMPI_MPI_AINT_TYPE ptrdiff_t
> >>>
> >>>
> >>> Is there a known fix for this?
> >>>
> >>> Thx...John Cary
> >>>
> >>>
> >>> More info:
> >>>
> >>> Configured with
> >>>
> >> A few notes on your configure line:
> >>
> >>> '/Users/cary/projects/ulixesall-llvm/builds/openmpi-4.0.1/nodl/../configure'
> >>>  \
> >>> --prefix=/Volumes/GordianStorage/opt/contrib-llvm7_appleclang/openmpi-4.0.1-nodl
> >>>  \
> >>> CC='/Volumes/GordianStorage/opt/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang'
> >>>  \
> >>> CXX='/Volumes/GordianStorage/opt/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++'
> >>>  \
> >> The MPI C++ bindings are no longer built by default, and you're not 
> >> enabling them (via --enable-mpi-cxx), so you don't need to specify CXX or 
> >> CXXFLAGS here.
> >>
> >>>   FC='/opt/homebrew/bin/gfortran-6' \
> >>>   F77='/opt/homebrew/bin/gfortran-6' \
> >> F77 is ignored these days; FC is the only one that matters.
> >>
> >>>   CFLAGS='-fvisibility=default -mmacosx-version-min=10.10 
> >>> -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include
> >>>  -fPIC -pipe -DSTDC_HEADERS' \
> >> Do you need all these CFLAGS?  E.g., does clang not -I that directory by 
> >> default (I don't actually know if it is necessary or not)?  What does 
> >> -DSTDC_HEADERS do?
> >>
> >>>   CXXFLAGS='-std=c++11 -stdlib=libc++ -fvisibility=default 
> >>> -mmacosx-version-min=10.10 
> >>> -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include
> >>>  -fPIC -pipe' \
> >>>   FCFLAGS='-fPIC -pipe' \
> >>>   --enable-static \
> >>>   --with-pic \
> >>>   --disable-dlopen \
> >>>   --enable-mpirun-prefix-by-default
> >> I don't have that version of clang, 

Re: [OMPI users] fatal error: ac_nonexistent.h: No such file or directory (openmpi-4.0.0)

2019-04-20 Thread Gilles Gouaillardet via users
The root cause is configure cannot run a simple Fortran program
(see the relevant log below)

I suggest you

export LD_LIBRARY_PATH=/share/apps/gcc-5.4.0/lib64:$LD_LIBRARY_PATH
and then try again.

Cheers,

Gilles

configure:44254: checking Fortran value of selected_int_kind(4)
configure:44281: /share/apps/gcc-5.4.0/bin/gfortran -o conftest
-L/share/apps/gcc-5.4.0/lib64  conftest.f90  -lz >&5
configure:44281: $? = 0
configure:44281: ./conftest
./conftest: /usr/lib64/libgfortran.so.3: version `GFORTRAN_1.4' not
found (required by ./conftest)
configure:44281: $? = 1
configure: program exited with status 1
configure: failed program was:
|   program main
|
| open(8, file="conftest.out")
| write(8, fmt="(I5)") selected_int_kind(4)
| close(8)
|
|   end
configure:44303: result: no
configure:44314: WARNING: Could not determine KIND value of C_SIGNED_CHAR
configure:44316: WARNING: See config.log for more details
configure:44318: error: Cannot continue

On Sun, Apr 21, 2019 at 9:32 AM Guido granda muñoz via users
 wrote:
>
> Hello openmpi users,
> I recently got an error during the instalation of openmpi-4.0.0. The error 
> showed up during the configuration.
> My configuration options are the following:
>   $ ./configure 
> --prefix=/home/guido/libraries/compiled_with_gcc-5.4.0/openmpi-4.0.0 
> --enable-fortran=all FC=/share/apps/gcc-5.4.0/bin/gfortran 
> CC=/share/apps/gcc-5.4.0/bin/gcc CXX=/share/apps/gcc-5.4.0/bin/g++ 
> CPPFLAGS=-I/share/apps/gcc-5.4.0/include LDFLAGS=-L/share/apps/gcc-5.4.0/lib64
> The gcc version used is : gcc version 5.4.0
>
> I got the following error:
>
> 1)
> configure:7047: /share/apps/gcc-5.4.0/bin/gcc -E 
> -I/share/apps/gcc-5.4.0/include conftest.c
> conftest.c:10:28: fatal error: ac_nonexistent.h: No such file or directory
> compilation terminated.
> 2)
> conftest.c:88:19: fatal error: /cuda.h: No such file or directory
>
> The full content of my log file can be download from:
>
> https://pastebin.com/9Gu62FTT?fbclid=IwAR3ajK2-S2t-wzRxBsq1g0_Fw54YC2kr0YwPuxO-xhJKKpb0QEY7uZkpirA
>
> Information about my system:
>
> $ uname -a
> Linux mouruka.crya.privado 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 
> 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> I hope you can help me to figure out what is wrong with my configuration 
> options or if something else
> is wrong. I am not an expert so please be answer accordingtly.
>
> Cheers,
>
>
> --
> Guido
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] TCP usage in MPI singletons

2019-04-17 Thread Gilles Gouaillardet

Daniel,


If your MPI singleton will never MPI_Comm_spawn(), then you can use the 
isolated mode like this


OMPI_MCA_ess_singleton_isolated=true ./program


You can also save some ports by blacklisting the btl/tcp component


OMPI_MCA_ess_singleton_isolated=true OMPI_MCA_pml=ob1 
OMPI_MCA_btl=vader,self ./program



Cheers,


Gilles

On 4/18/2019 3:51 AM, Daniel Hemberger wrote:

Hi everyone,

I've been trying to track down the source of TCP connections when 
running MPI singletons, with the goal of avoiding all TCP 
communication to free up ports for other processes. I have a local apt 
install of OpenMPI 2.1.1 on Ubuntu 18.04 which does not establish any 
TCP connections by default, either when run as "mpirun -np 1 
./program" or "./program". But it has non-TCP alternatives for both 
the BTL (vader, self, etc.) and OOB (ud and usock) frameworks, so I 
was not surprised by this result.


On a remote machine, I'm running the same test with an assortment of 
OpenMPI versions (1.6.4, 1.8.6, 4.0.0, 4.0.1 on RHEL6 and 1.10.7 on 
RHEL7). In all but 1.8.6 and 1.10.7, there is always a TCP connection 
established, even if I disable the TCP BTL on the command line (e.g. 
"mpirun --mca btl ^tcp"). Therefore, I assumed this was because `tcp` 
was the only OOB interface available in these installations. This TCP 
connection is established both for "mpirun -np 1 ./program" and 
"./program".


The confusing part is that the 1.8.6 and 1.10.7 installations only 
appear to establish a TCP connection when invoked with "mpirun -np 1 
./program", but _not_ with "./program", even though its only OOB 
interface was also `tcp`. This result was not consistent with my 
understanding, so now I am confused about when I should expect TCP 
communication to occur.


Is there a known explanation for what I am seeing? Is there actually a 
way to get singletons to forego all TCP communication, even if TCP is 
the only OOB available, or is there something else at play here? I'd 
be happy to provide any config.log files or ompi_info output if it 
would help.


For more context, the underlying issue I'm trying to resolve is that 
we are (unfortunately) running many short instances of mpirun, and the 
TCP connections are piling up in the TIME_WAIT state because they 
aren't cleaned up faster than we create them.


Any advice or pointers would be greatly appreciated!

Thanks,
-Dan

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] relocating an installation

2019-04-09 Thread Gilles Gouaillardet
Dave,

Can you please  post the configure command line of the Open MPI you are trying 
to relocate ?

Cheers,

Gilles

Dave Love  wrote:
>Reuti  writes:
>
>> export OPAL_PREFIX=
>>
>> to point it to the new location of installation before you start `mpiexec`.
>
>Thanks; that's now familiar, and I don't know how I missed it with
>strings.
>
>It should be documented.  I'd have expected --prefix to have the same
>effect, and for there to be an MCA variable.  Would there be some
>problem with either of those?
>___
>users mailing list
>users@lists.open-mpi.org
>https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Issues compiling HPL with OMPIv4.0.0

2019-04-03 Thread Gilles Gouaillardet

Do not get fooled by the symlinks to opal_wrapper !

opal_wrapper checks how it is invoked (e.g. check argv[0] in main()) and 
the behavior is different


if it is invoked as mpicc, mpiCC, mpifort and other


If the error persists with mpicc, you can manually extract the mpicc 
command line, and manually run it with the -showme parameter,


it will show you the full command line (and who knows, mpicc might 
invoke a C++ compiler after all, and that would be a config issue)



Cheers,


Gilles

On 4/4/2019 7:48 AM, afernan...@odyhpc.com wrote:


Sam and Jeff,

Thank you for your answers. My first attempts actually used mpicc 
rather than mpiCC, switching to mpiCC was simply to check out if the 
problem persisted. I noticed that both mpicc and mpiCC are linked to 
the same file (opal_wrapper) and didn't bother switching it back. I'm 
not sure if the wrapper figures out what compiler you call because I 
was getting the same error message. Jeff is right pointing out that 
'try' is reserved but the original file seems to be really old (think 
1970). Apparently, the new compiler (shipped with OMPIv4) is more 
sensitive and beeps when the older didn't.


Thanks again,

AFernandez

Indeed, you cannot use "try" as a variable name in C++ because it is a 
https://en.cppreference.com/w/cpp/keyword.


As already suggested, use a C compiler, or you can replace "try" with 
"xtry" or any other non-reserved word.


Jeff

On Wed, Apr 3, 2019 at 1:41 PM Gutierrez, Samuel K. via users 
mailto:users@lists.open-mpi.org>> wrote:


Hi,

It looks like you are using the C++ wrapper compiler (mpiCC)
instead of the C wrapper compiler (mpicc). Perhaps using mpicc
instead of mpiCC will resolve your issue.

Best,

Sam



On Apr 3, 2019, at 12:38 PM, afernan...@odyhpc.com
 wrote:

Hello,

I'm trying to compile HPL(v2.3) with OpenBLAS and OMPI. The
compilation succeeds when using the old OMPI (v1.10.8) but
fails with OMPI v4.0.0 (I'm still not using v4.0.1). The error
is for an old subroutine that determines machine-specific
arithmetic constants:

mpiCC -o HPL_dlamch.o -c
-I/home/centos/benchmarks/hpl-2.2/include
-I/home/centos/benchmarks/hpl-2.2/include/impetus03
-I/opt/openmpi/include  ../HPL_dlamch.c

../HPL_dlamch.c: In function ‘void HPL_dlamc5(int, int, int,
int, int*, double*)’:

../HPL_dlamch.c:749:67: error: expected unqualified-id before
‘try’

int    exbits=1, expsum, i, lexp=1, nbits,
try,

^

../HPL_dlamch.c:761:8: error: expected ‘{’ before ‘=’ token

    try = (int)( (unsigned int)(lexp) << 1 );

    ^

../HPL_dlamch.c:761:8: error: expected ‘catch’ before ‘=’ token

../HPL_dlamch.c:761:8: error: expected ‘(’ before ‘=’ token

../HPL_dlamch.c:761:8: error: expected type-specifier before
‘=’ token

../HPL_dlamch.c:761:8: error: expected ‘)’ before ‘=’ token

../HPL_dlamch.c:761:8: error: expected ‘{’ before ‘=’ token

../HPL_dlamch.c:761:8: error: expected primary-expression
before ‘=’ token

../HPL_dlamch.c:762:8: error: expected primary-expression
before ‘try’

    if( try <= ( -EMIN ) ) { lexp = try; exbits++; goto l_10; }

    ^

../HPL_dlamch.c:762:8: error: expected ‘)’ before ‘try’

../HPL_dlamch.c:762:36: error: expected primary-expression
before ‘try’

    if( try <= ( -EMIN ) ) { lexp = try; exbits++; goto l_10; }

^

../HPL_dlamch.c:762:36: error: expected ‘;’ before ‘try’

../HPL_dlamch.c:764:26: error: ‘uexp’ was not declared in this
scope

    if( lexp == -EMIN ) { uexp = lexp; } else { uexp = try;
exbits++; }

^

../HPL_dlamch.c:764:48: error: ‘uexp’ was not declared in this
scope

    if( lexp == -EMIN ) { uexp = lexp; } else { uexp = try;
exbits++; }

^

../HPL_dlamch.c:764:55: error: expected primary-expression
before ‘try’

    if( lexp == -EMIN ) { uexp = lexp; } else { uexp = try;
exbits++; }

^

../HPL_dlamch.c:764:55: error: expected ‘;’ before ‘try’

../HPL_dlamch.c:770:10: error: ‘uexp’ was not declared in this
scope

    if( ( uexp+EMIN ) > ( -lexp-EMIN ) )

  ^

make[2]: *** [HPL_dlamch.o] Error 1

make[2]: Leaving directory
`/home/centos/hpl-2.3/src/auxil/impetus03'

make[1]: *** [build_src] Error 2

make[1]: Leaving directory `/home/centos/hpl-2.3'

make: *** [build] Error 2

I don't understand the nature of the problem or why it works
with the old OMPI version and not with the new. Any help or
pointer would be appreciated.

Thanks.

AFernandez


Re: [OMPI users] Using strace with Open MPI on Cray

2019-03-30 Thread Gilles Gouaillardet
Christoph,

I do not know how to fix this, but here are some suggestions/thoughts

- do you need the -f flag ? if not, just remote it
- what if you mpirun strace -o /dev/null ... ?
- if the former works, then you might want to redirect the strace
output to a local file (mpirun wrapper.sh, in which wrapper.sh sets
the output file based on $PMIX_RANK or $$, and then exec strace ...

Cheers,

Gilles

On Sat, Mar 30, 2019 at 6:29 PM Christoph Niethammer  wrote:
>
> Hello,
>
> I was trying to investigate some processes with strace under Open MPI.
> However I have some issues when MPI I/O functionality is included writing 
> data to a NFS file system.
>
> mpirun -np 2 strace -f ./hello-world mpi-io
>
> does not return and strace is stuck reporting infinite "poll" calls.
> However, the program works fine without strace.
>
> I tried with Open MPI 3.x and 4.0.1 switching between ompi and romio on 
> different operating systems (CentOS 7.6, SLES 12).
>
> I'd appreciate any hints which help me to understand what is going on.
>
> Best
> Christoph
>
> --
>
> Christoph Niethammer
> High Performance Computing Center Stuttgart (HLRS)
> Nobelstrasse 19
> 70569 Stuttgart
>
> Tel: ++49(0)711-685-87203
> email: nietham...@hlrs.de
> http://www.hlrs.de/people/niethammer
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Possible buffer overflow on Recv rank

2019-03-27 Thread Gilles Gouaillardet
Carlos,

can you post a trimmed version of your code that evidences the issue ?

Keep in mind that if you want to write MPI code that is correct with respect to 
the standard, you should assume MPI_Send() might block until a matching receive 
is posted.

Cheers,

Gilles 

Sent from my iPod

> On Mar 27, 2019, at 20:46, carlos aguni  wrote:
> 
> Not "MPI_Send from 0"..
> MPI_Send from 1 to 0
> MPI_Send from 7 to 0
> And so on..
> 
>> On Wed, Mar 27, 2019, 8:43 AM carlos aguni  wrote:
>> Hi all.
>> 
>> I've an MPI application in which at one moment one rank receives a slice of 
>> an array from the other nodes.
>> Thing is that my application hangs there.
>> 
>> One thing I could get from printint out logs are:
>> (Rank 0) Starts MPI_Recv from source 4
>> But then it receives:
>> MPI_Send from 0
>> MPI_Send from 1
>> ... From 10
>> ... From 7
>> ... From 6
>> 
>> Then at one neither of them are responding.
>> The message is a double array type of size 100.000.
>> Later it would receive the message from 4.
>> 
>> So i assume the buffer on the Recv side is overflowing. 
>> 
>> Few tests:
>> - Using smaller array size works
>> - alreay tried using isend. Irecv. Bsend. And the ranks still get stuck.
>> 
>> So that leaves me to a few questions rather than how to solve this issue:
>> - how can i know the size of mpi's interbal buffer?
>> - how would one debug this?
>> 
>> Regards,
>> Carlos.
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] stack overflow in alloca() for Java programs in openmpi-master with pgcc-18.4

2019-03-26 Thread Gilles Gouaillardet
Siegmar,

Is this issue specific to the PGI compiler ?
What if you
ulimit -s
before invoking mpirun, is that good enough to workaround the problem ?

Cheers,

Gilles

On Tue, Mar 26, 2019 at 6:32 PM Siegmar Gross
 wrote:
>
> Hi,
>
> I've installed openmpi-v4.0.x-201903220241-97aa434 and
> openmpi-master-201903260242-dfbc144 on my "SUSE Linux Enterprise Server 12.3
> (x86_64)" with pgcc-18.4. Unfortunately, I still get the following error for
> my Java programs for openmpi-master. Everything works as expected for
> openmpi-4.0.x. I've already reported the error some time ago.
> https://users.open-mpi.narkive.com/qz90agAO/ompi-users-stack-overflow-in-routine-alloca-for-java-programs-in-openmpi-master-with-pgcc-18-4
>
>
> loki java 127 ompi_info | grep "Configure command line:"
>Configure command line: '--prefix=/usr/local/openmpi-master_64_pgcc'
> '--libdir=/usr/local/openmpi-master_64_pgcc/lib64'
> '--with-jdk-bindir=/usr/local/jdk-11/bin'
> '--with-jdk-headers=/usr/local/jdk-11/include' 'JAVA_HOME=/usr/local/jdk-11'
> 'LDFLAGS=-m64 -Wl,-z -Wl,noexecstack -L/usr/local/pgi/linux86-64/18.4/lib
> -R/usr/local/pgi/linux86-64/18.4/lib' 'LIBS=-lpgm' 'CC=pgcc' 'CXX=pgc++'
> 'FC=pgfortran' 'CFLAGS=-c11 -m64' 'CXXFLAGS=-m64' 'FCFLAGS=-m64' 'CPP=cpp'
> 'CXXCPP=cpp' '--enable-mpi-cxx' '--enable-cxx-exceptions' '--enable-mpi-java'
> '--with-valgrind=/usr/local/valgrind' '--with-hwloc=internal' 
> '--without-verbs'
> '--with-wrapper-cflags=-c11 -m64' '--with-wrapper-cxxflags=-m64'
> '--with-wrapper-fcflags=-m64' '--enable-debug'
>
>
> loki java 128 mpiexec -np 4 --host loki:4  java
> MatMultWithAnyProc2DarrayIn1DarrayMain
> Error: in routine alloca() there is a
> stack overflow: thread 0, max 8180KB, used 0KB, request 42B
> Error: in routine alloca() there is a
> stack overflow: thread 0, max 8180KB, used 0KB, request 42B
> Error: in routine alloca() there is a
> stack overflow: thread 0, max 8180KB, used 0KB, request 42B
> --
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code. Per user-direction, the job has been aborted.
> --
> --
> mpiexec detected that one or more processes exited with non-zero status, thus
> causing
> the job to be terminated. The first process to do so was:
>
>Process name: [[44592,1],1]
>Exit code:127
> --
> loki java 129
>
>
>
> I would be grateful, if somebody can fix the problem. Do you need anything
> else? Thank you very much for any help in advance.
>
>
> Kind regards
>
> Siegmar
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Network performance over TCP

2019-03-23 Thread Gilles Gouaillardet
bug here?  If I
>>> am misunderstanding the flag's intent, is there a different flag that would
>>> allow Open MPI to use multiple ports similar to what iperf is doing?
>>>
>>> Thanks.
>>> -Adam
>>>
>>> On Mon, Jul 10, 2017 at 9:31 PM, Adam Sylvester 
>>> wrote:
>>>
>>>> Thanks again Gilles.  Ahh, better yet - I wasn't familiar with the
>>>> config file way to set these parameters... it'll be easy to bake this into
>>>> my AMI so that I don't have to set them each time while waiting for the
>>>> next Open MPI release.
>>>>
>>>> Out of mostly laziness I try to keep to the formal releases rather than
>>>> applying patches myself, but thanks for the link to it (the commit comments
>>>> were useful to understand why this improved performance).
>>>>
>>>> -Adam
>>>>
>>>> On Mon, Jul 10, 2017 at 12:04 AM, Gilles Gouaillardet <
>>>> gil...@rist.or.jp> wrote:
>>>>
>>>>> Adam,
>>>>>
>>>>>
>>>>> Thanks for letting us know your performance issue has been resolved.
>>>>>
>>>>>
>>>>> yes, https://www.open-mpi.org/faq/?category=tcp is the best place to
>>>>> look for this kind of information.
>>>>>
>>>>> i will add a reference to these parameters. i will also ask folks at
>>>>> AWS if they have additional/other recommendations.
>>>>>
>>>>>
>>>>> note you have a few options before 2.1.2 (or 3.0.0) is released :
>>>>>
>>>>>
>>>>> - update your system wide config file (/.../etc/openmpi-mca-params.conf)
>>>>> or user config file
>>>>>
>>>>>   ($HOME/.openmpi/mca-params.conf) and add the following lines
>>>>>
>>>>> btl_tcp_sndbuf = 0
>>>>>
>>>>> btl_tcp_rcvbuf = 0
>>>>>
>>>>>
>>>>> - add the following environment variable to your environment
>>>>>
>>>>> export OMPI_MCA_btl_tcp_sndbuf=0
>>>>>
>>>>> export OMPI_MCA_btl_tcp_rcvbuf=0
>>>>>
>>>>>
>>>>> - use Open MPI 2.0.3
>>>>>
>>>>>
>>>>> - last but not least, you can manually download and apply the patch
>>>>> available at
>>>>>
>>>>> https://github.com/open-mpi/ompi/commit/b64fedf4f652cadc9bfc7c4693f9c1
>>>>> ef01dfb69f.patch
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Gilles
>>>>>
>>>>> On 7/9/2017 11:04 PM, Adam Sylvester wrote:
>>>>>
>>>>>> Gilles,
>>>>>>
>>>>>> Thanks for the fast response!
>>>>>>
>>>>>> The --mca btl_tcp_sndbuf 0 --mca btl_tcp_rcvbuf 0 flags you
>>>>>> recommended made a huge difference - this got me up to 5.7 Gb/s! I wasn't
>>>>>> aware of these flags... with a little Googling, is
>>>>>> https://www.open-mpi.org/faq/?category=tcp the best place to look
>>>>>> for this kind of information and any other tweaks I may want to try (or 
>>>>>> if
>>>>>> there's a better FAQ out there, please let me know)?
>>>>>> There is only eth0 on my machines so nothing to tweak there (though
>>>>>> good to know for the future). I also didn't see any improvement by
>>>>>> specifying more sockets per instance. But, your initial suggestion had a
>>>>>> major impact.
>>>>>> In general I try to stay relatively up to date with my Open MPI
>>>>>> version; I'll be extra motivated to upgrade to 2.1.2 so that I don't have
>>>>>> to remember to set these --mca flags on the command line. :o)
>>>>>> -Adam
>>>>>>
>>>>>> On Sun, Jul 9, 2017 at 9:26 AM, Gilles Gouaillardet <
>>>>>> gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>>
>>>>>> wrote:
>>>>>>
>>>>>> Adam,
>>>>>>
>>>>>> at first, you need to change the default send and receive socket
>>>>>> buffers :
>>>>>> mpirun --mca btl_tcp_sndbuf 0 --mca btl_tcp_rcvbuf 0 ...
>>>>>> /* 

Re: [OMPI users] MPI_Comm_spawn leads to pipe leak and other errors

2019-03-17 Thread Gilles Gouaillardet
FWIW I could observe some memory leaks on both mpirun and MPI task 0 with the 
latest master branch.

So I guess mileage varies depending on available RAM and number of iterations.

Sent from my iPod

> On Mar 17, 2019, at 20:47, Riebs, Andy  wrote:
> 
> Thomas, your test case is somewhat similar to a bash fork() bomb -- not the 
> same, but similar. After running one of your failing jobs, you might check to 
> see if the “out-of-memory” (“OOM”) killer has been invoked. If it has, that 
> can lead to unexpected consequences, such as what you’ve reported.
>  
> An easy way to check would be
> $ nodes=${ job’s node list }
> $  pdsh  -w $nodes dmesg  -T \|  grep  \"Out of memory\" 2>/dev/null
>  
> Andy
>  
> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Thomas Pak
> Sent: Saturday, March 16, 2019 4:14 PM
> To: Open MPI Users 
> Cc: Open MPI Users 
> Subject: Re: [OMPI users] MPI_Comm_spawn leads to pipe leak and other errors
>  
> Dear Jeff,
>  
> I did find a way to circumvent this issue for my specific application by 
> spawning less frequently. However, I wanted to at least bring attention to 
> this issue for the OpenMPI community, as it can be reproduced with an 
> alarmingly simple program.
>  
> Perhaps the user's mailing list is not the ideal place for this. Would you 
> recommend that I report this issue on the developer's mailing list or open a 
> GitHub issue?
>  
> Best wishes,
> Thomas Pak
>  
> On Mar 16 2019, at 7:40 pm, Jeff Hammond  wrote:
> Is there perhaps a different way to solve your problem that doesn’t spawn so 
> much as to hit this issue?
>  
> I’m not denying there’s an issue here, but in a world of finite human effort 
> and fallible software, sometimes it’s easiest to just avoid the bugs 
> altogether.
>  
> Jeff
>  
> On Sat, Mar 16, 2019 at 12:11 PM Thomas Pak  wrote:
> Dear all,
>  
> Does anyone have any clue on what the problem could be here? This seems to be 
> a persistent problem present in all currently supported OpenMPI releases and 
> indicates that there is a fundamental flaw in how OpenMPI handles dynamic 
> process creation.
>  
> Best wishes,
> Thomas Pak
>  
> From: "Thomas Pak" 
> To: users@lists.open-mpi.org
> Sent: Friday, 7 December, 2018 17:51:29
> Subject: [OMPI users] MPI_Comm_spawn leads to pipe leak and other errors
>  
> Dear all,
>  
> My MPI application spawns a large number of MPI processes using 
> MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced 
> that this results in problems for all currently supported OpenMPI versions 
> (2.1, 3.0, 3.1 and 4.0). I have written a short, self-contained program in C 
> (included below) that spawns child processes using MPI_Comm_spawn in an 
> infinite loop, where each child process exits after writing a message to 
> stdout. This short program leads to the following issues:
>  
> In versions 2.1.2 (Ubuntu package) and 2.1.5 (compiled from source), the 
> program leads to a pipe leak where pipes keep accumulating over time until my 
> MPI application crashes because the maximum number of pipes has been reached.
>  
> In versions 3.0.3 and 3.1.3 (both compiled from source), there appears to be 
> no pipe leak, but the program crashes with the following error message:
> PMIX_ERROR: UNREACHABLE in file ptl_tcp_component.c at line 1257
>  
> In version 4.0.0 (compiled from source), I have not been able to test this 
> issue very thoroughly because mpiexec ignores the --oversubscribe 
> command-line flag (as detailed in this GitHub issue 
> https://github.com/open-mpi/ompi/issues/6130). This prohibits the 
> oversubscription of processor cores, which means that spawning additional 
> processes immediately results in an error because "not enough slots" are 
> available. A fix for this was proposed recently 
> (https://github.com/open-mpi/ompi/pull/6139), but since the v4.0.x developer 
> branch is being actively developed right now, I decided not go into it.
>  
> I have found one e-mail thread on this mailing list about a similar problem 
> (https://www.mail-archive.com/users@lists.open-mpi.org/msg10543.html). In 
> this thread, Ralph Castain states that this is a known issue and suggests 
> that it is fixed in the then upcoming v1.3.x release. However, version 1.3 is 
> no longer supported and the issue has reappeared, hence this did not solve 
> the issue.
>  
> I have created a GitHub gist that contains the output from "ompi_info --all" 
> of all the OpenMPI installations mentioned here, as well as the config.log 
> files for the OpenMPI installations that I compiled from source: 
> https://gist.github.com/ThomasPak/1003160e396bb88dff27e53c53121e0c.
>  
> I have also attached the code for the short program that demonstrates these 
> issues. For good measure, I have included it directly here as well:
>  
> """
> #include 
> #include 
>  
> int main(int argc, char *argv[]) {
>  
> // Initialize MPI
> MPI_Init(NULL, NULL);
>  
> // Get parent
> MPI_Comm parent;
> 

Re: [OMPI users] Double free in collectives

2019-03-14 Thread Gilles Gouaillardet

Jeff,


The first location is indeed in ompi_coll_libnbc_iallreduce()


Lee Ann,


thanks for the bug report,

for the time being, can you please give the attached patch a try ?


Cheers,


Gilles



FWIW

NBC_Schedule_request() sets handle->tmpbuf = tmpbuf and call 
NBC_Start(handle, schedule)


then handle->schedule = schedule.

if NBC_Start_round() fails, then NBC_Return_handle(handle) calls 
NBC_Free(handle) that


OBJ_RELEASE(handle->schedule) and free(handle->tmpbuf)

Back to ompi_coll_libnbc_iallreduce(), we once again 
OBJ_RELEASE(schedule) and free(tmpbuf).





On 3/15/2019 7:27 AM, Jeff Squyres (jsquyres) via users wrote:

Lee Ann --

Thanks for your bug report.

I'm not able to find a call to NBC_Schedule_request() in 
ompi_coll_libnbc_iallreduce().

I see 2 calls to NBC_Schedule_request() in 
ompi/mca/coll/libnbc/nbc_iallreduce.c, but they are in different functions.

Can you clarify exactly which one(s) you're referring to?

Location 1:
https://github.com/open-mpi/ompi/blob/7412e88689ffd1dcc95a587e6ded9d7455fa031c/ompi/mca/coll/libnbc/nbc_iallreduce.c#L183

Location 2:
https://github.com/open-mpi/ompi/blob/7412e88689ffd1dcc95a587e6ded9d7455fa031c/ompi/mca/coll/libnbc/nbc_iallreduce.c#L247




On Mar 14, 2019, at 4:27 PM, Riesen, Lee Ann  wrote:

I'm trying to build OpenMPI 3.1.2 as part of Mellanox HPC-X and I'm having some 
problems with the underlying libraries.  The true problem was masked for awhile 
by an bug in error handling in OpenMPI.  In mca/coll/libnbc/nbc_iallreduce.c in 
function ompi_coll_libnbc_iallreduce() we have some error handling at the end 
that looks like:
  
  
   res = NBC_Schedule_request (schedule, comm, libnbc_module, request, tmpbuf);

   if (OPAL_UNLIKELY(OMPI_SUCCESS != res)) {
 OBJ_RELEASE(schedule);
 free(tmpbuf);
 return res;
   }
  
The Schedule_request call failed, and in that call the "schedule" and "tmpbuf" were freed.  Then we return and again, the "schedule" and "tmpbuf" are freed.  It looks like this occurs elsewhere in the source file too.
  
Lee Ann
  
  
-

Lee Ann Riesen, Enterprise and Government Group, Intel Corporation, Hillsboro, 
OR
Phone 503-613-1952
  
___

users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


diff --git a/ompi/mca/coll/libnbc/nbc.c b/ompi/mca/coll/libnbc/nbc.c
index dff6362..43eec5f 100644
--- a/ompi/mca/coll/libnbc/nbc.c
+++ b/ompi/mca/coll/libnbc/nbc.c
@@ -720,6 +720,9 @@ int NBC_Schedule_request(NBC_Schedule *schedule, 
ompi_communicator_t *comm, ompi
 
   res = NBC_Start (handle, schedule);
   if (OPAL_UNLIKELY(OMPI_SUCCESS != res)) {
+/* temporary workaround to avoid double free */
+handle->tmpbuf = NULL;
+handle->schedule = NULL;
 NBC_Return_handle (handle);
 return res;
   }
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Best way to send on mpi c, architecture dependent data type

2019-03-13 Thread Gilles Gouaillardet

Sergio,


I think your best option here is to install an aarch64 (read 64bits ARM) 
distro on your raspberry pi 3.


/* FWIW, the default raspberry distro is 32bits so it can run on all 
raspberry pi models */



If you cannot/do not wish to go that way, make sure Open MPI is built 
with --enable-heterogeneous


This is lightly tested in the release branches, so I suggest you use 
Open MPI 4.0.1rc1.



IIRC, Open MPI will transparently handle this scenario with the same 
datatype having different sizes (or endianness, but

that does not apply here) on different nodes.

Cheers,

Gilles

On 3/14/2019 7:10 AM, Sergio None wrote:

Hello.

I'm using OpenMPI 3.1.3 on x64 CPU  and two ARMv8( Raspberry pi 3).

But i'm having some issues with data types that are architecture 
dependent, like 'long'.


For example, if you send data in one process to other on this way:

if (my_rank != 0) {
   long size;
   char* array_data;

   array_data = malloc(size);
   //Initialize data...

   /* Send size first*/
   MPI_Ssend(, 1, MPI_LONG,  0, 0, MPI_COMM_WORLD);

   /* Send data*/
   MPI_Ssend(array_data, size, MPI_BYTE,  0, 0, MPI_COMM_WORLD);

}else{
char* data_of_procs[num_procs-1];
   long size_data_procs[num_procs-1];
   int i;
   for(i=0;i MPI_Recv(_data_procs[i], 1, MPI_LONG, i+1, 0, 
MPI_COMM_WORLD, );

 data_of_procs[i] = malloc(size_data_procs[i]);
 if (data_of_procs[i] == NULL) perror("Error on malloc");
 MPI_Recv(data_of_procs[i], size_data_procs[i], MPI_BYTE, i+1, 0, 
MPI_COMM_WORLD, );

   }
}

Probably you get an error of malloc saying to you: "can't allocate 
this memory". This fail because long in x86 are 8bytes and in arm 
compiler is 4bytes. For solve, instead of use long, you need to use 
int that have the same size in both architectures. Other option could 
be serialize long.


So my question is: there any way to pass data that don't depend of 
architecture?





___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Building PMIx and Slurm support

2019-03-12 Thread Gilles Gouaillardet

Passant,


you can first try a PMIx only program


for example, in the test directory of PMIx

srun --mpi=pmix -N 2 -n 4 .libs/pmix_client -n 4

should work just fine (otherwise, this is a non Open MPI related issue)


If it works, then you can build an other Open MPI and pass 
--enable-debug to the configure command line.


Hopefully, it will provide more information (or at least, you will have 
the option to ask some very verbose logs)



Cheers,


Gilles

On 3/12/2019 5:46 PM, Passant A. Hafez wrote:

Hi Gilles,

Yes it was just a typo in the last email, it was correctly spelled in the job 
script.

So I just tried to use 1 node * 2 tasks/node, I got the same error I posted 
before, just a copy for each process, here it is again:

*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***and potentially your MPI job)
[cn603-20-l:169109] Local abort before MPI_INIT completed completed 
successfully, but am not able to aggregate error messages, and not able to 
guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***and potentially your MPI job)
[cn603-20-l:169108] Local abort before MPI_INIT completed completed 
successfully, but am not able to aggregate error messages, and not able to 
guarantee that all other processes were killed!
srun: error: cn603-20-l: tasks 0-1: Exited with exit code 1


I'm suspecting Slurm, but anyways, how can I troubleshoot this?
The program is a simple MPI Hello World code.




All the best,
--
Passant A. Hafez | HPC Applications Specialist
KAUST Supercomputing Core Laboratory (KSL)
King Abdullah University of Science and Technology
Building 1, Al-Khawarizmi, Room 0123
Mobile : +966 (0) 55-247-9568
Mobile : +20 (0) 106-146-9644
Office  : +966 (0) 12-808-0367


From: users  on behalf of Gilles Gouaillardet 

Sent: Tuesday, March 12, 2019 8:22 AM
To: users@lists.open-mpi.org
Subject: Re: [OMPI users] Building PMIx and Slurm support

Passant,


Except the typo (it should be srun --mpi=pmix_v3), there is nothing
wrong with that, and it is working just fine for me

(same SLURM version, same PMIx version, same Open MPI version and same
Open MPI configure command line)

that is why I asked you some more information/logs in order to
investigate your issue.


You might want to try a single node job first in order to rule out
potential interconnect related issues.


Cheers,


Gilles


On 3/12/2019 1:54 PM, Passant A. Hafez wrote:

Hello Gilles,

Yes I do use srun --mpi=pmix_3 to run the app, what's the problem with
that?
Before that, when we tried to launch MPI apps directly with srun, we
got the error message saying Slurm missed the PMIx support, that's why
we proceeded with the installation.



All the best,

--

Passant

On Mar 12, 2019 6:53 AM, Gilles Gouaillardet  wrote:
Passant,


I built a similar environment, and had no issue running a simple MPI
program.


Can you please post your slurm script (I assume it uses srun to start
the MPI app),

the output of

scontrol show config | grep Mpi

and the full output of your job ?


Cheers,


Gilles


On 3/12/2019 7:59 AM, Passant A. Hafez wrote:

Hello,


So we now have Slurm 18.08.6-2 compiled with PMIx 3.1.2

then I installed openmpi 4.0.0 with:

--with-slurm  --with-pmix=internal --with-libevent=internal
--enable-shared --enable-
static  --with-x


(Following the thread, it was mentioned that building OMPI 4.0.0 with
PMIx 3.1.2 will fail with PMIX_MODEX and PMIX_INFO_ARRAY errors, so I
used internal PMIx)



The MPI program fails with:


*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***and potentially your MPI job)
[cn603-13-r:387088] Local abort before MPI_INIT completed completed
successfully, but am not able to aggregate error messages, and not
able to guarantee that all other processes were killed!


for each process, please advise! what's going wrong here?







All the best,
--
Passant A. Hafez | HPC Applications Specialist
KAUST Supercomputing Core Laboratory (KSL)
King Abdullah University of Science and Technology
Building 1, Al-Khawarizmi, Room 0123
Mobile : +966 (0) 55-247-9568
Mobile : +20 (0) 106-146-9644
Office : +966 (0) 12-808-0367

*From:* users  on behalf of Ralph H
Castain 
*Sent:* Monday, March 4, 2019 5:29 PM
*To:* Open MPI Users
*Subject:* Re: [OMPI users] Building PMIx and Slurm support



On Mar 4, 2019, at 5:34 AM, Daniel Letai mailto:d...@letai.org.il>> wrote:

Gilles,
On 3/4/19 8:28 AM, Gilles Gouaillardet wrote:

Daniel,


On 3/4/2019 3:18 PM, Daniel Letai wrote:

So unless you have a specific reason not to mix both, you might
also give the internal PMIx a try.

Does this hol

Re: [OMPI users] Building PMIx and Slurm support

2019-03-11 Thread Gilles Gouaillardet

Passant,


Except the typo (it should be srun --mpi=pmix_v3), there is nothing 
wrong with that, and it is working just fine for me


(same SLURM version, same PMIx version, same Open MPI version and same 
Open MPI configure command line)


that is why I asked you some more information/logs in order to 
investigate your issue.



You might want to try a single node job first in order to rule out 
potential interconnect related issues.



Cheers,


Gilles


On 3/12/2019 1:54 PM, Passant A. Hafez wrote:

Hello Gilles,

Yes I do use srun --mpi=pmix_3 to run the app, what's the problem with 
that?
Before that, when we tried to launch MPI apps directly with srun, we 
got the error message saying Slurm missed the PMIx support, that's why 
we proceeded with the installation.




All the best,

--

Passant

On Mar 12, 2019 6:53 AM, Gilles Gouaillardet  wrote:
Passant,


I built a similar environment, and had no issue running a simple MPI
program.


Can you please post your slurm script (I assume it uses srun to start
the MPI app),

the output of

scontrol show config | grep Mpi

and the full output of your job ?


Cheers,


Gilles


On 3/12/2019 7:59 AM, Passant A. Hafez wrote:
>
> Hello,
>
>
> So we now have Slurm 18.08.6-2 compiled with PMIx 3.1.2
>
> then I installed openmpi 4.0.0 with:
>
> --with-slurm  --with-pmix=internal --with-libevent=internal
> --enable-shared --enable-
> static  --with-x
>
>
> (Following the thread, it was mentioned that building OMPI 4.0.0 with
> PMIx 3.1.2 will fail with PMIX_MODEX and PMIX_INFO_ARRAY errors, so I
> used internal PMIx)
>
>
>
> The MPI program fails with:
>
>
> *** An error occurred in MPI_Init
> *** on a NULL communicator
> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
> ***    and potentially your MPI job)
> [cn603-13-r:387088] Local abort before MPI_INIT completed completed
> successfully, but am not able to aggregate error messages, and not
> able to guarantee that all other processes were killed!
>
>
> for each process, please advise! what's going wrong here?
>
>
>
>
>
>
>
> All the best,
> --
> Passant A. Hafez | HPC Applications Specialist
> KAUST Supercomputing Core Laboratory (KSL)
> King Abdullah University of Science and Technology
> Building 1, Al-Khawarizmi, Room 0123
> Mobile : +966 (0) 55-247-9568
> Mobile : +20 (0) 106-146-9644
> Office : +966 (0) 12-808-0367
> 
> *From:* users  on behalf of Ralph H
> Castain 
> *Sent:* Monday, March 4, 2019 5:29 PM
> *To:* Open MPI Users
> *Subject:* Re: [OMPI users] Building PMIx and Slurm support
>
>
>> On Mar 4, 2019, at 5:34 AM, Daniel Letai > <mailto:d...@letai.org.il>> wrote:
>>
>> Gilles,
>> On 3/4/19 8:28 AM, Gilles Gouaillardet wrote:
>>> Daniel,
>>>
>>>
>>> On 3/4/2019 3:18 PM, Daniel Letai wrote:
>>>>
>>>>> So unless you have a specific reason not to mix both, you might
>>>>> also give the internal PMIx a try.
>>>> Does this hold true for libevent too? Configure complains if
>>>> libevent for openmpi is different than the one used for the other
>>>> tools.
>>>>
>>>
>>> I am not exactly sure of which scenario you are running.
>>>
>>> Long story short,
>>>
>>>  - If you use an external PMIx, then you have to use an external
>>> libevent (otherwise configure will fail).
>>>
>>> It must be the same one used by PMIx, but I am not sure configure
>>> checks that.
>>>
>>> - If you use the internal PMIx, then it is up to you. you can either
>>> use the internal libevent, or an external one.
>>>
>> Thanks, that clarifies the issues I've experienced. Since PMIx
>> doesn't have to be the same for server and nodes, I can compile slurm
>> with external PMIx with system libevent, and compile openmpi with
>> internal PMIx and libevent, and that should work. Is that correct?
>
> Yes - that is indeed correct!
>
>>
>> BTW, building 4.0.1rc1 completed successfully using external for all,
>> will start testing in near future.
>>>
>>> Cheers,
>>>
>>>
>>> Gilles
>>>
>> Thanks,
>> Dani_L.
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>> ___
>> users mailing list
>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
>> https://lists.open-mpi.org/mailman/listinfo/users
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Building PMIx and Slurm support

2019-03-11 Thread Gilles Gouaillardet

Passant,


I built a similar environment, and had no issue running a simple MPI 
program.



Can you please post your slurm script (I assume it uses srun to start 
the MPI app),


the output of

scontrol show config | grep Mpi

and the full output of your job ?


Cheers,


Gilles


On 3/12/2019 7:59 AM, Passant A. Hafez wrote:


Hello,


So we now have Slurm 18.08.6-2 compiled with PMIx 3.1.2

then I installed openmpi 4.0.0 with:

--with-slurm  --with-pmix=internal --with-libevent=internal 
--enable-shared --enable-

static  --with-x


(Following the thread, it was mentioned that building OMPI 4.0.0 with 
PMIx 3.1.2 will fail with PMIX_MODEX and PMIX_INFO_ARRAY errors, so I 
used internal PMIx)




The MPI program fails with:


*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[cn603-13-r:387088] Local abort before MPI_INIT completed completed 
successfully, but am not able to aggregate error messages, and not 
able to guarantee that all other processes were killed!



for each process, please advise! what's going wrong here?







All the best,
--
Passant A. Hafez | HPC Applications Specialist
KAUST Supercomputing Core Laboratory (KSL)
King Abdullah University of Science and Technology
Building 1, Al-Khawarizmi, Room 0123
Mobile : +966 (0) 55-247-9568
Mobile : +20 (0) 106-146-9644
Office : +966 (0) 12-808-0367

*From:* users  on behalf of Ralph H 
Castain 

*Sent:* Monday, March 4, 2019 5:29 PM
*To:* Open MPI Users
*Subject:* Re: [OMPI users] Building PMIx and Slurm support


On Mar 4, 2019, at 5:34 AM, Daniel Letai <mailto:d...@letai.org.il>> wrote:


Gilles,
On 3/4/19 8:28 AM, Gilles Gouaillardet wrote:

Daniel,


On 3/4/2019 3:18 PM, Daniel Letai wrote:


So unless you have a specific reason not to mix both, you might 
also give the internal PMIx a try.
Does this hold true for libevent too? Configure complains if 
libevent for openmpi is different than the one used for the other 
tools.




I am not exactly sure of which scenario you are running.

Long story short,

 - If you use an external PMIx, then you have to use an external 
libevent (otherwise configure will fail).


It must be the same one used by PMIx, but I am not sure configure 
checks that.


- If you use the internal PMIx, then it is up to you. you can either 
use the internal libevent, or an external one.


Thanks, that clarifies the issues I've experienced. Since PMIx 
doesn't have to be the same for server and nodes, I can compile slurm 
with external PMIx with system libevent, and compile openmpi with 
internal PMIx and libevent, and that should work. Is that correct?


Yes - that is indeed correct!



BTW, building 4.0.1rc1 completed successfully using external for all, 
will start testing in near future.


Cheers,


Gilles


Thanks,
Dani_L.

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] local rank to rank comms

2019-03-11 Thread Gilles Gouaillardet

Michael,


this is odd, I will have a look.

Can you confirm you are running on a single node ?


At first, you need to understand which component is used by Open MPI for 
communications.


There are several options here, and since I do not know how Open MPI was 
built, nor which dependencies are installed,


I can only list a few


- pml/cm uses mtl/psm2 => omnipath is used for both inter and intra node 
communications


- pml/cm uses mtl/ofi => libfabric is used for both inter and intra node 
communications. it definitely uses libpsm2 for inter node 
communications, and I do not know enough about the internals to tell how 
inter communications are handled


- pml/ob1 is used, I guess it uses btl/ofi for inter node communications 
and btl/vader for intra node communications (in that case the NIC device 
is not used for intra node communications


there could be other I am missing (does UCX support OmniPath ? could 
btl/ofi also be used for intra node communications ?)



mpirun --mca pml_base_verbose 10 --mca btl_base_verbose 10 --mca 
mtl_base_verbose 10 ...


should tell you what is used (feel free to compress and post the full 
output if you have some hard time understanding the logs)



Cheers,


Gilles

On 3/12/2019 1:41 AM, Michael Di Domenico wrote:

On Mon, Mar 11, 2019 at 12:09 PM Gilles Gouaillardet
 wrote:

You can force
mpirun --mca pml ob1 ...
And btl/vader (shared memory) will be used for intra node communications ... 
unless MPI tasks are from different jobs (read MPI_Comm_spawn())

if i run

mpirun -n 16 IMB-MPI1 alltoallv
things run fine, 12us on average for all ranks

if i run

mpirun -n 16 --mca pml ob1 IMB-MPI1 alltoallv
the program runs, but then it hangs at "List of benchmarks to run:
#Alltoallv"  and no tests run
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] local rank to rank comms

2019-03-11 Thread Gilles Gouaillardet
Michael,

You can

mpirun --mca pml_base_verbose 10 --mca btl_base_verbose 10 --mca 
mtl_base_verbose 10 ...

It might show that pml/cm and mtl/psm2 are used. In that case, then yes, the 
OmniPath library is used even for intra node communications. If this library is 
optimized for intra node, then it will internally uses shared memory instead of 
the NIC.


You can force

mpirun --mca pml ob1 ...


And btl/vader (shared memory) will be used for intra node communications ... 
unless MPI tasks are from different jobs (read MPI_Comm_spawn())

Cheers,

Gilles

Michael Di Domenico  wrote:
>i have a user that's claiming when two ranks on the same node want to
>talk with each other, they're using the NIC to talk rather then just
>talking directly.
>
>i've never had to test such a scenario.  is there a way for me to
>prove one way or another whether two ranks are talking through say the
>kernel (or however it actually works) or using the nic?
>
>i didn't set any flags when i compiled openmpi to change this.
>
>i'm running ompi 3.1, pmix 2.2.1, and slurm 18.05 running atop omnipath
>___
>users mailing list
>users@lists.open-mpi.org
>https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Building PMIx and Slurm support

2019-03-03 Thread Gilles Gouaillardet

Daniel,


On 3/4/2019 3:18 PM, Daniel Letai wrote:


So unless you have a specific reason not to mix both, you might also 
give the internal PMIx a try.
Does this hold true for libevent too? Configure complains if libevent 
for openmpi is different than the one used for the other tools.




I am not exactly sure of which scenario you are running.

Long story short,

 - If you use an external PMIx, then you have to use an external 
libevent (otherwise configure will fail).


   It must be the same one used by PMIx, but I am not sure configure 
checks that.


- If you use the internal PMIx, then it is up to you. you can either use 
the internal libevent, or an external one.



Cheers,


Gilles

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Building PMIx and Slurm support

2019-03-03 Thread Gilles Gouaillardet

Daniel,


keep in mind PMIx was designed with cross-version compatibility in mind,

so a PMIx 3.0.2 client (read Open MPI 4.0.0 app with the internal 3.0.2 
PMIx) should be able


to interact with a PMIx 3.1.2 server (read SLURM pmix plugin built on 
top of PMIx 3.1.2).


So unless you have a specific reason not to mix both, you might also 
give the internal PMIx a try.



The 4.0.1 release candidate 1 was released a few days ago, and based on 
the feedback we receive,


the final 4.0.1 should be released in a very near future.


Cheers,


Gilles

On 3/4/2019 1:08 AM, Daniel Letai wrote:


Sent from my iPhone

On 3 Mar 2019, at 16:31, Gilles Gouaillardet 
 wrote:


Daniel,

PMIX_MODEX and PMIX_INFO_ARRAY have been removed from PMIx 3.1.2, and
Open MPI 4.0.0 was not ready for this.

You can either use the internal PMIx (3.0.2), or try 4.0.1rc1 (with
the external PMIx 3.1.2) that was published a few days ago.

Thanks, will try that tomorrow. I can’t use internal due to Slurm 
dependency, but I will try the rc.

Any idea when 4.0.1 will be released?

FWIW, you are right using --with-pmix=external (and not using 
--with-pmix=/usr)


Cheers,

Gilles


On Sun, Mar 3, 2019 at 10:57 PM Daniel Letai  wrote:

Hello,


I have built the following stack :

centos 7.5 (gcc 4.8.5-28, libevent 2.0.21-4)
MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz built with --all 
--without-32bit (this includes ucx 1.5.0)

hwloc from centos 7.5 : 1.11.8-4.el7
pmix 3.1.2
slurm 18.08.5-2 built --with-ucx --with-pmix
openmpi 4.0.0 : configure --with-slurm --with-pmix=external 
--with-pmi --with-libevent=external --with-hwloc=external 
--with-knem=/opt/knem-1.1.3.90mlnx1 --with-hcoll=/opt/mellanox/hcoll


The configure part succeeds, however 'make' errors out with:

ext3x.c: In function 'ext3x_value_unload':

ext3x.c:1109:10: error: 'PMIX_MODEX' undeclared (first use in this 
function)



And same for 'PMIX_INFO_ARRAY'


However, both are declared in the 
opal/mca/pmix/pmix3x/pmix/include/pmix_common.h file.


opal/mca/pmix/ext3x/ext3x.c does include pmix_common.h but as a 
system include #include  , while ext3x.h includes it as 
a local include #include "pmix_common". Neither seem to pull from 
the correct path.



Regards,

Dani_L.


On 2/24/19 3:09 AM, Gilles Gouaillardet wrote:

Passant,

you have to manually download and apply
https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch
to PMIx 2.2.1
that should likely fix your problem.

As a side note, it is a bad practice to configure --with-FOO=/usr
since it might have some unexpected side effects.
Instead, you can replace

configure --with-slurm --with-pmix=/usr --with-pmi=/usr 
--with-libevent=/usr


with

configure --with-slurm --with-pmix=external --with-pmi 
--with-libevent=external


to be on the safe side I also invite you to pass --with-hwloc=external
to the configure command line


Cheers,

Gilles

On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez
 wrote:

Hello Gilles,

Here are some details:

Slurm 18.08.4

PMIx 2.2.1 (as shown in /usr/include/pmix_version.h)

Libevent 2.0.21

srun --mpi=list
srun: MPI types are...
srun: none
srun: openmpi
srun: pmi2
srun: pmix
srun: pmix_v2

Open MPI versions tested: 4.0.0 and 3.1.2


For each installation to be mentioned a different MPI Hello World 
program was compiled.
Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun 
--mpi=pmix program


File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with 
configure options:

--with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
and configure log:
Libevent support: external
PMIx support: External (2x)

File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with 
configure options:

--with-slurm --with-pmix
and configure log:
Libevent support: internal (external libevent version is less that 
internal version 2.0.22)

PMIx support: Internal

Tested also different installations for 3.1.2 and got errors similar 
to 400ext_2x2.out

(NOT-SUPPORTED in file event/pmix_event_registration.c at line 101)





All the best,
--
Passant A. Hafez | HPC Applications Specialist
KAUST Supercomputing Core Laboratory (KSL)
King Abdullah University of Science and Technology
Building 1, Al-Khawarizmi, Room 0123
Mobile : +966 (0) 55-247-9568
Mobile : +20 (0) 106-146-9644
Office : +966 (0) 12-808-0367


From: users  on behalf of Gilles 
Gouaillardet 

Sent: Saturday, February 23, 2019 5:17 PM
To: Open MPI Users
Subject: Re: [OMPI users] Building PMIx and Slurm support

Hi,

PMIx has cross-version compatibility, so as long as the PMIx library
used by SLURM is compatible with the one (internal or external) used
by Open MPI, you should be fine.
If you want to minimize the risk of cross-version incompatibility,
then I encourage you to use the same (and hence external) PMIx that
was used to build SLURM with Open MPI.

Can you tell a bit more than "it didn't work" ?
(Open MPI version, PMIx v

Re: [OMPI users] Building PMIx and Slurm support

2019-03-03 Thread Gilles Gouaillardet
Daniel,

PMIX_MODEX and PMIX_INFO_ARRAY have been removed from PMIx 3.1.2, and
Open MPI 4.0.0 was not ready for this.

You can either use the internal PMIx (3.0.2), or try 4.0.1rc1 (with
the external PMIx 3.1.2) that was published a few days ago.

FWIW, you are right using --with-pmix=external (and not using --with-pmix=/usr)

Cheers,

Gilles

On Sun, Mar 3, 2019 at 10:57 PM Daniel Letai  wrote:
>
> Hello,
>
>
> I have built the following stack :
>
> centos 7.5 (gcc 4.8.5-28, libevent 2.0.21-4)
> MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz built with --all 
> --without-32bit (this includes ucx 1.5.0)
> hwloc from centos 7.5 : 1.11.8-4.el7
> pmix 3.1.2
> slurm 18.08.5-2 built --with-ucx --with-pmix
> openmpi 4.0.0 : configure --with-slurm --with-pmix=external --with-pmi 
> --with-libevent=external --with-hwloc=external 
> --with-knem=/opt/knem-1.1.3.90mlnx1 --with-hcoll=/opt/mellanox/hcoll
>
> The configure part succeeds, however 'make' errors out with:
>
> ext3x.c: In function 'ext3x_value_unload':
>
> ext3x.c:1109:10: error: 'PMIX_MODEX' undeclared (first use in this function)
>
>
> And same for 'PMIX_INFO_ARRAY'
>
>
> However, both are declared in the 
> opal/mca/pmix/pmix3x/pmix/include/pmix_common.h file.
>
> opal/mca/pmix/ext3x/ext3x.c does include pmix_common.h but as a system 
> include #include  , while ext3x.h includes it as a local include 
> #include "pmix_common". Neither seem to pull from the correct path.
>
>
> Regards,
>
> Dani_L.
>
>
> On 2/24/19 3:09 AM, Gilles Gouaillardet wrote:
>
> Passant,
>
> you have to manually download and apply
> https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch
> to PMIx 2.2.1
> that should likely fix your problem.
>
> As a side note,  it is a bad practice to configure --with-FOO=/usr
> since it might have some unexpected side effects.
> Instead, you can replace
>
> configure --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
>
> with
>
> configure --with-slurm --with-pmix=external --with-pmi 
> --with-libevent=external
>
> to be on the safe side I also invite you to pass --with-hwloc=external
> to the configure command line
>
>
> Cheers,
>
> Gilles
>
> On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez
>  wrote:
>
> Hello Gilles,
>
> Here are some details:
>
> Slurm 18.08.4
>
> PMIx 2.2.1 (as shown in /usr/include/pmix_version.h)
>
> Libevent 2.0.21
>
> srun --mpi=list
> srun: MPI types are...
> srun: none
> srun: openmpi
> srun: pmi2
> srun: pmix
> srun: pmix_v2
>
> Open MPI versions tested: 4.0.0 and 3.1.2
>
>
> For each installation to be mentioned a different MPI Hello World program was 
> compiled.
> Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun --mpi=pmix 
> program
>
> File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with configure 
> options:
> --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
> and configure log:
> Libevent support: external
> PMIx support: External (2x)
>
> File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with configure 
> options:
> --with-slurm --with-pmix
> and configure log:
> Libevent support: internal (external libevent version is less that internal 
> version 2.0.22)
> PMIx support: Internal
>
> Tested also different installations for 3.1.2 and got errors similar to 
> 400ext_2x2.out
> (NOT-SUPPORTED in file event/pmix_event_registration.c at line 101)
>
>
>
>
>
> All the best,
> --
> Passant A. Hafez | HPC Applications Specialist
> KAUST Supercomputing Core Laboratory (KSL)
> King Abdullah University of Science and Technology
> Building 1, Al-Khawarizmi, Room 0123
> Mobile : +966 (0) 55-247-9568
> Mobile : +20 (0) 106-146-9644
> Office  : +966 (0) 12-808-0367
>
> 
> From: users  on behalf of Gilles 
> Gouaillardet 
> Sent: Saturday, February 23, 2019 5:17 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Building PMIx and Slurm support
>
> Hi,
>
> PMIx has cross-version compatibility, so as long as the PMIx library
> used by SLURM is compatible with the one (internal or external) used
> by Open MPI, you should be fine.
> If you want to minimize the risk of cross-version incompatibility,
> then I encourage you to use the same (and hence external) PMIx that
> was used to build SLURM with Open MPI.
>
> Can you tell a bit more than "it didn't work" ?
> (Open MPI version, PMIx version used by SLURM, PMIx version used by
> Open MPI, error message, ...)
>
> Cheers,
>
> Gilles
>
> On Sat, Feb 23, 2019 at 

Re: [OMPI users] Building PMIx and Slurm support

2019-02-24 Thread Gilles Gouaillardet
Passant,

The fix is included in PMIx 2.2.2

The bug is in a public header file, so you might indeed have to
rebuild the SLURM plugin for PMIx.
I did not check the SLURM sources though, so assuming PMIx was built
as a shared library, there is still a chance
it might work even if you do not rebuild the SLURM plugin. I'd rebuild
at least the SLURM plugin for PMIx to be on the safe side though.

Cheers,

Gilles

On Sun, Feb 24, 2019 at 4:07 PM Passant A. Hafez
 wrote:
>
> Thanks Gilles.
>
> So do we have to rebuild Slurm after applying this patch?
>
> Another question, is this fix included in the PMIx 2.2.2 
> https://github.com/pmix/pmix/releases/tag/v2.2.2 ?
>
>
>
>
> All the best,
>
>
> ____
> From: users  on behalf of Gilles 
> Gouaillardet 
> Sent: Sunday, February 24, 2019 4:09 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] Building PMIx and Slurm support
>
> Passant,
>
> you have to manually download and apply
> https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch
> to PMIx 2.2.1
> that should likely fix your problem.
>
> As a side note,  it is a bad practice to configure --with-FOO=/usr
> since it might have some unexpected side effects.
> Instead, you can replace
>
> configure --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
>
> with
>
> configure --with-slurm --with-pmix=external --with-pmi 
> --with-libevent=external
>
> to be on the safe side I also invite you to pass --with-hwloc=external
> to the configure command line
>
>
> Cheers,
>
> Gilles
>
> On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez
>  wrote:
> >
> > Hello Gilles,
> >
> > Here are some details:
> >
> > Slurm 18.08.4
> >
> > PMIx 2.2.1 (as shown in /usr/include/pmix_version.h)
> >
> > Libevent 2.0.21
> >
> > srun --mpi=list
> > srun: MPI types are...
> > srun: none
> > srun: openmpi
> > srun: pmi2
> > srun: pmix
> > srun: pmix_v2
> >
> > Open MPI versions tested: 4.0.0 and 3.1.2
> >
> >
> > For each installation to be mentioned a different MPI Hello World program 
> > was compiled.
> > Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun 
> > --mpi=pmix program
> >
> > File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with 
> > configure options:
> > --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
> > and configure log:
> > Libevent support: external
> > PMIx support: External (2x)
> >
> > File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with 
> > configure options:
> > --with-slurm --with-pmix
> > and configure log:
> > Libevent support: internal (external libevent version is less that internal 
> > version 2.0.22)
> > PMIx support: Internal
> >
> > Tested also different installations for 3.1.2 and got errors similar to 
> > 400ext_2x2.out
> > (NOT-SUPPORTED in file event/pmix_event_registration.c at line 101)
> >
> >
> >
> >
> >
> > All the best,
> > --
> > Passant A. Hafez | HPC Applications Specialist
> > KAUST Supercomputing Core Laboratory (KSL)
> > King Abdullah University of Science and Technology
> > Building 1, Al-Khawarizmi, Room 0123
> > Mobile : +966 (0) 55-247-9568
> > Mobile : +20 (0) 106-146-9644
> > Office  : +966 (0) 12-808-0367
> >
> > 
> > From: users  on behalf of Gilles 
> > Gouaillardet 
> > Sent: Saturday, February 23, 2019 5:17 PM
> > To: Open MPI Users
> > Subject: Re: [OMPI users] Building PMIx and Slurm support
> >
> > Hi,
> >
> > PMIx has cross-version compatibility, so as long as the PMIx library
> > used by SLURM is compatible with the one (internal or external) used
> > by Open MPI, you should be fine.
> > If you want to minimize the risk of cross-version incompatibility,
> > then I encourage you to use the same (and hence external) PMIx that
> > was used to build SLURM with Open MPI.
> >
> > Can you tell a bit more than "it didn't work" ?
> > (Open MPI version, PMIx version used by SLURM, PMIx version used by
> > Open MPI, error message, ...)
> >
> > Cheers,
> >
> > Gilles
> >
> > On Sat, Feb 23, 2019 at 9:46 PM Passant A. Hafez
> >  wrote:
> > >
> > >
> > > Good day everyone,
> > >
> > > I've trying to build and use the PMIx support for Open MPI but I tried 
> > > many things that I can list 

Re: [OMPI users] Building PMIx and Slurm support

2019-02-23 Thread Gilles Gouaillardet
Passant,

you have to manually download and apply
https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch
to PMIx 2.2.1
that should likely fix your problem.

As a side note,  it is a bad practice to configure --with-FOO=/usr
since it might have some unexpected side effects.
Instead, you can replace

configure --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr

with

configure --with-slurm --with-pmix=external --with-pmi --with-libevent=external

to be on the safe side I also invite you to pass --with-hwloc=external
to the configure command line


Cheers,

Gilles

On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez
 wrote:
>
> Hello Gilles,
>
> Here are some details:
>
> Slurm 18.08.4
>
> PMIx 2.2.1 (as shown in /usr/include/pmix_version.h)
>
> Libevent 2.0.21
>
> srun --mpi=list
> srun: MPI types are...
> srun: none
> srun: openmpi
> srun: pmi2
> srun: pmix
> srun: pmix_v2
>
> Open MPI versions tested: 4.0.0 and 3.1.2
>
>
> For each installation to be mentioned a different MPI Hello World program was 
> compiled.
> Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun --mpi=pmix 
> program
>
> File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with configure 
> options:
> --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
> and configure log:
> Libevent support: external
> PMIx support: External (2x)
>
> File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with configure 
> options:
> --with-slurm --with-pmix
> and configure log:
> Libevent support: internal (external libevent version is less that internal 
> version 2.0.22)
> PMIx support: Internal
>
> Tested also different installations for 3.1.2 and got errors similar to 
> 400ext_2x2.out
> (NOT-SUPPORTED in file event/pmix_event_registration.c at line 101)
>
>
>
>
>
> All the best,
> --
> Passant A. Hafez | HPC Applications Specialist
> KAUST Supercomputing Core Laboratory (KSL)
> King Abdullah University of Science and Technology
> Building 1, Al-Khawarizmi, Room 0123
> Mobile : +966 (0) 55-247-9568
> Mobile : +20 (0) 106-146-9644
> Office  : +966 (0) 12-808-0367
>
> 
> From: users  on behalf of Gilles 
> Gouaillardet 
> Sent: Saturday, February 23, 2019 5:17 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Building PMIx and Slurm support
>
> Hi,
>
> PMIx has cross-version compatibility, so as long as the PMIx library
> used by SLURM is compatible with the one (internal or external) used
> by Open MPI, you should be fine.
> If you want to minimize the risk of cross-version incompatibility,
> then I encourage you to use the same (and hence external) PMIx that
> was used to build SLURM with Open MPI.
>
> Can you tell a bit more than "it didn't work" ?
> (Open MPI version, PMIx version used by SLURM, PMIx version used by
> Open MPI, error message, ...)
>
> Cheers,
>
> Gilles
>
> On Sat, Feb 23, 2019 at 9:46 PM Passant A. Hafez
>  wrote:
> >
> >
> > Good day everyone,
> >
> > I've trying to build and use the PMIx support for Open MPI but I tried many 
> > things that I can list if needed, but with no luck.
> > I was able to test the PMIx client but when I used OMPI specifying srun 
> > --mpi=pmix it didn't work.
> >
> > So if you please advise me with the versions of each PMIx and Open MPI that 
> > should be working well with Slurm 18.08, it'd be great.
> >
> > Also, what is the difference between using internal vs external PMIx 
> > installations?
> >
> >
> >
> > All the best,
> >
> > --
> >
> > Passant A. Hafez | HPC Applications Specialist
> > KAUST Supercomputing Core Laboratory (KSL)
> > King Abdullah University of Science and Technology
> > Building 1, Al-Khawarizmi, Room 0123
> > Mobile : +966 (0) 55-247-9568
> > Mobile : +20 (0) 106-146-9644
> > Office  : +966 (0) 12-808-0367
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Building PMIx and Slurm support

2019-02-23 Thread Gilles Gouaillardet
Hi,

PMIx has cross-version compatibility, so as long as the PMIx library
used by SLURM is compatible with the one (internal or external) used
by Open MPI, you should be fine.
If you want to minimize the risk of cross-version incompatibility,
then I encourage you to use the same (and hence external) PMIx that
was used to build SLURM with Open MPI.

Can you tell a bit more than "it didn't work" ?
(Open MPI version, PMIx version used by SLURM, PMIx version used by
Open MPI, error message, ...)

Cheers,

Gilles

On Sat, Feb 23, 2019 at 9:46 PM Passant A. Hafez
 wrote:
>
>
> Good day everyone,
>
> I've trying to build and use the PMIx support for Open MPI but I tried many 
> things that I can list if needed, but with no luck.
> I was able to test the PMIx client but when I used OMPI specifying srun 
> --mpi=pmix it didn't work.
>
> So if you please advise me with the versions of each PMIx and Open MPI that 
> should be working well with Slurm 18.08, it'd be great.
>
> Also, what is the difference between using internal vs external PMIx 
> installations?
>
>
>
> All the best,
>
> --
>
> Passant A. Hafez | HPC Applications Specialist
> KAUST Supercomputing Core Laboratory (KSL)
> King Abdullah University of Science and Technology
> Building 1, Al-Khawarizmi, Room 0123
> Mobile : +966 (0) 55-247-9568
> Mobile : +20 (0) 106-146-9644
> Office  : +966 (0) 12-808-0367
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3

2019-02-20 Thread Gilles Gouaillardet

Ryan,


as Edgar explained, that could be a compiler issue (fwiw, I am unable to 
reproduce the bug)


You can build Open MPI again and pass --disable-builtin-atomics to the 
configure command line.



That being said, the "Alarm clock" message looks a bit suspicious.

Does it always occur at 20+ minutes elapsed ?

Is there some mechanism that automatically kills a job if it does not 
write anything to stdout for some time ?


A quick way to rule that out is to

srun -- mpi=pmi2 -p main -t 1:00:00 -n6 -N1 sleep 1800

and see if that completes or get killed with the same error message.


You can also run use mpirun instead of srun, and even run mpirun outside 
of slurm


(if your cluster policy allows it, you can for example use mpirun and 
run on the frontend node)



Cheers,


Gilles

On 2/21/2019 3:01 AM, Ryan Novosielski wrote:

Does it make any sense that it seems to work fine when OpenMPI and HDF5 are 
built with GCC 7.4 and GCC 8.2, but /not/ when they are built with 
RHEL-supplied GCC 4.8.5? That appears to be the scenario. For the GCC 4.8.5 
build, I did try an XFS filesystem and it didn’t help. GPFS works fine for 
either of the 7.4 and 8.2 builds.

Just as a reminder, since it was reasonably far back in the thread, what I’m 
doing is running the “make check” tests in HDF5 1.10.4, in part because users 
use it, but also because it seems to have a good test suite and I can therefore 
verify the compiler and MPI stack installs. I get very little information, 
apart from it not working and getting that “Alarm clock” message.

I originally suspected I’d somehow built some component of this with a 
host-specific optimization that wasn’t working on some compute nodes. But I 
controlled for that and it didn’t seem to make any difference.

--

|| \\UTGERS, |---*O*---
||_// the State  | Ryan Novosielski - novos...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\of NJ  | Office of Advanced Research Computing - MSB C630, Newark
  `'


On Feb 18, 2019, at 1:34 PM, Ryan Novosielski  wrote:

It didn’t work any better with XFS, as it happens. Must be something else. I’m 
going to test some more and see if I can narrow it down any, as it seems to me 
that it did work with a different compiler.

--

|| \\UTGERS, |---*O*---
||_// the State  | Ryan Novosielski - novos...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\of NJ  | Office of Advanced Research Computing - MSB C630, Newark
 `'


On Feb 18, 2019, at 12:23 PM, Gabriel, Edgar  wrote:

While I was working on something else, I let the tests run with Open MPI master 
(which is for parallel I/O equivalent to the upcoming v4.0.1  release), and 
here is what I found for the HDF5 1.10.4 tests on my local desktop:

In the testpar directory, there is in fact one test that fails for both ompio 
and romio321 in exactly the same manner.
I used 6 processes as you did (although I used mpirun directly  instead of 
srun...) From the 13 tests in the testpar directory, 12 pass correctly 
(t_bigio, t_cache, t_cache_image, testphdf5, t_filters_parallel, t_init_term, 
t_mpi, t_pflush2, t_pread, t_prestart, t_pshutdown, t_shapesame).

The one tests that officially fails ( t_pflush1) actually reports that it 
passed, but then throws message that indicates that MPI_Abort has been called, 
for both ompio and romio. I will try to investigate this test to see what is 
going on.

That being said, your report shows an issue in t_mpi, which passes without 
problems for me. This is however not GPFS, this was an XFS local file system. 
Running the tests on GPFS are on my todo list as well.

Thanks
Edgar




-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of
Gabriel, Edgar
Sent: Sunday, February 17, 2019 10:34 AM
To: Open MPI Users 
Subject: Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI
3.1.3

I will also run our testsuite and the HDF5 testsuite on GPFS, I have access to a
GPFS file system since recently, and will report back on that, but it will take 
a
few days.

Thanks
Edgar


-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of
Ryan Novosielski
Sent: Sunday, February 17, 2019 2:37 AM
To: users@lists.open-mpi.org
Subject: Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI
3.1.3

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

This is on GPFS. I'll try it on XFS to see if it makes any difference.

On 2/16/19 11:57 PM, Gilles Gouaillardet wrote:

Ryan,

What filesystem are you running on ?

Open MPI defaults to the ompio component, except on Lustre
filesystem where ROMIO is used. (if the issue is related to ROMIO,
that can explain why you did not see any difference, in that case,
you might want to try an other files

Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3

2019-02-18 Thread Gilles Gouaillardet

Edgar,


t_pflush1 does not call MPI_Finalize(), that is why there is an error 
message regardless ompio or romio is used.


I naively tried to call MPI_Finalize(), but it causes the program to hang.


Cheers,


Gilles

On 2/19/2019 2:23 AM, Gabriel, Edgar wrote:

While I was working on something else, I let the tests run with Open MPI master 
(which is for parallel I/O equivalent to the upcoming v4.0.1  release), and 
here is what I found for the HDF5 1.10.4 tests on my local desktop:

In the testpar directory, there is in fact one test that fails for both ompio 
and romio321 in exactly the same manner.
I used 6 processes as you did (although I used mpirun directly  instead of 
srun...) From the 13 tests in the testpar directory, 12 pass correctly 
(t_bigio, t_cache, t_cache_image, testphdf5, t_filters_parallel, t_init_term, 
t_mpi, t_pflush2, t_pread, t_prestart, t_pshutdown, t_shapesame).

The one tests that officially fails ( t_pflush1) actually reports that it 
passed, but then throws message that indicates that MPI_Abort has been called, 
for both ompio and romio. I will try to investigate this test to see what is 
going on.

That being said, your report shows an issue in t_mpi, which passes without 
problems for me. This is however not GPFS, this was an XFS local file system. 
Running the tests on GPFS are on my todo list as well.

Thanks
Edgar




-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of
Gabriel, Edgar
Sent: Sunday, February 17, 2019 10:34 AM
To: Open MPI Users 
Subject: Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI
3.1.3

I will also run our testsuite and the HDF5 testsuite on GPFS, I have access to a
GPFS file system since recently, and will report back on that, but it will take 
a
few days.

Thanks
Edgar


-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of
Ryan Novosielski
Sent: Sunday, February 17, 2019 2:37 AM
To: users@lists.open-mpi.org
Subject: Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI
3.1.3

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

This is on GPFS. I'll try it on XFS to see if it makes any difference.

On 2/16/19 11:57 PM, Gilles Gouaillardet wrote:

Ryan,

What filesystem are you running on ?

Open MPI defaults to the ompio component, except on Lustre
filesystem where ROMIO is used. (if the issue is related to ROMIO,
that can explain why you did not see any difference, in that case,
you might want to try an other filesystem (local filesystem or NFS
for example)\


Cheers,

Gilles

On Sun, Feb 17, 2019 at 3:08 AM Ryan Novosielski
 wrote:

I verified that it makes it through to a bash prompt, but I’m a
little less confident that something make test does doesn’t clear it.
Any recommendation for a way to verify?

In any case, no change, unfortunately.

Sent from my iPhone


On Feb 16, 2019, at 08:13, Gabriel, Edgar

wrote:

What file system are you running on?

I will look into this, but it might be later next week. I just
wanted to emphasize that we are regularly running the parallel
hdf5 tests with ompio, and I am not aware of any outstanding items
that do not work (and are supposed to work). That being said, I
run the tests manually, and not the 'make test'
commands. Will have to check which tests are being run by that.

Edgar


-Original Message- From: users
[mailto:users-boun...@lists.open-mpi.org] On Behalf Of Gilles
Gouaillardet Sent: Saturday, February 16, 2019 1:49 AM To: Open
MPI Users  Subject: Re:
[OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI
3.1.3

Ryan,

Can you

export OMPI_MCA_io=^ompio

and try again after you made sure this environment variable is
passed by srun to the MPI tasks ?

We have identified and fixed several issues specific to the
(default) ompio component, so that could be a valid workaround
until the next release.

Cheers,

Gilles

Ryan Novosielski  wrote:

Hi there,

Honestly don’t know which piece of this puzzle to look at or how
to get more

information for troubleshooting. I successfully built HDF5
1.10.4 with RHEL system GCC 4.8.5 and OpenMPI 3.1.3. Running the
“make check” in HDF5 is failing at the below point; I am using a
value of RUNPARALLEL='srun -- mpi=pmi2 -p main -t
1:00:00 -n6 -N1’ and have a SLURM that’s otherwise properly
configured.

Thanks for any help you can provide.

make[4]: Entering directory
`/scratch/novosirj/install-files/hdf5-1.10.4-build-

gcc-4.8-openmpi-3.1.3/testpar'

 Testing  t_mpi
 t_mpi  Test Log
 srun: job 84126610 queued and

waiting

for resources srun: job 84126610 has been allocated resources
srun: error: slepner023: tasks 0-5: Alarm clock 0.01user
0.00system 20:03.95elapsed 0%CPU (0avgtext+0avgdata
5152maxresident)k 0inputs+0outputs

(0major+1529minor)pagefaults

0swaps make[4]: *** [t_mpi.chkexe_] Error 1 make[4]: Leaving
directory
`/scratch/novosirj/i

Re: [OMPI users] Segfault with OpenMPI 4 and dynamic window

2019-02-17 Thread Gilles Gouaillardet

Thanks Bart,


I opened https://github.com/open-mpi/ompi/issues/6394 to track this 
issue, and we should follow-up there from now.



FWIW, I added a more minimal example, and a possible fix.


Cheers,


Gilles

On 2/18/2019 12:43 AM, Bart Janssens wrote:
I just tried on master (commit 
91d05f91e28d3614d8b5da707df2505d8564ecd3), the same crash still 
happens there.
On 16 Feb 2019, 17:15 +0100, Open MPI Users 
, wrote:


Probably not. I think this is now fixed. Might be worth trying master 
to verify.


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3

2019-02-16 Thread Gilles Gouaillardet
Ryan,

What filesystem are you running on ?

Open MPI defaults to the ompio component, except on Lustre filesystem
where ROMIO is used.
(if the issue is related to ROMIO, that can explain why you did not
see any difference,
in that case, you might want to try an other filesystem (local
filesystem or NFS for example)\


Cheers,

Gilles

On Sun, Feb 17, 2019 at 3:08 AM Ryan Novosielski  wrote:
>
> I verified that it makes it through to a bash prompt, but I’m a little less 
> confident that something make test does doesn’t clear it. Any recommendation 
> for a way to verify?
>
> In any case, no change, unfortunately.
>
> Sent from my iPhone
>
> > On Feb 16, 2019, at 08:13, Gabriel, Edgar  wrote:
> >
> > What file system are you running on?
> >
> > I will look into this, but it might be later next week. I just wanted to 
> > emphasize that we are regularly running the parallel hdf5 tests with ompio, 
> > and I am not aware of any outstanding items that do not work (and are 
> > supposed to work). That being said, I run the tests manually, and not the 
> > 'make test' commands. Will have to check which tests are being run by that.
> >
> > Edgar
> >
> >> -----Original Message-
> >> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Gilles
> >> Gouaillardet
> >> Sent: Saturday, February 16, 2019 1:49 AM
> >> To: Open MPI Users 
> >> Subject: Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI
> >> 3.1.3
> >>
> >> Ryan,
> >>
> >> Can you
> >>
> >> export OMPI_MCA_io=^ompio
> >>
> >> and try again after you made sure this environment variable is passed by 
> >> srun
> >> to the MPI tasks ?
> >>
> >> We have identified and fixed several issues specific to the (default) ompio
> >> component, so that could be a valid workaround until the next release.
> >>
> >> Cheers,
> >>
> >> Gilles
> >>
> >> Ryan Novosielski  wrote:
> >>> Hi there,
> >>>
> >>> Honestly don’t know which piece of this puzzle to look at or how to get 
> >>> more
> >> information for troubleshooting. I successfully built HDF5 1.10.4 with RHEL
> >> system GCC 4.8.5 and OpenMPI 3.1.3. Running the “make check” in HDF5 is
> >> failing at the below point; I am using a value of RUNPARALLEL='srun --
> >> mpi=pmi2 -p main -t 1:00:00 -n6 -N1’ and have a SLURM that’s otherwise
> >> properly configured.
> >>>
> >>> Thanks for any help you can provide.
> >>>
> >>> make[4]: Entering directory 
> >>> `/scratch/novosirj/install-files/hdf5-1.10.4-build-
> >> gcc-4.8-openmpi-3.1.3/testpar'
> >>> 
> >>> Testing  t_mpi
> >>> 
> >>> t_mpi  Test Log
> >>> 
> >>> srun: job 84126610 queued and waiting for resources
> >>> srun: job 84126610 has been allocated resources
> >>> srun: error: slepner023: tasks 0-5: Alarm clock 0.01user 0.00system
> >>> 20:03.95elapsed 0%CPU (0avgtext+0avgdata 5152maxresident)k
> >>> 0inputs+0outputs (0major+1529minor)pagefaults 0swaps
> >>> make[4]: *** [t_mpi.chkexe_] Error 1
> >>> make[4]: Leaving directory 
> >>> `/scratch/novosirj/install-files/hdf5-1.10.4-build-
> >> gcc-4.8-openmpi-3.1.3/testpar'
> >>> make[3]: *** [build-check-p] Error 1
> >>> make[3]: Leaving directory 
> >>> `/scratch/novosirj/install-files/hdf5-1.10.4-build-
> >> gcc-4.8-openmpi-3.1.3/testpar'
> >>> make[2]: *** [test] Error 2
> >>> make[2]: Leaving directory 
> >>> `/scratch/novosirj/install-files/hdf5-1.10.4-build-
> >> gcc-4.8-openmpi-3.1.3/testpar'
> >>> make[1]: *** [check-am] Error 2
> >>> make[1]: Leaving directory 
> >>> `/scratch/novosirj/install-files/hdf5-1.10.4-build-
> >> gcc-4.8-openmpi-3.1.3/testpar'
> >>> make: *** [check-recursive] Error 1
> >>>
> >>> --
> >>> 
> >>> || \\UTGERS,   
> >>> |---*O*---
> >>> ||_// the State | Ryan Novosielski - novos...@rutgers.edu
> >>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS 
> >>> Campus
> >>> ||  \\of NJ | Office of Advanced Research Computing - MSB C630, 
> >>> Newark
> >>>  `'
> >> ___
> >> users mailing list
> >> users@lists.open-mpi.org
> >> https://lists.open-mpi.org/mailman/listinfo/users
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] What's the right approach to run a singleton MPI+OpenMP process

2019-02-16 Thread Gilles Gouaillardet
Simone,

If you want to run a single MPI task, you can either
 - mpirun -np 1 ./a.out (this is the most standard option)
 - ./a.out (this is the singleton mode. Note a.out will fork an
orted daemon under the hood, this is necessary for example if your app
will MPI_Comm_spawn().
 - OMPI_MCA_ess_singleton_isolated=1 ./a.out This is the singleton
mode, and no orted daemon is spawned. this is faster but the app will
abort if it invokes for example MPI_Comm_spawn().

mpirun does set an affinity mask by default. The default mask is one
core per task is you run 2 MPI tasks or less, and a NUMA domain
otherwise.

Since you run a single MPI task and the default affinity mask is not a
good fit, you can either run in singleton mode (and consider isolated
singleton if it is a fit), or mpirun --bind-to none ... -np 1.

Running one OpenMP thread per core vs hyperthread has to do with the
OpenMP runtime and not Open MPI.

Cheers,

Gilles

On Sun, Feb 17, 2019 at 12:15 PM Simone Atzeni  wrote:
>
> Hi,
>
>
>
> For testing purposes I run some MPI+OpenMP benchmarks with `mpirun -np 1 
> ./a.out`, and I am using OpenMPI 3.1.3.
>
> As far as I understand, `mpirun` sets an affinity mask, and the OpenMP 
> runtime (in my case the LLVM OpenMP RT) respects this mask and only sees 1 
> physical core.
>
> In my case, I am running in a POWER8 which has 8 logical cores per physical 
> core. The OpenMP runtime in this case creates always the max number of 
> available logical cores in the machines (160 in my machine), but because of 
> the mask in this case will create 8 threads.
>
> All this threads runs in the same physical cores making the program slower 
> than if it would run each thread in a different physical core.
>
>
>
> So my question is, what’s the right way to run a single MPI process such that 
> the OpenMP threads can run in different physical cores independently from the 
> mask set by mpirun?
>
>
>
> I know about the option `--bind-to none` and using that all the cores in the 
> system become available and the OpenMP runtime uses all of them.
>
> Otherwise, doing some web search I read that a singleton MPI program should 
> be executed with ` OMPI_MCA_ess_singleton_isolated=1 ./a.out` without 
> `mpirun` at all, but I couldn’t find a good explanation of it.
>
>
>
> Is there anyone that could clarify this?
>
>
>
> Thank you!
>
> Simone
>
>
>
> 
> This email message is for the sole use of the intended recipient(s) and may 
> contain confidential information.  Any unauthorized review, use, disclosure 
> or distribution is prohibited.  If you are not the intended recipient, please 
> contact the sender by reply email and destroy all copies of the original 
> message.
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Segfault with OpenMPI 4 and dynamic window

2019-02-16 Thread Gilles Gouaillardet
Bart,

It  looks like a bug that involves the osc/rdma component.

Meanwhile, you can
mpirun --mca osc ^rdma ...

Cheers,

Gilles

On Sat, Feb 16, 2019 at 8:43 PM b...@bartjanssens.org
 wrote:
>
> Hi,
>
> Running the following test code on two processes:
>
> #include 
> #include 
> #include 
>
> #define N 2
>
> int main(int argc, char **argv)
> {
> int i, rank, num_procs, len, received[N], buf[N];
> MPI_Aint addrbuf[1], recvaddr[1];
> MPI_Win win, awin;
>
> MPI_Init(, );
> MPI_Comm_rank(MPI_COMM_WORLD, );
> MPI_Comm_size(MPI_COMM_WORLD, _procs);
>
> MPI_Win_create_dynamic(MPI_INFO_NULL, MPI_COMM_WORLD, );
> MPI_Win_attach(win, buf, sizeof(int)*N);
> MPI_Win_create(addrbuf, sizeof(MPI_Aint), sizeof(MPI_Aint), MPI_INFO_NULL, 
> MPI_COMM_WORLD, );
>
> MPI_Win_lock(MPI_LOCK_EXCLUSIVE, rank, 0, awin);
> MPI_Get_address(buf, [0]);
> MPI_Win_unlock(rank,awin);
>
> if(rank == 0)
> {
> printf("Process %d is waiting for debugger attach\n", getpid());
> sleep(15);
> }
>
> MPI_Barrier(MPI_COMM_WORLD);
>
> if(rank == 0)
> {
> for(int r = 0; r != N; ++r)
> {
> MPI_Win_lock(MPI_LOCK_EXCLUSIVE, r, 0, awin);
> MPI_Get(recvaddr, 1, MPI_AINT, r, 0, 1, MPI_AINT, awin);
> MPI_Win_unlock(r, awin);
> MPI_Win_lock(MPI_LOCK_EXCLUSIVE, r, 0, win);
> MPI_Get(received, N, MPI_INT, r, recvaddr[0], N, MPI_INT, win);
> printf("First value from %d is %d\n", r, received[0]);
> MPI_Win_unlock(r, win);
> }
> }
>
> MPI_Barrier(MPI_COMM_WORLD);
>
> MPI_Win_free();
> MPI_Finalize();
> return 0;
> }
>
>
> results in a crash with this backtrace (starting at the second MPI_Get line 
> in my code above):
>
> #0  mca_btl_vader_get_cma (btl=0x7f44888d0220 , endpoint=0x0, 
> local_address=0x74a13c18, remote_address=, 
> local_handle=0x0,
> remote_handle=, size=8, flags=0, order=255, 
> cbfunc=0x7f4488231250 , cbcontext=0x555d01e1c060, 
> cbdata=0x0) at btl_vader_get.c:95
> #1  0x7f44882308c1 in ompi_osc_rdma_get_contig 
> (sync=sync@entry=0x555d01e1be90, peer=peer@entry=0x555d01e16f10, 
> source_address=,
> source_address@entry=140737297595424, 
> source_handle=source_handle@entry=0x7f448a747180, target_buffer= out>, target_buffer@entry=0x74a13c18, size=size@entry=8,
> request=) at osc_rdma_comm.c:698
> #2  0x7f44882354b6 in ompi_osc_rdma_master (alloc_reqs=true, 
> rdma_fn=0x7f4488230610 , max_rdma_len= out>, request=0x555d01e1c060,
> remote_datatype=0x555d0004a2c0 , remote_count= out>, remote_handle=0x7f448a747180, remote_address=, 
> peer=,
> local_datatype=0x555d0004a2c0 , local_count= out>, local_address=0x74a13c18, sync=0x555d01e1be90) at 
> osc_rdma_comm.c:349
> #3  ompi_osc_rdma_get_w_req (request=0x0, source_datatype=0x555d0004a2c0 
> , source_count=, source_disp=, 
> peer=,
> origin_datatype=0x555d0004a2c0 , origin_count= out>, origin_addr=0x74a13c18, sync=0x555d01e1be90) at osc_rdma_comm.c:803
> #4  ompi_osc_rdma_get (origin_addr=0x74a13c18, origin_count= out>, origin_datatype=0x555d0004a2c0 , source_rank= out>,
> source_disp=, source_count=, 
> source_datatype=0x555d0004a2c0 , win=0x555d01e0aae0) at 
> osc_rdma_comm.c:880
> #5  0x7f448b404b6b in PMPI_Get (origin_addr=0x74a13c18, 
> origin_count=2, origin_datatype=0x555d0004a2c0 , 
> target_rank=,
> target_disp=, target_count=, 
> target_datatype=0x555d0004a2c0 , win=0x555d01e0aae0) at 
> pget.c:81
> #6  0x555d00047430 in main (argc=1, argv=0x74a13d18) at 
> onesided_crash_report.c:41
>
> On OpenMPI 3.1.3 the code works fine. Am I doing something wrong, or is this 
> a bug?
>
> Kind regards,
>
> Bart ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] HDF5 1.10.4 "make check" problems w/OpenMPI 3.1.3

2019-02-15 Thread Gilles Gouaillardet
Ryan,

Can you

export OMPI_MCA_io=^ompio

and try again after you made sure this environment variable is passed by srun 
to the MPI tasks ?

We have identified and fixed several issues specific to the (default) ompio 
component, so that could be a valid workaround until the next release.

Cheers,

Gilles

Ryan Novosielski  wrote:
>Hi there,
>
>Honestly don’t know which piece of this puzzle to look at or how to get more 
>information for troubleshooting. I successfully built HDF5 1.10.4 with RHEL 
>system GCC 4.8.5 and OpenMPI 3.1.3. Running the “make check” in HDF5 is 
>failing at the below point; I am using a value of RUNPARALLEL='srun --mpi=pmi2 
>-p main -t 1:00:00 -n6 -N1’ and have a SLURM that’s otherwise properly 
>configured.
>
>Thanks for any help you can provide.
>
>make[4]: Entering directory 
>`/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-openmpi-3.1.3/testpar'
>
>Testing  t_mpi
>
>t_mpi  Test Log
>
>srun: job 84126610 queued and waiting for resources
>srun: job 84126610 has been allocated resources
>srun: error: slepner023: tasks 0-5: Alarm clock
>0.01user 0.00system 20:03.95elapsed 0%CPU (0avgtext+0avgdata 5152maxresident)k
>0inputs+0outputs (0major+1529minor)pagefaults 0swaps
>make[4]: *** [t_mpi.chkexe_] Error 1
>make[4]: Leaving directory 
>`/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-openmpi-3.1.3/testpar'
>make[3]: *** [build-check-p] Error 1
>make[3]: Leaving directory 
>`/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-openmpi-3.1.3/testpar'
>make[2]: *** [test] Error 2
>make[2]: Leaving directory 
>`/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-openmpi-3.1.3/testpar'
>make[1]: *** [check-am] Error 2
>make[1]: Leaving directory 
>`/scratch/novosirj/install-files/hdf5-1.10.4-build-gcc-4.8-openmpi-3.1.3/testpar'
>make: *** [check-recursive] Error 1
>
>--
>
>|| \\UTGERS,|---*O*---
>||_// the State | Ryan Novosielski - novos...@rutgers.edu
>|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
>||  \\of NJ | Office of Advanced Research Computing - MSB C630, 
>Newark
>   `'
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] v3.1.3 cannot suppress load errors

2019-02-11 Thread Gilles Gouaillardet

Jingchao,


The error message is not coming from Open MPI but from the PMIx component.


adding the following line in pmix-mca-params.conf should do the trick


mca_base_component_show_load_errors=0


Cheers,


Gilles

On 2/12/2019 7:31 AM, Jingchao Zhang wrote:


Hi,


We have both psm and psm2 interfaces on our cluster. Since show load 
errors was default to true from the v2.* series, we have been setting 
"mca_base_component_show_load_errors=0" in the openmpi-mca-params.conf 
file to suppress the load errors. But in version v3.1.3, this only 
works for the libpsm2 interface.



On a psm2 network, with mca_base_component_show_load_errors=0, no load 
errors were shown. But on a psm network, with 
mca_base_component_show_load_errors=0, I got


"pmix_mca_base_component_repository_open: unable to open mca_pnet_opa: 
libpsm2.so.2: cannot open shared object file: No such file or 
directory (ignored)".



Is there a new flag I should use?


Thanks,

Jingchao





___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] OpenMPI 3 without network connection

2019-01-28 Thread Gilles Gouaillardet

Patrick,


I double checked the code, and indeed, mpirun should have automatically 
felt back


on the loopback interface (and mpirun should have worked)

The virbr0 interface prevented that and this is a bug I fixed in 
https://github.com/open-mpi/ompi/pull/6315



Future releases of Open MPI will include this fix, meanwhile, you can 
either remove the virbr0 interface


or use the workaround I previously described


Cheers,


Gilles

On 1/29/2019 1:56 PM, Gilles Gouaillardet wrote:

Patrick,

The root cause is we do not include the localhost interface by default 
for OOB communications.



You should be able to run with

mpirun --mca oob_tcp_if_include lo -np 4 hostname


Cheers,

Gilles

On 1/28/2019 11:02 PM, Patrick Bégou wrote:


Hi,

I fall in a strange problem with OpenMPI 3.1 installed on a CentOS7 
laptop. If no  network is available I cannot launch a local mpi job 
on the laptop:


bash-4.2$ mpirun -np 4 hostname
-- 

No network interfaces were found for out-of-band communications. We 
require

at least one available network for out-of-band messaging.
-- 



OpenMPI is built localy with

    Open MPI: 3.1.3rc1
  Open MPI repo revision: v3.1.2-78-gc8e9819
  Configure command line: '--prefix=/opt/GCC73/openmpi31x'
'--enable-mpirun-prefix-by-default'
  '--disable-dlopen' 
'--enable-mca-no-build=openib'

  '--without-verbs' '--enable-mpi-cxx'
  '--without-slurm' 
'--enable-mpi-thread-multiple'


I've tested some btl setup found with google but none solve the problem.

bash-4.2$ mpirun -np 4 -mca btl ^tcp hostname

or

bash-4.2$ mpirun -np 4 -mca btl vader,self hostname

Sarting a wifi connection (when it is available):

bash-4.2$ mpirun -np 4 hostname
localhost.localdomain
localhost.localdomain
localhost.localdomain
localhost.localdomain

Any suggestion is welcome

Patrick


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 3 without network connection

2019-01-28 Thread Gilles Gouaillardet

Patrick,

The root cause is we do not include the localhost interface by default 
for OOB communications.



You should be able to run with

mpirun --mca oob_tcp_if_include lo -np 4 hostname


Cheers,

Gilles

On 1/28/2019 11:02 PM, Patrick Bégou wrote:


Hi,

I fall in a strange problem with OpenMPI 3.1 installed on a CentOS7 
laptop. If no  network is available I cannot launch a local mpi job on 
the laptop:


bash-4.2$ mpirun -np 4 hostname
--
No network interfaces were found for out-of-band communications. We 
require

at least one available network for out-of-band messaging.
--

OpenMPI is built localy with

    Open MPI: 3.1.3rc1
  Open MPI repo revision: v3.1.2-78-gc8e9819
  Configure command line: '--prefix=/opt/GCC73/openmpi31x'
'--enable-mpirun-prefix-by-default'
  '--disable-dlopen' 
'--enable-mca-no-build=openib'

  '--without-verbs' '--enable-mpi-cxx'
  '--without-slurm' '--enable-mpi-thread-multiple'

I've tested some btl setup found with google but none solve the problem.

bash-4.2$ mpirun -np 4 -mca btl ^tcp hostname

or

bash-4.2$ mpirun -np 4 -mca btl vader,self hostname

Sarting a wifi connection (when it is available):

bash-4.2$ mpirun -np 4 hostname
localhost.localdomain
localhost.localdomain
localhost.localdomain
localhost.localdomain

Any suggestion is welcome

Patrick


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 3 without network connection

2019-01-28 Thread Gilles Gouaillardet
Patrick,

Does “no network is available” means the lo interface (localhost 127.0.0.1)
is not even available ?

Cheers,

Gilles

On Monday, January 28, 2019, Patrick Bégou <
patrick.be...@legi.grenoble-inp.fr> wrote:

> Hi,
>
> I fall in a strange problem with OpenMPI 3.1 installed on a CentOS7
> laptop. If no  network is available I cannot launch a local mpi job on the
> laptop:
>
> bash-4.2$ mpirun -np 4 hostname
> --
> No network interfaces were found for out-of-band communications. We require
> at least one available network for out-of-band messaging.
> --
>
> OpenMPI is built localy with
>
> Open MPI: 3.1.3rc1
>   Open MPI repo revision: v3.1.2-78-gc8e9819
>   Configure command line: '--prefix=/opt/GCC73/openmpi31x'
>   '--enable-mpirun-prefix-by-default'
>   '--disable-dlopen' '--enable-mca-no-build=openib'
>   '--without-verbs' '--enable-mpi-cxx'
>   '--without-slurm' '--enable-mpi-thread-multiple'
>
> I've tested some btl setup found with google but none solve the problem.
> bash-4.2$ mpirun -np 4 -mca btl ^tcp hostname
>
> or
>
> bash-4.2$ mpirun -np 4 -mca btl vader,self hostname
>
> Sarting a wifi connection (when it is available):
>
> bash-4.2$ mpirun -np 4 hostname
> localhost.localdomain
> localhost.localdomain
> localhost.localdomain
> localhost.localdomain
>
> Any suggestion is welcome
>
> Patrick
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] rmaps_base_oversubscribe Option in Open MPI 4.0

2019-01-27 Thread Gilles Gouaillardet

Ben,


This is a bug that will be fixed in 4.0.1 (it is already fixed in the 
v4.0.x branch)


meanwhile, you can add

rmaps_base_mapping_policy=numa:OVERSUBSCRIBE

in your openmpi-mca-params.conf.


Note the default policy is to bind to NUMA domain if there are more than 
two MPI tasks, and bind to core otherwise,


so this option is not strictly equivalent to --oversubscribe.


Cheers,


Gilles

On 1/26/2019 2:47 AM, Benjamin Brock wrote:

I used to be able to (e.g. in Open MPI 3.1) put the line

rmaps_base_oversubscribe = true

in my `openmpi-mca-params.conf`, and this would enable 
oversubscription by default.  In 4.0.0, it appears that this option 
doesn't work anymore, and I have to use `--oversubscribe`.


Am I missing something, or has the parameter name changed?

Ben

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Open MPI installation problem

2019-01-25 Thread Gilles Gouaillardet
Great point from David !

As a side note, you can
configure --enable-mpirun-prefix-by-default ... && make install

If you choose to do so, you will not have to set LD_LIBRARY_PATH since
it will be "built in" the Open MPI binaries/libraries (via the -rpath
linker option)

Cheers,

Gilles

On Sat, Jan 26, 2019 at 1:50 AM Shrader, David Lee via users
 wrote:
>
> The error about not finding shared libraries can be fixed by adding the 
> proper path to LD_LIBRARY_PATH, which Ralph has already mentioned. If Open 
> MPI is installed in $HOME/openmpi, LD_LIBRARY_PATH needs to be set like this:
>
> export LD_LIBRARY_PATH=$HOME/openmpi/lib:$LD_LIBRARY_PATH
>
> The mpi libraries are not in the base install directory, but in a lib 
> directory inside the installation directory.
>
> Thanks,
> David
>
>
>
>
> From: gil...@rist.or.jp 
> Date: Friday, Jan 25, 2019, 8:42 AM
> To: Open MPI Users 
> Subject: Re: [OMPI users] Open MPI installation problem
>
> Serdar,
>
>
>
> you need to export PATH and LD_LIBRARY_PATH in your .bashrc
>
>
>
> (e.g. export PATH=$HOME/openmpi/bin:$PATH)
>
>
>
> Also, make sure you built your application with Open MPI installed in 
> $HOME/openmpi
>
>
>
>
>
> Cheers,
>
>
>
> Gilles
>
>
>
> - Original Message -
>
> Hi folks,
>
> After installing OpenMPI, I executed these lines
>
> echo 'PATH=$HOME/openmpi/bin:$PATH' >> ~/.bashrc
> echo 'LD_LIBRARY_PATH=$HOME/openmpi/' >> ~/.bashrc
> source. bashrc
>
> and run  a simple file by using
>
> mpirun np 1 helloworld
>
> The error message is
>
> helloworld: error while loading shared libraries: libmpi_cxx.so.1: cannot 
> open shared object file: No such file or directory
> --
> mpirun noticed that the job aborted, but has no info as to the process
> that caused that situation.
> --
>  It is about inaccurate linking the libraries but I could not fix it. When i 
> run ldd helloworld, this appears
>
> linux-vdso.so.1 (0x7fff8f2d2000)
> libmpi_cxx.so.1 => not found
> libmpi.so.1 => not found
> libm.so.6 => /lib64/libm.so.6 (0x7fbc21e55000)
> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x7fbc21acb000)
> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x7fbc218b3000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x7fbc21696000)
> libc.so.6 => /lib64/libc.so.6 (0x7fbc212f1000)
> /lib64/ld-linux-x86-64.so.2 (0x7fbc22152000)
>
> Do you have any idea to fix it ?
>
> Best,
> Serdar
>
>
> Serdar Hiçdurmaz , 23 Oca 2019 Çar, 16:26 
> tarihinde şunu yazdı:
>>
>> Thanks Ralph. It worked.
>>
>> Serdar
>>
>> Ralph H Castain , 23 Oca 2019 Çar, 15:48 tarihinde şunu 
>> yazdı:
>>>
>>> Your PATH and LD_LIBRARY_PATH setting is incorrect. You installed OMPI into 
>>> $HOME/openmpi, so you should have done:
>>>
>>> PATH=$HOME/openmpi/bin:$PATH
>>> LD_LIBRARY_PATH=$HOME/openmpi/lib:$LD_LIBRARY_PATH
>>>
>>> Ralph
>>>
>>>
>>> On Jan 23, 2019, at 6:36 AM, Serdar Hiçdurmaz  
>>> wrote:
>>>
>>> Hi All,
>>>
>>> I try to install Open MPI, which is prerequiste for liggghts (DEM 
>>> software). Some info about my current linux version :
>>>
>>> NAME="SLED"
>>> VERSION="12-SP3"
>>> VERSION_ID="12.3"
>>> PRETTY_NAME="SUSE Linux Enterprise Desktop 12 SP3"
>>> ID="sled"
>>>
>>> I installed Open MPI 1.6 by typing
>>>
>>> ./configure --prefix=$HOME/openmpi
>>> make all
>>> make install
>>>
>>> Here, it is discussed that openmpi 1.6 is compatible with OpenSuse 12.3 
>>> https://public.kitware.com/pipermail/paraview/2014-February/030487.html 
>>> https://build.opensuse.org/package/show/openSUSE:12.3/openmpi
>>>
>>> To add OpenMPI to my path and LD_LIBRARY_PATH, I execute the following 
>>> comands on terminal:
>>>
>>> export PATH=$PATH:/usr/lib64/mpi/gcc/openmpi/bin
>>> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/mpi/gcc/openmpi/lib64
>>>
>>> Then, in /liggghts/src directory, I execute make auto, this appears :
>>>
>>> Creating list of contact models completed.
>>> make[1]: Entering directory 
>>> '/home/serdarhd/liggghts/LIGGGHTS-PUBLIC/src/Obj_auto'
>>> Makefile:456: *** 'Could not compile a simple MPI example. Test was done 
>>> with MPI_INC="" and MPICXX="mpicxx"'. Stop.
>>> make[1]: Leaving directory 
>>> '/home/serdarhd/liggghts/LIGGGHTS-PUBLIC/src/Obj_auto'
>>> Makefile:106: recipe for target 'auto' failed
>>> make: *** [auto] Error 2
>>>
>>> Do you have any idea what the problem is here ? I went through the 
>>> "makefile" but it looks like quite complicated as linux beginner like me.
>>>
>>> Thanks in advance. Regards,
>>>
>>> Serdar
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>>>
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>
> 

Re: [OMPI users] Help Getting Started with Open MPI and PMIx and UCX

2019-01-19 Thread Gilles Gouaillardet
Matt,

There are two ways of using PMIx

- if you use mpirun, then the MPI app (e.g. the PMIx client) will talk
to mpirun and orted daemons (e.g. the PMIx server)
- if you use SLURM srun, then the MPI app will directly talk to the
PMIx server provided by SLURM. (note you might have to srun
--mpi=pmix_v2 or something)

In the former case, it does not matter whether you use the embedded or
external PMIx.
In the latter case, Open MPI and SLURM have to use compatible PMIx
libraries, and you can either check the cross-version compatibility
matrix,
or build Open MPI with the same PMIx used by SLURM to be on the safe
side (not a bad idea IMHO).


Regarding the hang, I suggest you try different things
- use mpirun in a SLURM job (e.g. sbatch instead of salloc so mpirun
runs on a compute node rather than on a frontend node)
- try something even simpler such as mpirun hostname (both with sbatch
and salloc)
- explicitly specify the network to be used for the wire-up. you can
for example mpirun --mca oob_tcp_if_include 192.168.0.0/24 if this is
the network subnet by which all the nodes (e.g. compute nodes and
frontend node if you use salloc) communicate.


Cheers,

Gilles

On Sat, Jan 19, 2019 at 3:31 AM Matt Thompson  wrote:
>
> On Fri, Jan 18, 2019 at 1:13 PM Jeff Squyres (jsquyres) via users 
>  wrote:
>>
>> On Jan 18, 2019, at 12:43 PM, Matt Thompson  wrote:
>> >
>> > With some help, I managed to build an Open MPI 4.0.0 with:
>>
>> We can discuss each of these params to let you know what they are.
>>
>> > ./configure --disable-wrapper-rpath --disable-wrapper-runpath
>>
>> Did you have a reason for disabling these?  They're generally good things.  
>> What they do is add linker flags to the wrapper compilers (i.e., mpicc and 
>> friends) that basically put a default path to find libraries at run time 
>> (that can/will in most cases override LD_LIBRARY_PATH -- but you can 
>> override these linked-in-default-paths if you want/need to).
>
>
> I've had these in my Open MPI builds for a while now. The reason was one of 
> the libraries I need for the climate model I work on went nuts if both of 
> them weren't there. It was originally the rpath one but then eventually (Open 
> MPI 3?) I had to add the runpath one. But I have been updating the libraries 
> more aggressively recently (due to OS upgrades) so it's possible this is no 
> longer needed.
>
>>
>>
>> > --with-psm2
>>
>> Ensure that Open MPI can include support for the PSM2 library, and abort 
>> configure if it cannot.
>>
>> > --with-slurm
>>
>> Ensure that Open MPI can include support for SLURM, and abort configure if 
>> it cannot.
>>
>> > --enable-mpi1-compatibility
>>
>> Add support for MPI_Address and other MPI-1 functions that have since been 
>> deleted from the MPI 3.x specification.
>>
>> > --with-ucx
>>
>> Ensure that Open MPI can include support for UCX, and abort configure if it 
>> cannot.
>>
>> > --with-pmix=/usr/nlocal/pmix/2.1
>>
>> Tells Open MPI to use the PMIx that is installed at /usr/nlocal/pmix/2.1 
>> (instead of using the PMIx that is bundled internally to Open MPI's source 
>> code tree/expanded tarball).
>>
>> Unless you have a reason to use the external PMIx, the internal/bundled PMIx 
>> is usually sufficient.
>
>
> Ah. I did not know that. I figured if our SLURM was built linked to a 
> specific PMIx v2 that I should build Open MPI with the same PMIx. I'll build 
> an Open MPI 4 without specifying this.
>
>>
>>
>> > --with-libevent=/usr
>>
>> Same as previous; change "pmix" to "libevent" (i.e., use the external 
>> libevent instead of the bundled libevent).
>>
>> > CC=icc CXX=icpc FC=ifort
>>
>> Specify the exact compilers to use.
>>
>> > The MPI 1 is because I need to build HDF5 eventually and I added psm2 
>> > because it's an Omnipath cluster. The libevent was probably a red herring 
>> > as libevent-devel wasn't installed on the system. It was eventually, and I 
>> > just didn't remove the flag. And I saw no errors in the build!
>>
>> Might as well remove the --with-libevent if you don't need it.
>>
>> > However, I seem to have built an Open MPI that doesn't work:
>> >
>> > (1099)(master) $ mpirun --version
>> > mpirun (Open MPI) 4.0.0
>> >
>> > Report bugs to http://www.open-mpi.org/community/help/
>> > (1100)(master) $ mpirun -np 4 ./helloWorld.mpi3.SLES12.OMPI400.exe
>> >
>> > It just sits there...forever. Can the gurus here help me figure out what I 
>> > managed to break? Perhaps I added too much to my configure line? Not 
>> > enough?
>>
>> There could be a few things going on here.
>>
>> Are you running inside a SLURM job?  E.g., in a "salloc" job, or in an 
>> "sbatch" script?
>
>
> I have salloc'd 8 nodes of 40 cores each. Intel MPI 18 and 19 work just fine 
> (as you'd hope on an Omnipath cluster), but for some reason Open MPI is 
> twitchy on this cluster. I once managed to get Open MPI 3.0.1 working (a few 
> months ago), and it had some interesting startup scaling I liked (slow at low 
> core count, but getting 

Re: [OMPI users] Fwd: Minimum time between MPI_Bcast or MPI_Reduce calls?

2019-01-18 Thread Gilles Gouaillardet
Jeff,

that could be a copy/paste error and/or an email client issue.

The syntax is
mpirun --mca variable value ...

(short hyphen, short hyphen, m, c, a)

The error message is about the missing —-mca executable
(long hyphen, short hyphen, m, c, a)

This is most likely the root cause of this issue.

An other option is to set these parameters via the environment
export OMPI_MCA_coll_sync_priority=100
export OMPI_MCA_coll_sync_barrier_after=10
and then invoke mpirun without the --mca options.

Cheers,

Gilles

On Sat, Jan 19, 2019 at 11:28 AM Jeff Wentworth via users
 wrote:
>
> Hi,
>
> Thanks for the quick response.  But it looks like I am missing something 
> because neither -mca nor --mca is being recognized by my mpirun command.
>
> % mpirun --mca coll_sync_priority 100 --mca coll_sync_barrier_after 10 -q -np 
> 2 a.out
> --
> mpirun was unable to find the specified executable file, and therefore
> did not launch the job.  This error was first reported for process
> rank 0; it may have occurred for other processes as well.
>
> NOTE: A common cause for this error is misspelling a mpirun command
>   line parameter option (remember that mpirun interprets the first
>   unrecognized command line token as the executable).
>
> Node:   mia
> Executable: —-mca
> --
> 4 total processes failed to start
>
> % which mpirun
> /usr/local/bin/mpirun
> % ls -l /usr/local/bin/mpirun
> lrwxrwxrwx. 1 root root 7 Jan 15 20:50 /usr/local/bin/mpirun -> orterun
>
> jw2002
>
> 
> On Fri, 1/18/19, Nathan Hjelm via users  wrote:
>
>  Subject: [OMPI users] Fwd: Minimum time between MPI_Bcast or MPI_Reduce 
> calls?
>  To: "Open MPI Users" 
>  Cc: "Nathan Hjelm" 
>  Date: Friday, January 18, 2019, 9:00 PM
>
>
>  Since neither bcast nor reduce acts as
>  a barrier it is possible to run out of resources if either
>  of these calls (or both) are used in a tight loop. The sync
>  coll component exists for this scenario. You can enable it
>  by  adding the following to mpirun (or setting these
>  variables through the environment or a file):
>
>  —mca coll_sync_priority 100 —mca
>  coll_sync_barrier_after 10
>
>
>  This will effectively throttle the
>  collective calls for you. You can also change the reduce to
>  an allreduce.
>
>
>  -Nathan
>
>  > On Jan 18, 2019, at 6:31 PM, Jeff
>  Wentworth via users 
>  wrote:
>  >
>  > Greetings everyone,
>  >
>  > I have a scientific code using
>  Open MPI (v3.1.3) that seems to work fine when MPI_Bcast()
>  and MPI_Reduce() calls are well spaced out in time.
>  Yet if the time between these calls is short, eventually one
>  of the nodes hangs at some random point, never returning
>  from the broadcast or reduce call.  Is there some
>  minimum time between calls that needs to be obeyed in order
>  for Open MPI to process these reliably?
>  >
>  > The reason this has come up is
>  because I am trying to run in a multi-node environment some
>  established acceptance tests in order to verify that the
>  Open MPI configured version of the code yields the same
>  baseline result as the original single node version of the
>  code.  These acceptance tests must pass in order for
>  the code to be considered validated and deliverable to the
>  customer.  One of these acceptance tests that hangs
>  does involve 90 broadcasts and 90 reduces in a short period
>  of time (less than .01 cpu sec), as in:
>  >
>  > Broadcast #89 in
>  >  Broadcast #89 out 8 bytes
>  >  Calculate angle #89
>  >  Reduce #89 in
>  >  Reduce #89 out 208 bytes
>  > Write result #89 to file on
>  service node
>  > Broadcast #90 in
>  >  Broadcast #90 out 8 bytes
>  >  Calculate angle #89
>  >  Reduce #90 in
>  >  Reduce #90 out 208 bytes
>  > Write result #90 to file on
>  service node
>  >
>  > If I slow down the above
>  acceptance test, for example by running it under valgrind,
>  then it runs to completion and yields the correct
>  result.  So it seems to suggest that something internal
>  to Open MPI is getting swamped.  I understand that
>  these acceptance tests might be pushing the limit, given
>  that they involve so many short calculations combined with
>  frequent, yet tiny, transfers of data among nodes.
>  >
>  > Would it be worthwhile for me to
>  enforce with some minimum wait time between the MPI calls,
>  say 0.01 or 0.001 sec via nanosleep()?  The only time
>  it would matter would be when acceptance tests are run, as
>  the situation doesn't arise when beefier runs are performed.
>
>  >
>  > Thanks.
>  >
>  > jw2002
>  >
>  ___
>  > users mailing list
>  > users@lists.open-mpi.org
>  > https://lists.open-mpi.org/mailman/listinfo/users
>
>
>  ___
>  users mailing list
>  users@lists.open-mpi.org
>  

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

2019-01-10 Thread Gilles Gouaillardet
Eduardo,

You have two options to use OmniPath

- “directly” via the psm2 mtl
mpirun —mca pml cm —mca mtl psm2 ...

- “indirectly” via libfabric
mpirun —mca pml cm —mca mtl ofi ...

I do invite you to try both. By explicitly requesting the mtl you will
avoid potential conflicts.

libfabric is used in production by Cisco and AWS (both major contributors
to both Open MPI and libfabric) so this is clearly not something to stay
away from. That being said, bug always happen and they could be related to
Open MPI, libfabric and/or OmniPath (and fwiw, Intel is a major contributor
to libfabric too)

Cheers,

Gilles

On Thursday, January 10, 2019, ROTHE Eduardo - externe <
eduardo-externe.ro...@edf.fr> wrote:

> Hi Gilles, thank you so much for your support!
>
> For now I'm just testing the software, so it's running on a single node.
>
> Your suggestion was very precise. In fact, choosing the ob1 component
> leads to a successfull execution! The tcp component had no effect.
>
> mpirun --mca pml ob1 —mca btl tcp,self -np 2 ./a.out *> Success*
> mpirun --mca pml ob1 -np 2 ./a.out *> Success*
>
> But... our cluster is equiped with Intel OMNI Path interconnects and we
> are aiming to use psm2 through ofi component in order to take full
> advantage of this technology.
>
> I believe your suggestion is showing that the problem is right here. But
> unfortunately I cannot see further.
>
> Meanwhile, I've also compiled Open MPI 3.1.3 and I have a successfull run
> with the same options and the same environment (no MPI_Send error). Could
> Open MPI 4.0.0 bring a different behaviour in this area? Eventually
> regarding ofi component?
>
> Do you have any idea that I could put in practice to narrow the problem
> further?
>
> Regards,
> Eduardo
>
> ps: I've recompiled Open MPI 4.0.0 using --with-hwloc=external, but with
> no different results (the same MPI_Send error);
>
> ps2: Yes, the configure line thing is really fishy, the original line was 
> --prefix=/opt/openmpi/4.0.0
> --with-pmix=/usr/lib/x86_64-linux-gnu/pmix --with-libevent=external
> --with-slurm --enable-mpi-cxx --with-ofi --with-verbs=no
> --disable-silent-rules --with-hwloc=/usr --enable-mpirun-prefix-by-default
> --with-devel-headers
>
>
> --
> *De :* users  de la part de
> gilles.gouaillar...@gmail.com 
> *Envoyé :* mercredi 9 janvier 2019 15:16
> *À :* Open MPI Users
> *Objet :* Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send
>
> Eduardo,
>
> The first part of the configure command line is for an install in /usr,
> but then there is ‘—prefix=/opt/openmpi/4.0.0’ and this is very fishy.
> You should also use ‘—with-hwloc=external’.
>
> How many nodes are you running on and which interconnect are you using ?
> What if you
> mpirun —mca pml ob1 —mca btl tcp,self -np 2 ./a.out
>
> Cheers,
>
> Gilles
>
> On Wednesday, January 9, 2019, ROTHE Eduardo - externe <
> eduardo-externe.ro...@edf.fr> wrote:
>
>> Hi.
>>
>> I'm testing Open MPI 4.0.0 and I'm struggling with a weird behaviour. In
>> a very simple example (very frustrating). I'm having the following error
>> returned by MPI_Send:
>>
>>
>>
>>
>>
>>
>> *  [gafront4:25692] *** An error occurred in MPI_Send
>> [gafront4:25692] *** reported by process [3152019457,0]
>> [gafront4:25692] *** on communicator MPI_COMM_WORLD
>> [gafront4:25692] *** MPI_ERR_OTHER: known error not in list
>> [gafront4:25692] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
>> will now abort,   [gafront4:25692] ***and potentially your MPI
>> job)*
>>
>> In the same machine I have other two instalations of Open MPI (2.0.2 and
>> 2.1.2) and they all run successfully this dummy program:
>>
>> #include 
>> #include 
>>
>> int main(int argc, char **argv) {
>> int process;
>> int population;
>>
>> MPI_Init(NULL, NULL);
>> MPI_Comm_rank(MPI_COMM_WORLD, );
>> MPI_Comm_size(MPI_COMM_WORLD, );
>> printf("Hello World from proccess %d out of %d\n", process,
>> population);
>>
>> int send_number = 10;
>> int recv_number;
>>
>> if (process == 0) {
>> MPI_Send(_number, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
>> printf("This is process 0 reporting::\n");
>> } else if (process == 1) {
>> MPI_Recv(_number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD,
>> MPI_STATUS_IGNORE);
>> printf("Process 1 received number %d from process 0\n",
>> recv_number);
>> }
>>
>> MPI_Finalize();
>> return 0;
>> }
>>
>> I'm really upset about recurring to you with this problem. I've been
>> arround it for days now and can't find any good solution. Can you please
>> take a look? I've enabled *FI_LOG_LEVEL=Debug* to see if I can trap any
>> information that could be of use but unfortunetly with no success. I've
>> also googled a lot, but I don't see where this error message might be
>> pointing at. Specially having two other working versions on the same
>> machine. The thing 

Re: [OMPI users] Increasing OpenMPI RMA win attach region count.

2019-01-10 Thread Gilles Gouaillardet
Jeff,

At first glance, a comment in the code suggests the rationale is to minimize 
the number of allocations and hence the time spent registering the memory.

Cheers,

Gilles

Jeff Hammond  wrote:
>Why is this allocated statically? I dont understand the difficulty of a 
>dynamically allocates and thus unrestricted implementation. Is there some 
>performance advantage to a bounded static allocation?  Or is it that you use 
>O(n) lookups and need to keep n small to avoid exposing that to users?
>
>
>I have usage models with thousands of attached segments, hence need to 
>understand how bad this will be with Open-MPI (yes I can amortize overhead but 
>it’s a pain).
>
>
>Thanks,
>
>
>Jeff
>
>
>On Wed, Jan 9, 2019 at 8:12 AM Nathan Hjelm via users 
> wrote:
>
>If you need to support more attachments you can set the value of that variable 
>either by setting:
>
>Environment:
>
>OMPI_MCA_osc_rdma_max_attach
>
>
>mpirun command line:
>
>—mca osc_rdma_max_attach
>
>
>Keep in mind that each attachment may use an underlying hardware resource that 
>may be easy to exhaust (hence the low default limit). It is recommended to 
>keep the total number as small as possible.
>
>-Nathan
>
>> On Jan 8, 2019, at 9:36 PM, Udayanga Wickramasinghe  wrote:
>> 
>> Hi,
>> I am running into an issue in open-mpi where it crashes abruptly during 
>> MPI_WIN_ATTACH. 
>> [nid00307:25463] *** An error occurred in MPI_Win_attach
>> [nid00307:25463] *** reported by process [140736284524545,140728898420736]
>> [nid00307:25463] *** on win rdma window 3
>> [nid00307:25463] *** MPI_ERR_RMA_ATTACH: Could not attach RMA segment
>> [nid00307:25463] *** MPI_ERRORS_ARE_FATAL (processes in this win will now 
>> abort,
>> [nid00307:25463] ***    and potentially your MPI job)
>> 
>> Looking more into this issue, it seems like open-mpi has a restriction on 
>> the maximum number of segments attached to 32. (OpenMpi3.0 spec doesn't spec 
>> doesn't say a lot about this scenario --"The argument win must be a window 
>> that was created with MPI_WIN_CREATE_DYNAMIC. Multiple (but nonoverlapping) 
>> memory regions may be attached to the same window")
>> 
>> To workaround this, I have temporarily modified the variable 
>> mca_osc_rdma_component.max_attach. Is there any way to configure this in 
>> open-mpi?
>> 
>> Thanks
>> Udayanga
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>
>___
>users mailing list
>users@lists.open-mpi.org
>https://lists.open-mpi.org/mailman/listinfo/users
>
>-- 
>
>Jeff Hammond
>jeff.scie...@gmail.com
>http://jeffhammond.github.io/
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

2019-01-09 Thread Gilles Gouaillardet
Eduardo,

The first part of the configure command line is for an install in /usr, but
then there is ‘—prefix=/opt/openmpi/4.0.0’ and this is very fishy.
You should also use ‘—with-hwloc=external’.

How many nodes are you running on and which interconnect are you using ?
What if you
mpirun —mca pml ob1 —mca btl tcp,self -np 2 ./a.out

Cheers,

Gilles

On Wednesday, January 9, 2019, ROTHE Eduardo - externe <
eduardo-externe.ro...@edf.fr> wrote:

> Hi.
>
> I'm testing Open MPI 4.0.0 and I'm struggling with a weird behaviour. In a
> very simple example (very frustrating). I'm having the following error
> returned by MPI_Send:
>
>
>
>
>
>
> *  [gafront4:25692] *** An error occurred in MPI_Send
> [gafront4:25692] *** reported by process [3152019457,0]
> [gafront4:25692] *** on communicator MPI_COMM_WORLD
> [gafront4:25692] *** MPI_ERR_OTHER: known error not in list
> [gafront4:25692] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
> will now abort,   [gafront4:25692] ***and potentially your MPI
> job)*
>
> In the same machine I have other two instalations of Open MPI (2.0.2 and
> 2.1.2) and they all run successfully this dummy program:
>
> #include 
> #include 
>
> int main(int argc, char **argv) {
> int process;
> int population;
>
> MPI_Init(NULL, NULL);
> MPI_Comm_rank(MPI_COMM_WORLD, );
> MPI_Comm_size(MPI_COMM_WORLD, );
> printf("Hello World from proccess %d out of %d\n", process,
> population);
>
> int send_number = 10;
> int recv_number;
>
> if (process == 0) {
> MPI_Send(_number, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
> printf("This is process 0 reporting::\n");
> } else if (process == 1) {
> MPI_Recv(_number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD,
> MPI_STATUS_IGNORE);
> printf("Process 1 received number %d from process 0\n",
> recv_number);
> }
>
> MPI_Finalize();
> return 0;
> }
>
> I'm really upset about recurring to you with this problem. I've been
> arround it for days now and can't find any good solution. Can you please
> take a look? I've enabled *FI_LOG_LEVEL=Debug* to see if I can trap any
> information that could be of use but unfortunetly with no success. I've
> also googled a lot, but I don't see where this error message might be
> pointing at. Specially having two other working versions on the same
> machine. The thing is that I see no reason why this code shouldn't run.
>
> The following is the configure command line, as given by ompi_info.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> * Configure command line: '--build=x86_64-linux-gnu'
> '--prefix=/usr'
> '--includedir=${prefix}/include'
> '--mandir=${prefix}/share/man'
> '--infodir=${prefix}/share/info'
> '--sysconfdir=/etc' '--localstatedir=/var'
> '--disable-silent-rules'
> '--libdir=${prefix}/lib/x86_64-linux-gnu'
> '--libexecdir=${prefix}/lib/x86_64-linux-gnu'
>  '--disable-maintainer-mode'
> '--disable-dependency-tracking'
> '--prefix=/opt/openmpi/4.0.0'
> '--with-pmix=/usr/lib/x86_64-linux-gnu/pmix'
> '--with-libevent=external' '--with-slurm'
> '--enable-mpi-cxx' '--with-ofi' '--with-verbs=no'
>   '--disable-silent-rules' '--with-hwloc=/usr'
>   '--enable-mpirun-prefix-by-default'
>  '--with-devel-headers'*
>
> Thank you for your time.
> Regards,
> Ed
>
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de
> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> 
>
> This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and notify the sender 

Re: [OMPI users] Querying/limiting OpenMPI memory allocations

2018-12-20 Thread Gilles Gouaillardet
Adam,

you can rewrite MPI_Allgatherv() in your app. it should simply invoke
PMPI_Allgatherv() (note the leading 'P') with the same arguments
followed by MPI_Barrier() in the same communicator (feel free to also
MPI_Barrier() before PMPI_Allgatherv()).
That can make your code slower, but it will force the unexpected
messages related to allgatherv being received.
If it helps with respect to memory consumption, that means we have a lead

Cheers,

Gilles

On Fri, Dec 21, 2018 at 5:00 AM Jeff Hammond  wrote:
>
> You might try replacing MPI_Allgatherv with the equivalent Send+Recv followed 
> by Broadcast.  I don't think MPI_Allgatherv is particularly optimized (since 
> it is hard to do and not a very popular function) and it might improve your 
> memory utilization.
>
> Jeff
>
> On Thu, Dec 20, 2018 at 7:08 AM Adam Sylvester  wrote:
>>
>> Gilles,
>>
>> It is btl/tcp (we'll be upgrading to newer EC2 types next year to take 
>> advantage of libfabric).  I need to write a script to log and timestamp the 
>> memory usage of the process as reported by /proc//stat and sync that up 
>> with the application's log of what it's doing to say this definitively, but 
>> based on what I've watched on 'top' so far, I think where these big 
>> allocations are happening are two areas where I'm doing MPI_Allgatherv() - 
>> every rank has roughly 1/numRanks of the data (but not divided exactly 
>> evenly so need to use MPI_Allgatherv)... the ranks are reusing that 
>> pre-allocated buffer to store their local results and then pass that same 
>> pre-allocated buffer into MPI_Allgatherv() to bring results in from all 
>> ranks.  So, there is a lot of communication across all ranks at these 
>> points.  So, does your comment about using the coll/sync module apply in 
>> this case?  I'm not familiar with this module - is this something I specify 
>> at OpenMPI compile time or a runtime option that
  I enable?
>>
>> Thanks for the detailed help.
>> -Adam
>>
>> On Thu, Dec 20, 2018 at 9:41 AM Gilles Gouaillardet 
>>  wrote:
>>>
>>> Adam,
>>>
>>> Are you using btl/tcp (e.g. plain TCP/IP) for internode communications
>>> ? Or are you using libfabric on top of the latest EC2 drivers ?
>>>
>>> There is no control flow in btl/tcp, which means for example if all
>>> your nodes send messages to rank 0, that can create a lot of
>>> unexpected messages on that rank..
>>> In the case of btl/tcp, this means a lot of malloc() on rank 0, until
>>> these messages are received by the app.
>>> If rank 0 is overflowed, then that will likely end up in the node
>>> swapping to death (or killing your app if you have little or no swap).
>>>
>>> If you are using collective operations, make sure the coll/sync module
>>> is selected.
>>> This module insert MPI_Barrier() every n collectives on a given
>>> communicator. This forces your processes to synchronize and can force
>>> message to be received. (Think of the previous example if you run
>>> MPI_Scatter(root=0) in a loop)
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On Thu, Dec 20, 2018 at 11:06 PM Adam Sylvester  wrote:
>>> >
>>> > This case is actually quite small - 10 physical machines with 18 physical 
>>> > cores each, 1 rank per machine.  These are AWS R4 instances (Intel Xeon 
>>> > E5 Broadwell processors).  OpenMPI version 2.1.0, using TCP (10 Gbps).
>>> >
>>> > I calculate the memory needs of my application upfront (in this case ~225 
>>> > GB per machine), allocate one buffer upfront, and reuse this buffer for 
>>> > valid and scratch throughout processing.  This is running on RHEL 7 - I'm 
>>> > measuring memory usage via top where I see it go up to 248 GB in an 
>>> > MPI-intensive portion of processing.
>>> >
>>> > I thought I was being quite careful with my memory allocations and there 
>>> > weren't any other stray allocations going on, but of course it's possible 
>>> > there's a large temp buffer somewhere that I've missed... based on what 
>>> > you're saying, this is way more memory than should be attributed to 
>>> > OpenMPI - is there a way I can query OpenMPI to confirm that?  If the OS 
>>> > is unable to keep up with the network traffic, is it possible there's 
>>> > some low-level system buffer that gets allocated to gradually work off 
>>> > the TCP traffic?
>>> >
>>> > Thanks.
>>> >
>>> > On Thu, Dec 20, 20

Re: [OMPI users] Querying/limiting OpenMPI memory allocations

2018-12-20 Thread Gilles Gouaillardet
Adam,

Are you using btl/tcp (e.g. plain TCP/IP) for internode communications
? Or are you using libfabric on top of the latest EC2 drivers ?

There is no control flow in btl/tcp, which means for example if all
your nodes send messages to rank 0, that can create a lot of
unexpected messages on that rank..
In the case of btl/tcp, this means a lot of malloc() on rank 0, until
these messages are received by the app.
If rank 0 is overflowed, then that will likely end up in the node
swapping to death (or killing your app if you have little or no swap).

If you are using collective operations, make sure the coll/sync module
is selected.
This module insert MPI_Barrier() every n collectives on a given
communicator. This forces your processes to synchronize and can force
message to be received. (Think of the previous example if you run
MPI_Scatter(root=0) in a loop)

Cheers,

Gilles

On Thu, Dec 20, 2018 at 11:06 PM Adam Sylvester  wrote:
>
> This case is actually quite small - 10 physical machines with 18 physical 
> cores each, 1 rank per machine.  These are AWS R4 instances (Intel Xeon E5 
> Broadwell processors).  OpenMPI version 2.1.0, using TCP (10 Gbps).
>
> I calculate the memory needs of my application upfront (in this case ~225 GB 
> per machine), allocate one buffer upfront, and reuse this buffer for valid 
> and scratch throughout processing.  This is running on RHEL 7 - I'm measuring 
> memory usage via top where I see it go up to 248 GB in an MPI-intensive 
> portion of processing.
>
> I thought I was being quite careful with my memory allocations and there 
> weren't any other stray allocations going on, but of course it's possible 
> there's a large temp buffer somewhere that I've missed... based on what 
> you're saying, this is way more memory than should be attributed to OpenMPI - 
> is there a way I can query OpenMPI to confirm that?  If the OS is unable to 
> keep up with the network traffic, is it possible there's some low-level 
> system buffer that gets allocated to gradually work off the TCP traffic?
>
> Thanks.
>
> On Thu, Dec 20, 2018 at 8:32 AM Nathan Hjelm via users 
>  wrote:
>>
>> How many nodes are you using? How many processes per node? What kind of 
>> processor? Open MPI version? 25 GB is several orders of magnitude more 
>> memory than should be used except at extreme scale (1M+ processes). Also, 
>> how are you calculating memory usage?
>>
>> -Nathan
>>
>> > On Dec 20, 2018, at 4:49 AM, Adam Sylvester  wrote:
>> >
>> > Is there a way at runtime to query OpenMPI to ask it how much memory it's 
>> > using for internal buffers?  Is there a way at runtime to set a max amount 
>> > of memory OpenMPI will use for these buffers?  I have an application where 
>> > for certain inputs OpenMPI appears to be allocating ~25 GB and I'm not 
>> > accounting for this in my memory calculations (and thus bricking the 
>> > machine).
>> >
>> > Thanks.
>> > -Adam
>> > ___
>> > users mailing list
>> > users@lists.open-mpi.org
>> > https://lists.open-mpi.org/mailman/listinfo/users
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Unable to build Open MPI with external PMIx library support

2018-12-17 Thread Gilles Gouaillardet

Eduardo,


By config.log, we mean the config.log automatically generated by your 
configure command


(e.g. not the output of the configure command)

this is a huge file, so please compress it


Cheers,


Gilles


this file should start with

This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by Open MPI configure 4.0.0, which was
generated by GNU Autoconf 2.69.  Invocation command line was




On 12/17/2018 7:29 PM, Eduardo Rothe via users wrote:

Hi Howard,

Thank you for you reply. I have just re-executed the whole process and 
here is the config.log (in attachment to this message)!


Just for restating, when I use internal PMIx I get the following error 
while running mpirun (using Open MPI 4.0.0):


--
We were unable to find any usable plugins for the BFROPS framework. 
This PMIx
framework requires at least one plugin in order to operate. This can 
be caused

by any of the following:

* we were unable to build any of the plugins due to some combination
  of configure directives and available system support

* no plugin was selected due to some combination of MCA parameter
  directives versus built plugins (i.e., you excluded all the plugins
  that were built and/or could execute)

* the PMIX_INSTALL_PREFIX environment variable, or the MCA parameter
  "mca_base_component_path", is set and doesn't point to any location
  that includes at least one usable plugin for this framework.

Please check your installation and environment.
--

Regards,
Eduardo


On Saturday, 15 December 2018, 18:35:44 CET, Howard Pritchard 
 wrote:



Hi Eduardo

Could you post the config.log for the build with internal PMIx so we 
can figure that out first.


Howard

Eduardo Rothe via users > schrieb am Fr. 14. Dez. 2018 um 09:41:


Open MPI: 4.0.0
PMIx: 3.0.2
OS: Debian 9

I'm building a debian package for Open MPI and either I get the
following error messages while configuring:

      undefined reference to symbol 'dlopen@@GLIBC_2.2.5'
  undefined reference to symbol 'lt_dlopen'

when using the configure option:

      ./configure --with-pmix=/usr/lib/x86_64-linux-gnu/pmix

or otherwise, if I use the following configure options:

  ./configure --with-pmix=external
--with-pmix-libdir=/usr/lib/x86_64-linux-gnu/pmix

I have a successfull compile, but when running mpirun I get the
following message:

--
We were unable to find any usable plugins for the BFROPS
framework. This PMIx
framework requires at least one plugin in order to operate. This
can be caused
by any of the following:

* we were unable to build any of the plugins due to some combination
  of configure directives and available system support

* no plugin was selected due to some combination of MCA parameter
  directives versus built plugins (i.e., you excluded all the plugins
  that were built and/or could execute)

* the PMIX_INSTALL_PREFIX environment variable, or the MCA parameter
  "mca_base_component_path", is set and doesn't point to any location
  that includes at least one usable plugin for this framework.

Please check your installation and environment.
--

What I find most strange is that I get the same error message
(unable to find
any usable plugins for the BFROPS framework) even if I don't
configure
external PMIx support!

Can someone please hint me about what's going on?

Cheers!
___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] questions about attribute caching

2018-12-16 Thread Gilles Gouaillardet
Almost :-)

seqBegin() and seqEnd() takes a communicator as a parameter.
So attribute caching is good for performance (only) if more than one
sequence *per communicator* is used in the application.

Cheers,

Gilles

On Sun, Dec 16, 2018 at 11:38 PM 邹海峰  wrote:
>
> Thank you very much for the reply.
>
> According to your explanation and the content from the website, if there is 
> only one sequential execution in the program, then it doesn't matter whether 
> using the attribute. But if there are multiple sequential execution,  each 
> process only needs to use MPI_Comm_dup() once with the help of attribute, 
> instead of duplicating the MPI_COMM_WORLD whenever there is a  sequential 
> execution. Am I right?
>
> Best wishes!
>
> Gilles Gouaillardet  于2018年12月16日周日 上午8:14写道:
>>
>> Hi,
>>
>> Your understanding is incorrect :
>> "Attributes are local to the process and specific to the communicator
>> to which they are attached."
>> (per 
>> https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-1.1/node119.htm)
>>
>> think an attribute is often a pointer, and really bad things can
>> happen if rank 0 uses a pointer that is valid on rank 1,
>> so if attributes were global, they would be virtually unusable.
>>
>> note the comment before the barrier is incorrect. this is a simple
>> barrier and all MPI tasks will block until all of them invoke seqEnd()
>>
>> The main goal of using attributes in this example is to invoke
>> MPI_Comm_dup() once per communicator (instead of once per sequence,
>> since this is an expensive operation).
>>
>>
>> Cheers,
>>
>> Gilles
>> On Sun, Dec 16, 2018 at 1:04 AM 邹海峰  wrote:
>> >
>> > Hi there,
>> >
>> > At first, I think attribute is just like global variable that attached to 
>> > a specific communicator. I can define and set the value on one process, 
>> > then get and modify the value on another process as long as those 
>> > processes belonging to the same communicator. But when I was reading 
>> > chapter 6 of the book using mpi: portable parallel programming with the 
>> > message-passing interface.  I was confused by the usage of caching 
>> > attribute.
>> >
>> > The purpose the code is to make the execution sequential. the main 
>> > function is
>> >
>> >   seqBegin( MPI_COMM_WORLD );
>> >   printf( "My rank is %d\n", wrank );
>> >   fflush( stdout );
>> >   seqEnd( MPI_COMM_WORLD );
>> >
>> > which is simple to understand. The program will print the rank in order. 
>> > The defination of the function "seqBegin()" is
>> >
>> > static int seqKeyval = MPI_KEYVAL_INVALID;
>> >
>> > void seqBegin( MPI_Comm comm )
>> > {
>> >   MPI_Comm lcomm;
>> >   int  flag, mysize, myrank;
>> >   seqInfo  *info;
>> >   if (seqKeyval == MPI_KEYVAL_INVALID) {
>> > MPI_Comm_create_keyval( MPI_NULL_COPY_FN, seqDelFn, , NULL );
>> >   }
>> >   MPI_Comm_get_attr( comm, seqKeyval, ,  );
>> >   if (!flag) {
>> > info = (seqInfo *)malloc( sizeof(seqInfo) );
>> > MPI_Comm_dup( comm, >lcomm );
>> > MPI_Comm_rank( info->lcomm,  );
>> > MPI_Comm_size( info->lcomm,  );
>> > info->prevRank = myrank - 1;
>> > if (info->prevRank < 0)   info->prevRank = MPI_PROC_NULL;
>> > info->nextRank = myrank + 1;
>> > if (info->nextRank >= mysize) info->nextRank = MPI_PROC_NULL;
>> > if (verbose) {
>> >   printf( "seqbegin: prev = %d, next = %d\n",
>> >   info->prevRank, info->nextRank );
>> > }
>> > MPI_Comm_set_attr( comm, seqKeyval, info );
>> >   }
>> >   MPI_Recv( NULL, 0, MPI_INT, info->prevRank, 0, info->lcomm,
>> > MPI_STATUS_IGNORE );
>> > }
>> >
>> > and the defination of function "seqEnd()" is
>> >
>> > void seqEnd( MPI_Comm comm )
>> > {
>> >   seqInfo *info;
>> >   int flag;
>> >
>> >   /* Sanity check */
>> >   if (seqKeyval == MPI_KEYVAL_INVALID)
>> > MPI_Abort( MPI_COMM_WORLD, 1 );
>> >   MPI_Comm_get_attr( comm, seqKeyval, ,  );
>> >   if (!info || !flag)
>> > MPI_Abort( MPI_COMM_WORLD, 1 );
>> >   if (verbose) {
>> > printf( "seqend: prev = %d, next = %d\n",
>> > info->prevRank, info->nextRank );
>

Re: [OMPI users] questions about attribute caching

2018-12-15 Thread Gilles Gouaillardet
Hi,

Your understanding is incorrect :
"Attributes are local to the process and specific to the communicator
to which they are attached."
(per 
https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-1.1/node119.htm)

think an attribute is often a pointer, and really bad things can
happen if rank 0 uses a pointer that is valid on rank 1,
so if attributes were global, they would be virtually unusable.

note the comment before the barrier is incorrect. this is a simple
barrier and all MPI tasks will block until all of them invoke seqEnd()

The main goal of using attributes in this example is to invoke
MPI_Comm_dup() once per communicator (instead of once per sequence,
since this is an expensive operation).


Cheers,

Gilles
On Sun, Dec 16, 2018 at 1:04 AM 邹海峰  wrote:
>
> Hi there,
>
> At first, I think attribute is just like global variable that attached to a 
> specific communicator. I can define and set the value on one process, then 
> get and modify the value on another process as long as those processes 
> belonging to the same communicator. But when I was reading chapter 6 of the 
> book using mpi: portable parallel programming with the message-passing 
> interface.  I was confused by the usage of caching attribute.
>
> The purpose the code is to make the execution sequential. the main function is
>
>   seqBegin( MPI_COMM_WORLD );
>   printf( "My rank is %d\n", wrank );
>   fflush( stdout );
>   seqEnd( MPI_COMM_WORLD );
>
> which is simple to understand. The program will print the rank in order. The 
> defination of the function "seqBegin()" is
>
> static int seqKeyval = MPI_KEYVAL_INVALID;
>
> void seqBegin( MPI_Comm comm )
> {
>   MPI_Comm lcomm;
>   int  flag, mysize, myrank;
>   seqInfo  *info;
>   if (seqKeyval == MPI_KEYVAL_INVALID) {
> MPI_Comm_create_keyval( MPI_NULL_COPY_FN, seqDelFn, , NULL );
>   }
>   MPI_Comm_get_attr( comm, seqKeyval, ,  );
>   if (!flag) {
> info = (seqInfo *)malloc( sizeof(seqInfo) );
> MPI_Comm_dup( comm, >lcomm );
> MPI_Comm_rank( info->lcomm,  );
> MPI_Comm_size( info->lcomm,  );
> info->prevRank = myrank - 1;
> if (info->prevRank < 0)   info->prevRank = MPI_PROC_NULL;
> info->nextRank = myrank + 1;
> if (info->nextRank >= mysize) info->nextRank = MPI_PROC_NULL;
> if (verbose) {
>   printf( "seqbegin: prev = %d, next = %d\n",
>   info->prevRank, info->nextRank );
> }
> MPI_Comm_set_attr( comm, seqKeyval, info );
>   }
>   MPI_Recv( NULL, 0, MPI_INT, info->prevRank, 0, info->lcomm,
> MPI_STATUS_IGNORE );
> }
>
> and the defination of function "seqEnd()" is
>
> void seqEnd( MPI_Comm comm )
> {
>   seqInfo *info;
>   int flag;
>
>   /* Sanity check */
>   if (seqKeyval == MPI_KEYVAL_INVALID)
> MPI_Abort( MPI_COMM_WORLD, 1 );
>   MPI_Comm_get_attr( comm, seqKeyval, ,  );
>   if (!info || !flag)
> MPI_Abort( MPI_COMM_WORLD, 1 );
>   if (verbose) {
> printf( "seqend: prev = %d, next = %d\n",
> info->prevRank, info->nextRank );
>   }
>   MPI_Send( NULL, 0, MPI_INT, info->nextRank, 0, info->lcomm );
>
>   /* Make everyone wait until all have completed their send */
>   MPI_Barrier( info->lcomm );
> }
>
> Other details are omitted. In fact, all the codes can be found in 
> https://www.mcs.anl.gov/research/projects/mpi/usingmpi/examples-usingmpi/libraries/index.html
>  which is provided by the author of the book.
>
> The program uses send and recv to block the execution. Only if the process 
> receive the message from last process, the process can continue to execute, 
> otherwise it is blocked, which resulting in the sequential execution.  The 
> part I don't understand is in function "seqBegin()". If my undertstanding 
> about attribute is right, only one process will enter the if condition and 
> set the value of attribute, other processes just get the value . Here comes 
> the question: since other processes don't set the value, how can they get the 
> prevRank and nextRank of their own.
>
> The code can be executed as expected. But I still can't get the rational 
> behind this and there is little reference about attribute caching, so I come 
> here for help. Thank you very much !
>
> Best Wishes !
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] singularity support

2018-12-12 Thread Gilles Gouaillardet
My understanding is that MPI tasks will be launched inside a singularity 
container.


In a typical environment, mpirun spawns an instance of the orted on each 
node, and then each orted daemon (or mpirun on the local node) fork 
the MPI tasks (a.out)



With singularity, orted would fork a container running a.out

(not sure if it is one container per task, or one container per node)


Cheers,


Gilles

On 12/10/2018 5:00 AM, Johnson, Glenn P wrote:

Greetings,

Could someone explain, or point to documentation that explains, what 
the --with-singularity option enables with regard to OpenMPI support 
for singularity containers?


Thanks.

Glenn Johnson

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] number of exchange messages

2018-12-12 Thread Gilles Gouaillardet

Hi,


Open MPI does not do this out of the box.


MPI profilers can achieve that. Some are commercials (ITAC from Intel, 
MAP from Allinea/ARM) and some are free (Score-P)



An other option is to build your own mini-profiler with the PMPI interface.

You can find an example at 
http://www.training.prace-ri.eu/uploads/tx_pracetmo/mpi-3.pdf (slides 7-9)



Cheers,


Gilles


On 12/12/2018 3:28 PM, saad alosaimi wrote:

Hello,

Is there any way to detect the number of exchange messages between the 
MPI processes ?


Many thanks in advance.

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] MPI_Reduce_Scatter Segmentation Fault with Intel 2019 Update 1 Compilers on OPA-1

2018-12-04 Thread Gilles Gouaillardet

Thanks Mikhail,


You have a good point.


With the current semantic used in the IMB benchmark, this cannot be 
equivalent to


MPI_Reduce() of N bytes followed by MPI_Scatterv() of N bytes.


So this is indeed a semantical question :

what should be a MPI_Reduce_scatter() of N bytes equivalent to ?

1) MPI_Reduce() of N bytes followed by MPI_Scatterv() in which each task 
receives N/commsize bytes


2) MPI_Reduce() of N*commsize bytes followed by MPI_Scatterv() in which 
each task receives N bytes.



I honestly have no opinion on that, and as long as there is no memory 
corruption, I am happy with both options.




Cheers,


Gilles

On 12/5/2018 12:25 PM, Mikhail Kurnosov wrote:

Hi,

The memory manager of IMB (IMB_mem_manager.c) do not support the 
MPI_Reduce_scatter operation. It allocates too small send buffer: 
sizeof(msg), but the operation requires commsize * sizeof(msg).

There are two possible solutions:

1) Fix computations of recvcounts (as proposed by Gilles)
2) Change memory allocation for send buffer in the memory manager of 
IMB. That approach was consistent with IMB style (for example, buffer 
allocation for MPI_Scatter operation)


WBR,
Mikhail Kurnosov
On 04.12.2018 17:06, Peter Kjellström wrote:

On Mon, 3 Dec 2018 19:41:25 +
"Hammond, Simon David via users"  wrote:

 > Hi Open MPI Users,
 >
 > Just wanted to report a bug we have seen with OpenMPI 3.1.3 and 4.0.0
 > when using the Intel 2019 Update 1 compilers on our
 > Skylake/OmniPath-1 cluster. The bug occurs when running the Github
 > master src_c variant of the Intel MPI Benchmarks.

I've noticed this also when using intel mpi (2018 and 2019u1). I
classified it as a bug in imb but didn't look too deep (new
reduce_scatter code).

/Peter K

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] MPI_Reduce_Scatter Segmentation Fault with Intel 2019 Update 1 Compilers on OPA-1

2018-12-04 Thread Gilles Gouaillardet

Thanks for the report.


As far as I am concerned, this is a bug in the IMB benchmark, and I 
issued a PR to fix that


https://github.com/intel/mpi-benchmarks/pull/11


Meanwhile, you can manually download and apply the patch at

https://github.com/intel/mpi-benchmarks/pull/11.patch



Cheers,


Gilles


On 12/4/2018 4:41 AM, Hammond, Simon David via users wrote:

Hi Open MPI Users,

Just wanted to report a bug we have seen with OpenMPI 3.1.3 and 4.0.0 when 
using the Intel 2019 Update 1 compilers on our Skylake/OmniPath-1 cluster. The 
bug occurs when running the Github master src_c variant of the Intel MPI 
Benchmarks.

Configuration:

./configure --prefix=/home/projects/x86-64-skylake/openmpi/3.1.3/intel/19.1.144 
--with-slurm --with-psm2 
CC=/home/projects/x86-64/intel/compilers/2019/compilers_and_libraries_2019.1.144/linux/bin/intel64/icc
 
CXX=/home/projects/x86-64/intel/compilers/2019/compilers_and_libraries_2019.1.144/linux/bin/intel64/icpc
 
FC=/home/projects/x86-64/intel/compilers/2019/compilers_and_libraries_2019.1.144/linux/bin/intel64/ifort
 --with-zlib=/home/projects/x86-64/zlib/1.2.11 
--with-valgrind=/home/projects/x86-64/valgrind/3.13.0

Operating System is RedHat 7.4 release and we utilize a local build of GCC 
7.2.0 for our Intel compiler (C++) header files. Everything makes correctly, 
and passes a make check without any issues.

We then compile IMB and run IMB-MPI1 on 24 nodes and get the following:

#
# Benchmarking Reduce_scatter
# #processes = 64
# ( 1088 additional processes waiting in MPI_Barrier)
#
#bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]
 0 1000 0.18 0.19 0.18
 4 1000 7.3910.37 8.68
 8 1000 7.8411.14 9.23
16 1000 8.5012.3710.14
32 100010.3714.6612.15
64 100013.7618.8216.17
   128 100021.6327.6124.87
   256 100039.9847.2743.96
   512 100072.9378.5975.15
  1024 1000   147.21   152.98   149.94
  2048 1000   413.41   426.90   420.15
  4096 1000   421.28   442.58   434.52
  8192 1000   418.31   450.20   438.51
 16384 1000  1082.85  1221.44  1140.92
 32768 1000  2434.11  2529.90  2476.72
 65536  640  5469.57  6048.60  5687.08
131072  320 11702.94 12435.06 12075.07
262144  160 19214.42 20433.83 19883.80
524288   80 49462.22 53896.43 52101.56
   1048576   40119422.53131922.20126920.99
   2097152   20256345.97288185.72275767.05
[node06:351648] *** Process received signal ***
[node06:351648] Signal: Segmentation fault (11)
[node06:351648] Signal code: Invalid permissions (2)
[node06:351648] Failing at address: 0x7fdb6efc4000
[node06:351648] [ 0] /lib64/libpthread.so.0(+0xf5e0)[0x7fdb8646c5e0]
[node06:351648] [ 1] ./IMB-MPI1(__intel_avx_rep_memcpy+0x140)[0x415380]
[node06:351648] [ 2] 
/home/projects/x86-64-skylake/openmpi/3.1.3/intel/19.1.144/lib/libopen-pal.so.40(opal_datatype_copy_content_same_ddt+0xca)[0x7fdb858d847a]
[node06:351648] [ 3] 
/home/projects/x86-64-skylake/openmpi/3.1.3/intel/19.1.144/lib/libmpi.so.40(ompi_coll_base_reduce_scatter_intra_ring+0x3f9)[0x7fdb86c43b29]
[node06:351648] [ 4] 
/home/projects/x86-64-skylake/openmpi/3.1.3/intel/19.1.144/lib/libmpi.so.40(PMPI_Reduce_scatter+0x1d7)[0x7fdb86c1de67]
[node06:351648] [ 5] ./IMB-MPI1[0x40d624]
[node06:351648] [ 6] ./IMB-MPI1[0x407d16]
[node06:351648] [ 7] ./IMB-MPI1[0x403356]
[node06:351648] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fdb860bbc05]
[node06:351648] [ 9] ./IMB-MPI1[0x402da9]
[node06:351648] *** End of error message ***
[node06:351649] *** Process received signal ***
[node06:351649] Signal: Segmentation fault (11)
[node06:351649] Signal code: Invalid permissions (2)
[node06:351649] Failing at address: 0x7f9b19c6f000
[node06:351649] [ 0] /lib64/libpthread.so.0(+0xf5e0)[0x7f9b311295e0]
[node06:351649] [ 1] ./IMB-MPI1(__intel_avx_rep_memcpy+0x140)[0x415380]
[node06:351649] [ 2] 
/home/projects/x86-64-skylake/openmpi/3.1.3/intel/19.1.144/lib/libopen-pal.so.40(opal_datatype_copy_content_same_ddt+0xca)[0x7f9b3059547a]
[node06:351649] [ 3] 
/home/projects/x86-64-skylake/openmpi/3.1.3/intel/19.1.144/lib/libmpi.so.40(ompi_coll_base_reduce_scatter_intra_ring+0x3f9)[0x7f9b31900b29]
[node06:351649] [ 4] 

Re: [OMPI users] Building OpenMPI with Lustre support using PGI fails

2018-11-27 Thread Gilles Gouaillardet

Folks,


sorry for the late follow-up. The config.log was indeed sent offline.


Here is the relevant part :


configure:294375: checking for required lustre data structures
configure:294394: pgcc -O -DNDEBUG   -Iyes/include -c conftest.c
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 157)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 157)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 158)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 158)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 159)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 159)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 160)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 160)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 161)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 161)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 162)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 162)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 163)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 163)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 164)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 164)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 211)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 211)
PGC-S-0040-Illegal use of symbol, u_int64_t (/usr/include/sys/quota.h: 212)
PGC-W-0156-Type not specified, 'int' assumed (/usr/include/sys/quota.h: 212)
PGC/x86-64 Linux 18.7-0: compilation completed with severe errors
configure:294401: $? = 2
configure:294415: result: no
configure:294424: error: Lustre support requested but not found. Aborting


Here is the conftest.c that triggers the error

#include "lustre/lustreapi.h"
void alloc_lum()
{
  int v1, v3;
  v1 = sizeof(struct lov_user_md_v1) +
   LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
  v3 = sizeof(struct lov_user_md_v3) +
    LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data_v1);
}


The same code was reported to work with gcc compiler, so at that stage, 
this looks like


a PGI or an environment issue (sometimes the sysadmin has to re-run 
makelocalrc if some dependencies have changed),


so I recommend this error is submitted to PGI support.


I reviewed the code and filled a PR that get rids of the "-Iyes/include" 
flag.


Merged or not, that does not fix the real issue here.


Cheers,


Gilles


On 11/28/2018 6:04 AM, Gabriel, Edgar wrote:

Gilles submitted a patch for that, and I approved it a couple of days back, I 
*think* it has not been merged however. This was a bug in the Open MPI Lustre 
configure logic, should be fixed after this one however.

https://github.com/open-mpi/ompi/pull/6080

Thanks
Edgar


-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Latham,
Robert J. via users
Sent: Tuesday, November 27, 2018 2:03 PM
To: users@lists.open-mpi.org
Cc: Latham, Robert J. ; gi...@rist.or.jp
Subject: Re: [OMPI users] Building OpenMPI with Lustre support using PGI fails

On Tue, 2018-11-13 at 21:57 -0600, gil...@rist.or.jp wrote:

Raymond,

can you please compress and post your config.log ?

I didn't see the config.log in response to this.  Maybe Ray and Giles took the
discusison off list?  As someone who might have introduced the offending
configure-time checks, I'm particularly interested in fixing lustre detection.

==rob



Cheers,

Gilles

- Original Message -

I am trying  to build OpenMPI with Lustre support using PGI 18.7 on
CentOS 7.5 (1804).

It builds successfully with Intel compilers, but fails to find the
necessary  Lustre components with the PGI compiler.

I have tried building  OpenMPI 4.0.0, 3.1.3 and 2.1.5.   I can
build
OpenMPI, but configure does not find the proper Lustre files.

Lustre is installed from current client RPMS, version 2.10.5

Include files are in /usr/include/lustre

When specifying --with-lustre, I get:

--- MCA component fs:lustre (m4 configuration macro) checking for
MCA component fs:lustre compile mode... dso checking --with-lustre
value... simple ok (unspecified value) looking for header without
includes checking lustre/lustreapi.h usability... yes checking
lustre/lustreapi.h presence... yes checking for
lustre/lustreapi.h... yes checking for library containing
llapi_file_create... -llustreapi checking if liblustreapi requires
libnl v1 or v3...
checking for required lustre data structures... no
configure: error: Lustre support requested but not found. Aborting


--

   Ray Muno
   IT Manager


University of Minnesota
   Aerospace Engineering and Mechanics Mechanical
Engineering
   110 Union St. S.E. 

Re: [OMPI users] OpenMPI2 + slurm

2018-11-23 Thread Gilles Gouaillardet
Lothar,

it seems you did not configure Open MPI with --with-pmi=

If SLURM was built with PMIx support, then an other option is to use that.
First, srun --mpi=list will show you the list of available MPI
modules, and then you could
srun --mpi=pmix_v2 ... MPI_Hellow
If you believe that should be the default, then you should contact
your sysadmin that can make that for you.

You you want to use PMIx, then I recommend you configure Open MPI with
the same external PMIx that was used to
build SLURM (e.g. configure --with-pmix=). Though PMIx
has cross version support, using the same PMIx will avoid you running
incompatible PMIx versions.


Cheers,

Gilles
On Fri, Nov 23, 2018 at 5:20 PM Lothar Brendel
 wrote:
>
> Hi guys,
>
> I've always been somewhat at a loss regarding slurm's idea about tasks vs. 
> jobs. That didn't cause any problems, though, until passing to OpenMPI2 
> (2.0.2 that is, with slurm 16.05.9).
>
> Running http://mpitutorial.com/tutorials/mpi-hello-world as an example with 
> just
>
> srun -n 2 MPI-hellow
>
> yields
>
> Hello world from processor node31, rank 0 out of 1 processors
> Hello world from processor node31, rank 0 out of 1 processors
>
> i.e. the two tasks don't see each other MPI-wise. Well, srun doesn't include 
> an mpirun.
>
> But running
>
> srun -n 2 mpirun MPI-hellow
>
> produces
>
> Hello world from processor node31, rank 1 out of 2 processors
> Hello world from processor node31, rank 0 out of 2 processors
> Hello world from processor node31, rank 1 out of 2 processors
> Hello world from processor node31, rank 0 out of 2 processors
>
> i.e. I get *two* independent MPI-tasks with 2 processors each. (The same 
> applies if stating explicitly "mpirun -np 2".)
> I never could make sense of this squaring, I rather used to run my jobs like
>
> srun -c 2 mpirun -np 2 MPI-hellow
>
> which provided the desired job with *one* task using 2 processors. Passing 
> from OpenMPI 1.6.5 to 2.0.2 (Debian Jessie -> Stretch), though, I'm getting 
> the error
> "There are not enough slots available in the system to satisfy the 2 slots
> that were requested by the application:
>   MPI-hellow" now.
>
> The environment on the node contains
>
> SLURM_CPUS_ON_NODE=2
> SLURM_CPUS_PER_TASK=2
> SLURM_JOB_CPUS_PER_NODE=2
> SLURM_NTASKS=1
> SLURM_TASKS_PER_NODE=1
>
> which looks fine to me, but mpirun infers slots=1 from that (confirmed by 
> ras_base_verbose 5). In deed, looking into 
> orte/mca/ras/slurm/ras_slurm_module.c, I find that while 
> orte_ras_slurm_allocate() reads the value of SLURM_CPUS_PER_TASK into its 
> local variable cpus_per_task, it doesn't use it anywhere. Rather, the number 
> of slots is determined from SLURM_TASKS_PER_NODE.
>
> Is this intended behaviour?
>
> What's wrong here? I know that I can use --oversubscribe, but that seems 
> rather a workaround.
>
> Thanks in advance,
> Lothar
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] OpenFabrics warning

2018-11-12 Thread Gilles Gouaillardet
Andrei,

you can

mpirun --mca btl ^openib ...

in order to "disable" infiniband


Cheers,

Gilles
On Mon, Nov 12, 2018 at 9:52 AM Andrei Berceanu
 wrote:
>
> The node has an IB card, but it is a stand-alone node, disconnected from the 
> rest of the cluster.
> I am using OMPI to communicate internally between the GPUs of this node (and 
> not between nodes).
> So how can I disable the IB?
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] issue compiling openmpi 3.2.1 with pmi and slurm

2018-10-10 Thread Gilles Gouaillardet

I digged a bit the configury logic and found


./configure --prefix=/usr/local/ --with-cuda --with-slurm 
--with-pmi=/usr/local/slurm



should do the trick, if not


./configure --prefix=/usr/local/ --with-cuda --with-slurm 
--with-pmi=/usr/local/slurm --with-pmi-libdir=/usr/local/slurm/lib64



should work.


Internally, if we --with-pmi=FOO and FOO/pmi.h exists, when we do not 
set slurm_pmi_found=yes, and the opal_pmi{1,2}_CPPFLAGS do not get set.



Cheers,


Gilles


On 10/11/2018 7:02 AM, Charles A Taylor wrote:
./configure --prefix=/usr/local/ --with-cuda --with-slurm 
--with-pmi=/usr/local/slurm/include/slurm 
--with-pmi-libdir=/usr/local/slurm/lib64


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-05 Thread Gilles Gouaillardet
Charles,

are you saying that even if you

mpirun --mca pml ob1 ...

(e.g. force the ob1 component of the pml framework) the memory leak is
still present ?

As a side note, we strongly recommend to avoid
configure --with-FOO=/usr
instead
configure --with-FOO
should be used (otherwise you will end up with -I/usr/include
-L/usr/lib64 and that could silently hide third party libraries
installed in a non standard directory). If --with-FOO fails for you,
then this is a bug we will appreciate you report.

Cheers,

Gilles
On Fri, Oct 5, 2018 at 6:42 AM Charles A Taylor  wrote:
>
>
> We are seeing a gaping memory leak when running OpenMPI 3.1.x (or 2.1.2, for 
> that matter) built with UCX support.   The leak shows up
> whether the “ucx” PML is specified for the run or not.  The applications in 
> question are arepo and gizmo but it I have no reason to believe
> that others are not affected as well.
>
> Basically the MPI processes grow without bound until SLURM kills the job or 
> the host memory is exhausted.
> If I configure and build with “--without-ucx” the problem goes away.
>
> I didn’t see anything about this on the UCX github site so I thought I’d ask 
> here.  Anyone else seeing the same or similar?
>
> What version of UCX is OpenMPI 3.1.x tested against?
>
> Regards,
>
> Charlie Taylor
> UF Research Computing
>
> Details:
> —
> RHEL7.5
> OpenMPI 3.1.2 (and any other version I’ve tried).
> ucx 1.2.2-1.el7 (RH native)
> RH native IB stack
> Mellanox FDR/EDR IB fabric
> Intel Parallel Studio 2018.1.163
>
> Configuration Options:
> —
> CFG_OPTS=""
> CFG_OPTS="$CFG_OPTS C=icc CXX=icpc FC=ifort FFLAGS=\"-O2 -g -warn -m64\" 
> LDFLAGS=\"\" "
> CFG_OPTS="$CFG_OPTS --enable-static"
> CFG_OPTS="$CFG_OPTS --enable-orterun-prefix-by-default"
> CFG_OPTS="$CFG_OPTS --with-slurm=/opt/slurm"
> CFG_OPTS="$CFG_OPTS --with-pmix=/opt/pmix/2.1.1"
> CFG_OPTS="$CFG_OPTS --with-pmi=/opt/slurm"
> CFG_OPTS="$CFG_OPTS --with-libevent=external"
> CFG_OPTS="$CFG_OPTS --with-hwloc=external"
> CFG_OPTS="$CFG_OPTS --with-verbs=/usr"
> CFG_OPTS="$CFG_OPTS --with-libfabric=/usr"
> CFG_OPTS="$CFG_OPTS --with-ucx=/usr"
> CFG_OPTS="$CFG_OPTS --with-verbs-libdir=/usr/lib64"
> CFG_OPTS="$CFG_OPTS --with-mxm=no"
> CFG_OPTS="$CFG_OPTS --with-cuda=${HPC_CUDA_DIR}"
> CFG_OPTS="$CFG_OPTS --enable-openib-udcm"
> CFG_OPTS="$CFG_OPTS --enable-openib-rdmacm"
> CFG_OPTS="$CFG_OPTS --disable-pmix-dstore"
>
> rpmbuild --ba \
>  --define '_name openmpi' \
>  --define "_version $OMPI_VER" \
>  --define "_release ${RELEASE}" \
>  --define "_prefix $PREFIX" \
>  --define '_mandir %{_prefix}/share/man' \
>  --define '_defaultdocdir %{_prefix}' \
>  --define 'mflags -j 8' \
>  --define 'use_default_rpm_opt_flags 1' \
>  --define 'use_check_files 0' \
>  --define 'install_shell_scripts 1' \
>  --define 'shell_scripts_basename mpivars' \
>  --define "configure_options $CFG_OPTS " \
>  openmpi-${OMPI_VER}.spec 2>&1 | tee rpmbuild.log
>
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Cannot run MPI code on multiple cores with PBS

2018-10-04 Thread Gilles Gouaillardet
In this case, some Open MPI plugins are missing some third party libraries,
so you would have to ldd all the plugins (e.g. the .so files) located
in /lib/openmpi
in order to evidence any issue.

Cheers,

Gilles

On Thu, Oct 4, 2018 at 4:34 PM John Hearns via users
 wrote:
>
> Michele  one tip:   log into a compute node using ssh and as your own 
> username.
> If you use the Modules envirnonment then load the modules you use in
> the job script
> then use the  ldd  utility to check if you can load all the libraries
> in the code.io executable
>
> Actually you are better to submit a short batch job which does not use
> mpirun but uses ldd
> A proper batch job will duplicate the environment you wish to run in.
>
> ldd ./code.io
>
> By the way, is the batch system PBSPro or OpenPBS?  Version 6 seems a bit old.
> Can you say what version of Redhat or CentOS this cluster is installed with?
>
>
>
> On Thu, 4 Oct 2018 at 00:02, Castellana Michele
>  wrote:
> >
> > I fixed it, the correct file was in /lib64, not in /lib.
> >
> > Thank you for your help.
> >
> > On Oct 3, 2018, at 11:30 PM, Castellana Michele 
> >  wrote:
> >
> > Thank you, I found some libcrypto files in /usr/lib indeed:
> >
> > $ ls libcry*
> > libcrypt-2.17.so  libcrypto.so.10  libcrypto.so.1.0.2k  libcrypt.so.1
> >
> > but I could not find libcrypto.so.0.9.8. Here they suggest to create a 
> > hyperlink, but if I do I still get an error from MPI. Is there another way 
> > around this?
> >
> > Best,
> >
> > On Oct 3, 2018, at 11:00 PM, Jeff Squyres (jsquyres) via users 
> >  wrote:
> >
> > It's probably in your Linux distro somewhere -- I'd guess you're missing a 
> > package (e.g., an RPM or a deb) out on your compute nodes...?
> >
> >
> > On Oct 3, 2018, at 4:24 PM, Castellana Michele 
> >  wrote:
> >
> > Dear Ralph,
> > Thank you for your reply. Do you know where I could find libcrypto.so.0.9.8 
> > ?
> >
> > Best,
> >
> > On Oct 3, 2018, at 9:41 PM, Ralph H Castain  wrote:
> >
> > Actually, I see that you do have the tm components built, but they cannot 
> > be loaded because you are missing libcrypto from your LD_LIBRARY_PATH
> >
> >
> > On Oct 3, 2018, at 12:33 PM, Ralph H Castain  wrote:
> >
> > Did you configure OMPI —with-tm=? It looks like we didn’t 
> > build PBS support and so we only see one node with a single slot allocated 
> > to it.
> >
> >
> > On Oct 3, 2018, at 12:02 PM, Castellana Michele 
> >  wrote:
> >
> > Dear all,
> > I am having trouble running an MPI code across multiple cores on a new 
> > computer cluster, which uses PBS. Here is a minimal example, where I want 
> > to run two MPI processes, each on  a different node. The PBS script is
> >
> > #!/bin/bash
> > #PBS -l walltime=00:01:00
> > #PBS -l mem=1gb
> > #PBS -l nodes=2:ppn=1
> > #PBS -q batch
> > #PBS -N test
> > mpirun -np 2 ./code.o
> >
> > and when I submit it with
> >
> > $qsub script.sh
> >
> > I get the following message in the PBS error file
> >
> > $ cat test.e1234
> > [shbli040:08879] mca_base_component_repository_open: unable to open 
> > mca_plm_tm: libcrypto.so.0.9.8: cannot open shared object file: No such 
> > file or directory (ignored)
> > [shbli040:08879] mca_base_component_repository_open: unable to open 
> > mca_oob_ud: libibverbs.so.1: cannot open shared object file: No such file 
> > or directory (ignored)
> > [shbli040:08879] mca_base_component_repository_open: unable to open 
> > mca_ras_tm: libcrypto.so.0.9.8: cannot open shared object file: No such 
> > file or directory (ignored)
> > --
> > There are not enough slots available in the system to satisfy the 2 slots
> > that were requested by the application:
> >  ./code.o
> >
> > Either request fewer slots for your application, or make more slots 
> > available
> > for use.
> > —
> >
> > The PBS version is
> >
> > $ qstat --version
> > Version: 6.1.2
> >
> > and here is some additional information on the MPI version
> >
> > $ mpicc -v
> > Using built-in specs.
> > COLLECT_GCC=/bin/gcc
> > COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
> > Target: x86_64-redhat-linux
> > […]
> > Thread model: posix
> > gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
> >
> > Do you guys know what may be the issue here?
> >
> > Thank you
> > Best,
> >
> >
> >
> >
> >
> >
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> >
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> >
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> >
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> 

Re: [OMPI users] How do I build 3.1.0 (or later) with mellanox's libraries

2018-09-14 Thread Gilles Gouaillardet
Alan,

Can you please compress and post your config.log ?

My understanding of the mentioned commit is it does not build the
reachable/netlink component if libnl version 1 is used (by third party libs
such as mxm).
I do not believe it should abort configure

Cheers,

Gilles

On Saturday, September 15, 2018, Alan Wild  wrote:

> I apologize if this has been discussed before but I've been unable to find
> discussion on the topic.
>
> I recently went to build 3.1.2 on our cluster only to have the build
> completely fail during configure due to issues with libnl versions.
>
> Specifically I was had requested support for  mellanox's libraries (mxm,
> hcoll, sharp, etc) which was fine for me in 3.0.0 and 3.0.1.  However it
> appears all of those libraries are built with libnl version 1 but the
> netlink component is now requiring netlink version 3 and aborts the build
> if it finds anything else in LIBS that using version 1.
>
> I don't believe mellanox's is providing releases of these libraries linked
> agsinst liblnl version 3 (love to find out I'm wrong on that) at least not
> for CentOS 6.9.
>
> According to github, it appears bwbarret's commit a543e7f (from one year
> ago today) which was merged into 3.1.0 is responsible.  However I'm having
> a hard time believing that openmpi would want to break support for these
> libraries or there isn't some other kind of workaround.
>
> I'm on a short timeline to deliver this build of openmpi to my users but I
> know they won't accept a build that doesn't support mellanox's libraries.
>
> Hoping there's an easy fix here (short of trying to reverse the commit in
> my build) that I'm overlooking here.
>
> Thanks,
>
> -Alan
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] stdout/stderr question

2018-09-10 Thread Gilles Gouaillardet

It seems I got it wrong :-(


Can you please give the attached patch a try ?


FWIW, an other option would be to opal_output(orte_help_output, ...) but 
we would have to make orte_help_output "public first.



Cheers,


Gilles




On 9/11/2018 11:14 AM, emre brookes wrote:

Gilles Gouaillardet wrote:
I investigated a this a bit and found that the (latest ?) v3 branches 
have the expected behavior


(e.g. the error messages is sent to stderr)


Since it is very unlikely Open MPI 2.1 will ever be updated, I can 
simply encourage you to upgrade to a newer Open MPI version.


Latest fully supported versions are currently such as 3.1.2 or 3.0.2



Cheers,

Gilles



So you tested 3.1.2 or something newer with this error?


But the originally reported error still goes to stdout:

$ /src/ompi-3.1.2/bin/mpicxx test_without_mpi_abort.cpp
$ /src/ompi-3.1.2/bin/mpirun -np 2 a.out > stdout
-- 

mpirun detected that one or more processes exited with non-zero 
status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[22380,1],0]
  Exit code:    255
-- 


$ cat stdout
hello from 0
hello from 1
---
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
---
$

-Emre





On 9/11/2018 2:27 AM, Ralph H Castain wrote:
I’m not sure why this would be happening. These error outputs go 
through the “show_help” functionality, and we specifically target it 
at stderr:


 /* create an output stream for us */
 OBJ_CONSTRUCT(, opal_output_stream_t);
 lds.lds_want_stderr = true;
 orte_help_output = opal_output_open();

Jeff: is it possible the opal_output system is ignoring the request 
and pushing it to stdout??

Ralph



On Sep 5, 2018, at 4:11 AM, emre brookes  wrote:

Thanks Gilles,

My goal is to separate openmpi errors from the stdout of the MPI 
program itself so that errors can be identified externally (in 
particular in an external framework running MPI jobs from various 
developers).


My not so "well written MPI program" was doing this:
   MPI_Finalize();
   exit( errorcode );
Which I assume you are telling me was bad practice & will replace with
   MPI_Abort( MPI_COMM_WORLD, errorcode );
   MPI_Finalize();
   exit( errorcode );
I was previously a bit put off of MPI_Abort due to the vagueness of 
the man page:

_Description_
This routine makes a "best attempt" to abort all tasks in the 
group of comm. This function does not require that the invoking 
environment take any action with the error code. However, a UNIX 
or POSIX environment should handle this as a return errorcode from 
the main program or an abort (errorcode).
& I didn't really have an MPI issue to "Abort", but had used this 
for a user input or parameter issue.

Nevertheless, I accept your best practice recommendation.

It was not only the originally reported message, other messages 
went to stdout.
Initially used the Ubuntu 16 LTS  "$ apt install openmpi-bin 
libopenmpi-dev" which got me version (1.10.2),
but this morning compiled and tested 2.1.5, with the same behavior, 
e.g.:


$ /src/ompi-2.1.5/bin/mpicxx test_using_mpi_abort.cpp
$ /src/ompi-2.1.5/bin/mpirun -np 2 a.out > stdout
[domain-name-embargoed:26078] 1 more process has sent help message 
help-mpi-api.txt / mpi-abort
[domain-name-embargoed:26078] Set MCA parameter 
"orte_base_help_aggregate" to 0 to see all help / error messages

$ cat stdout
hello from 0
hello from 1
-- 


MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
-- 


$

Tested 3.1.2, where this has been *somewhat* fixed:

$ /src/ompi-3.1.2/bin/mpicxx test_using_mpi_abort.cpp
$ /src/ompi-3.1.2/bin/mpirun -np 2 a.out > stdout
-- 


MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
-- 

[domain-name-embargoed:19784] 1 more process has sent help message 
help-mpi-api.txt / mpi-abort
[domain-name-embargoed:19784] Set MCA parameter 
"orte_base_help_aggregate" to 0 to see all help / error messages

$ cat stdout
h

Re: [OMPI users] stdout/stderr question

2018-09-10 Thread Gilles Gouaillardet
I investigated a this a bit and found that the (latest ?) v3 branches 
have the expected behavior


(e.g. the error messages is sent to stderr)


Since it is very unlikely Open MPI 2.1 will ever be updated, I can 
simply encourage you to upgrade to a newer Open MPI version.


Latest fully supported versions are currently such as 3.1.2 or 3.0.2



Cheers,

Gilles




On 9/11/2018 2:27 AM, Ralph H Castain wrote:

I’m not sure why this would be happening. These error outputs go through the 
“show_help” functionality, and we specifically target it at stderr:

 /* create an output stream for us */
 OBJ_CONSTRUCT(, opal_output_stream_t);
 lds.lds_want_stderr = true;
 orte_help_output = opal_output_open();

Jeff: is it possible the opal_output system is ignoring the request and pushing 
it to stdout??
Ralph



On Sep 5, 2018, at 4:11 AM, emre brookes  wrote:

Thanks Gilles,

My goal is to separate openmpi errors from the stdout of the MPI program itself 
so that errors can be identified externally (in particular in an external 
framework running MPI jobs from various developers).

My not so "well written MPI program" was doing this:
   MPI_Finalize();
   exit( errorcode );
Which I assume you are telling me was bad practice & will replace with
   MPI_Abort( MPI_COMM_WORLD, errorcode );
   MPI_Finalize();
   exit( errorcode );
I was previously a bit put off of MPI_Abort due to the vagueness of the man 
page:

_Description_
This routine makes a "best attempt" to abort all tasks in the group of comm. 
This function does not require that the invoking environment take any action with the 
error code. However, a UNIX or POSIX environment should handle this as a return errorcode 
from the main program or an abort (errorcode).

& I didn't really have an MPI issue to "Abort", but had used this for a user 
input or parameter issue.
Nevertheless, I accept your best practice recommendation.

It was not only the originally reported message, other messages went to stdout.
Initially used the Ubuntu 16 LTS  "$ apt install openmpi-bin libopenmpi-dev" 
which got me version (1.10.2),
but this morning compiled and tested 2.1.5, with the same behavior, e.g.:

$ /src/ompi-2.1.5/bin/mpicxx test_using_mpi_abort.cpp
$ /src/ompi-2.1.5/bin/mpirun -np 2 a.out > stdout
[domain-name-embargoed:26078] 1 more process has sent help message 
help-mpi-api.txt / mpi-abort
[domain-name-embargoed:26078] Set MCA parameter "orte_base_help_aggregate" to 0 
to see all help / error messages
$ cat stdout
hello from 0
hello from 1
--
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--
$

Tested 3.1.2, where this has been *somewhat* fixed:

$ /src/ompi-3.1.2/bin/mpicxx test_using_mpi_abort.cpp
$ /src/ompi-3.1.2/bin/mpirun -np 2 a.out > stdout
--
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--
[domain-name-embargoed:19784] 1 more process has sent help message 
help-mpi-api.txt / mpi-abort
[domain-name-embargoed:19784] Set MCA parameter "orte_base_help_aggregate" to 0 
to see all help / error messages
$ cat stdout
hello from 1
hello from 0
$

But the originally reported error still goes to stdout:

$ /src/ompi-3.1.2/bin/mpicxx test_without_mpi_abort.cpp
$ /src/ompi-3.1.2/bin/mpirun -np 2 a.out > stdout
--
mpirun detected that one or more processes exited with non-zero status, thus 
causing
the job to be terminated. The first process to do so was:

  Process name: [[22380,1],0]
  Exit code:255
--
$ cat stdout
hello from 0
hello from 1
---
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
---
$

Summary:
1.10.2, 2.1.5 both send most openmpi generated messages to stdout.
3.1.2 sends at least one type of openmpi generated messages to stdout.
I'll continue with my "wrapper" strategy for now, as it seems safest and
most broadly deployable [e.g. on compute resources where I need to use admin 
installed versions of MPI],
but it would be nice for openmpi to ensure

Re: [OMPI users] stdout/stderr question

2018-09-04 Thread Gilles Gouaillardet
Open MPI should likely write this message on stderr, I will have a look 
at that.



That being said, and though I have no intention to dodge the question, 
this case should not happen.


A well written (MPI) program should either exit(0) or have main() return 
0, so this scenario


(e.g. all MPI tasks call MPI_Finalize() and then at least one MPI task 
exit with a non zero error code)


should not happen.


If your program might fail, it should call MPI_Abort() with a non zero 
error code *before* calling MPI_Finalize().


note this error can occur if your main() subroutine does not return any 
value (e.g. it returns an undefined value, that might be non zero)



Cheers,


Gilles


On 9/5/2018 6:08 AM, emre brookes wrote:

Background:
---
Running on ubuntu 16.04 with apt install openmpi-bin libopenmpi-dev
$  mpirun --version
mpirun (Open MPI) 1.10.2

I did search thru the docs a bit (ok, maybe I missed something 
obvious, my apologies if so)

---
Question:

Is there some setting to turn off the extra messages generated by 
openmpi ?


e.g.
$ mpirun -np 2 my_job > my_job.stdout
adds this message to my_job.stdout
---
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
---
which strangely goes to stdout and not stderr.
I would intuitively expect error or notice messages to go to stderr.
Is there a way to redirect these messages to stderr or some specified 
file?


I need to separate this from the collected stdout of the job processes 
themselves.


Somewhat kludgy options that come to mind:

1. I can use --output-filename outfile, which does separate the 
"openmpi" messages,
but this creates a file for each process and I'd rather keep them as 
produced in one file,

but without any messages from openmpi, which I'd like to keep separately.

2. Or I could write a script to filter the output and separate. A bit 
risky as someone could conceivably put something that looks like a 
openmpi message pattern in the mpi executable output.


3. hack the source code of openmpi.

Any suggestions as to a more elegant or standard way of dealing with 
this?


TIA,
Emre.



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] MPI_MAXLOC problems

2018-08-24 Thread Gilles Gouaillardet
Diego,

Try calling allreduce with count=1

Cheers,

Gilles

On Wednesday, August 22, 2018, Diego Avesani 
wrote:

> Dear all,
>
> I am going to start again the discussion about MPI_MAXLOC. We had one a
> couple of week before with George, Ray, Nathan, Jeff S, Jeff S., Gus.
>
> This because I have a problem. I have two groups and two communicators.
> The first one takes care of compute the maximum vale and to which
> processor it belongs:
>
> nPart = 100
>
> IF(MPI_COMM_NULL .NE. MPI_MASTER_COMM)THEN
>
> CALL MPI_ALLREDUCE( EFFMAX, EFFMAXW, 2, MPI_2DOUBLE_PRECISION, MPI_MAXLOC,
> MPI_MASTER_COMM,MPImaster%iErr )
> whosend = INT(EFFMAXW(2))
> gpeff   = EFFMAXW(1)
> CALL MPI_BCAST(whosend,1,MPI_INTEGER,whosend,MPI_MASTER_
> COMM,MPImaster%iErr)
>
> ENDIF
>
> If I perform this,  the program set to zero one variable, specifically
> nPart.
>
> if I print:
>
>  IF(MPI_COMM_NULL .NE. MPI_MASTER_COMM)THEN
>   WRITE(*,*) MPImaster%rank,nPart
>  ELSE
>   WRITE(*,*) MPIlocal%rank,nPart
>  ENDIF
>
> I get;
>
> 1 2
> 1 2
> 3 2
> 3 2
> 2 2
> 2 2
> 1 2
> 1 2
> 3 2
> 3 2
> 2 2
> 2 2
>
>
> 1 0
> 1 0
> 0 0
> 0 0
>
> This seems some typical memory allocation problem.
>
> What do you think?
>
> Thanks for any kind of help.
>
>
>
>
> Diego
>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] MPI group and stuck in communication

2018-08-20 Thread Gilles Gouaillardet

Diego,


first, try using MPI_IN_PLACE when sendbuffer and recvbuffer are identical


at first glance, the second allreduce should be in MPI_COMM_WORLD (with 
counter=0 when master_comm is null),


or you have to add an extra broadcast in local_comm


Cheers,


Gilles


On 8/20/2018 3:56 PM, Diego Avesani wrote:

Dear George, Dear Gilles, Dear Jeff, Deal all,

Thank for all the suggestions.
The problem is that I do not want to FINALIZE, but only to exit from a 
cycle.

This is my code:
I have:
master_group;
each master sends to its slaves only some values;
the slaves perform something;
according to a counter, every processor has to leave a cycle.

Here an example, if you want I can give you more details.

DO iRun=1,nRun
   !
   IF(MPI_COMM_NULL .NE. MPI_MASTER_COMM)THEN
      VARS(1) = REAL(iRun+1)
      VARS(2) = REAL(iRun+100)
      VARS(3) = REAL(iRun+200)
      VARS(4) = REAL(iRun+300)
   ENDIF
   !
   CALL MPI_BCAST(VARS,4,MPI_DOUBLE_PRECISION,0,MPI_LOCAL_COMM,iErr)
   !
   test = SUM(VARS)
   !
   CALL MPI_ALLREDUCE(test, test, 1, MPI_DOUBLE_PRECISION, MPI_SUM, 
MPI_LOCAL_COMM,iErr)

   !
   !
   counter = test
   !
   CALL MPI_ALLREDUCE(counter, counter, 1, MPI_DOUBLE_PRECISION, 
MPI_SUM, MPI_MASTER_COMM,iErr)

   !
   IF(counter.GT.1)THEN
      EXIT
   ENDIF
ENDDO

My original code stucks on the cycle and I do not know why.

Thanks





Diego


On 13 August 2018 at 23:44, George Reeke > wrote:



>         On Aug 12, 2018, at 2:18 PM, Diego Avesani
>         mailto:diego.aves...@gmail.com>> wrote:
>         >
>         > For example, I have to exit to a cycle, according to a
>         check:
>         >
>         > IF(counter.GE.npercstop*nParticles)THEN
>         >         flag2exit=1
>         >         WRITE(*,*) '-Warning PSO has been exit'
>         >         EXIT pso_cycle
>         >      ENDIF
>         >
>         > But this is difficult to do since I have to exit only
after
>         all the threats inside a set have finish their task.
>         >
>         > Do you have some suggestions?
>         > Do you need other information?
>
Dear Diego et al,
Assuming I understand your problem:
The way I do this is set up one process that is responsible for normal
and error exits.  It sits looking for messages from all the other
ranks
that are doing work.  Certain messages are defined to indicate an
error
exit with an error number or some text.  The exit process is
spawned by
the master process at startup and is told how many working
processes are
there.  Each process either sends an OK exit when it is done or an
error
message.  The exit process counts these exit messages and when the
count
equals the number of working processes, it prints any/all errors, then
sends messages back to all the working processes, which, at this time,
should be waiting for these and they can terminate with MPI_Finalize.
   Of course it is more complicated than that to handle special cases
like termination before everything has really started or when the
protocol is not followed, debug messages that do not initiate
termination, etc. but maybe this will give you an idea for one
way to deal with this issue.
George Reeke




___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users





___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] MPI group and stuck in communication

2018-08-13 Thread Gilles Gouaillardet
Diego,

Since this question is not Open MPI specific, Stack Overflow (or similar
forum) is a better place to ask.
Make sure you first read https://stackoverflow.com/help/mcve

Feel free to post us a link to your question.


Cheers,

Gilles

On Monday, August 13, 2018, Diego Avesani  wrote:

> dear Jeff, dear all,
>
> its my fault.
>
> Can I send an attachment?
> thanks
>
> Diego
>
>
> On 13 August 2018 at 19:06, Jeff Squyres (jsquyres) 
> wrote:
>
>> On Aug 12, 2018, at 2:18 PM, Diego Avesani 
>> wrote:
>> >
>> > Dear all, Dear Jeff,
>> > I have three communicator:
>> >
>> > the standard one:
>> > MPI_COMM_WORLD
>> >
>> > and other two:
>> > MPI_LOCAL_COMM
>> > MPI_MASTER_COMM
>> >
>> > a sort of two-level MPI.
>> >
>> > Suppose to have 8 threats,
>> > I use 4 threats for run the same problem with different value. These
>> are the LOCAL_COMM.
>> > In addition I have a MPI_MASTER_COMM to allow the master of each group
>> to communicate.
>>
>> I don't understand what you're trying to convey here, sorry.  Can you
>> draw it, perhaps?
>>
>> (I am assuming you mean "threads", not "threats")
>>
>> > These give me some problem.
>> >
>> >
>> > For example, I have to exit to a cycle, according to a check:
>> >
>> > IF(counter.GE.npercstop*nParticles)THEN
>> > flag2exit=1
>> > WRITE(*,*) '-Warning PSO has been exit'
>> > EXIT pso_cycle
>> >  ENDIF
>> >
>> > But this is difficult to do since I have to exit only after all the
>> threats inside a set have finish their task.
>> >
>> > Do you have some suggestions?
>> > Do you need other information?
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>>
>>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] openMPI and ifort debuggin flags, is it possible?

2018-08-03 Thread Gilles Gouaillardet
Diego,

Yes, that would be clearly an issue.

Cheers,

Gilles

On Friday, August 3, 2018, Diego Avesani  wrote:

> Dear Gilles, dear all,
>
> I do not remember.
> I use -r8 when I compile.
>
> What do you think?
> It could be a problem?
>
>
> Thanks a lot
>
> Diego
>
>
> On 27 July 2018 at 16:05, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
>> Diego,
>>
>> Did you build OpenMPI with FCFLAGS=-r8 ?
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On Friday, July 27, 2018, Diego Avesani  wrote:
>>
>>> Dear all,
>>>
>>> I am developing a code for hydrological applications. It is written in
>>> FORTRAN and I am using ifort combine with openMPI.
>>>
>>> In this moment, I am debugging my code due to the fact that I have some
>>> NaN errors. As a consequence, I have introduce in my Makefile some flags
>>> for the ifort compiler. In particular:
>>>
>>> -c -r8 -align *-CB -traceback -check all -check uninit -ftrapuv -debug
>>> all* -fpp
>>>
>>> However, this produce some unexpected errors/warning with mpirun.  This
>>> is the error\warning:
>>>
>>> Image  PCRoutineLine
>>>Source
>>> MPIHyperStrem  005AA3F0  Unknown   Unknown
>>>  Unknown
>>> MPIHyperStrem  00591A5C  mod_lathyp_mp_lat 219
>>>  LATHYP.f90
>>> MPIHyperStrem  005A0C2A  mod_optimizer_mp_ 279
>>>  OPTIMIZER.f90
>>> MPIHyperStrem  005986F2  mod_optimizer_mp_  34
>>>  OPTIMIZER.f90
>>> MPIHyperStrem  005A1F84  MAIN__114
>>>  MAIN.f90
>>> MPIHyperStrem  0040A46E  Unknown   Unknown
>>>  Unknown
>>> libc-2.23.so   7FEA758B8830  __libc_start_main Unknown
>>>  Unknown
>>> MPIHyperStrem  0040A369  Unknown   Unknown
>>>  Unknown
>>> forrtl: warning (406): fort: (1): In call to SHUFFLE, an array temporary
>>> was created for argument #1
>>>
>>>
>>>
>>> My questions is:
>>> It is possible to use ifort debugging flags with openMPI?
>>>
>>> thanks a lot
>>>
>>> Diego
>>>
>>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>>
>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] openMPI and ifort debuggin flags, is it possible?

2018-07-27 Thread Gilles Gouaillardet
Diego,

Did you build OpenMPI with FCFLAGS=-r8 ?

Cheers,

Gilles

On Friday, July 27, 2018, Diego Avesani  wrote:

> Dear all,
>
> I am developing a code for hydrological applications. It is written in
> FORTRAN and I am using ifort combine with openMPI.
>
> In this moment, I am debugging my code due to the fact that I have some
> NaN errors. As a consequence, I have introduce in my Makefile some flags
> for the ifort compiler. In particular:
>
> -c -r8 -align *-CB -traceback -check all -check uninit -ftrapuv -debug
> all* -fpp
>
> However, this produce some unexpected errors/warning with mpirun.  This is
> the error\warning:
>
> Image  PCRoutineLineSource
>
> MPIHyperStrem  005AA3F0  Unknown   Unknown
>  Unknown
> MPIHyperStrem  00591A5C  mod_lathyp_mp_lat 219
>  LATHYP.f90
> MPIHyperStrem  005A0C2A  mod_optimizer_mp_ 279
>  OPTIMIZER.f90
> MPIHyperStrem  005986F2  mod_optimizer_mp_  34
>  OPTIMIZER.f90
> MPIHyperStrem  005A1F84  MAIN__114
>  MAIN.f90
> MPIHyperStrem  0040A46E  Unknown   Unknown
>  Unknown
> libc-2.23.so   7FEA758B8830  __libc_start_main Unknown
>  Unknown
> MPIHyperStrem  0040A369  Unknown   Unknown
>  Unknown
> forrtl: warning (406): fort: (1): In call to SHUFFLE, an array temporary
> was created for argument #1
>
>
>
> My questions is:
> It is possible to use ifort debugging flags with openMPI?
>
> thanks a lot
>
> Diego
>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Error in file base/plm_base_launch_support.c: OPAL_HWLOC_TOPO

2018-07-21 Thread Gilles Gouaillardet
Brian,

As Ralph already stated, this is likely a hwloc API issue.
From debian9, you can
lstopo --of xml | ssh debian8 lstopo --if xml -i -

that will likely confirm the API error.

If you are willing to get a bit more details, you can add some printf
in opal_hwloc_unpack (from opal/mca/hwloc/base/hwloc_base_dt.c) to
figure out where exactly the failure occurs.

Meanwhile, you can move forward by using the embedded hwloc on both
distros (--with-hwloc=internal or no --with-hwloc option at all).


Note we strongly discourage you configure --with-FOO=/usr
(it explicitly add /usr/include and /usr/lib[64] in the search path,
and might hide some other external libraries installed in a non
standard location). In order to force the external hwloc lib installed
in the default location, --with-hwloc=external is what you need (same
thing applies to libevent and pmix)


Cheers,

Gilles
On Sun, Jul 22, 2018 at 7:52 AM r...@open-mpi.org  wrote:
>
> More than likely the problem is the difference in hwloc versions - sounds 
> like the topology to/from xml is different between the two versions, and the 
> older one doesn’t understand the new one.
>
> > On Jul 21, 2018, at 12:04 PM, Brian Smith  
> > wrote:
> >
> > Greetings,
> >
> > I'm having trouble getting openmpi 2.1.2 to work when launching a
> > process from debian 8 on a remote debian 9 host. To keep things simple
> > in this example, I'm just launching date on the remote host.
> >
> > deb8host$ mpirun -H deb9host date
> > [deb8host:01552] [[32763,0],0] ORTE_ERROR_LOG: Error in file
> > base/plm_base_launch_support.c at line 954
> >
> > It works fine when executed from debian 9:
> > deb9host$ mpirun -H deb8host date
> > Sat Jul 21 13:40:43 CDT 2018
> >
> > Also works when executed from debian 8 against debian 8:
> > deb8host:~$ mpirun -H deb8host2 date
> > Sat Jul 21 13:55:57 CDT 2018
> >
> > The failure results from an error code returned by:
> > opal_dss.unpack(buffer, , , OPAL_HWLOC_TOPO)
> >
> > openmpi was built with the same configure flags on both hosts.
> >
> >--prefix=$(PREFIX) \
> >--with-verbs \
> >--with-libfabric \
> >--disable-silent-rules \
> >--with-hwloc=/usr \
> >--with-libltdl=/usr \
> >--with-devel-headers \
> >--with-slurm \
> >--with-sge \
> >--without-tm \
> >--disable-heterogeneous \
> >--with-contrib-vt-flags=--disable-iotrace \
> >--sysconfdir=$(PREFIX)/etc \
> >--libdir=$(PREFIX)/lib\
> >--includedir=$(PREFIX)/include
> >
> >
> > deb9host libhwloc and libhwloc-plugins is 1.11.5-1
> > deb8host libhwloc and libhwloc-plugins is 1.10.0-3
> >
> > I've been trying to debug this for the past few days and would
> > appreciate any help on determining why this failure is occurring
> > and/or resolving the problem.
> >
> > --
> > Brian T. Smith
> > System Fabric Works
> > Senior Technical Staff
> > bsm...@systemfabricworks.com
> > GPG Key: B3C2C7B73BA3CD7F
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Java: wrong extent value for resized derived data type

2018-07-19 Thread Gilles Gouaillardet
Siegmar,


I must admit I am having a hard time understanding why MPI.DOUBLE.getSize()
returns 1 instead of 8 ...

Anyway, for the time being, I think you can get the expected result by
createResized(..., ..., 8).

If there is a consensus that your program is correct, I will be happy to
issue a PR
(Simply multiply lb and extent by baseSize before passing them to the
native function)

Cheers,

Gilles

On Thursday, July 19, 2018, Siegmar Gross <
siegmar.gr...@informatik.hs-fulda.de> wrote:

> Hi,
>
> I've installed openmpi-3.1.0 on my "SUSE Linux Enterprise Server
> 12.3 (x86_64)" with gcc-6.4.0. Why do I get "extent: 0" instead of
> "extent: 1" for my small program. In my opinion the extent of the
> old data type and the resized extent of the new data type should
> be the same. Am I wrong, is something wrong with my program, or
> results the unexpected value from an error of the MPI Java method
> getExtent() for a derived data type? I get the value 1, if I use
> MPI.DOUBLE.getSize() or MPI.DOUBLE.getExtent().
>
> loki java 130 which \mpijavac
> /usr/local/openmpi-3.1.0_64_gcc/bin/mpijavac
> loki java 131 \mpijavac SizeExtentMain.java
> loki java 132 mpiexec -np 1 java SizeExtentMain
> strided vector:
>   size of old data type: 1
>   count: 2
>   blocklength:   2
>   stride:4
>   size:  4
>   lower bound:   0
>   extent:0
>   true lower bound:  0
>   true extent:   6
> loki java 133
>
> I would be grateful, if somebody can fix the problem, if it is a
> problem of the MPI Java method or if somebody knows, what I'm doing
> wrong in my program. Thank you very much for any help in advance.
>
>
> Kind regards
>
> Siegmar
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Fw: Open MPI not building

2018-07-09 Thread Gilles Gouaillardet

Henry,


this is what I have


$ md5sum openmpi-3.1.0/configure

78f5308c46545a9cf1583593328592ac  openmpi-3.1.0/configure


from

0895e268ca27735d7654bf64cee6c256  openmpi-3.1.0.tar.bz2
(as advertised on https://www.open-mpi.org/software/ompi/v3.1/)



Just to be clear, if you choose to install Open MPI from a tarball, you 
do not need (and you likely do not want)


to run autogen.pl --force

note autogen.pl will regenerate configure (and hence the md5sum will 
very likely change)




Best regards,


Gilles


On 7/9/2018 10:14 PM, Lovelace III, Henry wrote:


Hi,

    I did not make any changes to the configure file, below is the out 
put of the***md5sum*:



*1f1ccf59888dc739fad0f9fcff6175d5 configure*

*
*

Thanks,


Henry


*From:* users  on behalf of Gilles 
Gouaillardet 

*Sent:* Sunday, July 8, 2018 8:25:58 PM
*To:* users@lists.open-mpi.org
*Subject:* Re: [OMPI users] Fw: Open MPI not building
Henry,



By any change, did you manually edit 'configure' and remove that line

PACKAGE_VERSION=3.1.0


or even replace it with

PACKAGE_VERSION=



would you mind running

md5sum configure

and post the output ?



note you did not configure --prefix=/.../, Open MPI will be installed in
/usr/local

so since you are building it from a non root account, you should

sudo make install


Cheers,


Gilles


On 7/9/2018 8:49 AM, Lovelace III, Henry wrote:
>
> Hi,
>
>    opal_crs.7 does not generate, *package-version* requires argument.
> How do I fix this error?
>
> Thanks
>
>
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Fw: Open MPI not building

2018-07-09 Thread Gilles Gouaillardet
Henry,

I have a different md5sum from the 3.1.0 tarball.

Did you run ?
autogen.pl --force 

The easiest path is likely you download again a tarball and run configure 
without autogen.pl


Cheers,

Gilles

"Lovelace III, Henry"  wrote:
> 
>
>Hi,
>
>    I did not make any changes to the configure file, below is the out put of 
>the md5sum:
>
>
>1f1ccf59888dc739fad0f9fcff6175d5  configure
>
>
>Thanks,
>
>
>Henry
>
>From: users  on behalf of Gilles 
>Gouaillardet 
>Sent: Sunday, July 8, 2018 8:25:58 PM
>To: users@lists.open-mpi.org
>Subject: Re: [OMPI users] Fw: Open MPI not building 
>
> 
>
>Henry,
>
>
>
>By any change, did you manually edit 'configure' and remove that line
>
>PACKAGE_VERSION=3.1.0
>
>
>or even replace it with
>
>PACKAGE_VERSION=
>
>
>
>would you mind running
>
>md5sum configure
>
>and post the output ?
>
>
>
>note you did not configure --prefix=/.../, Open MPI will be installed in 
>/usr/local
>
>so since you are building it from a non root account, you should
>
>sudo make install
>
>
>Cheers,
>
>
>Gilles
>
>
>On 7/9/2018 8:49 AM, Lovelace III, Henry wrote:
>>
>> Hi,
>>
>>    opal_crs.7 does not generate, *package-version* requires argument. 
>> How do I fix this error?
>>
>> Thanks
>>
>>
>>
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>
>___
>users mailing list
>users@lists.open-mpi.org
>https://lists.open-mpi.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Fw: Open MPI not building

2018-07-08 Thread Gilles Gouaillardet

Henry,



By any change, did you manually edit 'configure' and remove that line

PACKAGE_VERSION=3.1.0


or even replace it with

PACKAGE_VERSION=



would you mind running

md5sum configure

and post the output ?



note you did not configure --prefix=/.../, Open MPI will be installed in 
/usr/local


so since you are building it from a non root account, you should

sudo make install


Cheers,


Gilles


On 7/9/2018 8:49 AM, Lovelace III, Henry wrote:


Hi,

   opal_crs.7 does not generate, *package-version* requires argument. 
How do I fix this error?


Thanks





___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] MPI_Ialltoallv

2018-07-06 Thread Gilles Gouaillardet
Clyde,

thanks for reporting the issue.

Can you please give the attached patch a try ?


Cheers,

Gilles

FWIW, the nbc module was not initially specific to Open MPI, and hence used
standard MPI subroutines.
In this case, we can avoid the issue by calling internal Open MPI
subroutines.
This is an intermediate patch, since similar issues might potentially occur
in other places


On Fri, Jul 6, 2018 at 11:12 PM Stanfield, Clyde <
clyde.stanfi...@radiantsolutions.com> wrote:

> We are using MPI_Ialltoallv for an image processing algorithm. When doing
> this we pass in an MPI_Type_contiguous with an MPI_Datatype of
> MPI_C_FLOAT_COMPLEX which ends up being the size of multiple rows of the
> image (based on the number of nodes used for distribution). In addition
> sendcounts, sdispls, resvcounts, and rdispls all fit within a signed int.
> Usually this works without any issues, but when we lower our number of
> nodes we sometimes see failures.
>
>
>
> What we found is that even though we can fit everything into signed ints,
> line 528 of nbc_internal.h ends up calling a malloc with an int that
> appears to be the size of the (num_distributed_rows * num_columns  *
> sizeof(std::complex)) which in very large cases wraps back to
> negative.  As a result we end up seeing “Error in malloc()” (line 530 of
> nbc_internal.h) throughout our output.
>
>
>
> We can get around this issue by ensuring the sum of our contiguous type
> never exceeds 2GB. However, this was unexpected to us as our understanding
> was that all long as we can fit all the parts into signed ints we should be
> able to transfer more than 2GB at a time. Is it intended that
> MPI_Ialltoallv requires your underlying data to be less than 2GB or is this
> in error in how malloc is being called (should be called with a size_t
> instead of an int)?
>
>
>
> Thanks,
>
> Clyde Stanfield
>
>
>
> [image:
> https://digitalglobe-marketing.s3.amazonaws.com/files/signature/radiantsolutions-sig-logo.png]
> 
>
> *Clyde Stanfield*
> Software Engineer
> 734-480-5100 office 
> clyde.stanfi...@mdaus.com
>
> [image:
> https://digitalglobe-marketing.s3.amazonaws.com/files/signature/radiantsolutions-twitter-wide.png]
>  [image:
> https://digitalglobe-marketing.s3.amazonaws.com/files/signature/radiantsolutions-linkedin-wide.png]
> 
>
>
>
>
>
>
> The information contained in this communication is confidential, is
> intended only for the use of the recipient(s) named above, and may be
> legally privileged. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution, or
> copying of this communication is strictly prohibited. If you have received
> this communication in error, please re-send this communication to the
> sender and delete the original message or any copy of it from your computer
> system.
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users


nbc_copy.diff
Description: Binary data
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Disable network interface selection

2018-07-01 Thread Gilles Gouaillardet

Carlos,


Open MPI 3.0.2 has been released, and it contains several bug fixes, so I do

encourage you to upgrade and try again.



if it still does not work, can you please run

mpirun --mca oob_base_verbose 10 ...

and then compress and post the output ?


out of curiosity, would

mpirun --mca routed_radix 1 ...

work in your environment ?


once we can analyze the logs, we should be able to figure out what is 
going wrong.



Cheers,

Gilles

On 6/29/2018 4:10 AM, carlos aguni wrote:

Just realized my email wasn't sent to the archive.

On Sat, Jun 23, 2018 at 5:34 PM, carlos aguni > wrote:


Hi!

Thank you all for your reply Jeff, Gilles and rhc.

Thank you Jeff and rhc for clarifying to me some of the openmpi's
internals.

>> FWIW: we never send interface names to other hosts - just dot
addresses
> Should have clarified - when you specify an interface name for the
MCA param, then it is the interface name that is transferred as
that is the value of the MCA param. However, once we determine our
address, we only transfer dot addresses between ourselves

If only dot addresses are sent to the hosts then why doesn't
openmpi use the default route like `ip route get `
instead of choosing a random one? Is it an expected behaviour? Can
it be changed?

Sorry. As Gilles pointed out I forgot to mention which openmpi
version I was using. I'm using openmpi 3.0.0 gcc 7.3.0 from
openhpc. Centos 7.5.

> mpirun—mca oob_tcp_if_exclude192.168.100.0/24
...

I cannot just exclude that interface cause after that I want to
add another computer that's on a different network. And this is
where things get messy :( I cannot just include and exclude
networks cause I have different machines on different networks.
This is what I want to achieve:




compute01



compute02



compute03

ens3



192.168.100.104/24 



10.0.0.227/24 



192.168.100.105/24 

ens8



10.0.0.228/24 



172.21.1.128/24 



---

ens9



172.21.1.155/24 



---



---


So I'm in compute01 MPI_spawning another process on compute02 and
compute03.
With both MPI_Spawn and `mpirun -n 3 -host
compute01,compute02,compute03 hostname`

Then when I include the mca parameters I get this:
`mpirun --oversubscribe --allow-run-as-root -n 3 --mca
oob_tcp_if_include 10.0.0.0/24,192.168.100.0/24
 -host
compute01,compute02,compute03 hostname`
WARNING: An invalid value was given for oob_tcp_if_include. This
value will be ignored.
...
Message:    Did not find interface matching this subnet

This would all work if it were to use the system's internals like
`ip route`.

Best regards,
Carlos.




___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Disable network interface selection

2018-06-22 Thread Gilles Gouaillardet
Carlos,

By any chance, could

mpirun—mca oob_tcp_if_exclude 192.168.100.0/24 ...

work for you ?

Which Open MPI version are you running ?


IIRC, subnets are internally translated to interfaces, so that might be an
issue if
the translation if made on the first host, and then the interface name is
sent to the other hosts.

Cheers,

Gilles

On Saturday, June 23, 2018, carlos aguni  wrote:

> Hi all,
>
> I'm trying to run a code on 2 machines that has at least 2 network
> interfaces in it.
> So I have them as described below:
>
>
> compute01
>
> compute02
>
> ens3
>
> 192.168.100.104/24
>
> 10.0.0.227/24
>
> ens8
>
> 10.0.0.228/24
>
> 172.21.1.128/24
>
> ens9
>
> 172.21.1.155/24
>
> ---
>
> Issue is. When I execute `mpirun -n 2 -host compute01,compute02 hostname`
> on them what I get is the correct output after a very long delay..
>
> What I've read so far is that OpenMPI performs a greedy algorithm on each
> interface that timeouts if it doesn't find the desired IP.
> Then I saw here (https://www.open-mpi.org/faq/?category=tcp#tcp-selection)
> that I can run commands like:
> `$ mpirun -n 2 --mca oob_tcp_if_include 10.0.0.0/24 -n 2 -host
> compute01,compute02 hosname`
> But this configuration doesn't reach the other host(s).
> In the end I sometimes I get the same timeout.
>
> So is there a way to let it to use the system's default route?
>
> Regards,
> Carlos.
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Enforcing specific interface and subnet usage

2018-06-19 Thread Gilles Gouaillardet
What is exactly the issue you are facing ?

You also need to force the subnet used by oob/tcp
mpirun —mca oob_tcp_if_include 10.233.0.0/19 ...

iirc, Open MPI might discard addresses from a bridge interface, but I do
not exactly remember if it affects both btl/tcp and/or oob/tcp and/or none
by default.

 Cheers,

Gilles

On Tuesday, June 19, 2018, Maksym Planeta 
wrote:

> Hello,
>
> I want to force OpenMPI to use TCP and in particular use a particular
> subnet. Unfortunately, I can't manage to do that.
>
> Here is what I try:
>
> $BIN/mpirun --mca pml ob1 --mca btl tcp,self --mca
> ptl_tcp_remote_connections 1 --mca btl_tcp_if_include '10.233.0.0/19' -np
> 4  --oversubscribe -H ib1n,ib2n bash -c 'echo $PMIX_SERVER_URI2'
>
> The expected result would be a list of IP addresses in 10.233.0.0 subnet,
> but instead I get this:
>
> 2659516416.2;tcp4://127.0.0.1:46777
> 2659516416.2;tcp4://127.0.0.1:46777
> 2659516416.1;tcp4://127.0.0.1:45055
> 2659516416.1;tcp4://127.0.0.1:45055
>
> Could you help me to debug this problem somehow?
>
> The IP addresses are completely available in the desired subnet
>
> $BIN/mpirun --mca pml ob1 --mca btl tcp,self  --mca
> ptl_tcp_remote_connections 1 --mca btl_tcp_if_include '10.233.0.0/19' -np
> 4  --oversubscribe -H ib1n,ib2n ip addr show dev br0
>
> Returns a set of bridges looking like:
>
> 9: br0:  mtu 1500 qdisc noqueue state UP
> group default qlen 1000
> link/ether 94:de:80:ba:37:e4 brd ff:ff:ff:ff:ff:ff
> inet 141.76.49.17/26 brd 141.76.49.63 scope global br0
>valid_lft forever preferred_lft forever
> inet 10.233.0.82/19 scope global br0
>valid_lft forever preferred_lft forever
> inet6 2002:8d4c:3001:48:40de:80ff:feba:37e4/64 scope global
> deprecated mngtmpaddr dynamic
>valid_lft 59528sec preferred_lft 0sec
> inet6 fe80::96de:80ff:feba:37e4/64 scope link tentative dadfailed
>valid_lft forever preferred_lft forever
> 
>
> What is more boggling is that if I attache with a debugger at
> opal/mca/pmix/pmix3x/pmix/src/mca/ptl/tcp/ptl_tcp_components.c around
> line 500 I see that mca_ptl_tcp_component.remote_connections is false.
> This means that the way I set up component parameters is ignored.
>
> --
> Regards,
> Maksym Planeta
>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] A couple of general questions

2018-06-14 Thread Gilles Gouaillardet
Charles,


If you are using infiniband hardware, the recommended way is to use UCX.


Cheers,

Gilles

On Thursday, June 14, 2018, Charles A Taylor  wrote:

> Because of the issues we are having with OpenMPI and the openib BTL
> (questions previously asked), I’ve been looking into what other transports
> are available.  I was particularly interested in OFI/libfabric support but
> cannot find any information on it more recent than a reference to the usNIC
> BTL from 2015 (Jeff Squyres, Cisco).  Unfortunately, the openmpi-org
> website FAQ’s covering OpenFabrics support don’t mention anything beyond
> OpenMPI 1.8.  Given that 3.1 is the current stable version, that seems odd.
>
> That being the case, I thought I’d ask here. After laying down the
> libfabric-devel RPM and building (3.1.0) with —with-libfabric=/usr, I end
> up with an “ofi” MTL but nothing else.   I can run with OMPI_MCA_mtl=ofi
> and OMPI_MCA_btl=“self,vader,openib” but it eventually crashes in
> libopen-pal.so.   (mpi_waitall() higher up the stack).
>
> GIZMO:9185 terminated with signal 11 at PC=2b4d4b68a91d SP=7ffcfbde9ff0.
> Backtrace:
> /apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libopen-
> pal.so.40(+0x9391d)[0x2b4d4b68a91d]
> /apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libopen-
> pal.so.40(opal_progress+0x24)[0x2b4d4b632754]
> /apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libmpi.so.
> 40(ompi_request_default_wait_all+0x11f)[0x2b4d47be2a6f]
> /apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libmpi.so.
> 40(PMPI_Waitall+0xbd)[0x2b4d47c2ce4d]
>
> Questions: Am I using the OFI MTL as intended?   Should there be an “ofi”
> BTL?   Does anyone use this?
>
> Thanks,
>
> Charlie Taylor
> UF Research Computing
>
> PS - If you could use some help updating the FAQs, I’d be willing to put
> in some time.  I’d probably learn a lot.
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --oversubscribe option

2018-06-06 Thread Gilles Gouaillardet
Mahmoud,

By default 1 slot = 1 core, that is why you need —oversubscribe or
—use-hwthread-cpus to run 16 MPI tasks.

It seems your lammps job benefits from hyper threading. Some applications
behave like this, and this is not odd a priori.

Cheers,

Gilles

On Wednesday, June 6, 2018, r...@open-mpi.org  wrote:

> I’m not entirely sure what you are asking here. If you use oversubscribe,
> we do not bind your processes and you suffer some performance penalty for
> it. If you want to run one process/thread and retain binding, then do not
> use --oversubscribe and instead use --use-hwthread-cpus
>
>
> On Jun 6, 2018, at 3:06 AM, Mahmood Naderan  wrote:
>
> Hi,
> On a Ryzen 1800x which has 8 cores and 16 threads, when I run "mpirun -np
> 16 lammps..." I get an error that there is not enough slot. It seems that
> --oversubscribe option will fix that.
>
> Odd thing is that when I run "mpirun -np 8 lammps" it takes about 46
> minutes to complete the job while with "mpirun --oversubscribe -np 16
> lammps" it takes about 39 minutes.
>
> I want to be sure that I "-np 16" uses are logical cores. Is that
> confirmed with --oversubscribe?
>
> Regards,
> Mahmood
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Sockets half-broken in Open MPI 2.0.2?

2018-06-06 Thread Gilles Gouaillardet
Alexander,

Note the v2.0 series is no more supported, and you should upgrade to v3.1,
v3.0 or v2.1

You might have to force the tcp buffers size to 0 for optimal performances
iirc, mpirun —mca btl_tcp_sndbuf_size 0 —mca btl_tcp_rcvbuf_size 0 ...
(I am afk, so please confirm both parameter names and default values with
ompi_info —all)
If you upgrade to the latest version, default parameter is already optimal.

Last but not least, the btl/tcp component uses all the available interfaces
by default, so you might want to first restrict to a single interface
mpirun —mca btl_tcp_if_include 192.168.0.0/24 ...

Hope this helps !

Gilles

On Wednesday, June 6, 2018, Alexander Supalov 
wrote:

> Hi everybody,
>
> I noticed that sockets do not seem to work properly in the Open MPI
> version mentioned above. Intranode runs are OK. Internode, over 100-MBit
> Ethernet, I can go only as high as 32 KiB in a simple MPI ping-pong kind of
> benchmark. Before I start composing a full bug report: is this another
> known issue?
>
> Here are the diagnostics under Ubuntu 16.04 LTS:
>
> papa@pete:~/Documents/Projects/Books/Inside/MPI/Source/openmpi-2.0.2-plain$
> mpirun --prefix 
> /home/papa/Documents/Projects/Books/Inside/MPI/Source/openmpi-2.0.2-installed
> -map-by node -hostfile mpi.hosts -n 2 $PWD/pingpong1
> r = 0bytes = 0   iters = 1   time = 0.0156577
>  lat = 0.00782887  bw = 0
> r = 1bytes = 0   iters = 1   time = 0.011045
>  lat = 0.00552249  bw = 0
> r = 0bytes = 1   iters = 1   time = 0.000459942
>  lat = 0.000229971 bw = 4348.37
> r = 1bytes = 1   iters = 1   time = 0.00026
>  lat = 0.00013 bw = 7438.04
> r = 0bytes = 2   iters = 1   time = 0.000386158
>  lat = 0.000193079 bw = 10358.5
> r = 1bytes = 2   iters = 1   time = 0.000253175
>  lat = 0.000126587 bw = 15799.3
> r = 0bytes = 4   iters = 1   time = 0.000388046
>  lat = 0.000194023 bw = 20616.1
> r = 1bytes = 4   iters = 1   time = 0.000235434
>  lat = 0.000117717 bw = 33979.8
> r = 0bytes = 8   iters = 1   time = 0.000354141
>  lat = 0.00017707  bw = 45179.8
> r = 1bytes = 8   iters = 1   time = 0.000240324
>  lat = 0.000120162 bw = 66576.8
> r = 0bytes = 16  iters = 1   time = 0.000350701
>  lat = 0.000175351 bw = 91245.8
> r = 1bytes = 16  iters = 1   time = 0.000184242
>  lat = 9.2121e-05  bw = 173685
> r = 0bytes = 32  iters = 1   time = 0.000351037
>  lat = 0.000175518 bw = 182317
> r = 1bytes = 32  iters = 1   time = 0.00025953
>  lat = 0.000129765 bw = 246600
> r = 0bytes = 64  iters = 1   time = 0.000425288
>  lat = 0.000212644 bw = 300973
> r = 1bytes = 64  iters = 1   time = 0.000241162
>  lat = 0.000120581 bw = 530764
> r = 0bytes = 128 iters = 1   time = 0.000401526
>  lat = 0.000200763 bw = 637568
> r = 1bytes = 128 iters = 1   time = 0.000279226
>  lat = 0.000139613 bw = 916820
> r = 0bytes = 256 iters = 1   time = 0.000436665
>  lat = 0.000218332 bw = 1.17252e+06
> r = 1bytes = 256 iters = 1   time = 0.000269657
>  lat = 0.000134829 bw = 1.89871e+06
> r = 0bytes = 512 iters = 1   time = 0.000496634
>  lat = 0.000248317 bw = 2.06188e+06
> r = 1bytes = 512 iters = 1   time = 0.000291029
>  lat = 0.000145514 bw = 3.51855e+06
> r = 1bytes = 1024iters = 1   time = 0.000405219 r =
> 0bytes = 1024iters = 1   time = 0.000672843 lat =
> 0.000336421 bw = 3.0438e+06
> lat = 0.000202609 bw = 5.05406e+06
> r = 0bytes = 2048iters = 1   time = 0.000874569
>  lat = 0.000437284 bw = 4.68345e+06
> r = 1bytes = 2048iters = 1   time = 0.000489308
>  lat = 0.000244654 bw = 8.37101e+06
> r = 1bytes = 4096iters = 1   time = 0.000853111
>  lat = 0.000426556 r = 0bytes = 4096iters = 1   time =
> 0.00142215  lat = 0.000711077 bw = 5.76027e+06
> bw = 9.6025e+06
> r = 0bytes = 8192iters = 1   time = 0.00239346
>  lat = 0.00119673  bw = 6.84531e+06
> r = 1bytes = 8192iters = 1   time = 0.00132503
>  lat = 0.000662515 bw = 1.2365e+07
> r = 0bytes = 16384   iters = 1   time = 0.004443
>  lat = 0.0022215   bw = 7.37519e+06
> r = 1bytes = 16384   iters = 1   time = 0.00255605
>  lat = 0.00127803  bw = 1.28198e+07
> r = 1bytes = 32768   iters = 1   r = 0bytes = 32768
>  iters = 1   time = 0.00812741  lat = 0.0040637   bw =
> 8.06358e+06
> time = 0.0046272   lat = 0.0023136  

Re: [OMPI users] need help finding mpi for Raspberry pi Raspian Stretch

2018-05-30 Thread Gilles Gouaillardet

Neil,


config.log is in the directory in which you ran configure.

Please also compress and post the output of make


Cheers,


Gilles


On 5/31/2018 10:03 AM, Neil k8it wrote:

ok here is a progress report.
first I want to thank you both for getting me this far.
gilles I think you were right about setting the CFLAGS and LDFLAGS 
=-march=armv6
it got to the first make command and after about an hour going good, 
It crashed when it started make 3.
I was going to compress and post the config.log file , but I can not 
find it.
I checked /var/log folder.can somebody please tell me where this is 
located?




Thanks
73 Neil Sablatzky  K8IT


--
From: "Gilles Gouaillardet" 
Sent: Tuesday, May 29, 2018 8:26 PM
To: 
Subject: Re: [OMPI users] need help finding mpi for Raspberry pi 
Raspian Streach



Neil,


If that does not work, please compress and post your config.log


There used to be an issue with raspberry pi3 which is detected as an 
ARMv8 processor but the raspbian compilers only generate


ARMv6 compatible binaries.


If such an issue occurs, you might want to

configure CFLAGS=-march=armv6 LDFLAGS=-march=armv6


and try again


FWIW, I run openSuSE Leap for aarch64 (e.g. native ARMv8 processor) 
and have no issue building/using Open MPI




Cheers,

Gilles

On 5/30/2018 9:03 AM, Jeff Squyres (jsquyres) wrote:
If your Linux distro does not have an Open MPI package readily 
available, you can build Open MPI from source for an RPi fairly 
easily.  Something like this (not tested on an RPi / YMMV):


wget 
https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.0.tar.bz2

tar xf openmpi-3.1.0.tar.bz2
cd openmpi-3.1.0
./configure |& tee config.out
make -j |& tee make.out
sudo make install |& tee install.out

This will download, configure, build, and install Open MPI into the 
/usr/local tree.


You can optionally specify a prefix to have it install elsewhere, e.g.:

./configure --prefix=/path/to/where/you/want/it/to/install |& tee 
config.out


Then do the make/sudo make again.



On May 29, 2018, at 6:43 PM, Neil k8it  wrote:

I  am starting to build a Raspberry pi cluster with MPI and I want 
to use the latest Raspian Streach Lite version from the 
raspberrypi.org website. After a lot of trials of watching youtubes 
on how to do this, I have found that the new version of Raspian 
Streach LITE is not compatible . I am looking for details 
instructions on how to install MPIwith this latest version of 
Raspian Streach Lite. I am using the newset hardware,RPI 3 B+ which 
requires this OS to use the features on the  -new chipset

  Thanks
Neil
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users




___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users 


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] need help finding mpi for Raspberry pi Raspian Streach

2018-05-29 Thread Gilles Gouaillardet

Neil,


If that does not work, please compress and post your config.log


There used to be an issue with raspberry pi3 which is detected as an 
ARMv8 processor but the raspbian compilers only generate


ARMv6 compatible binaries.


If such an issue occurs, you might want to

configure CFLAGS=-march=armv6 LDFLAGS=-march=armv6


and try again


FWIW, I run openSuSE Leap for aarch64 (e.g. native ARMv8 processor) and 
have no issue building/using Open MPI




Cheers,

Gilles

On 5/30/2018 9:03 AM, Jeff Squyres (jsquyres) wrote:

If your Linux distro does not have an Open MPI package readily available, you 
can build Open MPI from source for an RPi fairly easily.  Something like this 
(not tested on an RPi / YMMV):

wget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.0.tar.bz2
tar xf openmpi-3.1.0.tar.bz2
cd openmpi-3.1.0
./configure |& tee config.out
make -j |& tee make.out
sudo make install |& tee install.out

This will download, configure, build, and install Open MPI into the /usr/local 
tree.

You can optionally specify a prefix to have it install elsewhere, e.g.:

./configure --prefix=/path/to/where/you/want/it/to/install |& tee config.out

Then do the make/sudo make again.



On May 29, 2018, at 6:43 PM, Neil k8it  wrote:

I  am starting to build a Raspberry pi cluster with MPI and I want to use the 
latest Raspian Streach Lite version from the raspberrypi.org website. After a 
lot of trials of watching youtubes on how to do this, I have found that the new 
version of Raspian Streach LITE is not compatible . I am looking for details 
instructions on how to install MPIwith this latest version of Raspian Streach 
Lite. I am using the newset hardware,RPI 3 B+ which requires this OS to use the 
features on the  -new chipset
  
  
  
  
  
Thanks

Neil
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users




___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] slurm configuration override mpirun command line process mapping

2018-05-17 Thread Gilles Gouaillardet
Nicolas,

This looks odd at first glance, but as stated before, 1.6 is an obsolete
series.
A workaround could be to
mpirun—mca ess ...
And replace ... with a comma separated list of ess components that excludes
both slurm and slurmd.

An other workaround could be to remove SLURM related environment variables
before calling mpirun.


Cheers,

Gilles

On Thursday, May 17, 2018, Nicolas Deladerriere <
nicolas.deladerri...@gmail.com> wrote:

> Hi all,
>
> Thanks for your feedback,
>
> about using " mpirun --mca ras ^slurm --mca plm ^slurm --mca ess
> ^slurm,slurmd ...". I am a bit confused since syntax sounds good, but I
> keep getting following error at run time :
>
>
>
>
>
>
>
>
>
>
>
>
>
> *--MCA
> framework parameters can only take a single negation operator("^"), and it
> must be at the beginning of the value.  The followingvalue violates this
> rule:env,^slurm,slurmdWhen used, the negation operator sets the
> "exclusive" behavior mode,meaning that it will exclude all specified
> components (and implicitlyinclude all others). ..You cannot mix
> inclusive and exclusive behavior.*
>
> Is there other mca setting that could violate command line setting ?
>
> here is full mpirun command line :
>
> */../openmpi/1.6.5/bin/mpirun -prefix /.../openmpi/1.6.5 -tag-output -H
> r01n05 -x OMP_NUM_THREADS -np 1 --mca ras ^slurm --mca plm ^slurm --mca ess
> ^slurm,slurmd  master_exe.x: -H r01n06,r01n07 -x OMP_NUM_THREADS -np 2
> slave_exe.x*
>
> and ompi default setting :
>
>
>
>
>
>
>
>
>
>
>
> *host% ompi_info --all | grep slurm MCA ras: slurm (MCA
> v2.0, API v2.0, Component v1.6.5) MCA plm: slurm (MCA v2.0,
> API v2.0, Component v1.6.5) MCA ess: slurm (MCA v2.0, API
> v2.0, Component v1.6.5) MCA ess: slurmd (MCA v2.0, API
> v2.0, Component v1.6.5) MCA ras: parameter
> "ras_slurm_priority" (current value: <75>, data source: default
> value)  Priority of the slurm ras
> component MCA plm: parameter "plm_slurm_args" (current
> value: , data source: default value) MCA plm:
> parameter "plm_slurm_priority" (current value: <0>, data source: default
> value) MCA ess: parameter "ess_slurm_priority" (current
> value: <0>, data source: default value) MCA ess: parameter
> "ess_slurmd_priority" (current value: <0>, data source: default value)*
>
>
> About "-H" option and using --bynode option:
>
> In my case, I do not specify number of slots by node to openmpi (see
> mpirun command just above). From what I see the only place I define number
> of slots in this case is actually through SLURM configuration
> (SLURM_JOB_CPUS_PER_NODE=4(x3)). And I was not expected this to be taken
> when running mpi processes.
>
> Using --bynode is probably the easiest solution in my case, even if I am
> scared that it will not necessary fit all my running configuration. Better
> solution would be to review my management script for better integration
> with slurm resources manager, but this is another story.
>
> Thanks for your help.
> Regards,
> Nicolas
>
>
> 2018-05-16 9:47 GMT+02:00 r...@open-mpi.org <r...@open-mpi.org>:
>
>> The problem here is that you have made an incorrect assumption. In the
>> older OMPI versions, the -H option simply indicated that the specified
>> hosts were available for use - it did not imply the number of slots on that
>> host. Since you have specified 2 slots on each host, and you told mpirun to
>> launch 2 procs of your second app_context (the “slave”), it filled the
>> first node with the 2 procs.
>>
>> I don’t recall the options for that old a version, but IIRC you should
>> add --pernode to the cmd line to get exactly 1 proc/node
>>
>> Or upgrade to a more recent OMPI version where -H can also be used to
>> specify the #slots on a node :-)
>>
>>
>> > On May 15, 2018, at 11:58 PM, Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com> wrote:
>> >
>> > You can try to disable SLURM :
>> >
>> > mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ...
>> >
>> > That will require you are able to SSH between compute nodes.
>> > Keep in mind this is far form ideal since it might leave some MPI
>> > processes on nodes if you cancel a job, and mess SLURM accounting too.
>> >
>> >
>> > Cheers

Re: [OMPI users] slurm configuration override mpirun command line process mapping

2018-05-16 Thread Gilles Gouaillardet
You can try to disable SLURM :

mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ...

That will require you are able to SSH between compute nodes.
Keep in mind this is far form ideal since it might leave some MPI
processes on nodes if you cancel a job, and mess SLURM accounting too.


Cheers,

Gilles

On Wed, May 16, 2018 at 3:50 PM, Nicolas Deladerriere
 wrote:
> Hi all,
>
>
>
> I am trying to run mpi application through SLURM job scheduler. Here is my
> running sequence
>
>
> sbatch --> my_env_script.sh --> my_run_script.sh --> mpirun
>
>
> In order to minimize modification of my production environment, I had to
> setup following hostlist management in different scripts:
>
>
> my_env_script.sh
>
>
> build host list from SLURM resource manager information
>
> Example: node01 nslots=2 ; node02 nslots=2 ; node03 nslots=2
>
>
> my_run_script.sh
>
>
> Build host list according to required job (process mapping depends on job
> requirement).
>
> Nodes are always fully dedicated to my job, but I have to manage different
> master-slave situation with corresponding mpirun command:
>
> as many process as number of slots:
>
> mpirun -H node01 -np 1 process_master.x : -H node02,node02,node03,node03 -np
> 4 process_slave.x
>
> only one process per node (slots are usually used through openMP threading)
>
> mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2
> other_process_slave.x
>
>
>
> However, I realized that whatever I specified through my mpirun command,
> process mapping is overridden at run time by slurm according to slurm
> setting (either default setting or sbatch command line). For example, if I
> run with:
>
>
> sbatch -N 3 --exclusive my_env_script.sh myjob
>
>
> where final mpirun command (depending on myjob) is:
>
>
> mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2
> other_process_slave.x
>
>
> It will be run with process mapping corresponding to:
>
>
> mpirun -H node01 -np 1 other_process_master.x : -H node02,node02 -np 2
> other_process_slave.x
>
>
> So far I did not find a way to force mpirun to run with host mapping from
> command line instead of slurm one. Is there a way to do it (either by using
> MCA parameters of slurm configuration or …) ?
>
>
> openmpi version : 1.6.5
>
> slurm version : 17.11.2
>
>
>
> Ragards,
>
> Nicolas
>
>
> Note 1: I know, it would be better to let slurm manage my process mapping by
> only using slurm parameters and not specifying host mapping in my mpirun
> command, but in order to minimize modification in my production environment
> I had to use such solution.
>
> Note 2: I know I am using old openmpi version !
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] peformance abnormality with openib and tcp framework

2018-05-15 Thread Gilles Gouaillardet
The long story is you need always need a subnet manager to initialize 
the fabric.


That means you can run the subnet manager and stop it once so each HCA 
is assigned a LID.


In that case, the commands that interact with the SM (ibhosts, 
ibdiagnet) will obviously fail.



Cheers,


Gilles


On 5/15/2018 4:51 PM, John Hearns via users wrote:

Xie,   as far as I know you need to run OpenSM even on two hosts.

On 15 May 2018 at 03:29, Blade Shieh > wrote:


Hi, John:

You are right on the network framework. I do have no IB switch and
just connect the servers with an IB cable. I did not even open the
opensmd service because it seems unnecessary in this situation.
Can this be the reason why IB performs poorer?

Interconnection details are in the attachment.

Best Regards,

Xie Bin



John Hearns via users > 于 2018年5月14日 周一 17:45写道:

Xie Bin,  I do hate to ask this.  You say  "in a two-node
cluster (IB direcet-connected). "
Does that mean that you have no IB switch, and that there is a
single IB cable joining up these two servers?
If so please run:    ibstatus ibhosts   ibdiagnet
I am trying to check if the IB fabric is functioning properly
in that situation.
(Also need to check if there is o Subnet Manager  - so run  
sminfo)

But you do say that the IMB test gives good results for IB, so
you must have IB working properly.
Therefore I am an idiot...



On 14 May 2018 at 11:04, Blade Shieh > wrote:


Hi, Nathan:
    Thanks for you reply.
1) It was my mistake not to notice usage of osu_latency.
Now it worked well, but still poorer in openib.
2) I did not use sm or vader because I wanted to check
performance between tcp and openib. Besides, I will run
the application in cluster, so vader is not so important.
3) Of course, I tried you suggestions. I used ^tcp/^openib
and set btl_openib_if_include to mlx5_0 in a two-node
cluster (IB direcet-connected).  The result did not change
-- IB still better in MPI benchmark but poorer in my
applicaion.

Best Regards,
Xie Bin

___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users





___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] peformance abnormality with openib and tcp framework

2018-05-14 Thread Gilles Gouaillardet

Xie Bin,


According to the man page, -N is equivalent to npernode, which is 
equivalent to --map-by ppr:N:node.


This is *not* equivalent to -map-by node :

The former packs tasks to the same node, and the latter scatters tasks 
accross the nodes



[gilles@login ~]$ mpirun --host n0:2,n1:2 -N 2 --tag-output hostname | sort
[1,0]:n0
[1,1]:n0
[1,2]:n1
[1,3]:n1


[gilles@login ~]$ mpirun --host n0:2,n1:2 -np 4 --tag-output -map-by 
node hostname | sort

[1,0]:n0
[1,1]:n1
[1,2]:n0
[1,3]:n1


I am pretty sure a subnet manager was ran at some point in time (so your 
HCA can get their identifier).


/* feel free to reboot your nodes and see if ibstat still shows the 
adapters as active */



Note you might also use --mca pml ob1 in order to make sure mxm nor ucx 
are used



Cheers,


Gilles



On 5/15/2018 10:45 AM, Blade Shieh wrote:

Hi, George:
My command lines are:
1) single node
mpirun --allow-run-as-root -mca btl self,tcp(or openib) -mca 
btl_tcp_if_include eth2 -mca btl_openib_if_include mlx5_0 -x 
OMP_NUM_THREADS=2 -n 32 myapp

2) 2-node cluster
mpirun --allow-run-as-root -mca btl ^tcp(or ^openib) -mca 
btl_tcp_if_include eth2 -mca btl_openib_if_include mlx5_0 -x 
OMP_NUM_THREADS=4 -N 16 myapp


In 2nd condition, I used -N, which is equal to --map-by node.

Best regards,
Xie Bin


George Bosilca > 于 
2018年5月15日 周二 02:07写道:


Shared memory communication is important for multi-core platforms,
especially when you have multiple processes per node. But this is
only part of your issue here.

You haven't specified how your processes will be mapped on your
resources. As a result rank 0 and 1 will be on the same node, so
you are testing the shared memory support of whatever BTL you
allow. In this case the performance will be much better for TCP
than for IB, simply because you are not using your network, but
its capacity to move data across memory banks. In such an
environment, TCP translated to a memcpy plus a system call, which
is much faster than IB. That being said, it should not matter
because shared memory is there to cover this case.

Add "--map-by node" to your mpirun command to measure the
bandwidth between nodes.

  George.



On Mon, May 14, 2018 at 5:04 AM, Blade Shieh > wrote:


Hi, Nathan:
    Thanks for you reply.
1) It was my mistake not to notice usage of osu_latency. Now
it worked well, but still poorer in openib.
2) I did not use sm or vader because I wanted to check
performance between tcp and openib. Besides, I will run the
application in cluster, so vader is not so important.
3) Of course, I tried you suggestions. I used ^tcp/^openib and
set btl_openib_if_include to mlx5_0 in a two-node cluster (IB
direcet-connected). The result did not change -- IB still
better in MPI benchmark but poorer in my applicaion.

Best Regards,
Xie Bin

___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 3.0.1 - mpirun hangs with 2 hosts

2018-05-14 Thread Gilles Gouaillardet

In the initial report, the /usr/bin/ssh process was in the 'T' state
(it generally hints the process is attached by a debugger)

/usr/bin/ssh -x b09-32 orted

did behave as expected (e.g. orted was executed, exited with an error 
since the command line is invalid, and error message was received)



can you try to run

/home/user/openmpi_install/bin/mpirun --host b09-30,b09-32 hostname

and see how things go ? (since you simply 'ssh orted', an other orted 
might be used)


If you are still facing the same hang with ssh in the 'T' state, can you 
check the logs on b09-32 and see
if the sshd server was even contacted ? I can hardly make sense of this 
error fwiw.



Cheers,

Gilles

On 5/15/2018 5:27 AM, r...@open-mpi.org wrote:
You got that error because the orted is looking for its rank on the 
cmd line and not finding it.



On May 14, 2018, at 12:37 PM, Max Mellette > wrote:


Hi Gus,

Thanks for the suggestions. The correct version of openmpi seems to 
be getting picked up; I also prepended .bashrc with the installation 
path like you suggested, but it didn't seemed to help:


user@b09-30:~$ cat .bashrc
export 
PATH=/home/user/openmpi_install/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

export LD_LIBRARY_PATH=/home/user/openmpi_install/lib
user@b09-30:~$ which mpicc
/home/user/openmpi_install/bin/mpicc
user@b09-30:~$ /usr/bin/ssh -x b09-32 orted
[b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file 
ess_env_module.c at line 147
[b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad parameter in 
file util/session_dir.c at line 106
[b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad parameter in 
file util/session_dir.c at line 345
[b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad parameter in 
file base/ess_base_std_orted.c at line 270

--
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_session_dir failed
  --> Returned value Bad parameter (-5) instead of ORTE_SUCCESS
--

Thanks,
Max


On Mon, May 14, 2018 at 11:41 AM, Gus Correa > wrote:


Hi Max

Just in case, as environment mix often happens.
Could it be that you are inadvertently picking another
installation of OpenMPI, perhaps installed from packages
in /usr , or /usr/local?
That's easy to check with 'which mpiexec' or
'which mpicc', for instance.

Have you tried to prepend (as opposed to append) OpenMPI
to your PATH? Say:

export

PATH='/home/user/openmpi_install/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'

I hope this helps,
Gus Correa


___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users




___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] Building Open MPI and default hostfile change does not go through

2018-05-13 Thread Gilles Gouaillardet

Konstantinos,


You need to double check that, your OS might have done it out of the box 
for you already.



Once logged, you can

which mpirun

If it resolves to /usr/local/bin/mpirun, then there is no need to update 
$PATH, and then


ldd /usr/local/bin/mpirun

If it correctly resolves to /usr/local/lib/libopen-pal.so and friends, 
then there is no need


to update $LD_LIBRARY_PATH as well


Cheers,


Gilles


On 5/14/2018 12:46 PM, Konstantinos Konstantinidis wrote:

Thank you Gilles,

One more question. Do I need to add the following lines to the .bashrc 
file after installing Open MPI?


export PATH="$PATH:/usr/local/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"


On Sun, May 13, 2018 at 8:48 PM, Gilles Gouaillardet 
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>> wrote:


Konstantinos,


Since you ran

configure --prefix=/usr/local

the system-wide config file should be in

/usr/local/etc/openmpi-default-hostfile


Note /usr/local is the default prefix, so you do not even need the
--prefix=/usr/local option


Cheers,


Gilles


On 5/12/2018 6:58 AM, Konstantinos Konstantinidis wrote:

Yeap, exactly the hostfile I have is of the form

node1 slots=1
node2 slots=1
node3 slots=1

where the above hostnames are resolved in ~/.ssh/config file
which has entries of the form

Host node1
 HostName 192.168.0.100
 User ubuntu
 IdentityFile ~/.ssh/mykey.pem

and so on.

So the mpirun cannot pickup the hostfile by itself and I have
to specify it each time.


On Fri, May 11, 2018 at 4:02 PM, Jeff Squyres (jsquyres)
<jsquy...@cisco.com <mailto:jsquy...@cisco.com>
<mailto:jsquy...@cisco.com <mailto:jsquy...@cisco.com>>> wrote:

    Can you provide some more detail?  I'm not able to get this to
    fail (i.e., it seems to be working as expected for me).

    For example, what's the contents of your
    /etc/openmpi/openmpi-default-hostfile -- did you list some
    hostnames in there?


    > On May 11, 2018, at 4:43 AM, Konstantinos Konstantinidis
    <kostas1...@gmail.com <mailto:kostas1...@gmail.com>
<mailto:kostas1...@gmail.com <mailto:kostas1...@gmail.com>>>
wrote:
    >
    > Hi,
    >
    > I have built Open MPI 2.1.2 multiple times on Ubuntu
16.04 and
    then I add the line
    >
    > orte_default_hostfile=/etc/openmpi/openmpi-default-hostfile
    >
    > to the file
    >
    > /etc/openmpi/openmpi-mca-params.conf
    >
    > and I execute
    >
    > sudo chown myUsername /etc/openmpi/openmpi-default-hostfile
    >
    > For some reason this change never goes through and each
time I
    run a program with mpirun only one local process runs. So
I have
    to manually specify my hostname with the --hostfile argument.
    >
    > What can be the cause of this?
    >
    > The exact series of commands I use for building is the
following
    >
    > sudo apt-get update
    > sudo apt-get upgrade
    > sudo apt-get install g++
    > sudo apt-get install valgrind
    > sudo apt-get install libopenmpi-dev
    > sudo apt-get install gfortran
    > sudo apt-get install make
    > wget

https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.2.tar.gz

<https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.2.tar.gz>
   

<https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.2.tar.gz

<https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.2.tar.gz>>

    > tar -xvf openmpi-* && cd openmpi-*
    > ./configure --prefix=/usr/local --enable-mpi-cxx
--enable-debug
    --enable-memchecker --with-valgrind=/usr
    > sudo make all install
    >
    > Then, I add the following lines to the .bashrc file (Is this
    necessary?)
    >
    > export PATH="$PATH:/usr/local/bin"
    > export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
    >
    > for setting the path and library path, respectively and of
    course reload .bashrc.
    >
    > Is the above way of installing Open MPI correct? I am really
    wondering since I have no solid Linux knowledge.
    > ___
  

Re: [OMPI users] Building Open MPI and default hostfile change does not go through

2018-05-13 Thread Gilles Gouaillardet

Konstantinos,


Since you ran

configure --prefix=/usr/local

the system-wide config file should be in

/usr/local/etc/openmpi-default-hostfile


Note /usr/local is the default prefix, so you do not even need the 
--prefix=/usr/local option



Cheers,


Gilles


On 5/12/2018 6:58 AM, Konstantinos Konstantinidis wrote:

Yeap, exactly the hostfile I have is of the form

node1 slots=1
node2 slots=1
node3 slots=1

where the above hostnames are resolved in ~/.ssh/config file which has 
entries of the form


Host node1
 HostName 192.168.0.100
 User ubuntu
 IdentityFile ~/.ssh/mykey.pem

and so on.

So the mpirun cannot pickup the hostfile by itself and I have to 
specify it each time.



On Fri, May 11, 2018 at 4:02 PM, Jeff Squyres (jsquyres) 
> wrote:


Can you provide some more detail?  I'm not able to get this to
fail (i.e., it seems to be working as expected for me).

For example, what's the contents of your
/etc/openmpi/openmpi-default-hostfile -- did you list some
hostnames in there?


> On May 11, 2018, at 4:43 AM, Konstantinos Konstantinidis
> wrote:
>
> Hi,
>
> I have built Open MPI 2.1.2 multiple times on Ubuntu 16.04 and
then I add the line
>
> orte_default_hostfile=/etc/openmpi/openmpi-default-hostfile
>
> to the file
>
> /etc/openmpi/openmpi-mca-params.conf
>
> and I execute
>
> sudo chown myUsername /etc/openmpi/openmpi-default-hostfile
>
> For some reason this change never goes through and each time I
run a program with mpirun only one local process runs. So I have
to manually specify my hostname with the --hostfile argument.
>
> What can be the cause of this?
>
> The exact series of commands I use for building is the following
>
> sudo apt-get update
> sudo apt-get upgrade
> sudo apt-get install g++
> sudo apt-get install valgrind
> sudo apt-get install libopenmpi-dev
> sudo apt-get install gfortran
> sudo apt-get install make
> wget
https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.2.tar.gz


> tar -xvf openmpi-* && cd openmpi-*
> ./configure --prefix=/usr/local --enable-mpi-cxx --enable-debug
--enable-memchecker --with-valgrind=/usr
> sudo make all install
>
> Then, I add the following lines to the .bashrc file (Is this
necessary?)
>
> export PATH="$PATH:/usr/local/bin"
> export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
>
> for setting the path and library path, respectively and of
course reload .bashrc.
>
> Is the above way of installing Open MPI correct? I am really
wondering since I have no solid Linux knowledge.
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/users



-- 
Jeff Squyres

jsquy...@cisco.com 




___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 3.0.1 - mpirun hangs with 2 hosts

2018-05-12 Thread Gilles Gouaillardet
Max,

the 'T' state of the ssh process is very puzzling.

can you try to run
/usr/bin/ssh -x b09-32 orted
on b09-30 and see what happens ?
(it should fail with an error message, instead of hanging)

In order to check there is no firewall, can you run instead
iptables -L
Also, is 'selinux' enabled ? there could be some rules that prevent
'ssh' from working as expected


Cheers,

Gilles

On Sat, May 12, 2018 at 7:38 AM, Max Mellette  wrote:
> Hi Jeff,
>
> Thanks for the reply. FYI since I originally posted this, I uninstalled
> OpenMPI 3.0.1 and installed 3.1.0, but I'm still experiencing the same
> problem.
>
> When I run the command without the `--mca plm_base_verbose 100` flag, it
> hangs indefinitely with no output.
>
> As far as I can tell, these are the additional processes running on each
> machine while mpirun is hanging (printed using `ps -aux | less`):
>
> On executing host b09-30:
>
> user 361714  0.4  0.0 293016  8444 pts/0Sl+  15:10   0:00 mpirun
> --host b09-30,b09-32 hostname
> user 361719  0.0  0.0  37092  5112 pts/0T15:10   0:00
> /usr/bin/ssh -x b09-32  orted -mca ess "env" -mca ess_base_jobid "638517248"
> -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca orte_node_regex
> "b[2:9]-30,b[2:9]-32@0(2)" -mca orte_hnp_uri
> "638517248.0;tcp://169.228.66.102,10.1.100.30:55090" -mca plm "rsh" -mca
> pmix "^s1,s2,cray,isolated"
>
> On remote host b09-32:
>
> root 175273  0.0  0.0  61752  5824 ?Ss   15:10   0:00 sshd:
> [accepted]
> sshd 175274  0.0  0.0  61752   708 ?S15:10   0:00 sshd:
> [net]
>
> I only see orted showing up in the ssh flags on b09-30. Any ideas what I
> should try next?
>
> Thanks,
> Max
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] problem

2018-05-09 Thread Gilles Gouaillardet
Ankita,

Do you have any reason to suspect the root cause of the crash is Open MPI ?

Cheers,

Gilles

On Wednesday, May 9, 2018, Ankita m  wrote:

> MPI "Hello World" program is also working
>
> please see this error file attached below. its of a different program
>
> On Wed, May 9, 2018 at 4:10 PM, John Hearns via users <
> users@lists.open-mpi.org> wrote:
>
>> Ankita, looks like your program is not launching correctly.
>> I would try the following:
>> define two hosts in a machinefile.  Use mpirun -np 2  machinefile  date
>> Ie can you use mpirun just to run the command 'date'
>>
>> Secondly compile up and try to run an MPI 'Hello World' program
>>
>>
>> On 9 May 2018 at 12:28, Ankita m  wrote:
>>
>>> I am using ompi -3.1.0 version in my program and compiler is mpicc
>>>
>>> its a parallel program which uses multiple nodes with 16 cores in each
>>> node.
>>>
>>> but its not working and generates a error file . i Have attached the
>>> error file below.
>>>
>>> can anyone please tell what is the issue actually
>>>
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>>>
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>>
>
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OMPI sendrecv bugs?

2018-04-12 Thread Gilles Gouaillardet
Which version of Open MPI are you running ?
This reminds me of a bug in CMA that has already been fixed.


can you try again with


mpirun --mca btl_vader_single_copy_mechanism none ...


Cheers,

Gilles

On Fri, Apr 13, 2018 at 1:51 PM, Kaiming Ouyang  wrote:
> Hi all,
> I am trying to test the bandwidth of intra-MPI send and recv. The code is
> attached here. When I give the input 2048 (namely each process will send and
> receive 2GB data), the program reported:
> Read 2147479552, expected 2147483648, errno = 95
> Read 2147479552, expected 2147483648, errno = 98
> Read 2147479552, expected 2147483648, errno = 98
> Read 2147479552, expected 2147483648, errno = 98
>
> Does this mean Openmpi does not support the send and recv where data size is
> larger than 2GB, or is there a bug in my code? Thank you.
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] libmpi_cxx.so doesn't exist in lib path when installing 3.0.1

2018-04-08 Thread Gilles Gouaillardet

Ajith,


Note the C++ bindings have not been removed yet from Open MPI.

You need to configure --enable-mpi-cxx in order to build them (this is 
no more the default option)



As Nathan pointed out, the C++ bindings will likely be removed from Open 
MPI 4, so you will have to modernize your code at some point in time.




On this mailing list, C++ users have previously pointed/recommended

 - boostMPI https://github.com/boostorg/mpi

- Elemental http://libelemental.org

Though these are not part of the MPI standard, these third party 
libraries provide C++ bindings.


And of course, using the C binding is a portable option that does not 
require any external dependencies.



Cheers,


Gilles



On 4/9/2018 2:48 AM, Ajith Subramanian wrote:
Thanks for the quick reply. One of the third-party packages we're 
using still depends on the C++ bindings, but I'll look to migrating 
them to the C bindings.


Ajith

On Sat, Apr 7, 2018 at 4:40 PM, Nathan Hjelm > wrote:


The MPI C++ bindings were depricated more than 10 years ago and
were removed from the standard 6 years ago. They have been
disabled by default for a couple of years in Open MPI. They will
likely be removed in Open MPI 4.0. You should migrate your code to
use the C bindings.

-Nathan

> On Apr 7, 2018, at 2:48 PM, Ajith Subramanian
> wrote:
>
> Hello,
>
> Installing version 3.0.1 using --prefix=/usr/local doesn't seem
to place the libmpi_cxx.so library in /usr/local/lib. I've
attached the requested output, in addition to ls.out which shows
the existing libmpi* files in /usr/local/lib after install.
>
> The machine is running Ubuntu 13.10 LTS and a previous
installation of OpenMPI 1.8.4 (which has now been uninstalled)
does place libmpi_cxx.so in /usr/local/lib.
>
> Thanks,
> Ajith
> 
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org 
https://lists.open-mpi.org/mailman/listinfo/users





___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] disabling libraries?

2018-04-05 Thread Gilles Gouaillardet
Michael,

in this case, you can
mpirun --mca oob ^ud ...
in order to blacklist the oob/ud component.

an alternative is to add
oob = ^ud
in /.../etc/openmpi-mca-params.conf

If Open MPI is installed on a local filesystem, then this setting can
be node specific.


That being said, the error suggest mca_oob_ud.so is a module from a
previous install,
Open MPI was not built on the system it is running, or libibverbs.so.1
has been removed after
Open MPI was built.
So I do encourage you to take a step back, and think if you can find a
better solution for your site.


Cheers,

Gilles

On Fri, Apr 6, 2018 at 3:37 AM, Michael Di Domenico
 wrote:
> i'm trying to compile openmpi to support all of our interconnects,
> psm/openib/mxm/etc
>
> this works fine, openmpi finds all the libs, compiles and runs on each
> of the respective machines
>
> however, we don't install the libraries for everything everywhere
>
> so when i run things like ompi_info and mpirun i get
>
> mca_base_component_reposity_open: unable to open mca_oob_ud:
> libibverbs.so.1: cannot open shared object file: no such file or
> directory (ignored)
>
> and so on, for a bunch of other libs.
>
> i understand how the lib linking works so this isn't unexpected and
> doesn't stop the mpi programs from running.
>
> here's the part i don't understand, how can i trace the above warning
> and others like it back the required --mca parameters i need to add
> into the configuration to make the warnings go away?
>
> as an aside, i believe i can set most of them via environment
> variables as well as the command, but what i really like to do is set
> them from a file.  i know i can create a default param file, but is
> there a way to feed a param file at invocation depending where mpirun
> is being run?
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] mpi send/recv pair hangin

2018-04-05 Thread Gilles Gouaillardet
Noam,

you might also want to try

mpirun --mca btl tcp,self ...

to rule out btl (shared memory and/or infiniband) related issues.


Once you rebuild Open MPI with --enable-debug, I recommend you first
check the arguments of the MPI_Send() and MPI_Recv() functions and
make sure
 - same communicator is used (in C, check comm->c_contextid)
 - same tag
 - double check the MPI tasks do wait for each other (in C, check
comm->c_my_rank, source and dest)


Cheers,

Gilles

On Fri, Apr 6, 2018 at 5:31 AM, George Bosilca  wrote:
> Yes, you can do this by adding --enable-debug to OMPI configure (and make
> sure your don't have the configure flag --with-platform=optimize).
>
>   George.
>
>
> On Thu, Apr 5, 2018 at 4:20 PM, Noam Bernstein 
> wrote:
>>
>>
>> On Apr 5, 2018, at 4:11 PM, George Bosilca  wrote:
>>
>> I attach with gdb on the processes and do a "call mca_pml_ob1_dump(comm,
>> 1)". This allows the debugger to make a call our function, and output
>> internal information about the library status.
>>
>>
>> Great.  But I guess I need to recompile ompi in debug mode?  Is that just
>> a flag to configure?
>>
>> thanks,
>> Noam
>>
>>
>> 
>> |
>> |
>> |
>> U.S. NAVAL
>> |
>> |
>> _RESEARCH_
>> |
>> LABORATORY
>>
>> Noam Bernstein, Ph.D.
>> Center for Materials Physics and Technology
>> U.S. Naval Research Laboratory
>> T +1 202 404 8628  F +1 202 404 7546
>> https://www.nrl.navy.mil
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] running mpi program between my PC and an ARM-architektur raspberry

2018-04-03 Thread Gilles Gouaillardet

Let me shed a different light on that.


Once in a while, I run Open MPI between x86_64 and sparcv9, and it works 
quite well as far as I am concerned.


Note this is the master branch, and I never try older nor releases branches.


Note you likely need to configure Open MPI with --enable-heterogeneous 
on both arch.


If you are still unlucky, then I suggest you download and build the 
3.1.0rc3 version (with --enable-debug --enable-heterogeneous)


and then

mpirun --mca oob_base_verbose 10 --mca pml_base_verbose 10 --host ... 
hostname


and then

mpirun --mca oob_base_verbose 10 --mca pml_base_verbose 10 --mca 
btl_base_verbose 10 --host ... mpi_helloworld


and either open a github issue or post the logs on this ML


Cheers,


Gilles




On 4/4/2018 8:39 AM, Jeff Squyres (jsquyres) wrote:

On Apr 2, 2018, at 1:39 PM, dpchoudh .  wrote:

Sorry for a pedantic follow up:

Is this (heterogeneous cluster support) something that is specified by
the MPI standard (perhaps as an optional component)?

The MPI standard states that if you send a message, you should receive the same 
values at the receiver.  E.g., if you sent int=3, you should receive int=3, 
even if one machine is big endian and the other machine is little endian.

It does not specify what happens when data sizes are different (e.g., if type X 
is 4 bits on one side and 8 bits on the other) -- there's no good answers on 
what to do there.


Do people know if
MPICH. MVAPICH, Intel MPI etc support it? (I do realize this is an
OpenMPI forum)

I don't know offhand.  I know that this kind of support is very unpopular with 
MPI implementors because:

1. Nearly nobody uses it (we get *at most* one request a year to properly support 
BE<-->LE transformation).
2. It's difficult to implement BE<-->LE transformation properly without causing 
at least some performance loss and/or code complexity in the main datatype engine.
3. It is very difficult for MPI implementors to test properly (especially in 
automated environments).

#1 is probably the most important reason.  If lots of people were asking for 
this, MPI implementors would take the time to figure out #2 and #3.  But since 
almost no one asks for it, it gets pushed (waay) down on the priority list 
of things to implement.

Sorry -- just being brutally honest here.  :-\


The reason I ask is that I have a mini Linux lab of sort that consists
of Linux running on many architectures, both 32 and 64 bit and both LE
and BE. Some have advanced fabrics, but all have garden variety
Ethernet. I mostly use this for software porting work, but I'd love to
set it up as a test bench for testing OpenMPI in a heterogeneous
environment and report issues, if that is something that the
developers want to achieve.

Effectively, the current set of Open MPI developers have not put up any resources to 
fix, update, and maintain the BE<-->LE transformation in the Open MPI datatype 
engine.  I don't think that there are any sane answers for what to do when datatypes 
are different sizes.

However, that being said, Open MPI is an open source community -- if someone 
wants to contribute pull requests and/or testing to support this feature, that 
would be great!



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] can not compile Openmpi-3.0

2018-04-03 Thread Gilles Gouaillardet

In that case, you need to

configure --with-cuda=/usr/local/cuda-8.0


Cheers,


Gilles


On 4/3/2018 4:12 PM, abhisek Mondal wrote:

I have cuda installed in /usr/local/cuda-8.0.

On Tue, Apr 3, 2018 at 12:37 PM, Gilles Gouaillardet 
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>> wrote:


Did you install CUDA and where ?


On 4/3/2018 3:51 PM, abhisek Mondal wrote:

Thanks a lot. Installing the headers worked like a charm !

I have a question though. Why does configuration says:

 Version: 3.0.1
Build MPI C bindings: yes
Build MPI C++ bindings (deprecated): no
Build MPI Fortran bindings: mpif.h, use mpi
MPI Build Java bindings (experimental): no
Build Open SHMEM support: yes
Debug build: no
Platform file: (none)

Miscellaneous
---
*CUDA support: no*
*
*
*
*
I have a single GPU on my machine. Do I need to define it
manually while configuring openmpi?

On Tue, Apr 3, 2018 at 11:21 AM, Gilles Gouaillardet
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>
<mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>>> wrote:

    It looks like kernel-headers is required too


    Cheers,


    Gilles


    On 4/3/2018 2:28 PM, abhisek Mondal wrote:

        Hello,

        Installing glibc moved the installation little bit but
again
        stuck at sanity check.
        Location of config.log:

https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>
       

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>>

        $ cpp --version
        cpp (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
        Copyright (C) 2015 Free Software Foundation, Inc.
        This is free software; see the source for copying
conditions.
        There is NO
        warranty; not even for MERCHANTABILITY or FITNESS FOR A
        PARTICULAR PURPOSE.

        I have gcc and gcc-c++ installed.

            On Tue, Apr 3, 2018 at 10:08 AM, Gilles Gouaillardet
        <gil...@rist.or.jp <mailto:gil...@rist.or.jp>
<mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>>
        <mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>
<mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>>>> wrote:

            This is the relevant part related to this error

            configure:6620: checking whether we are cross
compiling
            configure:6628: gcc -o conftest conftest.c >&5
            conftest.c:10:19: fatal error: stdio.h: No such
file or
        directory
             #include 
               ^
            compilation terminated.
            configure:6632: $? = 1
            configure:6639: ./conftest
            ../configure: line 6641: ./conftest: No such file
or directory
            configure:6643: $? = 127


            at the very least, you need to install the
glibc-headers
        package


            Cheers,


            Gilles


            On 4/3/2018 1:22 PM, abhisek Mondal wrote:

                Hello,

                I have uploaded the config.log here:

https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>
       

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>>

       

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>
       

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/

Re: [OMPI users] can not compile Openmpi-3.0

2018-04-03 Thread Gilles Gouaillardet

Did you install CUDA and where ?


On 4/3/2018 3:51 PM, abhisek Mondal wrote:

Thanks a lot. Installing the headers worked like a charm !

I have a question though. Why does configuration says:

 Version: 3.0.1
Build MPI C bindings: yes
Build MPI C++ bindings (deprecated): no
Build MPI Fortran bindings: mpif.h, use mpi
MPI Build Java bindings (experimental): no
Build Open SHMEM support: yes
Debug build: no
Platform file: (none)

Miscellaneous
---
*CUDA support: no*
*
*
*
*
I have a single GPU on my machine. Do I need to define it manually 
while configuring openmpi?


On Tue, Apr 3, 2018 at 11:21 AM, Gilles Gouaillardet 
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>> wrote:


It looks like kernel-headers is required too


Cheers,


Gilles


On 4/3/2018 2:28 PM, abhisek Mondal wrote:

Hello,

Installing glibc moved the installation little bit but again
stuck at sanity check.
Location of config.log:

https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>

$ cpp --version
cpp (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. 
There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

I have gcc and gcc-c++ installed.

On Tue, Apr 3, 2018 at 10:08 AM, Gilles Gouaillardet
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>
<mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>>> wrote:

    This is the relevant part related to this error

    configure:6620: checking whether we are cross compiling
    configure:6628: gcc -o conftest    conftest.c >&5
    conftest.c:10:19: fatal error: stdio.h: No such file or
directory
     #include 
       ^
    compilation terminated.
    configure:6632: $? = 1
    configure:6639: ./conftest
    ../configure: line 6641: ./conftest: No such file or directory
    configure:6643: $? = 127


    at the very least, you need to install the glibc-headers
package


    Cheers,


    Gilles


    On 4/3/2018 1:22 PM, abhisek Mondal wrote:

        Hello,

        I have uploaded the config.log here:

https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>
       

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>>




        On Tue, Apr 3, 2018 at 9:47 AM, Gilles Gouaillardet
        <gil...@rist.or.jp <mailto:gil...@rist.or.jp>
<mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>>
        <mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>
<mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>>>> wrote:

            Can you please compress and attach your config.log ?


            You might also want to double check you can
compile *and*
        run a
            simple C hello world program


            Cheers,


            Gilles


            On 4/3/2018 1:06 PM, abhisek Mondal wrote:

                Hi,

                I need some help regarding compiling
Openmpi-3.0. I have
                perfectly working C compiler, however, during
        configuration
                I'm keep getting this error:


       

//
                /== Configuring Open MPI/

       

//
                /
                /
                /*** Startup tests/
                /checking build system type...
x86_64-unknown-linux-gnu/
                /checking host system type...
x86_64-unknown-linux-gnu/
                /checking target system type...
x86_64-unknown-linux-gnu/
                /checking for gcc... gcc/
                /checking whether the C compiler works... yes/
                /checking for

Re: [OMPI users] can not compile Openmpi-3.0

2018-04-02 Thread Gilles Gouaillardet

It looks like kernel-headers is required too


Cheers,


Gilles


On 4/3/2018 2:28 PM, abhisek Mondal wrote:

Hello,

Installing glibc moved the installation little bit but again stuck at 
sanity check.
Location of config.log: 
https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU


$ cpp --version
cpp (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR 
PURPOSE.


I have gcc and gcc-c++ installed.

On Tue, Apr 3, 2018 at 10:08 AM, Gilles Gouaillardet 
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>> wrote:


This is the relevant part related to this error

configure:6620: checking whether we are cross compiling
configure:6628: gcc -o conftest    conftest.c  >&5
conftest.c:10:19: fatal error: stdio.h: No such file or directory
 #include 
   ^
compilation terminated.
configure:6632: $? = 1
configure:6639: ./conftest
../configure: line 6641: ./conftest: No such file or directory
configure:6643: $? = 127


at the very least, you need to install the glibc-headers package


Cheers,


Gilles


On 4/3/2018 1:22 PM, abhisek Mondal wrote:

Hello,

I have uploaded the config.log here:

https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU

<https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU>




On Tue, Apr 3, 2018 at 9:47 AM, Gilles Gouaillardet
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>
<mailto:gil...@rist.or.jp <mailto:gil...@rist.or.jp>>> wrote:

    Can you please compress and attach your config.log ?


    You might also want to double check you can compile *and*
run a
    simple C hello world program


    Cheers,


    Gilles


    On 4/3/2018 1:06 PM, abhisek Mondal wrote:

        Hi,

        I need some help regarding compiling Openmpi-3.0. I have
        perfectly working C compiler, however, during
configuration
        I'm keep getting this error:

       

//
        /== Configuring Open MPI/
       

//
        /
        /
        /*** Startup tests/
        /checking build system type... x86_64-unknown-linux-gnu/
        /checking host system type... x86_64-unknown-linux-gnu/
        /checking target system type... x86_64-unknown-linux-gnu/
        /checking for gcc... gcc/
        /checking whether the C compiler works... yes/
        /checking for C compiler default output file name...
a.out/
        /checking for suffix of executables... /
        /*checking whether we are cross compiling...
configure: error:
        in `/home/user/openmpi-2.0.4':*/
        /configure: error: cannot run C compiled programs./
        /If you meant to cross compile, use `--host'./
        /See `config.log' for more details/

        I do not know waht is cross compiling but how can I
avoid this
        error ?
        Please help me out here.

        Thanks

        --         Abhisek Mondal
        /Senior Research Fellow
        /
        /Structural Biology and Bioinformatics Division
        /
        /CSIR-Indian Institute of Chemical Biology/
        /Kolkata 700032
        /
        /INDIA
        /


        ___
        users mailing list
users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
<mailto:users@lists.open-mpi.org
<mailto:users@lists.open-mpi.org>>
https://lists.open-mpi.org/mailman/listinfo/users
<https://lists.open-mpi.org/mailman/listinfo/users>
        <https://lists.open-mpi.org/mailman/listinfo/users
<https://lists.open-mpi.org/mailman/listinfo/users>>


    ___
    users mailing list
users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
<mailto:users@lists.open-mpi.org
<mailto:users@lists.open-mpi.org>>
https://lists.open-mpi.org/mailman/listinfo/users
<https://lists.open-mpi.org/mailman/listinfo/users>
    

Re: [OMPI users] can not compile Openmpi-3.0

2018-04-02 Thread Gilles Gouaillardet

This is the relevant part related to this error

configure:6620: checking whether we are cross compiling
configure:6628: gcc -o conftest    conftest.c  >&5
conftest.c:10:19: fatal error: stdio.h: No such file or directory
 #include 
   ^
compilation terminated.
configure:6632: $? = 1
configure:6639: ./conftest
../configure: line 6641: ./conftest: No such file or directory
configure:6643: $? = 127


at the very least, you need to install the glibc-headers package


Cheers,


Gilles


On 4/3/2018 1:22 PM, abhisek Mondal wrote:

Hello,

I have uploaded the config.log here: 
https://drive.google.com/drive/u/0/folders/0B6O-L5Y7BiGJfmQ4N2FpblBEcFNxaDZnaGpsUFFEUlotVWFjajR0UFFHNk5aYlhoSHVTWkU





On Tue, Apr 3, 2018 at 9:47 AM, Gilles Gouaillardet <gil...@rist.or.jp 
<mailto:gil...@rist.or.jp>> wrote:


Can you please compress and attach your config.log ?


You might also want to double check you can compile *and* run a
simple C hello world program


Cheers,


Gilles


On 4/3/2018 1:06 PM, abhisek Mondal wrote:

Hi,

I need some help regarding compiling Openmpi-3.0. I have
perfectly working C compiler, however, during configuration
I'm keep getting this error:


//
/== Configuring Open MPI/

//
/
/
/*** Startup tests/
/checking build system type... x86_64-unknown-linux-gnu/
/checking host system type... x86_64-unknown-linux-gnu/
/checking target system type... x86_64-unknown-linux-gnu/
/checking for gcc... gcc/
/checking whether the C compiler works... yes/
/checking for C compiler default output file name... a.out/
/checking for suffix of executables... /
/*checking whether we are cross compiling... configure: error:
in `/home/user/openmpi-2.0.4':*/
/configure: error: cannot run C compiled programs./
/If you meant to cross compile, use `--host'./
/See `config.log' for more details/

I do not know waht is cross compiling but how can I avoid this
error ?
Please help me out here.

Thanks

-- 
Abhisek Mondal

/Senior Research Fellow
/
/Structural Biology and Bioinformatics Division
/
/CSIR-Indian Institute of Chemical Biology/
/Kolkata 700032
/
/INDIA
/


___
users mailing list
users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
https://lists.open-mpi.org/mailman/listinfo/users
<https://lists.open-mpi.org/mailman/listinfo/users>


___
users mailing list
users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
https://lists.open-mpi.org/mailman/listinfo/users
<https://lists.open-mpi.org/mailman/listinfo/users>




--
Abhisek Mondal
/Senior Research Fellow
/
/Structural Biology and Bioinformatics Division
/
/CSIR-Indian Institute of Chemical Biology/
/Kolkata 700032
/
/INDIA
/


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] can not compile Openmpi-3.0

2018-04-02 Thread Gilles Gouaillardet

Can you please compress and attach your config.log ?


You might also want to double check you can compile *and* run a simple C 
hello world program



Cheers,


Gilles


On 4/3/2018 1:06 PM, abhisek Mondal wrote:

Hi,

I need some help regarding compiling Openmpi-3.0. I have perfectly 
working C compiler, however, during configuration I'm keep getting 
this error:


//
/== Configuring Open MPI/
//
/
/
/*** Startup tests/
/checking build system type... x86_64-unknown-linux-gnu/
/checking host system type... x86_64-unknown-linux-gnu/
/checking target system type... x86_64-unknown-linux-gnu/
/checking for gcc... gcc/
/checking whether the C compiler works... yes/
/checking for C compiler default output file name... a.out/
/checking for suffix of executables... /
/*checking whether we are cross compiling... configure: error: in 
`/home/user/openmpi-2.0.4':*/

/configure: error: cannot run C compiled programs./
/If you meant to cross compile, use `--host'./
/See `config.log' for more details/

I do not know waht is cross compiling but how can I avoid this error ?
Please help me out here.

Thanks

--
Abhisek Mondal
/Senior Research Fellow
/
/Structural Biology and Bioinformatics Division
/
/CSIR-Indian Institute of Chemical Biology/
/Kolkata 700032
/
/INDIA
/


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] libmpi_cxx

2018-03-28 Thread Gilles Gouaillardet
Arthur,

Try to
configure --enable-mpi-xxx

Note c++ bindings have been removed from the MPI standard long time ago, so you 
might want to consider modernizing your app.

Cheers,

Gilles

"Arthur H. Edwards"  wrote:
>I have built openmpi 3.0 on an ubuntu 16.04 system I have used --with-cuda. 
>There is no libmpi_cxx.so generated, yet the code I'm using requires it.  
>There is a libmpi_cxx.so in the ubuntu installed version. Any insight, or 
>instruction on how to configure so that the build generates this library would 
>be greatly  appreciated.
>
>Art Edwards
>
>-- 
>  Arthur H. Edwards
>  edwards...@fastmail.fm
>___
>users mailing list
>users@lists.open-mpi.org
>https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


  1   2   3   4   5   6   7   8   9   >