Re: [OMPI users] growing memory use from MPI application

2019-06-19 Thread George Bosilca via users
To completely disable UCX you need to disable the UCX MTL and not only the
BTL. I would use "--mca pml ob1 --mca btl ^ucx —mca btl_openib_allow_ib 1".

As you have a gdb session on the processes you can try to break on some of
the memory allocations function (malloc, realloc, calloc).

  George.


On Wed, Jun 19, 2019 at 2:37 PM Noam Bernstein via users <
users@lists.open-mpi.org> wrote:

> I tried to disable ucx (successfully, I think - I replaced the “—mca btl
> ucx —mca btl ^vader,tcp,openib” with “—mca btl_openib_allow_ib 1”, and
> attaching gdb to a running process shows no ucx-related routines active).
> It still has the same fast growing (1 GB/s) memory usage problem.
>
>
>   Noam
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Can displs in Scatterv/Gatherv/etc be a GPU array for CUDA-aware MPI?

2019-06-11 Thread George Bosilca via users
Leo,

In a UMA system having the displacement and/or recvcounts arrays on managed
GPU memory should work, but it will incur overheads for at least 2 reasons:
1. the MPI API arguments are checked for correctness (here recvcounts)
2. the collective algorithm part that executes on the CPU uses the
displacements and recvcounts to issue and manage communications and it
therefore need access to both.

Moreover, as you mention your code will not be portable anymore.

  George.


On Tue, Jun 11, 2019 at 11:27 AM Fang, Leo via users <
users@lists.open-mpi.org> wrote:

> Hello,
>
>
> I understand that once Open MPI is built against CUDA, sendbuf/recvbuf can
> be pointers to GPU memory. I wonder whether or not the “displs" argument of
> the collective calls on variable data (Scatterv/Gatherv/etc) can also live
> on GPU. CUDA awareness isn’t part of the MPI standard (yet), so I suppose
> it’s worth asking or even documenting.
>
> Thank you.
>
>
> Sincerely,
> Leo
>
> ---
> Yao-Lung Leo Fang
> Assistant Computational Scientist
> Computational Science Initiative
> Brookhaven National Laboratory
> Bldg. 725, Room 2-169
> P.O. Box 5000, Upton, NY 11973-5000
> Office: (631) 344-3265
> Email: leof...@bnl.gov
> Website: https://leofang.github.io/
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Open questions on MPI_Allreduce background implementation

2019-06-08 Thread George Bosilca via users
There is an ongoing discussion about this on issue #4067 (
https://github.com/open-mpi/ompi/issues/4067). Also the mailing list
contains few examples on how to tweak the collective algorithms to your
needs.

  George.


On Thu, Jun 6, 2019 at 7:42 PM hash join via users 
wrote:

> Hi all,
>
>
> I want to use the MPI_Allreduce to synchronize results from all the
> processes:
> https://github.com/open-mpi/ompi/blob/master/ompi/mpi/c/allreduce.c
>
> But I want to know the background algorithm to implement Allreduce. When I
> check the coll base of MPI_Allreduce:
> https://github.com/open-mpi/ompi/blob/2ae3cfd9bc9aa8cab80986a1921fd7ad9d198d07/ompi/mca/coll/base/coll_base_allreduce.c
>
> There are implementations like:
> ompi_coll_base_allreduce_intra_nonoverlapping,
> ompi_coll_base_allreduce_intra_recursivedoubling,
> ompi_coll_base_allreduce_intra_ring..etc. So how can I know which exact
> implement I am using when I use MPI_Allreduce. Or can I configure these?
> For example, I only want to use ring allreduce. How should I configure it?
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OMPI 4.0.1 valgrind error on simple MPI_Send()

2019-04-30 Thread George Bosilca via users
Depending on the alignment of the different types there might be small
holes in the low-level headers we exchange between processes It should not
be a concern for users.

valgrind should not stop on the first detected issue except
if --exit-on-first-error has been provided (the default value should be
no), so the SIGTERM might be generated for some other reason. What is
at jackhmmer.c:1597 ?

  George.


On Tue, Apr 30, 2019 at 2:27 PM David Mathog via users <
users@lists.open-mpi.org> wrote:

> Attempting to debug a complex program (99.9% of which is others' code)
> which stops running when run in valgrind as follows:
>
> mpirun -np 10 \
>--hostfile /usr/common/etc/openmpi.machines.LINUX_INTEL_newsaf_rev2 \
>--mca plm_rsh_agent rsh \
>/usr/bin/valgrind \
>  --leak-check=full \
>  --leak-resolution=high \
>  --show-reachable=yes \
>  --log-file=nc.vg.%p \
>  --suppressions=/opt/ompi401/share/openmpi/openmpi-valgrind.supp \
> /usr/common/tmp/jackhmmer  \
>--tformat ncbi \
>-T 150  \
>--chkhmm jackhmmer_test \
>--mpi \
>~safrun/a1hu.pfa \
>/usr/common/tmp/testing/nr_lcl \
>>jackhmmer_test_mpi.out 2>jackhmmer_test_mpi.stderr &
>
> Every one of the nodes has a variant of this in the log file (followed
> by a long list
> of memory allocation errors, since it exits without being able to clean
> anything up):
>
> ==5135== Memcheck, a memory error detector
> ==5135== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==5135== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright
> info
> ==5135== Command: /usr/common/tmp/jackhmmer --tformat ncbi -T 150
> --chkhmm jackhmmer_test --mpi /ulhhmi/safrun
> /a1hu.pfa /usr/common/tmp/testing/nr_lcl
> ==5135== Parent PID: 5119
> ==5135==
> ==5135== Syscall param socketcall.sendto(msg) points to uninitialised
> byte(s)
> ==5135==at 0x5459BFB: send (in /usr/lib64/libpthread-2.17.so)
> ==5135==by 0xF84A282: mca_btl_tcp_send_blocking (in
> /opt/ompi401/lib/openmpi/mca_btl_tcp.so)
> ==5135==by 0xF84E414: mca_btl_tcp_endpoint_send_handler (in
> /opt/ompi401/lib/openmpi/mca_btl_tcp.so)
> ==5135==by 0x5D6E4EF: event_persist_closure (event.c:1321)
> ==5135==by 0x5D6E4EF: event_process_active_single_queue
> (event.c:1365)
> ==5135==by 0x5D6E4EF: event_process_active (event.c:1440)
> ==5135==by 0x5D6E4EF: opal_libevent2022_event_base_loop
> (event.c:1644)
> ==5135==by 0x5D2465F: opal_progress (in
> /opt/ompi401/lib/libopen-pal.so.40.20.1)
> ==5135==by 0xF36A9CC: ompi_request_wait_completion (in
> /opt/ompi401/lib/openmpi/mca_pml_ob1.so)
> ==5135==by 0xF36C30E: mca_pml_ob1_send (in
> /opt/ompi401/lib/openmpi/mca_pml_ob1.so)
> ==5135==by 0x51BC581: PMPI_Send (in
> /opt/ompi401/lib/libmpi.so.40.20.1)
> ==5135==by 0x40B46E: mpi_worker (jackhmmer.c:1560)
> ==5135==by 0x406726: main (jackhmmer.c:413)
> ==5135==  Address 0x1ffefff8d5 is on thread 1's stack
> ==5135==  in frame #2, created by mca_btl_tcp_endpoint_send_handler
> (???:)
> ==5135==
> ==5135==
> ==5135== Process terminating with default action of signal 15 (SIGTERM)
> ==5135==at 0x5459EFD: ??? (in /usr/lib64/libpthread-2.17.so)
> ==5135==by 0x408817: mpi_failure (jackhmmer.c:887)
> ==5135==by 0x40B708: mpi_worker (jackhmmer.c:1597)
> ==5135==by 0x406726: main (jackhmmer.c:413)
>
> jackhmmer line 1560 is just this:
>
>
>  MPI_Send(, 1, MPI_INT, 0, HMMER_SETUP_READY_TAG,
> MPI_COMM_WORLD);
>
> preceded at varying distances by:
>
>int  status   = eslOK;
>status = 0;
>
> I can see why MPI might have some uninitialized bytes in that send, for
> instance, if it has a minimum buffer size it will send or something like
> that.  The problem is that it completely breaks valgrind in this
> application because valgrind exits immediately when it sees this error.
> The suppression file supplied with the release does not prevent that.
>
> How do I work around this?
>
> Thank you,
>
> David Mathog
> mat...@caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] 3.0.4, 4.0.1 build failure on OSX Mojave with LLVM

2019-04-24 Thread George Bosilca via users
Jon,

The configure AC_HEADER_STDC macro is considered obsolete [1] as most of
the OSes are STDC compliant nowadays. To have it failing on a recent
version of OSX, is therefore something unexpected. Moreover, many of the
OMPI developers work on OSX Mojave with the default compiler but with the
same system headers as you, so I expect the problem is not coming from the
OS but from how your compiler has been installed. To add to this assumption
I can successfully compile master and 4.0 on my laptop (Mojave) with the
default compile, the macport clang 7.0 package, the macport gcc 7 and 8,
and icc 19 20180804.

I would start investigating with the most basic config.log you can provide
(no STDC related flags), to see if we can figure out why autoconf fails to
detect the STDC compliance.

  George.

[1]
https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/Particular-Headers.html

On Wed, Apr 24, 2019 at 10:07 AM JR Cary via users 
wrote:

>
>
> On 4/24/19 7:25 AM, Gilles Gouaillardet wrote:
> > John,
> >
> > what if you move some parameters to CPPFLAGS and CXXCPPFLAGS (see the
> > new configure command line below)
>
> Thanks, Gilles, that did in fact work for 3.3.4.  I was writing up an email
> when yours came in, so I'll move my response to here:
>
> Jeff, I've now tried several things, so I have some ideas, and I can
> send you one
> or more config.log's, as you wish.  Will mention options at end of this.
>
> First, I got the compile to work, below, with -DOPAL_STDC_HEADERS added
> to CFLAGS,
> but then subsequent packages failed to build for not knowing ptrdiff_t.
> So it
> appeared that I would have to have -DOPAL_STDC_HEADERS set for all
> dependent
> packaged.  So then I configured with CPPFLAGS set to the same as CFLAGS,
> and
> for 3.1.4, it configured, built, installed, and dependent packages did
> as well.
> For 4.0.1, it did not.
>
> Some responses below:
>
>  A few notes on your configure line:
> 
> >
> '/Users/cary/projects/ulixesall-llvm/builds/openmpi-4.0.1/nodl/../configure'
> \
> >
> --prefix=/Volumes/GordianStorage/opt/contrib-llvm7_appleclang/openmpi-4.0.1-nodl
> \
> >
> CC='/Volumes/GordianStorage/opt/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang'
> \
> >
> CXX='/Volumes/GordianStorage/opt/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++'
> \
>  The MPI C++ bindings are no longer built by default, and you're not
> enabling them (via --enable-mpi-cxx), so you don't need to specify CXX or
> CXXFLAGS here.
>
> Thanks for this and your other comments on how to minimize our
> configuration
> line.  Will adopt.
>
>
> >FC='/opt/homebrew/bin/gfortran-6' \
> >F77='/opt/homebrew/bin/gfortran-6' \
>  F77 is ignored these days; FC is the only one that matters.
>
> Thanks.  Will clean up.
>
>
> >CFLAGS='-fvisibility=default -mmacosx-version-min=10.10
> -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include
> -fPIC -pipe -DSTDC_HEADERS' \
>  Do you need all these CFLAGS?  E.g., does clang not -I that directory
> by default (I don't actually know if it is necessary or not)?  What does
> -DSTDC_HEADERS do?
>
> -fvisibility=default: found needed for Boost, and kept through entire
> chain.  May revisit.
>
> -mmacosx-version-min=10.10: Gives a build that is guaranteed to work back
> through Yosemite, when
> we distribute our binaries.
>
> -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include:
> Apple is now putting nothing in /usr/include.  The sdk then provides its
> unique /usr/include above.  However, the LLVM latest (which gives openmp),
> does not know
> that, and so it has to be told which /usr/include to use.
>
> One can install command-line tools, but then one cannot have several sdk's
> co-exist.
>
> -fPIC: duplicative given --with-pic, but no harm, passed to all our
> packages.
>
> -pipe: supposedly build performance, but I have not tested.
>
> I *think* -DSTDC_HEADERS is a hint to Apple (but not to openmpi) to use
> only headers
> consistent with stdc.
>
>  Can you send all the information listed here:
> 
>   https://www.open-mpi.org/community/help/
>
> I have the following builds/build attempts:
>
> 3.3.4:
>
> 1. Not adding -DOPAL_STDC_HEADERS: build fails.
>
> 2. Adding -DOPAL_STDC_HEADERS: build succeeds, hdf5 parallel build fails.
>
> 3. Not adding -DOPAL_STDC_HEADERS, but setting CPPFLAGS to be identical
> to CFLAGS:
> openmpi and hdf5-parallel builds both succeed.
>
> 4.0.1, best formula above (CPPFLAGS set to same as CFLAGS):
>
> 4. Fails to build for a different reason.
>
> Which would you like?
>
> Best..John
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org