Josh and Valentin, thanks a lot for your answers! Your understanding of my case is essentially correct, but let me briefly refine.

In the case at hand I am running a 3D solver, which computes on grids with at least 26-neighbors. But that also depends on the grid refinement etc, so there could be much more than that. The code runs with a single MPI rank per node, with OpenMP for intra-node parallelism, and some of the neighbors might well reside on the same node. Admittedly, in that case they do not need to participate in the Bcast at all, because the threads can access each other's memory.

So the size of the communicator might be small in practice, but up to ~20 (?) in extreme case. I will have to evaluate that by experiments, but it also depends on the problem solved. But my main concern when thinking about this is not the efficiency of communication as such, but the local CPU memory bandwidth. If I use an Isend, say, 20 times, then with each send the IB hardware will have to read the message from the main memory. My hope was to send it to the hardware once, and be done with it. From this point of view it doesn't really matter how the hardware handles the message, what is the complexity of the Bcast / Mcast algorithm, and if the communication will or not be faster than the trivial case. I just don't want to pollute the local memory bus with the same data over and over again.

About the RMA Put, I was wondering if it is possible to achieve the same result: make a call to some *_put function (not necessarily MPI_Put, could be UCX or any other) with multiple destinations, which reads the data from the host memory once, and sends it to the hardware. The HW then propagates it to all other receivers. So, a kind of an RMA broad-write, or multi-write. Not sure if something like that even exists, but it would be a perfect choice.


On 3/22/19 8:22 AM, Valentin Petrov wrote:


One more comment. Regarding the IB Mcast and its usage. The key advantage of IB Mcast (enabled with hpcx+hcoll when user utilizes MPI_Bcast collective call) is nearly constant scaling. So, it gives most advantage when many multiple nodes are used at the same time for a collective. By default, hcoll will use mcast based algorithm when the MPI communicator used to launch MPI_Bcast spans at least 8 nodes (does not matter how many ranks-per-node in this case). As far as I understood your case, you have a Neighbor-Exchange pattern, i.e. limited number of neighbors (usually 4 in 2D regular grid, or 6 in 3D) that you need to send the data to. If this is the case then your application most likely has many multiple small (5,7 ranks) communicators. IB mcast is unlikely to give a lot of advantage in this case (scale is small). Moreover, if you run multiple ranks per node then a lot of communication happens entirely innode where the data goes through shared memory w/o IB.

Finally, if you want to do RDMA, it is possible. You could do MPI_Put/get. Alternative is to use open_shmem which could be a good fit for that case (this is the standard that is designed for one-sided put/get semantics) but this will require some study if you are not familiar with it.

*From:* devel <> *On Behalf Of *George Bosilca
*Sent:* Thursday, March 21, 2019 7:31 PM
*To:* Open MPI Developers <>
*Subject:* Re: [OMPI devel] Memory performance with Bcast


I am not sure I understand your question, a bcast is a collective operation that must be posted by all participants. Independently at what level the bcast is serviced, if some of the participants have not posted their participation to the collective, only partial progress can be made.


On Thu, Mar 21, 2019 at 12:24 PM Joshua Ladd < <>> wrote:


    HPC-X implements the MPI BCAST operation by leveraging hardware
    multicast capabilities. Starting with HPC-X v2.3 we introduced a
    new multicast based algorithm for large messages as well. Hardware
    multicast scales as O(1) modulo switch hops. It is the most
    efficient way to broadcast a message in an IB network.

    Hope this helps.



    On Thu, Mar 21, 2019 at 5:01 AM marcin.krotkiewski
    <>> wrote:

        Thanks, George! So, the function you mentioned is used when I
        turn off HCOLL and use OpenMPI's tuned coll instead. That
        helps a lot. Another thing that makes me think is that in my
        case the data is sent to the targets asynchronously, or rather
        - it is a 'put' operation in nature, and the targets don't
        know, when the data is ready. I guess the tree algorithms you
        mentioned require active participation of all nodes, otherwise
        the algorithm will not progress? Is it enough to call any MPI
        routine to assure progression, or do I have to call the
        matching Bcast?

        Anyone from Mellanox here, who knows how HCOLL does this
        internally? Especially on the EDR architecture. Is there any
        hardware aid?



        On 3/20/19 5:10 PM, George Bosilca wrote:

            If you have support for FCA then it might happen that the
            collective will use the hardware support. In any case,
            most of the bcast algorithms have a logarithmic behavior,
            so there will be at most O(log(P)) memory accesses on the

            If you want to take a look at the code in OMPI to
            understand what function is called in your specific case
            head to ompi/mca/coll/tuned/ and search for the
            ompi_coll_tuned_bcast_intra_dec_fixed function
            in coll_tuned_decision_fixed.c.


            On Wed, Mar 20, 2019 at 4:53 AM marcin.krotkiewski
            <>> wrote:


                I'm wondering about the details of Bcast
                implementation in OpenMPI. I'm
                specifically interested in IB interconnects, but
                information about other
                architectures (and OpenMPI in general) would also be
                very useful.

                I am working with a code, which sends the same 
                (large) message to a
                bunch of 'neighboring' processes. Somewhat like a
                ghost-zone exchange,
                but the message is the same for all neighbors. Since
                memory bandwidth is
                a scarce resource, I'd like to make sure we send the
                message with fewest
                possible memory accesses.

                Hence the question: what does OpenMPI (and
                specifically for the IB case
                - the HPCX) do in such case? Does it get the buffer
                from memory O(1)
                times to send it to n peers, and the broadcast is
                orchestrated by the
                hardware? Or does it have to read the memory O(n)
                times? Is it more
                efficient to use Bcast, or is it the same as
                implementing the operation
                by n distinct send / put operations? Finally, is there
                any way to use
                the RMA put method with multiple targets, so that I
                only have to read
                the host memory once, and the switches / HCA take care
                of the rest?

                Thanks a lot for any insights!


                devel mailing list


            devel mailing list



        devel mailing list <>

    devel mailing list <>

devel mailing list
devel mailing list

Reply via email to