Does UCT support iWARP?
> On Apr 6, 2018, at 9:45 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> > wrote: > > That would be sweet. Are you aiming for v4.0.0, perchance? > > I.e., should we have "remove openib BTL / add uct BTL" to the target feature > list for v4.0.0? > > (I did laugh at the Kylo name, but I really, really don't want to keep the > propagate the idea of meaningless names for BTLs -- especially since BTL > names, more so than most other component names, are visible to the user. > ...unless someone wants to finally finish the ideas and implement the whole > network-transport-name system that we've talked about for a few years... :-) ) > > >> On Apr 5, 2018, at 1:49 PM, Thananon Patinyasakdikul <tpati...@vols.utk.edu> >> wrote: >> >> Just more information to help with the decision: >> >> I am working on Nathan’s uct btl to make it work with ob1 and infiniband. So >> this could be a replacement for openib and honestly we should totally call >> this new uct btl Kylo. >> >> Arm >> >>> On Apr 5, 2018, at 1:37 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> >>> wrote: >>> >>> Below is an email exchange from the users mailing list. >>> >>> I'm moving this over to devel to talk among the developer community. >>> >>> Multiple times recently on the users list, we've told people with problems >>> with the openib BTL that they should be using UCX (per Mellanox's >>> publicly-stated support positions). >>> >>> Is it time to deprecate / print warning messages / remove the openib BTL? >>> >>> >>> >>>> Begin forwarded message: >>>> >>>> From: Nathan Hjelm <hje...@me.com> >>>> Subject: Re: [OMPI users] Eager RDMA causing slow osu_bibw with 3.0.0 >>>> Date: April 5, 2018 at 12:48:08 PM EDT >>>> To: Open MPI Users <us...@lists.open-mpi.org> >>>> Cc: Open MPI Users <us...@lists.open-mpi.org> >>>> Reply-To: Open MPI Users <us...@lists.open-mpi.org> >>>> >>>> >>>> Honestly, this is a configuration issue with the openib btl. There is no >>>> reason to keep either eager RDMA nor is there a reason to pipeline RDMA. I >>>> haven't found an app where either of these "features" helps you with >>>> infiniband. You have the right idea with the parameter changes but Howard >>>> is correct, for Mellanox the future is UCX not verbs. I would try it and >>>> see if it works for you but if it doesn't I would set those two parameters >>>> in your /etc/openmpi-mca-params.conf and run like that. >>>> >>>> -Nathan >>>> >>>> On Apr 05, 2018, at 01:18 AM, Ben Menadue <ben.mena...@nci.org.au> wrote: >>>> >>>>> Hi, >>>>> >>>>> Another interesting point. I noticed that the last two message sizes >>>>> tested (2MB and 4MB) are lower than expected for both osu_bw and >>>>> osu_bibw. Increasing the minimum size to use the RDMA pipeline to above >>>>> these sizes brings those two data-points up to scratch for both >>>>> benchmarks: >>>>> >>>>> 3.0.0, osu_bw, no rdma for large messages >>>>> >>>>>> mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -map-by ppr:1:node >>>>>> -np 2 -H r6,r7 ./osu_bw -m 2097152:4194304 >>>>> # OSU MPI Bi-Directional Bandwidth Test v5.4.0 >>>>> # Size Bandwidth (MB/s) >>>>> 2097152 6133.22 >>>>> 4194304 6054.06 >>>>> >>>>> 3.0.0, osu_bibw, eager rdma disabled, no rdma for large messages >>>>> >>>>>> mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -mca >>>>>> btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw >>>>>> -m 2097152:4194304 >>>>> # OSU MPI Bi-Directional Bandwidth Test v5.4.0 >>>>> # Size Bandwidth (MB/s) >>>>> 2097152 11397.85 >>>>> 4194304 11389.64 >>>>> >>>>> This makes me think something odd is going on in the RDMA pipeline. >>>>> >>>>> Cheers, >>>>> Ben >>>>> >>>>> >>>>> >>>>>> On 5 Apr 2018, at 5:03 pm, Ben Menadue <ben.mena...@nci.org.au> wrote: >>>>>> Hi, >>>>>> >>>>>> We’ve just been running some OSU benchmarks with OpenMPI 3.0.0 and >>>>>> noticed that osu_bibw gives nowhere near the bandwidth I’d expect (this >>>>>> is on FDR IB). However, osu_bw is fine. >>>>>> >>>>>> If I disable eager RDMA, then osu_bibw gives the expected numbers. >>>>>> Similarly, if I increase the number of eager RDMA buffers, it gives the >>>>>> expected results. >>>>>> >>>>>> OpenMPI 1.10.7 gives consistent, reasonable numbers with default >>>>>> settings, but they’re not as good as 3.0.0 (when tuned) for large >>>>>> buffers. The same option changes produce no different in the performance >>>>>> for 1.10.7. >>>>>> >>>>>> I was wondering if anyone else has noticed anything similar, and if this >>>>>> is unexpected, if anyone has a suggestion on how to investigate further? >>>>>> >>>>>> Thanks, >>>>>> Ben >>>>>> >>>>>> >>>>>> Here’s are the numbers: >>>>>> >>>>>> 3.0.0, osu_bw, default settings >>>>>> >>>>>>> mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw >>>>>> # OSU MPI Bandwidth Test v5.4.0 >>>>>> # Size Bandwidth (MB/s) >>>>>> 1 1.13 >>>>>> 2 2.29 >>>>>> 4 4.63 >>>>>> 8 9.21 >>>>>> 16 18.18 >>>>>> 32 36.46 >>>>>> 64 69.95 >>>>>> 128 128.55 >>>>>> 256 250.74 >>>>>> 512 451.54 >>>>>> 1024 829.44 >>>>>> 2048 1475.87 >>>>>> 4096 2119.99 >>>>>> 8192 3452.37 >>>>>> 16384 2866.51 >>>>>> 32768 4048.17 >>>>>> 65536 5030.54 >>>>>> 131072 5573.81 >>>>>> 262144 5861.61 >>>>>> 524288 6015.15 >>>>>> 1048576 6099.46 >>>>>> 2097152 989.82 >>>>>> 4194304 989.81 >>>>>> >>>>>> 3.0.0, osu_bibw, default settings >>>>>> >>>>>>> mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw >>>>>> # OSU MPI Bi-Directional Bandwidth Test v5.4.0 >>>>>> # Size Bandwidth (MB/s) >>>>>> 1 0.00 >>>>>> 2 0.01 >>>>>> 4 0.01 >>>>>> 8 0.02 >>>>>> 16 0.04 >>>>>> 32 0.09 >>>>>> 64 0.16 >>>>>> 128 135.30 >>>>>> 256 265.35 >>>>>> 512 499.92 >>>>>> 1024 949.22 >>>>>> 2048 1440.27 >>>>>> 4096 1960.09 >>>>>> 8192 3166.97 >>>>>> 16384 127.62 >>>>>> 32768 165.12 >>>>>> 65536 312.80 >>>>>> 131072 1120.03 >>>>>> 262144 4724.01 >>>>>> 524288 4545.93 >>>>>> 1048576 5186.51 >>>>>> 2097152 989.84 >>>>>> 4194304 989.88 >>>>>> >>>>>> 3.0.0, osu_bibw, eager RDMA disabled >>>>>> >>>>>>> mpirun -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H >>>>>>> r6,r7 ./osu_bibw >>>>>> # OSU MPI Bi-Directional Bandwidth Test v5.4.0 >>>>>> # Size Bandwidth (MB/s) >>>>>> 1 1.49 >>>>>> 2 2.97 >>>>>> 4 5.96 >>>>>> 8 11.98 >>>>>> 16 23.95 >>>>>> 32 47.39 >>>>>> 64 93.57 >>>>>> 128 153.82 >>>>>> 256 304.69 >>>>>> 512 572.30 >>>>>> 1024 1003.52 >>>>>> 2048 1083.89 >>>>>> 4096 1879.32 >>>>>> 8192 2785.18 >>>>>> 16384 3535.77 >>>>>> 32768 5614.72 >>>>>> 65536 8113.69 >>>>>> 131072 9666.74 >>>>>> 262144 10738.97 >>>>>> 524288 11247.02 >>>>>> 1048576 11416.50 >>>>>> 2097152 989.88 >>>>>> 4194304 989.88 >>>>>> >>>>>> 3.0.0, osu_bibw, increased eager RDMA buffer count >>>>>> >>>>>>> mpirun -mca btl_openib_eager_rdma_num 32768 -map-by ppr:1:node -np 2 -H >>>>>>> r6,r7 ./osu_bibw >>>>>> # OSU MPI Bi-Directional Bandwidth Test v5.4.0 >>>>>> # Size Bandwidth (MB/s) >>>>>> 1 1.42 >>>>>> 2 2.84 >>>>>> 4 5.67 >>>>>> 8 11.18 >>>>>> 16 22.46 >>>>>> 32 44.65 >>>>>> 64 83.10 >>>>>> 128 154.00 >>>>>> 256 291.63 >>>>>> 512 537.66 >>>>>> 1024 942.35 >>>>>> 2048 1433.09 >>>>>> 4096 2356.40 >>>>>> 8192 1998.54 >>>>>> 16384 3584.82 >>>>>> 32768 5523.08 >>>>>> 65536 7717.63 >>>>>> 131072 9419.50 >>>>>> 262144 10564.77 >>>>>> 524288 11104.71 >>>>>> 1048576 11130.75 >>>>>> 2097152 7943.89 >>>>>> 4194304 5270.00 >>>>>> >>>>>> 1.10.7, osu_bibw, default settings >>>>>> >>>>>>> mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw >>>>>> # OSU MPI Bi-Directional Bandwidth Test v5.4.0 >>>>>> # Size Bandwidth (MB/s) >>>>>> 1 1.70 >>>>>> 2 3.45 >>>>>> 4 6.95 >>>>>> 8 13.68 >>>>>> 16 27.41 >>>>>> 32 53.80 >>>>>> 64 105.34 >>>>>> 128 164.40 >>>>>> 256 324.63 >>>>>> 512 623.95 >>>>>> 1024 1127.35 >>>>>> 2048 1784.58 >>>>>> 4096 3305.45 >>>>>> 8192 3697.55 >>>>>> 16384 4935.75 >>>>>> 32768 7186.28 >>>>>> 65536 8996.94 >>>>>> 131072 9301.78 >>>>>> 262144 4691.36 >>>>>> 524288 7039.18 >>>>>> 1048576 7213.33 >>>>>> 2097152 9601.41 >>>>>> 4194304 9281.31 >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@lists.open-mpi.org >>>>> https://lists.open-mpi.org/mailman/listinfo/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@lists.open-mpi.org >>>> https://lists.open-mpi.org/mailman/listinfo/users >>> >>> >>> -- >>> Jeff Squyres >>> jsquy...@cisco.com >>> >>> _______________________________________________ >>> devel mailing list >>> devel@lists.open-mpi.org >>> https://lists.open-mpi.org/mailman/listinfo/devel >> >> _______________________________________________ >> devel mailing list >> devel@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/devel > > > -- > Jeff Squyres > jsquy...@cisco.com > > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/devel -- Jeff Squyres jsquy...@cisco.com _______________________________________________ devel mailing list devel@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/devel
