Dennis, I know you're using the OSU benchmark to test the allreduce, but if you want an alternative benchmark then NetPIPE does test more message sizes and allows you to test at the InfiniBand as well as MPI levels. I haven't run the IB tests in a few years, but they should still work. It doesn't run global tests, but does point-to-point unidirectional, bi-directional, and aggregate and may give you some information about the performance change at 16 KB and whether it is coming from OpenMPI or IB.
https://netpipe.cs.ksu.edu Dave Turner On Tue, Feb 8, 2022 at 1:01 PM <users-requ...@lists.open-mpi.org> wrote: > Send users mailing list submissions to > users@lists.open-mpi.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.open-mpi.org/mailman/listinfo/users > or, via email, send a message with subject or body 'help' to > users-requ...@lists.open-mpi.org > > You can reach the person managing the list at > users-ow...@lists.open-mpi.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users digest..." > > > Today's Topics: > > 1. Re: Using OSU benchmarks for checking Infiniband network > (Joseph Schuchart) > 2. Re: Using OSU benchmarks for checking Infiniband network > (Bertini, Denis Dr.) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 8 Feb 2022 10:02:53 -0500 > From: Joseph Schuchart <schuch...@icl.utk.edu> > To: users@lists.open-mpi.org > Subject: Re: [OMPI users] Using OSU benchmarks for checking Infiniband > network > Message-ID: <d31c2d1a-0fad-f8b8-e9c8-7cd6a0fab...@icl.utk.edu> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Hi Denis, > > Sorry if I missed it in your previous messages but could you also try > running a different MPI implementation (MVAPICH) to see whether Open MPI > is at fault or the system is somehow to blame for it? > > Thanks > Joseph > > On 2/8/22 03:06, Bertini, Denis Dr. via users wrote: > > > > Hi > > > > Thanks for all these informations ! > > > > > > But i have to confess that in this?multi-tuning-parameter space, > > > > i got somehow lost. > > > > Furthermore it is somtimes mixing between user-space and kernel-space. > > > > I have only possibility to act on the user space. > > > > > > 1)?So i have on the system max locked memory: > > > > ? ? ? ? ? ? ? ? ? ? ? ? - ulimit -l unlimited (default ) > > > > ? and i do not see any warnings/errors related to that when launching > MPI. > > > > > > 2) I tried differents algorithms for MPI_all_reduce op.? all showing > > drop in > > > > bw for size=16384 > > > > > > 4) I disable openIB ( no RDMA, ) and used only TCP,? and i noticed > > > > the same behaviour. > > > > > > 3) i realized that increasing the so-called warm up parameter? in the > > > > OSU benchmark (argument -x 200 as default) the discrepancy. > > > > At the contrary putting lower threshold ( -x 10 ) can increase this BW > > > > discrepancy up to factor 300 at message size 16384 compare to > > > > message size 8192 for example. > > > > So does it means that there are some caching effects > > > > in the internode communication? > > > > > > From my experience, to?tune parameters is a time-consuming and cumbersome > > > > task. > > > > > > Could it also be the problem is not really on the openMPI > > implemenation but on the > > > > system? > > > > > > Best > > > > Denis > > > > ------------------------------------------------------------------------ > > *From:* users <users-boun...@lists.open-mpi.org> on behalf of Gus > > Correa via users <users@lists.open-mpi.org> > > *Sent:* Monday, February 7, 2022 9:14:19 PM > > *To:* Open MPI Users > > *Cc:* Gus Correa > > *Subject:* Re: [OMPI users] Using OSU benchmarks for checking > > Infiniband network > > This may have changed since, but these used to be relevant points. > > Overall, the Open MPI FAQ have lots of good suggestions: > > https://www.open-mpi.org/faq/ > > some specific for performance tuning: > > https://www.open-mpi.org/faq/?category=tuning > > https://www.open-mpi.org/faq/?category=openfabrics > > > > 1) Make sure you are not using the Ethernet TCP/IP, which is widely > > available in compute nodes: > > mpirun --mca btl self,sm,openib ... > > > > https://www.open-mpi.org/faq/?category=tuning#selecting-components > > > > However, this may have changed lately: > > https://www.open-mpi.org/faq/?category=tcp#tcp-auto-disable > > 2) Maximum locked memory used by IB and their system limit. Start > > here: > > > https://www.open-mpi.org/faq/?category=openfabrics#limiting-registered-memory-usage > > 3) The eager vs. rendezvous message size threshold. I wonder if it may > > sit right where you see the latency spike. > > https://www.open-mpi.org/faq/?category=all#ib-locked-pages-user > > 4) Processor and memory locality/affinity and binding (please check > > the current options and syntax) > > https://www.open-mpi.org/faq/?category=tuning#using-paffinity-v1.4 > > > > On Mon, Feb 7, 2022 at 11:01 AM Benson Muite via users > > <users@lists.open-mpi.org> wrote: > > > > Following https://www.open-mpi.org/doc/v3.1/man1/mpirun.1.php > > > > mpirun --verbose --display-map > > > > Have you tried newer OpenMPI versions? > > > > Do you get similar behavior for the osu_reduce and osu_gather > > benchmarks? > > > > Typically internal buffer sizes as well as your hardware will affect > > performance. Can you give specifications similar to what is > > available at: > > http://mvapich.cse.ohio-state.edu/performance/collectives/ > > where the operating system, switch, node type and memory are > > indicated. > > > > If you need good performance, may want to also specify the algorithm > > used. You can find some of the parameters you can tune using: > > > > ompi_info --all > > > > A particular helpful parameter is: > > > > MCA coll tuned: parameter "coll_tuned_allreduce_algorithm" (current > > value: "ignore", data source: default, level: 5 tuner/detail, > > type: int) > > ? ? ? ? ? ? ? ? ? ? ? ? ? ?Which allreduce algorithm is used. Can be > > locked down to any of: 0 ignore, 1 basic linear, 2 nonoverlapping > > (tuned > > reduce + tuned bcast), 3 recursive doubling, 4 ring, 5 segmented ring > > ? ? ? ? ? ? ? ? ? ? ? ? ? ?Valid values: 0:"ignore", > > 1:"basic_linear", > > 2:"nonoverlapping", 3:"recursive_doubling", 4:"ring", > > 5:"segmented_ring", 6:"rabenseifner" > > ? ? ? ? ? ?MCA coll tuned: parameter > > "coll_tuned_allreduce_algorithm_segmentsize" (current value: "0", > > data > > source: default, level: 5 tuner/detail, type: int) > > > > For OpenMPI 4.0, there is a tuning program [2] that might also be > > helpful. > > > > [1] > > > https://stackoverflow.com/questions/36635061/how-to-check-which-mca-parameters-are-used-in-openmpi > > [2] https://github.com/open-mpi/ompi-collectives-tuning > > > > On 2/7/22 4:49 PM, Bertini, Denis Dr. wrote: > > > Hi > > > > > > When i repeat i always got the huge?discrepancy at the > > > > > > message size of 16384. > > > > > > May be there is a way to run mpi in verbose mode in order > > > > > > to further investigate this behaviour? > > > > > > Best > > > > > > Denis > > > > > > > > > ------------------------------------------------------------------------ > > > *From:* users <users-boun...@lists.open-mpi.org> on behalf of > > Benson > > > Muite via users <users@lists.open-mpi.org> > > > *Sent:* Monday, February 7, 2022 2:27:34 PM > > > *To:* users@lists.open-mpi.org > > > *Cc:* Benson Muite > > > *Subject:* Re: [OMPI users] Using OSU benchmarks for checking > > Infiniband > > > network > > > Hi, > > > Do you get similar results when you repeat the test? Another job > > could > > > have interfered with your run. > > > Benson > > > On 2/7/22 3:56 PM, Bertini, Denis Dr. via users wrote: > > >> Hi > > >> > > >> I am using OSU microbenchmarks compiled with?openMPI 3.1.6 in > > order to > > >> check/benchmark > > >> > > >> the infiniband network for our cluster. > > >> > > >> For that i use the collective all_reduce benchmark and run over > > 200 > > >> nodes, using 1 process per node. > > >> > > >> And this is the results i obtained ? > > >> > > >> > > >> > > >> ################################################################ > > >> > > >> # OSU MPI Allreduce Latency Test v5.7.1 > > >> # Size? ? ? ?Avg Latency(us)? ?Min Latency(us) ?Max > > Latency(us)? Iterations > > >> 4? ? ? ? ? ? ? ? ? ? ?114.65? ? ? ? ? ? ?83.22 ? ? ? 147.98? ? > > ? ? 1000 > > >> 8? ? ? ? ? ? ? ? ? ? ?133.85? ? ? ? ? ? 106.47 ? ? ? 164.93? ? > > ? ? 1000 > > >> 16? ? ? ? ? ? ? ? ? ? 116.41? ? ? ? ? ? ?87.57 ? ? ? 150.58? ? > > ? ? 1000 > > >> 32? ? ? ? ? ? ? ? ? ? 112.17? ? ? ? ? ? ?93.25 ? ? ? 130.23? ? > > ? ? 1000 > > >> 64? ? ? ? ? ? ? ? ? ? 106.85? ? ? ? ? ? ?81.93 ? ? ? 134.74? ? > > ? ? 1000 > > >> 128? ? ? ? ? ? ? ? ? ?117.53? ? ? ? ? ? ?87.50 ? ? ? 152.27? ? > > ? ? 1000 > > >> 256? ? ? ? ? ? ? ? ? ?143.08? ? ? ? ? ? 115.63 ? ? ? 173.97? ? > > ? ? 1000 > > >> 512? ? ? ? ? ? ? ? ? ?130.34? ? ? ? ? ? 100.20 ? ? ? 167.56? ? > > ? ? 1000 > > >> 1024? ? ? ? ? ? ? ? ? 155.67? ? ? ? ? ? 111.29 ? ? ? 188.20? ? > > ? ? 1000 > > >> 2048? ? ? ? ? ? ? ? ? 151.82? ? ? ? ? ? 116.03 ? ? ? 198.19? ? > > ? ? 1000 > > >> 4096? ? ? ? ? ? ? ? ? 159.11? ? ? ? ? ? 122.09 ? ? ? 199.24? ? > > ? ? 1000 > > >> 8192? ? ? ? ? ? ? ? ? 176.74? ? ? ? ? ? 143.54 ? ? ? 221.98? ? > > ? ? 1000 > > >> 16384? ? ? ? ? ? ? ?48862.85? ? ? ? ? 39270.21 ? ? 54970.96? ? > > ? ? 1000 > > >> 32768? ? ? ? ? ? ? ? 2737.37? ? ? ? ? ?2614.60 ? ? ?2802.68? ? > > ? ? 1000 > > >> 65536? ? ? ? ? ? ? ? 2723.15? ? ? ? ? ?2585.62 ? ? ?2813.65? ? > > ? ? 1000 > > >> > > >> > > #################################################################### > > >> > > >> Could someone explain me what is happening for message = 16384 ? > > >> One can notice a?huge latency (~ 300 time larger) compare to > > message > > >> size = 8192. > > >> I do not really understand what could? create such an increase > > in the > > >> latency. > > >> The reason i use the OSU?microbenchmarks?is that we > > >> sporadically?experience?a drop > > >> in the bandwith for typical collective operations such as > > MPI_Reduce in > > >> our cluster > > >> which is difficult to understand. > > >> I would be grateful if somebody can share its expertise or such > > problem > > >> with me. > > >> > > >> Best, > > >> Denis > > >> > > >> > > >> > > >> --------- > > >> Denis Bertini > > >> Abteilung:?CIT > > >> Ort: SB3 2.265a > > >> > > >> Tel: +49 6159 71 2240 > > >> Fax: +49 6159 71 2986 > > >> E-Mail: d.bert...@gsi.de > > >> > > >> GSI Helmholtzzentrum f?r Schwerionenforschung GmbH > > >> Planckstra?e 1, 64291 Darmstadt, Germany, www.gsi.de > > <http://www.gsi.de> > > >> > > >> Commercial Register / Handelsregister: Amtsgericht Darmstadt, > > HRB 1528 > > >> Managing Directors / Gesch?ftsf?hrung: > > >> Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, J?rg Blaurock > > >> Chairman of the GSI Supervisory Board / Vorsitzender des > > GSI-Aufsichtsrats: > > >> Ministerialdirigent Dr. Volkmar Dietz > > >> > > > > > > > > > ------------------------------ > > Message: 2 > Date: Tue, 8 Feb 2022 15:37:28 +0000 > From: "Bertini, Denis Dr." <d.bert...@gsi.de> > To: Open MPI Users <users@lists.open-mpi.org> > Subject: Re: [OMPI users] Using OSU benchmarks for checking Infiniband > network > Message-ID: <c52cbd1f11c2477d907da27d18d47...@gsi.de> > Content-Type: text/plain; charset="utf-8" > > Hi > > > I do not have so much experience with MVAPICH. > > Since we work with singularity container, i can create a > > container and install this version to compare. > > > Cheers, > > Denis > > ________________________________ > From: users <users-boun...@lists.open-mpi.org> on behalf of Joseph > Schuchart via users <users@lists.open-mpi.org> > Sent: Tuesday, February 8, 2022 4:02:53 PM > To: users@lists.open-mpi.org > Cc: Joseph Schuchart > Subject: Re: [OMPI users] Using OSU benchmarks for checking Infiniband > network > > Hi Denis, > > Sorry if I missed it in your previous messages but could you also try > running a different MPI implementation (MVAPICH) to see whether Open MPI > is at fault or the system is somehow to blame for it? > > Thanks > Joseph > > On 2/8/22 03:06, Bertini, Denis Dr. via users wrote: > > > > Hi > > > > Thanks for all these informations ! > > > > > > But i have to confess that in this multi-tuning-parameter space, > > > > i got somehow lost. > > > > Furthermore it is somtimes mixing between user-space and kernel-space. > > > > I have only possibility to act on the user space. > > > > > > 1) So i have on the system max locked memory: > > > > - ulimit -l unlimited (default ) > > > > and i do not see any warnings/errors related to that when launching > MPI. > > > > > > 2) I tried differents algorithms for MPI_all_reduce op. all showing > > drop in > > > > bw for size=16384 > > > > > > 4) I disable openIB ( no RDMA, ) and used only TCP, and i noticed > > > > the same behaviour. > > > > > > 3) i realized that increasing the so-called warm up parameter in the > > > > OSU benchmark (argument -x 200 as default) the discrepancy. > > > > At the contrary putting lower threshold ( -x 10 ) can increase this BW > > > > discrepancy up to factor 300 at message size 16384 compare to > > > > message size 8192 for example. > > > > So does it means that there are some caching effects > > > > in the internode communication? > > > > > > From my experience, to tune parameters is a time-consuming and cumbersome > > > > task. > > > > > > Could it also be the problem is not really on the openMPI > > implemenation but on the > > > > system? > > > > > > Best > > > > Denis > > > > ------------------------------------------------------------------------ > > *From:* users <users-boun...@lists.open-mpi.org> on behalf of Gus > > Correa via users <users@lists.open-mpi.org> > > *Sent:* Monday, February 7, 2022 9:14:19 PM > > *To:* Open MPI Users > > *Cc:* Gus Correa > > *Subject:* Re: [OMPI users] Using OSU benchmarks for checking > > Infiniband network > > This may have changed since, but these used to be relevant points. > > Overall, the Open MPI FAQ have lots of good suggestions: > > https://www.open-mpi.org/faq/ > > some specific for performance tuning: > > https://www.open-mpi.org/faq/?category=tuning > > https://www.open-mpi.org/faq/?category=openfabrics > > > > 1) Make sure you are not using the Ethernet TCP/IP, which is widely > > available in compute nodes: > > mpirun --mca btl self,sm,openib ... > > > > https://www.open-mpi.org/faq/?category=tuning#selecting-components > > > > However, this may have changed lately: > > https://www.open-mpi.org/faq/?category=tcp#tcp-auto-disable > > 2) Maximum locked memory used by IB and their system limit. Start > > here: > > > https://www.open-mpi.org/faq/?category=openfabrics#limiting-registered-memory-usage > > 3) The eager vs. rendezvous message size threshold. I wonder if it may > > sit right where you see the latency spike. > > https://www.open-mpi.org/faq/?category=all#ib-locked-pages-user > > 4) Processor and memory locality/affinity and binding (please check > > the current options and syntax) > > https://www.open-mpi.org/faq/?category=tuning#using-paffinity-v1.4 > > > > On Mon, Feb 7, 2022 at 11:01 AM Benson Muite via users > > <users@lists.open-mpi.org> wrote: > > > > Following https://www.open-mpi.org/doc/v3.1/man1/mpirun.1.php > > > > mpirun --verbose --display-map > > > > Have you tried newer OpenMPI versions? > > > > Do you get similar behavior for the osu_reduce and osu_gather > > benchmarks? > > > > Typically internal buffer sizes as well as your hardware will affect > > performance. Can you give specifications similar to what is > > available at: > > http://mvapich.cse.ohio-state.edu/performance/collectives/ > > where the operating system, switch, node type and memory are > > indicated. > > > > If you need good performance, may want to also specify the algorithm > > used. You can find some of the parameters you can tune using: > > > > ompi_info --all > > > > A particular helpful parameter is: > > > > MCA coll tuned: parameter "coll_tuned_allreduce_algorithm" (current > > value: "ignore", data source: default, level: 5 tuner/detail, > > type: int) > > Which allreduce algorithm is used. Can be > > locked down to any of: 0 ignore, 1 basic linear, 2 nonoverlapping > > (tuned > > reduce + tuned bcast), 3 recursive doubling, 4 ring, 5 segmented ring > > Valid values: 0:"ignore", > > 1:"basic_linear", > > 2:"nonoverlapping", 3:"recursive_doubling", 4:"ring", > > 5:"segmented_ring", 6:"rabenseifner" > > MCA coll tuned: parameter > > "coll_tuned_allreduce_algorithm_segmentsize" (current value: "0", > > data > > source: default, level: 5 tuner/detail, type: int) > > > > For OpenMPI 4.0, there is a tuning program [2] that might also be > > helpful. > > > > [1] > > > https://stackoverflow.com/questions/36635061/how-to-check-which-mca-parameters-are-used-in-openmpi > > [2] https://github.com/open-mpi/ompi-collectives-tuning > > > > On 2/7/22 4:49 PM, Bertini, Denis Dr. wrote: > > > Hi > > > > > > When i repeat i always got the huge discrepancy at the > > > > > > message size of 16384. > > > > > > May be there is a way to run mpi in verbose mode in order > > > > > > to further investigate this behaviour? > > > > > > Best > > > > > > Denis > > > > > > > > > ------------------------------------------------------------------------ > > > *From:* users <users-boun...@lists.open-mpi.org> on behalf of > > Benson > > > Muite via users <users@lists.open-mpi.org> > > > *Sent:* Monday, February 7, 2022 2:27:34 PM > > > *To:* users@lists.open-mpi.org > > > *Cc:* Benson Muite > > > *Subject:* Re: [OMPI users] Using OSU benchmarks for checking > > Infiniband > > > network > > > Hi, > > > Do you get similar results when you repeat the test? Another job > > could > > > have interfered with your run. > > > Benson > > > On 2/7/22 3:56 PM, Bertini, Denis Dr. via users wrote: > > >> Hi > > >> > > >> I am using OSU microbenchmarks compiled with openMPI 3.1.6 in > > order to > > >> check/benchmark > > >> > > >> the infiniband network for our cluster. > > >> > > >> For that i use the collective all_reduce benchmark and run over > > 200 > > >> nodes, using 1 process per node. > > >> > > >> And this is the results i obtained ? > > >> > > >> > > >> > > >> ################################################################ > > >> > > >> # OSU MPI Allreduce Latency Test v5.7.1 > > >> # Size Avg Latency(us) Min Latency(us) Max > > Latency(us) Iterations > > >> 4 114.65 83.22 147.98 > > 1000 > > >> 8 133.85 106.47 164.93 > > 1000 > > >> 16 116.41 87.57 150.58 > > 1000 > > >> 32 112.17 93.25 130.23 > > 1000 > > >> 64 106.85 81.93 134.74 > > 1000 > > >> 128 117.53 87.50 152.27 > > 1000 > > >> 256 143.08 115.63 173.97 > > 1000 > > >> 512 130.34 100.20 167.56 > > 1000 > > >> 1024 155.67 111.29 188.20 > > 1000 > > >> 2048 151.82 116.03 198.19 > > 1000 > > >> 4096 159.11 122.09 199.24 > > 1000 > > >> 8192 176.74 143.54 221.98 > > 1000 > > >> 16384 48862.85 39270.21 54970.96 > > 1000 > > >> 32768 2737.37 2614.60 2802.68 > > 1000 > > >> 65536 2723.15 2585.62 2813.65 > > 1000 > > >> > > >> > > #################################################################### > > >> > > >> Could someone explain me what is happening for message = 16384 ? > > >> One can notice a huge latency (~ 300 time larger) compare to > > message > > >> size = 8192. > > >> I do not really understand what could create such an increase > > in the > > >> latency. > > >> The reason i use the OSU microbenchmarks is that we > > >> sporadically experience a drop > > >> in the bandwith for typical collective operations such as > > MPI_Reduce in > > >> our cluster > > >> which is difficult to understand. > > >> I would be grateful if somebody can share its expertise or such > > problem > > >> with me. > > >> > > >> Best, > > >> Denis > > >> > > >> > > >> > > >> --------- > > >> Denis Bertini > > >> Abteilung: CIT > > >> Ort: SB3 2.265a > > >> > > >> Tel: +49 6159 71 2240 > > >> Fax: +49 6159 71 2986 > > >> E-Mail: d.bert...@gsi.de > > >> > > >> GSI Helmholtzzentrum f?r Schwerionenforschung GmbH > > >> Planckstra?e 1, 64291 Darmstadt, Germany, www.gsi.de > > <http://www.gsi.de> > > >> > > >> Commercial Register / Handelsregister: Amtsgericht Darmstadt, > > HRB 1528 > > >> Managing Directors / Gesch?ftsf?hrung: > > >> Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, J?rg Blaurock > > >> Chairman of the GSI Supervisory Board / Vorsitzender des > > GSI-Aufsichtsrats: > > >> Ministerialdirigent Dr. Volkmar Dietz > > >> > > > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://lists.open-mpi.org/mailman/private/users/attachments/20220208/ddf3f7b3/attachment.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > > ------------------------------ > > End of users Digest, Vol 4718, Issue 3 > ************************************** > -- Work: davetur...@ksu.edu Kansas State University Beocat Application Scientist cell: (785) 770-5929 Working entirely from home Home: drdavetur...@gmail.com cell: (785) 770-5929
