On Thu, 2008-05-29 at 11:19 +0200, Marcel Heinz wrote: > Hi, > > Dotan Barak wrote: > > Marcel Heinz wrote: > >> [low multicast throughput of ~250MB/s with own benchmark tool] > > > > 1) I know that ib_send_bw supports multicast as well, can you please > > check that you can reproduce your problem > > on this benchmark too? > > Well, the last time I've checked this, ib_send_bw didn't support > multicast, but this was some months ago. That multicast support seems > a bit odd, since it doesn't create/join the multicast groups and there > is still a 1:1 TCP connection used to establish the IB connection, so > one cannot benchmark "real" multicast scenarios with more than one > receiver. > > However, here are the results (I just used ipoib to let it create some > multicast groups for me): > > | [EMAIL PROTECTED]:~$ ib_send_bw -c UD -g mhtest1 > | ------------------------------------------------------------------ > | Send BW Multicast Test > | Connection type : UD > | Max msg size in UD is 2048 changing to 2048 > | Inline data is used up to 400 bytes message > | local address: LID 0x01, QPN 0x4a0405, PSN 0x8667a7 > | remote address: LID 0x03, QPN 0x4a0405, PSN 0x5d41b6 > | Mtu : 2048 > | ------------------------------------------------------------------ > | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] > | 2048 1000 301.12 247.05 > | ------------------------------------------------------------------ > > This is the same result as my own benchmark showed in that scenario. > > > 2) You should expect that multicast messages will be slower than > > unicast because the HCA/switch treat them in different way > > (message duplication need to be done if needed). > > Yes, but 250MB/s vs. 1100MB/s (UD unicast throughput) seems to be a bit > too much of overhead, don't you think?
Agreed. > Especially if I take into account > that with my own benchmark, I can get ~950MB/s when I start another > receiver on the same host as the sender. Note that both of the > receivers, the local and the remote one, are seeing all packets at that > rate, so the HCAs and the switch must be able to handle multicast > packets with this throughput. Perhaps this is a static rate issue. What SM is being used ? -- Hal > The other strange thing is that multicast traffics slows down other > traffic way more than the bandwith it consumes. Moreover, it seems like > it limits any other connections to the same throughput than that of the > multicast traffic, which looks suspicious to me. > > The same behavior can be reproduced with ib_send_bw, by starting an > unicast and multicast run in parallel: > > | [EMAIL PROTECTED]:~$ ib_send_bw -c UD mhtest1 & ib_send_bw -c UD -g mhtest1\ > | -p 18516 > | ./ib_send_bw -c UD -g mhtest1 -p 18516 > | [1] 4927 > | ------------------------------------------------------------------ > | Send BW Test > | Connection type : UD > | Max msg size in UD is 2048 changing to 2048 > | ------------------------------------------------------------------ > | Send BW Multicast Test > | Connection type : UD > | Max msg size in UD is 2048 changing to 2048 > | Inline data is used up to 400 bytes message > | Inline data is used up to 400 bytes message > | local address: LID 0x01, QPN 0x530405, PSN 0xe98523 > | local address: LID 0x01, QPN 0x530406, PSN 0x3b338e > | remote address: LID 0x03, QPN 0x540405, PSN 0x5c53e2 > | Mtu : 2048 > | remote address: LID 0x03, QPN 0x540406, PSN 0xff883f > | Mtu : 2048 > | ------------------------------------------------------------------ > | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] > | ------------------------------------------------------------------ > | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] > | 2048 1000 692.41 270.26 > | 2048 1000 246.00 244.68 > | ------------------------------------------------------------------ > | ------------------------------------------------------------------ > > Doing 2 unicast UD runs in parallel, I'm getting ~650MB/s average > bandwith for each, which sounds reasonable. > > Also, when using bidircetional mode, I'm getting ~1900MB/s (amlost > doubled) throughput for unicast, but still ~250MBs for multicast. > > Regards, > Marcel > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
