Hello, I've been observing strange TCP bandwidth performance behaviors with varying IPoIB-CM MTU sizes. For a fixed ttcp/TCP transfer size (-n 2048 -l 8192), one would expect bandwidth to increase as MTU size increases.
Using a heavily debug/checked IPoIB-CM driver on twin Windows 2008 R2 x64 systems with Infinihost HCAs (MT25208), the observed behavior is: When the MTU is > 8KB ... 64KB, TCP performance (ttcp -n 2048 -l 8192) is around 0.04 MB/sec? When the MTU is 2048 ... 8KB, TCP performance scales from 85 to 113 MB/sec. MTU (bytes) sender's BW MB/sec ------------------------------------- 2048 2K 85.11 4096 4K 102.56 8192 8K 113.48 9216 9K 000.04 12288 12K 000.04 16384 16K 000.04 65536 64K 000.04 65536 64K (UDP) 128.00 My local definition of Round-Trip Time (RTT) is defined as the time measured from a TCP segment/packet send (ib_post_send) to the time when the TCP ACK arrives as viewed from Wireshark. When the IPOIB-CM MTU <= 8KB, such that the NDIS send SGL (scatter gather list) size is <= 8KB, RTT is around 90 us (microseconds). When the IPoIB-CM MTU is > 8KB, RTT climbs to around 203 ms (milliseconds); ouch! The same ttcp transfer size using UDP instead of TCP does not suffer the performance drop @ 8KB. Are you aware of any TCP protocol settings/properties which could/should be adjusted to bring TCP bandwidth performance up to or > 8KB MTU BW levels? thanks, Stan. _______________________________________________ ofw mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
