Tzachi Dar wrote: > In short you need to disable Nagle's algorithm when running this test > (or use bigger send buffers), or run with ntttcp.
Do you have a windows version of nttcp.exe you could share? Or source + build env? > > When Nagle is running it means that there is no more than one unacked > packet in the air, or that you have a complete packet to send. > > On the other hand there is a tcp property known as delayed acks. > Delayed acks means that if one receives a packet it does not send an > ack until either the ack contains data or 200 ms have elapsed. > > In other words, you get into buffer combination that don't allow you > to send more data until there is ack, but ack only comes ones in > 200ms. > > See also: > http://www.stuartcheshire.org/papers/NagleDelayedAck/ > http://en.wikipedia.org/wiki/Nagle's_algorithm > > > Thanks > Tzachi > Thanks for the excellent education. ttcp -D (sockopt TCP_NODELAY) cures the symptoms....not so sure about the problem. Jumbo packets tends to embrace the old saying "bigger is not always better". stan. > > > > > >> -----Original Message----- >> From: Smith, Stan [mailto:[email protected]] >> Sent: Tuesday, November 23, 2010 12:42 AM >> To: Alex Naslednikov; Tzachi Dar >> Cc: [email protected] >> Subject: IPoIB-CM MTU and bandwidth scaling issue - TCP only, UDP OK. >> >> Hello, >> I've been observing strange TCP bandwidth performance behaviors >> with varying IPoIB-CM MTU sizes. For a fixed ttcp/TCP transfer size >> (-n 2048 -l 8192), one would expect bandwidth to increase as MTU >> size increases. >> >> Using a heavily debug/checked IPoIB-CM driver on twin Windows 2008 R2 >> x64 systems with Infinihost HCAs (MT25208), the observed behavior is: >> >> When the MTU is > 8KB ... 64KB, TCP performance (ttcp -n 2048 -l >> 8192) is around 0.04 MB/sec? >> >> When the MTU is 2048 ... 8KB, TCP performance scales from 85 to 113 >> MB/sec. >> >> MTU (bytes) sender's BW MB/sec >> ------------------------------------- >> 2048 2K 85.11 >> 4096 4K 102.56 >> 8192 8K 113.48 >> 9216 9K 000.04 >> 12288 12K 000.04 >> 16384 16K 000.04 >> 65536 64K 000.04 >> >> 65536 64K (UDP) 128.00 >> >> My local definition of Round-Trip Time (RTT) is defined as the time >> measured from a TCP segment/packet send (ib_post_send) to the time >> when the TCP ACK arrives as viewed from Wireshark. >> >> When the IPOIB-CM MTU <= 8KB, such that the NDIS send SGL (scatter >> gather list) size is <= 8KB, RTT is around 90 us (microseconds). >> >> When the IPoIB-CM MTU is > 8KB, RTT climbs to around 203 ms >> (milliseconds); ouch! >> >> The same ttcp transfer size using UDP instead of TCP does not suffer >> the performance drop @ 8KB. >> >> Are you aware of any TCP protocol settings/properties which >> could/should be adjusted to bring TCP bandwidth performance up to or >> > 8KB MTU BW levels? >> >> thanks, >> >> Stan. _______________________________________________ ofw mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
