On 5/13/2010 1:34 PM, Steve Simmons wrote: > > On May 12, 2010, at 10:48 AM, Jeffrey Altman wrote: > >> On 5/11/2010 1:01 PM, Hartmut Reuter wrote: >>> >>> I have now ported AFS/RXOSD to 1.5.74. >>> I am testing the client against servers running 1.4.12-osd. >>> >>> My client configuration is memcache with chunksize 18 (256K). >>> Writing to the fileserver and also to the rxosd is as fast as with the >>> 1.4.12 >>> client (I am testing on my laptop with the whole test-cell in the laptop, >>> so 30 >>> MB/s is already an acceptable value). >>> >>> But reading large files is terribly slow (~ 1 MB/s). With the 1.4.12 client >>> read >>> and write have about the same speed. >>> >>> with rxdebug -peer -long I can also see that some times in the rtt the >>> fileserver or rxosd see is growing up to several seconds and then again >>> going >>> down to a ms. So there seems to be something strange in the rx-layer. >>> It also shows a lot of resends (about 1 %). >>> >>> Any idea? >>> >>> Hartmut >> >> The first thing that comes to mind is that we found the rx library was >> dropping packets on the floor under high load. This was fixed in the >> 1.5.74 release but has not been fixed in the 1.4.12 release. > > Any feel for the net effect on throughput? Is the improvement significant > enough to consider backporting to 1.4.13?
Its a serious bug. It causes the Windows client stress test to have an average SMB request RTT of 51 seconds instead of under 5 seconds due to repeated RPC failures. The problem is that the Rx when under load drops packets on the floor by clearing the transmit queue before the packets have been sent and acknowledged. Since the queue is empty, the sender things there is no more work to do and will never retry. The receiver can only timeout. Regardless of which side of the connection the packets are dropped the RPC issuer must perform a retry of the request. These fixes do need to be back ported to 1.4. It has not been done yet. Before a 1.4.13 it will be done. >> The 1.5.74 client should be able to put more load on the file server >> as it does a much better job of efficiently reading from the page >> cache. > > I don't understand the connection here. Hartmut seems to be saying that a > 1.4.12osd client reading from a 1.4.12osd server runs 30x faster than a > 1.5.74osd client reading from a 1.4.12osd server. Improvements in 1.5.74 > client-side reading of the page cache shouldn't have any affect - presumably > one is only reading the file because it isn't in page or disk cache. 30-fold > is a pretty big number, too. The connection is this. If the client is able to issue more RPCs to the server in a shorter period of time, that puts stress on Rx and the likelihood of packets being discarded increases. As soon as packets are discarded, the transfer rate hits the floor. There are additional bottlenecks in Rx that have recently been removed as well. 1.5.74 no longer contains a bottleneck that prevented calls from ending while a request to allocate a new call was in flight. Allocations of new calls can block on outstanding packet processing so this reduced the ability of Rx to process calls in parallel. That in turn prevented multiple threads in the calling application to process multiple calls in parallel effectively. Another problem that has been fixed on master. The rx_rpc_stats mutex is supposed to be a fined grained lock. Unfortunately, the way it was used the mutex would halt all rx processing whenever a ReapConnections operation was performed. At some point of course, backporting the changes gets silly. We need to cut the 1.6 branch and begin to get off the 1.4 series. Its been nearly five years since 1.4.0 was released. Jeffrey Altman
smime.p7s
Description: S/MIME Cryptographic Signature
