We recently ran into what seemed to be a bizarre performance problem with our application. On Solaris 8, a given REST call was taking 10 minutes to complete where it would normally complete in 1 second.
At first glance, we had no idea what the problem was. We normally test our application on Solaris 10, Red Hat Linux4/5 and Windows 2003 and the application ran fine on those platforms. I thought it might be a problem with the Restlet "internal" client connector, so as a test, I dropped in the HttpClient connector and it solved our problem! While using the HttpClient connector fixed the symptom, we still had no idea what was causing the problem. I did a little digging and found that by default, HttpClient disables Nagle's algorithm (by setting the TCP_NODELAY flag on the Socket). The "internal" Restlet client connector left the socket option to be its default value (which was false, meaning Nagle's algorithm was enabled). I recompiled the "internal" Restlet client connector with an option (and later submitted as a patch RESTLET-600) to enable the TCP_NODELAY flag. It worked! Now, my knowledge of low level TCP is rudimentary, so I had to read up a bit on Nagle's algorithm and why it could be causing this problem. After finding the following article, it seems that if you get an unlucky combination of your tcp segment size and your data size, you could run into big problems: http://www.stuartcheshire.org/papers/NagleDelayedAck/ I encourage everyone to read that article as it is extremely informative and provides a great example as to how the combination of Nagle's algorithm and TCP delayed ack caused a delay many orders of magnitude higher than it should have been. From the article: > Consider sending 100,000-bytes. You send a stream of 69 full-sized packets. > Nagle holds onto the last 88 bytes. Then: > > - The receiver ACKs every second packet, up to packet 68. > - One more data packet arrives, packet 69. > - Delayed ACK means that the receiver won't ACK this packet until it > gets > (a) some response data from the local process, or > (b) another packet from the sender. > - The local process won't generate any response data (a) because it > hasn't got the full 100,000 bytes yet. > - The sender won't send the last packet (b) because Nagle won't let it > until it gets an ACK from the receiver. > > Now we have a brief deadlock, with performance-killing results: > > - Nagle won't send the last bit of data until it gets an ACK > - Delayed ACK won't send that ACK until it gets some response data > - Server process won't generate any response until it gets all the data > - but, Nagle won't send the last bit of data... and so on. > > So why did the problem affect Solaris 8 and not the other operating systems? The TCP segment size was slightly smaller and caused a number of extra packets to be waiting in the wings for data that wouldn't come. Sorry for the long post, I hope the information in here helps someone else down the line. We certainly had a hell of a time (and some measure of fun afterwards) solving this issue. Sincerely, Kevin Conaway
