Hello Paul, On 07/21/2014 06:41 PM, Paul Henry via smartos-discuss wrote: > Hello! > > We are on joyent and are having an issue with large machines that receive > thousands of short-lived connections. We're using a base64 smartos image, > joyent version 13.1.0. We're seeing a tcp retransmit rate of about 3 - 6 > per second. This drives up our request queueing time by a > minimum of 1 second (which is the default tcp initial retransmit timeout) > for the requests that trigger a tcp retransmit.
What's the version of the underlying platform that this is occurring on? You can get that output from uname -v. The platform version will influence this much more than the image version. > We've been able to reproduce this issue on a smaller app server that's > receiving no traffic by sending a high enough number of requests so that we > get over 8,000 total connections (TIME_WAIT and ESTABLISHED). Then about 1 > out of every 15 to 20 requests has a delay of 1 second. > > Any suggested tunings? It seems as though the OS occasionally pauses > sending ACKs under a high connection rate, possibly to do some cleanup of > TIME_WAITs that need to be closed. These then cause the retransmits. So, are all of these connections generally coming from over the local network or from the outside world? Are you employing ipf on this server? If so, have you looked at your ipf state tables? We've had issues like that before that have happened because the state tables are too small. There are some kstats and ipf statistics that can be used to see if this is happening. Another issue that we've seen is that if you have a lot of stuff in time wait, it almost sounds like the client isn't using a sequence number that's large enough so it gets reset and then tries to make a connection again perhaps. Getting some snoop/tcpdump output may help with something like that, though that'd be a lot to sift through. If this is happening, changing the strong iss tunable to 2 may help, though it'll depend on a lot of the specifics. You can do that with `ndd -set /dev/tcp tcp_strong_iss 2`. Robert ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
