Michael, it is not that the connections are being dropped, it is that the connections are not being dropped.
These server side sockets are ESTABLISHED, even though the client connection on the other side of the network device is long gone. This may well be an issue with the network device (it is valiantly trying to keep the connection alive it seems). That said KEEPALIVE on the server side would not be a bad idea. At least then the OS on the server would eventually (probably after 2 hours of inactivity) attempt to ping the client. At that point hopefully something interesting would happen perhaps causing an error and destroying the server side socket (note KEEPALIVE is also good for preventing idle connections from being dropped by other network devices along the way) rpc_keepalive on the server sets keep alive on the server side sockets for thrift, and is true by default There doesn’t seem to be a setting for the native protocol Note this isn’t a huge issue for us, they can be cleaned up by a rolling restart, and this particular case is not production, but related to development/testing against alpha by people working remotely over VPN - and it may well be the VPNs fault in this case… that said and maybe this is a dev list question, it seems like the option to set keepalive should exist. On Apr 9, 2014, at 12:25 PM, Michael Shuler <mich...@pbandjelly.org> wrote: > On 04/09/2014 11:39 AM, graham sanderson wrote: >> Thanks, but I would think that just sets keep alive from the client end; >> I’m talking about the server end… this is one of those issues where >> there is something (e.g. switch, firewall, VPN in between the client and >> the server) and we get left with orphaned established connections to the >> server when the client is gone. > > There would be no server setting for any service, not just c*, that would > correct mis-configured connection-assassinating network gear between the > client and server. Fix the gear to allow persistent connections. > > Digging through the various timeouts in c*.yaml didn't lead me to a simple > answer for something tunable, but I think this may be more basic networking > related. I believe it's up to the client to keep the connection open as Duy > indicated. I don't think c* will arbitrarily sever connections - something > that disconnects the client may happen. In that case, the TCP connection on > the server should drop to TIME_WAIT. Is this what you are seeing in `netstat > -a` on the server - a bunch of TIME_WAIT connections hanging around? Those > should eventually be recycled, but that's tunable in the network stack, if > they are being generated at a high rate. > > -- > Michael
smime.p7s
Description: S/MIME cryptographic signature
