Willy: On the load balancer we have iptables rate limits on ssh connections only. I have removed iptables and netfilter entirely on the server side to ensure that resets are not coming from iptables. I can watch the resets on the app server and see them increasing steadily and incrementing faster when the 503s happen. Is it possible that keepalives have something to do with this? I have searched the apache error logs and do not see anything there. 90% of the app will not make use of keepalive with the exception of digest auth.
-Jerry Jerry Champlin Absolute Performance Inc. Phone: 303-565-4401 -- Enabling businesses to deliver critical applications at lower cost and higher value to their customers. On Mon, Aug 2, 2010 at 1:34 AM, Willy Tarreau <[email protected]> wrote: > On Mon, Aug 02, 2010 at 01:11:48AM -0600, Jerry Champlin wrote: > > Willy: > > > > iptables is enabled on the load balancer as it also serves as part of a > > firewall pair for the environment. I just noticed that conntrack is also > > enabled on the backend servers. Could these tables be filling up and > > causing the resets? There are none of the usual errors from conntrack in > > dmesg so probably not. Apache is actively processing other requests > during > > this period and we have not seen it crash. > > OK. Apache is special in that it works with multiple processes and is > able to restart them, making it extremely resilient. But sometimes this > can also hide bugs that cause a process to crash and another one to > immediately start. > > > We have seen some odd things > > with Rails and Merb apps which are attached to apache through passenger > but > > those were all related to mid-level xml-library problems. What is the > > difference between the [R] and [R.] lines in this tcpdump output? > > 'R' alone is a sole reset flag. There is no ACK so it's a response to a > packet which had an ACK. It is very common on late packets after a session > clsoes. > > 'R.' is a 'reset + ACK'. It only happens in response to a SYN, indicating > a closed port. The ACK field then matches the seq field of the SYN + 1. > For instance, you have some 'R.' from port 113, which is normal since it's > usually closed these days. > > However, if 58322 is your application's port, then it means that the server > considers the port as closed ! So maybe we're in fact seeing a packet > forged > by iptables as a REJECT action in response to the SYN, but I have no idea > why. Just in case, check that you don't have rules to prevent too high > connection rates from a same IP or something like this. > > Regards, > Willy > >
