Re: 25% of requests shown as error-req
Hi Dennis, On Fri, Oct 31, 2014 at 04:35:59PM +0100, Dennis Jacobfeuerborn wrote: > On 31.10.2014 10:05, Willy Tarreau wrote: > > Hi Dennis, > > > > On Fri, Oct 31, 2014 at 12:51:21AM +0100, Dennis Jacobfeuerborn wrote: > >> On 30.10.2014 19:01, Dennis Jacobfeuerborn wrote: > >> ... > >>> > >>> [30/Oct/2014:18:46:36.035] front-http front-http/ > >>> -1/-1/-1/-1/19117 400 187 - - CR-- 49/49/0/0/0 0/0 "" > >> ... > >> > >> So after a bit more googling I found the following mail thread that > >> mentions this as some sort of tcp pre-connect behaviour by browsers: > >> > >> http://comments.gmane.org/gmane.comp.web.haproxy/10426 > >> > >> Has anyone come of the idea to be able to not count a connection that > >> gets opened but never used before it times out as an error? > >> Right now the problem is that while technically these are errors > >> according to the specs they kind of aren't given the real-world behavior > >> of browsers these days. > > > > In the case above, you had no traffic over the connection so > > "option dontlognull" will silently hide this case. > > This will hide the log entries but not deal with the error count. What I > would like to do is to monitor the error count in order to detect when > errors begin to show up more frequently but with the current behavior > this isn't really possible. > What I was hoping for is something like "option donterrornull" so these > connections are not counted as errors. > I realize that this might hide legitimate "null" requests in the error > count but for this particular frontend the benefit of not counting these > connections as errors would outweigh the downsides. > > In short with such an option some legitimate errors might get hidden but > without it the error count becomes just noise effectively hiding *all* > legitimate errors. That was something I was thinking about yesterday before the 1.5.7 release, in fact something like "silent-probes" which would have the following effects : - not sending 400 bad req to sender on close - not sending 408 timeout to sender on timeout - not increasing error counters - not logging This should be backportable to 1.5 as well as I think it can make a lot of people's life easier. What do others think ? Willy
Re: 25% of requests shown as error-req
On 31.10.2014 10:05, Willy Tarreau wrote: > Hi Dennis, > > On Fri, Oct 31, 2014 at 12:51:21AM +0100, Dennis Jacobfeuerborn wrote: >> On 30.10.2014 19:01, Dennis Jacobfeuerborn wrote: >> ... >>> >>> [30/Oct/2014:18:46:36.035] front-http front-http/ >>> -1/-1/-1/-1/19117 400 187 - - CR-- 49/49/0/0/0 0/0 "" >> ... >> >> So after a bit more googling I found the following mail thread that >> mentions this as some sort of tcp pre-connect behaviour by browsers: >> >> http://comments.gmane.org/gmane.comp.web.haproxy/10426 >> >> Has anyone come of the idea to be able to not count a connection that >> gets opened but never used before it times out as an error? >> Right now the problem is that while technically these are errors >> according to the specs they kind of aren't given the real-world behavior >> of browsers these days. > > In the case above, you had no traffic over the connection so > "option dontlognull" will silently hide this case. This will hide the log entries but not deal with the error count. What I would like to do is to monitor the error count in order to detect when errors begin to show up more frequently but with the current behavior this isn't really possible. What I was hoping for is something like "option donterrornull" so these connections are not counted as errors. I realize that this might hide legitimate "null" requests in the error count but for this particular frontend the benefit of not counting these connections as errors would outweigh the downsides. In short with such an option some legitimate errors might get hidden but without it the error count becomes just noise effectively hiding *all* legitimate errors. Regards, Dennis
Re: 25% of requests shown as error-req
Hi Dennis, On Fri, Oct 31, 2014 at 12:51:21AM +0100, Dennis Jacobfeuerborn wrote: > On 30.10.2014 19:01, Dennis Jacobfeuerborn wrote: > ... > > > > [30/Oct/2014:18:46:36.035] front-http front-http/ > > -1/-1/-1/-1/19117 400 187 - - CR-- 49/49/0/0/0 0/0 "" > ... > > So after a bit more googling I found the following mail thread that > mentions this as some sort of tcp pre-connect behaviour by browsers: > > http://comments.gmane.org/gmane.comp.web.haproxy/10426 > > Has anyone come of the idea to be able to not count a connection that > gets opened but never used before it times out as an error? > Right now the problem is that while technically these are errors > according to the specs they kind of aren't given the real-world behavior > of browsers these days. In the case above, you had no traffic over the connection so "option dontlognull" will silently hide this case. Regards, Willy
Re: 25% of requests shown as error-req
On 30.10.2014 19:01, Dennis Jacobfeuerborn wrote: ... > > [30/Oct/2014:18:46:36.035] front-http front-http/ > -1/-1/-1/-1/19117 400 187 - - CR-- 49/49/0/0/0 0/0 "" ... So after a bit more googling I found the following mail thread that mentions this as some sort of tcp pre-connect behaviour by browsers: http://comments.gmane.org/gmane.comp.web.haproxy/10426 Has anyone come of the idea to be able to not count a connection that gets opened but never used before it times out as an error? Right now the problem is that while technically these are errors according to the specs they kind of aren't given the real-world behavior of browsers these days. Right now this makes detecting real errors much harder and it would be great if haproxy would allow an option in a frontend to not count this specific behavior as an error. Regards, Dennis
Re: 25% of requests shown as error-req
On 30.10.2014 17:12, Dennis Jacobfeuerborn wrote: > Hi, > I just put haproxy into use on a site and while things seem to work I > noticed that the frontend shows 20 mio. sessions handled total but under > errors/req it shows a number of 5 mio. These 5 mio. seem to correspond > to the number of 4xx reponses shown when i hover over the sessions/total > entry on the stats page. > > When I send "show errors" to the stats socket the output rarely changes > (maybe once every 30 seconds) even though the session rate is 700. Given > the high number of request errors reported and the rate I would expect > the output here to rapidly change. > > I also went ahead and analyzed a tcpdump in wireshark but there I only > see 22 404 responses and one 304 and all others are the expected 200 and > 302. > > Does anyone have an idea what is going on here? Why is haproxy reporting > so many 4xx responses yet I cannot see these on the wire? > What other ways are there to debug this issue? > > Regards, >Dennis > After getting a bit more intimate with the logging settings I was able to isolate the type of request I think. It looks like this: [30/Oct/2014:18:46:36.035] front-http front-http/ -1/-1/-1/-1/19117 400 187 - - CR-- 49/49/0/0/0 0/0 "" The documentation says that "C" means the client unexpectedly closed by the client and "R" means that some resource on the haproxy system has been exhausted. I'm not sure what that resource could be though. The system has not firewalling or connection tracking, ss/netstat only shows about 6500 connections and I also increased net.ipv4.ip_local_port_range and set net.ipv4.tcp_tw_reuse to combat any exhaustion there. The system has 2G RAM and of that 1G are free and 300M used for the page cache. Any ideas what the source of this problem could be? Regards, Dennis
