Willy, I tried fetching 10000 packets into a file with tcpdump, then seeing which IPs in the haproxy log were doing the BADREQ errors at the time I was running tcpdump. I used -A -vvn to see the packets content. This is a sample of a packet I see a lot, and I saw the IP+port combination in haproxy error log correlating with the same combination in the tcpdump log, this is the packet: E..([email protected]..:..%.2t.V.m.P.|......P...(d..
There are many more pretty similar. Any idea what is it? Thanks! Bar. On Sat, May 12, 2012 at 10:39 AM, Willy Tarreau <[email protected]> wrote: > Hi Bar, > > On Thu, May 10, 2012 at 07:02:58PM +0300, Bar Ziony wrote: > > Hey, > > > > We're running haproxy 1.4.20 as our LB, nginx is listening on the same > > machine on port 443 and terminating SSL, proxying the unencrypted > requests > > to haproxy on localhost:80. > > > > I see many of these errors on the haproxy log: > > May 10 15:54:06 lb-01 haproxy[6563]: > > 1.1.1.1:50929[10/May/2012:15:54:01.113] public public/<NOSRV> > > -1/-1/-1/-1/5519 400 187 - > > - CR-- 3045/3045/0/0/0 0/0 "<BADREQ>" > > > > * I changed the source IP for the sake of this example. > > > > We get around 5-15 of these per second, and I checked some of the IPs and > > it seems at least some of them are IPs that users registered with (maybe > > it's a very big proxy or something so it's not actually those users). > > As you can see, the client took 5.5 seconds to send an incomplete request > then closed the connection ('CR'). It is possible that some users have > developped monitoring scripts which are targetting your site. I sometimes > get a number of these on the haproxy web site too. While the ones sending > valid requests are just a bandwidth annoyance, the ones sending invalid > requests are much more annoying. > > If the requests are completely invalid, you can find a capture of them on > the stats socket using "show errors" : > > echo "show errors" | socat stdio /var/run/haproxy.stat > (or whereever you put it, check "stats socket" in your global section) > > > We're running on a pretty fast Linode VPS (1GB RAM), it handled 5000 > > requests per second on testing (which is low, I know, but it is still a > > VPS). We are doing much less than 5000 req/sec ... > > The CPU usage is 10-20% for haproxy alone (10% more for nginx), and > 10-20% > > RAM usage for haproxy (~150MB RES, ~180MB VIRT). Does that make sense? > > Yes, nothing sounds strange here. > > > What are these requests? Is it possible these are regular users trying to > > somehow get to our web app and not succeeding ? > > That's really unlikely, because such invalid requests happen at a layer > which > is only controlled by the browser. A normal browser cannot emit invalid > requests. Only bots do. It's possible that some of your users are running > crappy site sucking plugins, or home-made search engines which emit invalid > requests. Some of the invalid requests I'm used to observe are those where > the user forget to send the last CR/LF, so the request is incomplete. > > Well, if you have 15 of these a second, just run tcpdump for a few seconds > to capture some of them and you'll know what these are. > > Regards, > Willy > >
