There are clearly a lot of junk bytes in those URI which are not allowed by
the HTTP specs. If you really want to be passing unencoded binary control
characters, spaces, and nulls to your backends in HTTP request and header
lines, then HTTP mode is probably not going to work for you.
TCP mode will allow them to get through but if your backends actually
expect the requests to be valid HTTP, you will likely be opening up a huge
can of worms and exposing your apps to a host of protocol level attacks.
Also, your connection limits seem pretty ambitious if there really are 2
php servers in that backend and not 2000.
-Bryan
On Mon, Nov 24, 2014 at 10:22 PM, Alexey Zilber <alexeyzil...@gmail.com>
wrote:
> Hi Willy and Lukas,
>
> Here's snippets of the new config:
>
>
> -----------------------------
>
> global
>
> maxconn 645000
>
> maxpipes 645000
>
> ulimit-n 645120
>
> user haproxy
>
> group haproxy
>
> tune.bufsize 49152
>
> spread-checks 10
>
> daemon
>
> quiet
>
> stats socket /var/run/haproxy.sock level admin
>
> pidfile /var/run/haproxy.pid
>
>
> defaults
>
> log global
>
> mode http
>
> option accept-invalid-http-request
>
> option accept-invalid-http-response
>
> option httplog
>
> option dontlognull
>
> option dontlog-normal
>
> option log-separate-errors
>
> option http-server-close
>
> option tcp-smart-connect
>
> option tcp-smart-accept
>
> option forwardfor except 127.0.0.1
>
> option dontlog-normal
>
> retries 3
>
> option redispatch
>
> maxconn 2000000
>
> contimeout 5000
>
> clitimeout 60000
>
> srvtimeout 60000
>
> listen www 0.0.0.0:80
>
> mode http
>
> capture response header Via len 20
>
> capture response header Content-Length len 10
>
> capture response header Cache-Control len 8
>
> capture response header Location len 40
>
> balance roundrobin
>
> # Haproxy status page
>
> stats uri /haproxy-status
>
> stats auth fb:phoo
>
> # when cookie persistence is required
>
> cookie SERVERID insert indirect nocache
>
> # When internal servers support a status page
>
> option httpchk GET /xyzzyx.php
>
> bind 0.0.0.0:443 ssl crt /etc/lighttpd/ssl_certs/xxxx.co.pem
>
> http-request add-header X-FORWARDED-PROTO https if { ssl_fc }
>
> server app1 10.1.1.6:85 check inter 40000 rise 2 fall 3 maxconn
> 16384
>
> server app2 10.1.1.7:85 check inter 40000 rise 2 fall 3 maxconn
> 16384
>
> -----------------------------
>
>
> The old config did NOT have the following items, and had about 500x more
> errors:
> -----------------------------
> tune.bufsize 49152
>
> option accept-invalid-http-request
>
> option accept-invalid-http-response
> -----------------------------
>
> Here's what the 'show errors' shows on a sampling of the server. It looks
> like 90% of the errors are the second error (25/Nov/2014:00:06:30.753):
>
>
>
>
> Total events captured on [24/Nov/2014:23:31:52.468] : 151
>
>
>
> [22/Nov/2014:21:55:56.597] frontend www (#2): invalid request
>
> backend www (#2), server <NONE> (#-1), event #150
>
> src 166.137.247.239:8949, session #3883610, session flags 0x00000080
>
> HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
>
> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>
> buffer flags 0x00808002, out 0 bytes, total 1183 bytes
>
> pending 1183 bytes, wrapping at 49152, error at position 227:
>
>
>
> 00000
> sited%22%3A1416713764%2C%22times_visited%22%3A6%2C%22device%22%3A%22We
>
> 00070+
> b%22%2C%22lastsource%22%3A%22bottomxpromo%22%2C%22language%22%3A%22en%
>
> 00140+
> 22%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22di
>
> 00210+ d_watch%22%3A1%7D; yXXXXX=5167136038811769837; _vhist=%7B%22visito
>
> 00280+
> r_id%22%3A%225024165909427731336%22%2C%22seen_articles%22%3A%22%7B%5C%
>
> 00350+
> 22950%5C%22%3A1402416590%2C%5C%22685%5C%22%3A1402416675%2C%5C%22799%5C
>
> 00420+
> %22%3A1402416789%2C%5C%22954%5C%22%3A1402416997%2C%5C%22939%5C%22%3A14
>
> 00490+
> 02417098%2C%5C%222334%5C%22%3A1407162586%2C%5C%222055%5C%22%3A14071626
>
> 00560+
> 91%2C%5C%223888%5C%22%3A1409938121%2C%5C%223020%5C%22%3A1409938211%2C%
>
> 00630+
> 5C%223773%5C%22%3A1409938340%2C%5C%222163%5C%22%3A1409938389%2C%5C%222
>
> 00700+
> 569%5C%22%3A1409938872%2C%5C%222426%5C%22%3A1409938959%2C%5C%2213984%5
>
> 00770+
> C%22%3A1411916274%2C%5C%221675%5C%22%3A1411916466%2C%5C%2214950%5C%22%
>
> 00840+
> 3A1412432461%2C%5C%2219759%5C%22%3A1416714580%7D%22%2C%22num_articles%
>
> 00910+ 22%3A17%7D; lastlargeleaderboard=1416713603;
> lastlike-264755123648776=
>
> 00980+ 1416713764; lastlike-703681396382652=1416713919;
> lastlike-939184096095
>
> 01050+ 676=1416714070; lastlike-293377080869393=1416714282;
> lastlike-14790568
>
> 01120+ 05705686=1416714412; popped_19731=1\r\n
>
> 01157 Connection: keep-alive\r\n
>
> 01181 \r\n
>
>
> -----------
>
>
> Total events captured on [25/Nov/2014:00:06:30.753] : 480
>
>
>
> [25/Nov/2014:00:06:26.417] frontend www (#2): invalid request
>
> backend www (#2), server <NONE> (#-1), event #479
>
> src 50.92.157.165:53649, session #14629125, session flags 0x00000080
>
> HTTP msg state 26, msg flags 0x00000000, tx flags 0x84000000
>
> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>
> buffer flags 0x00808002, out 0 bytes, total 45 bytes
>
> pending 45 bytes, wrapping at 49152, error at position 0:
>
>
>
> 00000
> \x17\x03\x03\x00(t\xF42\x9D\x96\x19[(\xF6\xFEK\xB7\x9Br\x00C\xDBn<\x98
>
> 00025+ \x12"\xC2\xBC\x91\x91\x0E5\xEA,\xCE0t\xF3\x90\x16\x1Dd\x99\xC4
>
>
> ------------------
>
> Total events captured on [25/Nov/2014:00:08:45.356] : 128
>
>
>
> [24/Nov/2014:22:38:34.626] frontend www (#2): invalid request
>
> backend www (#2), server <NONE> (#-1), event #127
>
> src 157.55.39.106:29900, session #4147419, session flags 0x00000080
>
> HTTP msg state 27, msg flags 0x00000000, tx flags 0x00000000
>
> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>
> buffer flags 0x00808002, out 0 bytes, total 304 bytes
>
> pending 304 bytes, wrapping at 49152, error at position 15:
>
>
>
> 00000 GET
> /?id=551&sr\xC3\x83\xC2\xAF\xC3\x82\xC2\xBF\xC3\x82\xC2\xBD=share_
>
> 00034+ fb_new_551 HTTP/1.1\r\n
>
> 00055 Cache-Control: no-cache\r\n
>
> 00080 Connection: Keep-Alive\r\n
>
> 00104 Pragma: no-cache\r\n
>
> 00122 Accept: */*\r\n
>
> 00135 Accept-Encoding: gzip, deflate\r\n
>
> 00167 From: bingbot(at)microsoft.com\r\n
>
> 00199 Host: xxxxxx.co\r\n
>
> 00217 User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +
> http://www.bing.com
>
> 00287+ /bingbot.htm)\r\n
>
> 00302 \r\n
>
> ------------------------
>
> Total events captured on [25/Nov/2014:00:11:16.030] : 557
>
>
>
> [25/Nov/2014:00:08:19.250] frontend www (#2): invalid request
>
> backend www (#2), server <NONE> (#-1), event #556
>
> src 199.59.148.211:40148, session #14688643, session flags 0x00000080
>
> HTTP msg state 26, msg flags 0x00000000, tx flags 0x84000000
>
> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>
> buffer flags 0x00808002, out 0 bytes, total 140 bytes
>
> pending 140 bytes, wrapping at 49152, error at position 5:
>
>
>
> 00000 GET /\x00/?id=16278&src=shorturl_16278 HTTP/1.1\r\n
>
> 00046 Host: xxxxxx.com\r\n
>
> 00065 User-Agent: Twitterbot/1.0\r\n
>
> 00093 Accept-Encoding: gzip, deflate\r\n
>
> 00125 Accept: */*\r\n
>
> 00138 \r\n
>
>
> ---------------------
>
> Total events captured on [25/Nov/2014:00:12:57.050] : 577
>
>
>
> [25/Nov/2014:00:09:20.036] frontend www (#2): invalid request
>
> backend www (#2), server <NONE> (#-1), event #576
>
> src 173.209.211.205:51013, session #15108367, session flags 0x00000080
>
> HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
>
> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>
> buffer flags 0x00808002, out 0 bytes, total 60 bytes
>
> pending 60 bytes, wrapping at 49152, error at position 6:
>
>
>
> 00000 action=video_ready&value=1&time_on_page=140.87&article=18539
>
> ----------------
>
> Total events captured on [25/Nov/2014:00:14:54.259] : 526
>
>
>
> [25/Nov/2014:00:08:22.991] frontend www (#2): invalid request
>
> backend www (#2), server <NONE> (#-1), event #525
>
> src 67.231.32.73:59810, session #15100093, session flags 0x00000080
>
> HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
>
> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>
> buffer flags 0x00808002, out 0 bytes, total 1460 bytes
>
> pending 1460 bytes, wrapping at 49152, error at position 6:
>
>
>
> 00000 Cookie:
> _vis=%7B%22visitor_id%22%3A%225108882918757752765%22%2C%22sour
>
> 00070+
> ce%22%3A%22fbfan_2081%22%2C%22joined%22%3A1410888291%2C%22last_visited
>
> 00140+
> %22%3A1412459722%2C%22times_visited%22%3A23%2C%22language%22%3A%22en%2
>
> 00210+
> 2%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22dev
>
> 00280+
> ice%22%3A%22Web%22%2C%22lastsource%22%3A%22fbfan_15040%22%2C%22did_wat
>
> 00350+ ch%22%3A1%7D; yXXXXX=5108882918757752765; _vhist=%7B%22visitor_id%
>
> 00420+
> 22%3A%225108882918757752765%22%2C%22seen_articles%22%3A%22%7B%5C%22208
>
> 00490+
> 1%5C%22%3A1410888291%2C%5C%2213102%5C%22%3A1410888488%2C%5C%2213225%5C
>
> 00560+
> %22%3A1410956287%2C%5C%2213369%5C%22%3A1410996782%2C%5C%2212535%5C%22%
>
> 00630+
> 3A1410997026%2C%5C%223020%5C%22%3A1410997306%2C%5C%222913%5C%22%3A1410
>
> 00700+
> 997471%2C%5C%222426%5C%22%3A1410997631%2C%5C%2213208%5C%22%3A141099779
>
> 00770+
> 0%2C%5C%222350%5C%22%3A1410997918%2C%5C%2212995%5C%22%3A1410998005%2C%
>
> 00840+
> 5C%2212777%5C%22%3A1410998345%2C%5C%222996%5C%22%3A1410998730%2C%5C%22
>
> 00910+
> 3471%5C%22%3A1410999158%2C%5C%222778%5C%22%3A1410999435%2C%5C%221840%5
>
> 00980+
> C%22%3A1410999880%2C%5C%222662%5C%22%3A1411000322%2C%5C%222655%5C%22%3
>
> 01050+
> A1411000534%2C%5C%222777%5C%22%3A1411000586%2C%5C%222398%5C%22%3A14110
>
> 01120+
> 00887%2C%5C%223787%5C%22%3A1411001258%2C%5C%2212977%5C%22%3A1411001402
>
> 01190+
> %2C%5C%222447%5C%22%3A1411001612%2C%5C%222450%5C%22%3A1411001901%2C%5C
>
> 01260+
> %222463%5C%22%3A1411002161%2C%5C%2213213%5C%22%3A1411029331%2C%5C%2236
>
> 01330+
> 83%5C%22%3A1411029512%2C%5C%22655%5C%22%3A1411029963%2C%5C%222217%5C%2
>
> 01400+ 2%3A1411030200%2C%5C%222464%5C%22%3A1411030432%2C%5C%222767%
>
>
> -----------------------------
>
>
> Thanks!
> -Alex
>
>
>
> On Mon, Nov 24, 2014 at 12:52 AM, Willy Tarreau <w...@1wt.eu> wrote:
>
>> On Sun, Nov 23, 2014 at 11:57:50PM +0800, Alexey Zilber wrote:
>> > Hi Willy,
>> >
>> > I already do "option dontlognull". These get logged anyway. We've
>> > actually seen a high amount of 400 errors happening. I'll post the
>> configs
>> > in a day or two (the pre/post) configs. I've been able to knock down
>> the
>> > 400 and 502 errors to about 10%, with 400 errors being around .6% of
>> total
>> > connections.
>>
>> Then there are data received over these connections, so please follow
>> Lukas' advice and issue "show errors" on the stats socket, you'll see
>> what is sent there. Maybe some bogus browsers sending an SSL hello over
>> this connection or something like this...
>>
>> Willy
>>
>>
>
