Re: Significant number of 400 errors..
Hi Alexey, On Tue, Nov 25, 2014 at 02:22:50PM +0800, Alexey Zilber wrote: > Hi Willy and Lukas, > > Here's snippets of the new config: A few points on your config below : > - > > global > >maxconn 645000 keep in mind that under linux, you're limitd to 1 million FD per process, thus 645k conn will roughly mean 1.29M FD end-to-end, thus you won't reach this limit unless you're limiting the server side connectionn counts, of course. >maxpipes 645000 Here you won't reach that limit, each pipe needs 2 fd in addition to the send/recv ones. I generally suggest not to set this value and to let haproxy set it to maxconn/4 which is the highest optimal value (half of FDs for sockets, other half for pipes). >ulimit-n 645120 This one is wrong, better not set it and let haproxy set it by itself. >user haproxy > >group haproxy > > tune.bufsize 49152 OK so you're ready to have haproxy eat up to about 60 GB of RAM for buffers (48k*2*645000), I hope the system is correctly sized, especially since you need at least 4kB per direction per socket for kernel buffers, which means 10 GB. > defaults >maxconn 200 It does not make sense to have a higher maxconn in frontends than the global one :-/ (...) > listen www 0.0.0.0:80 > bind 0.0.0.0:443 ssl crt /etc/lighttpd/ssl_certs/.co.pem Be careful, SSL can eat up to 92kB (measured) during connection setup or any handshake, so that's an additional 60 GB of RAM in the worst case for 645000 conns. > - > > > The old config did NOT have the following items, and had about 500x more > errors: > - > tune.bufsize 49152 > > option accept-invalid-http-request > > option accept-invalid-http-response > - Have you tried to split them apart ? If it's bufsize only which saves your traffic, it means that you're getting huge request or response headers, and that something is wrong in the application. Moreover, many corporate proxies and ISP's transparent proxies will not let that large requests pass through so some users will regularly experience some errors while browsing the site. > Here's what the 'show errors' shows on a sampling of the server. It looks > like 90% of the errors are the second error (25/Nov/2014:00:06:30.753): OK, some comments below : > Total events captured on [24/Nov/2014:23:31:52.468] : 151 > [22/Nov/2014:21:55:56.597] frontend www (#2): invalid request > backend www (#2), server (#-1), event #150 > src 166.137.247.239:8949, session #3883610, session flags 0x0080 > HTTP msg state 26, msg flags 0x, tx flags 0x > HTTP chunk len 0 bytes, HTTP body len 0 bytes > buffer flags 0x00808002, out 0 bytes, total 1183 bytes > pending 1183 bytes, wrapping at 49152, error at position 227: > > 0 > sited%22%3A1416713764%2C%22times_visited%22%3A6%2C%22device%22%3A%22We > 00070+ > b%22%2C%22lastsource%22%3A%22bottomxpromo%22%2C%22language%22%3A%22en% > 00140+ > 22%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22di > 00210+ d_watch%22%3A1%7D; yX=5167136038811769837; _vhist=%7B%22visito > 00280+ > r_id%22%3A%225024165909427731336%22%2C%22seen_articles%22%3A%22%7B%5C% > 00350+ > 22950%5C%22%3A1402416590%2C%5C%22685%5C%22%3A1402416675%2C%5C%22799%5C > 00420+ > %22%3A1402416789%2C%5C%22954%5C%22%3A1402416997%2C%5C%22939%5C%22%3A14 > 00490+ > 02417098%2C%5C%222334%5C%22%3A1407162586%2C%5C%222055%5C%22%3A14071626 > 00560+ > 91%2C%5C%223888%5C%22%3A1409938121%2C%5C%223020%5C%22%3A1409938211%2C% > 00630+ > 5C%223773%5C%22%3A1409938340%2C%5C%222163%5C%22%3A1409938389%2C%5C%222 > 00700+ > 569%5C%22%3A1409938872%2C%5C%222426%5C%22%3A1409938959%2C%5C%2213984%5 > 00770+ > C%22%3A1411916274%2C%5C%221675%5C%22%3A1411916466%2C%5C%2214950%5C%22% > 00840+ > 3A1412432461%2C%5C%2219759%5C%22%3A1416714580%7D%22%2C%22num_articles% > 00910+ 22%3A17%7D; lastlargeleaderboard=1416713603; > lastlike-264755123648776= > 00980+ 1416713764; lastlike-703681396382652=1416713919; > lastlike-939184096095 > 01050+ 676=1416714070; lastlike-293377080869393=1416714282; > lastlike-14790568 > 01120+ 05705686=1416714412; popped_19731=1\r\n > 01157 Connection: keep-alive\r\n > 01181 \r\n As you can see above, this request starts in the middle of a Cookie header. It definitely is invalid. If you're getting huge requests, it's very possible that a bogus client or a downstream proxy has too small headers and truncates requests then sends them in two parts... That's one of the trouble to expect from huge requests. > > Total events captured on [25/Nov/2014:00:06:30.753] : 480 > [25/Nov/2014:00:06:26.417] frontend www (#2): invalid request > backend www (#2), server (#-1), event #479 > src 50.92.157.165:53649, session #14629125, session flags 0x0080 > HTTP msg state 26, msg flags 0x00
Re: Significant number of 400 errors..
On 27 November 2014 at 10:39, Alexey Zilber wrote: > That's part of what I'm trying to figure out.. where are the junk bytes > coming from. Is it from the client, server, haproxy, or networking issue? That's what tcpdump is useful for. Use it at different places in your end-to-end client/backend path, and you'll discover where the junk originates. Jonathan
Re: Significant number of 400 errors..
That's part of what I'm trying to figure out.. where are the junk bytes
coming from. Is it from the client, server, haproxy, or networking issue?
I'd rather not be passing any kind of crap to the backend, but the
changes I made have reduced the errors by around 500%. I'm guessing the
issue may have been the header size, and the errors we're seeing now are
legitimate junk.. although I have no idea where it's coming from.I may
want to remove "option accept-invalid-http-request and "option
accept-invalid-http-response".
So the point is, not to switch to TCP to mask the issue, but to figure out
where this 'junk' is coming from.
There's 30 php servers btw, I figured it would be a bit redundant to list
all of them, then mask their ip. They all have the same config.
Thanks!
-Alex
On Thu, Nov 27, 2014 at 4:06 AM, Bryan Talbot
wrote:
> There are clearly a lot of junk bytes in those URI which are not allowed
> by the HTTP specs. If you really want to be passing unencoded binary
> control characters, spaces, and nulls to your backends in HTTP request and
> header lines, then HTTP mode is probably not going to work for you.
>
> TCP mode will allow them to get through but if your backends actually
> expect the requests to be valid HTTP, you will likely be opening up a huge
> can of worms and exposing your apps to a host of protocol level attacks.
>
> Also, your connection limits seem pretty ambitious if there really are 2
> php servers in that backend and not 2000.
>
> -Bryan
>
>
>
> On Mon, Nov 24, 2014 at 10:22 PM, Alexey Zilber
> wrote:
>
>> Hi Willy and Lukas,
>>
>> Here's snippets of the new config:
>>
>>
>> -
>>
>> global
>>
>>maxconn 645000
>>
>>maxpipes 645000
>>
>>ulimit-n 645120
>>
>>user haproxy
>>
>>group haproxy
>>
>> tune.bufsize 49152
>>
>>spread-checks 10
>>
>>daemon
>>
>>quiet
>>
>>stats socket /var/run/haproxy.sock level admin
>>
>>pidfile /var/run/haproxy.pid
>>
>>
>> defaults
>>
>>log global
>>
>>modehttp
>>
>> option accept-invalid-http-request
>>
>> option accept-invalid-http-response
>>
>>option httplog
>>
>>option dontlognull
>>
>>option dontlog-normal
>>
>>option log-separate-errors
>>
>> option http-server-close
>>
>> option tcp-smart-connect
>>
>> option tcp-smart-accept
>>
>> option forwardfor except 127.0.0.1
>>
>>option dontlog-normal
>>
>>retries 3
>>
>>option redispatch
>>
>>maxconn 200
>>
>>contimeout 5000
>>
>>clitimeout 6
>>
>>srvtimeout 6
>>
>> listen www 0.0.0.0:80
>>
>>mode http
>>
>> capture response header Via len 20
>>
>> capture response header Content-Length len 10
>>
>> capture response header Cache-Control len 8
>>
>> capture response header Location len 40
>>
>>balance roundrobin
>>
>># Haproxy status page
>>
>>stats uri /haproxy-status
>>
>>stats auth fb:phoo
>>
>># when cookie persistence is required
>>
>>cookie SERVERID insert indirect nocache
>>
>># When internal servers support a status page
>>
>>option httpchk GET /xyzzyx.php
>>
>> bind 0.0.0.0:443 ssl crt /etc/lighttpd/ssl_certs/.co.pem
>>
>> http-request add-header X-FORWARDED-PROTO https if { ssl_fc }
>>
>> server app1 10.1.1.6:85 check inter 4 rise 2 fall 3
>> maxconn 16384
>>
>> server app2 10.1.1.7:85 check inter 4 rise 2 fall 3
>> maxconn 16384
>>
>> -
>>
>>
>> The old config did NOT have the following items, and had about 500x more
>> errors:
>> -
>> tune.bufsize 49152
>>
>> option accept-invalid-http-request
>>
>> option accept-invalid-http-response
>> -
>>
>> Here's what the 'show errors' shows on a sampling of the server. It
>> looks like 90% of the errors are the second error
>> (25/Nov/2014:00:06:30.753):
>>
>>
>>
>>
>> Total events captured on [24/Nov/2014:23:31:52.468] : 151
>>
>>
>>
>> [22/Nov/2014:21:55:56.597] frontend www (#2): invalid request
>>
>> backend www (#2), server (#-1), event #150
>>
>> src 166.137.247.239:8949, session #3883610, session flags 0x0080
>>
>> HTTP msg state 26, msg flags 0x, tx flags 0x
>>
>> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>>
>> buffer flags 0x00808002, out 0 bytes, total 1183 bytes
>>
>> pending 1183 bytes, wrapping at 49152, error at position 227:
>>
>>
>>
>> 0
>> sited%22%3A1416713764%2C%22times_visited%22%3A6%2C%22device%22%3A%22We
>>
>> 00070+
>> b%22%2C%22lastsource%22%3A%22bottomxpromo%22%2C%22language%22%3A%22en%
>>
>> 00140+
>> 22%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22di
>>
>> 00210+ d_watch%22%3A1%7D; yX
Re: Significant number of 400 errors..
There are clearly a lot of junk bytes in those URI which are not allowed by
the HTTP specs. If you really want to be passing unencoded binary control
characters, spaces, and nulls to your backends in HTTP request and header
lines, then HTTP mode is probably not going to work for you.
TCP mode will allow them to get through but if your backends actually
expect the requests to be valid HTTP, you will likely be opening up a huge
can of worms and exposing your apps to a host of protocol level attacks.
Also, your connection limits seem pretty ambitious if there really are 2
php servers in that backend and not 2000.
-Bryan
On Mon, Nov 24, 2014 at 10:22 PM, Alexey Zilber
wrote:
> Hi Willy and Lukas,
>
> Here's snippets of the new config:
>
>
> -
>
> global
>
>maxconn 645000
>
>maxpipes 645000
>
>ulimit-n 645120
>
>user haproxy
>
>group haproxy
>
> tune.bufsize 49152
>
>spread-checks 10
>
>daemon
>
>quiet
>
>stats socket /var/run/haproxy.sock level admin
>
>pidfile /var/run/haproxy.pid
>
>
> defaults
>
>log global
>
>modehttp
>
> option accept-invalid-http-request
>
> option accept-invalid-http-response
>
>option httplog
>
>option dontlognull
>
>option dontlog-normal
>
>option log-separate-errors
>
> option http-server-close
>
> option tcp-smart-connect
>
> option tcp-smart-accept
>
> option forwardfor except 127.0.0.1
>
>option dontlog-normal
>
>retries 3
>
>option redispatch
>
>maxconn 200
>
>contimeout 5000
>
>clitimeout 6
>
>srvtimeout 6
>
> listen www 0.0.0.0:80
>
>mode http
>
> capture response header Via len 20
>
> capture response header Content-Length len 10
>
> capture response header Cache-Control len 8
>
> capture response header Location len 40
>
>balance roundrobin
>
># Haproxy status page
>
>stats uri /haproxy-status
>
>stats auth fb:phoo
>
># when cookie persistence is required
>
>cookie SERVERID insert indirect nocache
>
># When internal servers support a status page
>
>option httpchk GET /xyzzyx.php
>
> bind 0.0.0.0:443 ssl crt /etc/lighttpd/ssl_certs/.co.pem
>
> http-request add-header X-FORWARDED-PROTO https if { ssl_fc }
>
> server app1 10.1.1.6:85 check inter 4 rise 2 fall 3 maxconn
> 16384
>
> server app2 10.1.1.7:85 check inter 4 rise 2 fall 3 maxconn
> 16384
>
> -
>
>
> The old config did NOT have the following items, and had about 500x more
> errors:
> -
> tune.bufsize 49152
>
> option accept-invalid-http-request
>
> option accept-invalid-http-response
> -
>
> Here's what the 'show errors' shows on a sampling of the server. It looks
> like 90% of the errors are the second error (25/Nov/2014:00:06:30.753):
>
>
>
>
> Total events captured on [24/Nov/2014:23:31:52.468] : 151
>
>
>
> [22/Nov/2014:21:55:56.597] frontend www (#2): invalid request
>
> backend www (#2), server (#-1), event #150
>
> src 166.137.247.239:8949, session #3883610, session flags 0x0080
>
> HTTP msg state 26, msg flags 0x, tx flags 0x
>
> HTTP chunk len 0 bytes, HTTP body len 0 bytes
>
> buffer flags 0x00808002, out 0 bytes, total 1183 bytes
>
> pending 1183 bytes, wrapping at 49152, error at position 227:
>
>
>
> 0
> sited%22%3A1416713764%2C%22times_visited%22%3A6%2C%22device%22%3A%22We
>
> 00070+
> b%22%2C%22lastsource%22%3A%22bottomxpromo%22%2C%22language%22%3A%22en%
>
> 00140+
> 22%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22di
>
> 00210+ d_watch%22%3A1%7D; yX=5167136038811769837; _vhist=%7B%22visito
>
> 00280+
> r_id%22%3A%225024165909427731336%22%2C%22seen_articles%22%3A%22%7B%5C%
>
> 00350+
> 22950%5C%22%3A1402416590%2C%5C%22685%5C%22%3A1402416675%2C%5C%22799%5C
>
> 00420+
> %22%3A1402416789%2C%5C%22954%5C%22%3A1402416997%2C%5C%22939%5C%22%3A14
>
> 00490+
> 02417098%2C%5C%222334%5C%22%3A1407162586%2C%5C%222055%5C%22%3A14071626
>
> 00560+
> 91%2C%5C%223888%5C%22%3A1409938121%2C%5C%223020%5C%22%3A1409938211%2C%
>
> 00630+
> 5C%223773%5C%22%3A1409938340%2C%5C%222163%5C%22%3A1409938389%2C%5C%222
>
> 00700+
> 569%5C%22%3A1409938872%2C%5C%222426%5C%22%3A1409938959%2C%5C%2213984%5
>
> 00770+
> C%22%3A1411916274%2C%5C%221675%5C%22%3A1411916466%2C%5C%2214950%5C%22%
>
> 00840+
> 3A1412432461%2C%5C%2219759%5C%22%3A1416714580%7D%22%2C%22num_articles%
>
> 00910+ 22%3A17%7D; lastlargeleaderboard=1416713603;
> lastlike-264755123648776=
>
> 00980+ 1416713764; lastlike-703681396382652=1416713919;
> lastlike-939184096095
>
> 01050+ 676=1416714070; lastlike-293377080869393=14
Re: Significant number of 400 errors..
Hi Willy and Lukas,
Here's snippets of the new config:
-
global
maxconn 645000
maxpipes 645000
ulimit-n 645120
user haproxy
group haproxy
tune.bufsize 49152
spread-checks 10
daemon
quiet
stats socket /var/run/haproxy.sock level admin
pidfile /var/run/haproxy.pid
defaults
log global
modehttp
option accept-invalid-http-request
option accept-invalid-http-response
option httplog
option dontlognull
option dontlog-normal
option log-separate-errors
option http-server-close
option tcp-smart-connect
option tcp-smart-accept
option forwardfor except 127.0.0.1
option dontlog-normal
retries 3
option redispatch
maxconn 200
contimeout 5000
clitimeout 6
srvtimeout 6
listen www 0.0.0.0:80
mode http
capture response header Via len 20
capture response header Content-Length len 10
capture response header Cache-Control len 8
capture response header Location len 40
balance roundrobin
# Haproxy status page
stats uri /haproxy-status
stats auth fb:phoo
# when cookie persistence is required
cookie SERVERID insert indirect nocache
# When internal servers support a status page
option httpchk GET /xyzzyx.php
bind 0.0.0.0:443 ssl crt /etc/lighttpd/ssl_certs/.co.pem
http-request add-header X-FORWARDED-PROTO https if { ssl_fc }
server app1 10.1.1.6:85 check inter 4 rise 2 fall 3 maxconn
16384
server app2 10.1.1.7:85 check inter 4 rise 2 fall 3 maxconn
16384
-
The old config did NOT have the following items, and had about 500x more
errors:
-
tune.bufsize 49152
option accept-invalid-http-request
option accept-invalid-http-response
-
Here's what the 'show errors' shows on a sampling of the server. It looks
like 90% of the errors are the second error (25/Nov/2014:00:06:30.753):
Total events captured on [24/Nov/2014:23:31:52.468] : 151
[22/Nov/2014:21:55:56.597] frontend www (#2): invalid request
backend www (#2), server (#-1), event #150
src 166.137.247.239:8949, session #3883610, session flags 0x0080
HTTP msg state 26, msg flags 0x, tx flags 0x
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 1183 bytes
pending 1183 bytes, wrapping at 49152, error at position 227:
0
sited%22%3A1416713764%2C%22times_visited%22%3A6%2C%22device%22%3A%22We
00070+
b%22%2C%22lastsource%22%3A%22bottomxpromo%22%2C%22language%22%3A%22en%
00140+
22%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22di
00210+ d_watch%22%3A1%7D; yX=5167136038811769837; _vhist=%7B%22visito
00280+
r_id%22%3A%225024165909427731336%22%2C%22seen_articles%22%3A%22%7B%5C%
00350+
22950%5C%22%3A1402416590%2C%5C%22685%5C%22%3A1402416675%2C%5C%22799%5C
00420+
%22%3A1402416789%2C%5C%22954%5C%22%3A1402416997%2C%5C%22939%5C%22%3A14
00490+
02417098%2C%5C%222334%5C%22%3A1407162586%2C%5C%222055%5C%22%3A14071626
00560+
91%2C%5C%223888%5C%22%3A1409938121%2C%5C%223020%5C%22%3A1409938211%2C%
00630+
5C%223773%5C%22%3A1409938340%2C%5C%222163%5C%22%3A1409938389%2C%5C%222
00700+
569%5C%22%3A1409938872%2C%5C%222426%5C%22%3A1409938959%2C%5C%2213984%5
00770+
C%22%3A1411916274%2C%5C%221675%5C%22%3A1411916466%2C%5C%2214950%5C%22%
00840+
3A1412432461%2C%5C%2219759%5C%22%3A1416714580%7D%22%2C%22num_articles%
00910+ 22%3A17%7D; lastlargeleaderboard=1416713603;
lastlike-264755123648776=
00980+ 1416713764; lastlike-703681396382652=1416713919;
lastlike-939184096095
01050+ 676=1416714070; lastlike-293377080869393=1416714282;
lastlike-14790568
01120+ 05705686=1416714412; popped_19731=1\r\n
01157 Connection: keep-alive\r\n
01181 \r\n
---
Total events captured on [25/Nov/2014:00:06:30.753] : 480
[25/Nov/2014:00:06:26.417] frontend www (#2): invalid request
backend www (#2), server (#-1), event #479
src 50.92.157.165:53649, session #14629125, session flags 0x0080
HTTP msg state 26, msg flags 0x, tx flags 0x8400
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 45 bytes
pending 45 bytes, wrapping at 49152, error at position 0:
0
\x17\x03\x03\x00(t\xF42\x9D\x96\x19[(\xF6\xFEK\xB7\x9Br\x00C\xDBn<\x98
00025+ \x12"\xC2\xBC\x91\x91\x0E5\xEA,\xCE0t\xF3\x90\x16\x1Dd\x99\xC4
--
Total events captured on [25/Nov/2014:00:08:45.356] : 128
[24/Nov/2014:22:38:34.626] frontend www (#2): invalid request
backend www (#2), server (#-1), event #127
src 157.55.39.106
Re: Significant number of 400 errors..
On Sun, Nov 23, 2014 at 11:57:50PM +0800, Alexey Zilber wrote: > Hi Willy, > > I already do "option dontlognull". These get logged anyway. We've > actually seen a high amount of 400 errors happening. I'll post the configs > in a day or two (the pre/post) configs. I've been able to knock down the > 400 and 502 errors to about 10%, with 400 errors being around .6% of total > connections. Then there are data received over these connections, so please follow Lukas' advice and issue "show errors" on the stats socket, you'll see what is sent there. Maybe some bogus browsers sending an SSL hello over this connection or something like this... Willy
Re: Significant number of 400 errors..
Hi Willy, I already do "option dontlognull". These get logged anyway. We've actually seen a high amount of 400 errors happening. I'll post the configs in a day or two (the pre/post) configs. I've been able to knock down the 400 and 502 errors to about 10%, with 400 errors being around .6% of total connections. -Alex On Sun, Nov 23, 2014 at 7:53 PM, Willy Tarreau wrote: > Hi guys, > > On Sat, Nov 22, 2014 at 09:06:57PM +0100, Lukas Tribus wrote: > > > Hi Lukas, > > > > > > I had decoded the error message and it didn't make sense. There is > no > > > connection limit reached, there are no filters. If you look at the > > > rest of the log line, there were no cookies. In fact, the last part "a > > > security check which detected and blocked a dangerous error in server > > > response which might have caused information leak" is very ambiguous. > > > Is there any detailed explanation? > > > > Since, this error as-is can indeed have a lot of different causes, we > will > > only be able to provide a more detailed explanation when we have the > > info from "show errors" and equally important, the configuration. > > Given the high timeouts, I suspect that these are simply aborted > pre-connects > from browsers who finally decide they won't use their connection. Alex, > just > add "option dontlognull" and they'll go away if that's it. This option > ensures > that connections without any data exchange do not produce any log. > > Regards, > Willy > >
Re: Significant number of 400 errors..
Hi guys, On Sat, Nov 22, 2014 at 09:06:57PM +0100, Lukas Tribus wrote: > > Hi Lukas, > > > > I had decoded the error message and it didn't make sense. There is no > > connection limit reached, there are no filters. If you look at the > > rest of the log line, there were no cookies. In fact, the last part "a > > security check which detected and blocked a dangerous error in server > > response which might have caused information leak" is very ambiguous. > > Is there any detailed explanation? > > Since, this error as-is can indeed have a lot of different causes, we will > only be able to provide a more detailed explanation when we have the > info from "show errors" and equally important, the configuration. Given the high timeouts, I suspect that these are simply aborted pre-connects from browsers who finally decide they won't use their connection. Alex, just add "option dontlognull" and they'll go away if that's it. This option ensures that connections without any data exchange do not produce any log. Regards, Willy
RE: Significant number of 400 errors..
> Hi Lukas, > > I had decoded the error message and it didn't make sense. There is no > connection limit reached, there are no filters. If you look at the > rest of the log line, there were no cookies. In fact, the last part "a > security check which detected and blocked a dangerous error in server > response which might have caused information leak" is very ambiguous. > Is there any detailed explanation? Since, this error as-is can indeed have a lot of different causes, we will only be able to provide a more detailed explanation when we have the info from "show errors" and equally important, the configuration. Regards, Lukas
Re: Significant number of 400 errors..
Hi Lukas, I had decoded the error message and it didn't make sense. There is no connection limit reached, there are no filters. If you look at the rest of the log line, there were no cookies. In fact, the last part "a security check which detected and blocked a dangerous error in server response which might have caused information leak" is very ambiguous. Is there any detailed explanation? Thanks for the links btw, I completely missed the socket info, and that it was possible to get more detail on the errors via the sockets. I'm going to dig deeper with that and will post a followup. -Alex On Sat, Nov 22, 2014 at 9:37 PM, Lukas Tribus wrote: > Hi Alexey, > > > > > All, > > > > I've tripled the default buffer size, doubled maxconn and added > > accept invalid http request from client and server. This got rid of a > > large number of the 400 ' s but not all. Any ideas what it could be? > > There's nothing else specific in the logs and haproxy-status is all > > good. > > > First of all, lets decode the original error message first. > > In the log we see "PRNN", which means (according to the docs [1]): > P : the session was prematurely aborted by the proxy, because of a > connection limit enforcement, because a DENY filter was matched, > because of a security check which detected and blocked a dangerous > error in server response which might have caused information leak > (eg: cacheable cookie). > > > Subsequent characters (RNN) don't really matter at that point. > > To find out why exactly the proxy decided to close the connection, I > suggest you enable the unix admin socket and provide show errors [3] > output from there. This will tell us more about the last error. > > Also, you need to provide your configuration, especially timeout, acls > maxconn values etc. Feel free to obfuscate hostnames and IP addresses, > but make sure that everything else remains intact. > > > > Regards, > > Lukas > > > > [1] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.5 > [2] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2 > [3] > http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2-show%20errors > > >
RE: Significant number of 400 errors..
Hi Alexey, > All, > > I've tripled the default buffer size, doubled maxconn and added > accept invalid http request from client and server. This got rid of a > large number of the 400 ' s but not all. Any ideas what it could be? > There's nothing else specific in the logs and haproxy-status is all > good. First of all, lets decode the original error message first. In the log we see "PRNN", which means (according to the docs [1]): P : the session was prematurely aborted by the proxy, because of a connection limit enforcement, because a DENY filter was matched, because of a security check which detected and blocked a dangerous error in server response which might have caused information leak (eg: cacheable cookie). Subsequent characters (RNN) don't really matter at that point. To find out why exactly the proxy decided to close the connection, I suggest you enable the unix admin socket and provide show errors [3] output from there. This will tell us more about the last error. Also, you need to provide your configuration, especially timeout, acls maxconn values etc. Feel free to obfuscate hostnames and IP addresses, but make sure that everything else remains intact. Regards, Lukas [1] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.5 [2] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2 [3] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2-show%20errors
Re: Significant number of 400 errors..
All,
I've tripled the default buffer size, doubled maxconn and added accept
invalid http request from client and server. This got rid of a large
number of the 400 ' s but not all. Any ideas what it could be? There's
nothing else specific in the logs and haproxy-status is all good.
-Alex
On Nov 22, 2014 1:06 PM, "Alexey Zilber" wrote:
> Hi All,
>
> I'm running v1.5.4 and I'm seeing a large amount of 400 erros:
>
> Nov 21 22:46:06 srvr1 haproxy[28293]: 10.10.10.10:51184
> [21/Nov/2014:22:45:50.323] www www/ -1/-1/-1/-1/16350 400 187 - -
> PRNN 445/445/0/0/3 0/0 {|||} ""
>
> Nov 21 22:47:46 srvr1 haproxy[28293]: 10.10.10.10:54924
> [21/Nov/2014:22:47:43.572] www www/ -1/-1/-1/-1/2680 400 187 - -
> PRNN 366/366/0/0/3 0/0 {|||} ""
>
> Nov 21 22:48:20 srvr1 haproxy[28293]: 10.10.10.10:47761
> [21/Nov/2014:22:48:20.707] www www/ -1/-1/-1/-1/0 400 187 - - PRNN
> 417/417/0/0/3 0/0 {|||} ""
>
> Nov 21 22:52:55 srvr1 haproxy[28457]: 10.10.10.10:58249
> [21/Nov/2014:22:52:51.616] www www/ -1/-1/-1/-1/3536 400 187 - -
> PRNN 534/534/2/0/3 0/0 {|||} ""
>
> Nov 21 22:55:09 srvr1 haproxy[28457]: 10.10.10.10:49728
> [21/Nov/2014:22:55:06.827] www www/ -1/-1/-1/-1/2819 400 187 - -
> PRNN 381/381/1/0/3 0/0 {|||} ""
>
> Nov 21 22:55:12 srvr1 haproxy[28457]: 10.10.10.10:49727
> [21/Nov/2014:22:55:06.828] www www/ -1/-1/-1/-1/5766 400 187 - -
> PRNN 368/368/1/0/3 0/0 {|||} ""
>
> Nov 21 23:00:01 srvr1 haproxy[28457]: 10.10.10.10:64153
> [21/Nov/2014:22:59:08.964] www www/ -1/-1/-1/-1/52680 400 187 - -
> PRNN 409/409/1/0/3 0/0 {|||} ""
>
>
>
> We're seeing about 15 per minute. Legitimate users are reporting this
> error as well, so it doesn't seem to be a malicious user. Any ideas?
>
>
> -Alex
>
