sorry, sent that before it was ready. here's the complete message:
Some more info:
I am getting reports from users of high numbers of 504s.
My timeouts are pretty high (while trying to debug this problem), so it
doesn't seem like they are the issue:
timeout connect 20s
timeout client 80s
timeout server 80s
timeout http-keep-alive 50s
I have http logging on, but I am not seeing any 5xx responses (almost all
200s, with a low number of 4xx, which seems about right).
I am tracking the count of all status codes, and it seems that the ereq
count tracks pretty closely to the number of 400s that are in the haproxy
log (though the logs have a couple more 400s than are reported by stats).
Logs for these 400s look like one of three things:
1: Mar 29 15:13:42.000 haproxy-k49 haproxy[19450]: xx.xx.xx.xx:45381
[29/Mar/2014:15:13:42.181] frontend_https~ frontend_https/<NOSRV>
-1/-1/-1/-1/46 400 187 - - CR-- 1031/1031/0/0/0 0/0 "<BADREQ>"
2: Mar 29 15:13:41.000 haproxy-k49 haproxy[19450]: xx.xx.xx.xx:38440
[29/Mar/2014:15:13:40.874] frontend_https~ tapp_http/tapp-p8b
394/0/1/29/424 400 183 - - ---- 1046/1046/23/9/0 0/0 "GET
/v1/data/?key=k1&interval=1min&function=mean&start=2014-03-29T20%3A13%3A44.000Z&end=2014-03-29T20%3A13%3A40.623Z
HTTP/1.1"
3: Mar 29 15:09:19.000 haproxy-k49 haproxy[19450]: xx.xx.xx.xx:51969
[29/Mar/2014:15:09:19.213] frontend_https~ frontend_https/<NOSRV>
-1/-1/-1/-1/118 400 187 - - PR-- 1087/1087/0/0/0 0/0 "<BADREQ>"
These are spread across a variety of customers and don't seem related to
SSL (since some of the errors are on the http frontend). The counts for the
various types of 400s are here:
[patrick@haproxy-k49 ~]$ sudo grep haproxy /var/log/messages | grep -E
"[0-9] 400 [0-9]" | awk '{print $6 " " $9 " " $11 " " $15}' | sed
s/:[0-9]*// | sed s/tapp-.../tapp-abc/ | sort | uniq -c | sed
s/[0-9][0-9][0-9]\\?\\./x./g
37 x.x.x.245 tapp_http/tapp-abc 400 ----
12 x.x.x.25 frontend_http/<NOSRV> 400 CR--
1182 x.x.x.35 frontend_https/<NOSRV> 400 CR--
1 x.x.x.94 frontend_http/<NOSRV> 400 CR--
35 x.x.x.65 tapp_http/tapp-abc 400 ----
8 x.x.x.29 frontend_http/<NOSRV> 400 PR--
89 x.x.x.96 frontend_https/<NOSRV> 400 PR--
My guess is that requests like (2) are the ones that end up as 400s but
don't register as ereq's (just do to the low frequency of them).
The lines like (1) (the CR lines) I'm assuming as premature closes by the
client, and there's maybe nothing I can do about that.
For lines like (3) (the PR lines), I don't understand why the proxy is
denying them. Is there anyway to see exactly what is being sent for these
connections?
Thanks,
Patrick
On Sat, Mar 29, 2014 at 3:12 PM, Patrick Schless
<[email protected]>wrote:
> Some more info:
>
> I am getting reports from users of high numbers of 504s.
>
> My timeouts are pretty high (while trying to debug this problem), so it
> doesn't seem like they are the issue:
> timeout connect 20s
> timeout client 80s
> timeout server 80s
> timeout http-keep-alive 50s
>
> I have http logging on, but I am not seeing any 5xx responses (almost all
> 200s, with a low number of 4xx, which seems about right).
>
>
> On Fri, Mar 28, 2014 at 7:23 PM, Patrick Schless <
> [email protected]> wrote:
>
>> I am running on 1.5 dev22, and doing SSL termination. Traffic seems to be
>> handled fine, but my ereq is steadily rising. Poking at the source, it
>> looks like this can be caused by a number of different errors.
>>
>> What's the next step for trying to determine what's causing these? I
>> tried bumping my connect and cli timeouts, but that didn't change anything.
>>
>>
>> Thanks,
>> Patrick
>>
>
>
