On Thu, Jan 07, 2010 at 08:40:59PM -0500, Hector Danniel Paz Trillo wrote:
> Thanks Willy!
>
> I recompile haproxy with:
>
> "-DBUFSIZE=16000 -DMAXREWRITE=1024 -DSYSTEM_MAXCONN=40000"
>
> It doesn't work. Then I recompile with:
>
> "-DBUFSIZE=64000 -DMAXREWRITE=1024 -DSYSTEM_MAXCONN=40000"
>
> And it works. But I'm very worried about this value.
wow! I'd be worried too !
> This is the difference in haproxy logs:
>
> DBUFSIZE=default
>
> Jan 7 20:07:41 127.0.0.1 haproxy[15042]: X.X.X.X:36464
> [07/Jan/2010:20:07:41.338] main bdefault/proxy1 111/0/0/-1/182 502
> 8396 - - PHVN 0/0/0/0/0 0/0 {1643} "POST /user/config HTTP/1.1"
>
> DBUFSIZE=64000
>
> Jan 7 20:14:42 127.0.0.1 haproxy[15129]: X.X.X.X:31434
> [07/Jan/2010:20:14:41.531] main bdefault/proxy1 104/0/0/586/1206 200
> 46025 - - --VN 0/0/0/0/0 0/0 {102721} "POST /user/config HTTP/1.1"
>
> "show errors" doesn't show anything after the 502 (show info and show
> stat are working fine).
Ah, sorry, you're right. I've just checked the code, and we don't capture
too large requests/responses simply because we don't want to flush any
really wrong one.
> So i'm going to test BUFSIZE=64000 very carefully. Meanwhile i'm
> checking why the headers are so big. I know haproxy can capture
> headers, but it seems can not capture all so I don't know where to
> start.
If you don't have too much traffic or can try it by hand, start it
in debug mode using "-d" on the command line, and redirect its output
to a file. It will dump all headers it receives. That can be huge.
If you're on production and cannot do that, your best friend is
tcpdump, especially if you know that the problem only affects one
of your customers and can filter on the IP address. Simply don't
forget to dump full packets (-s0).
If you don't have too much traffic, strace can help a lot too :
strace -o haproxy.trace -s 1000 -e trace=recv,send -p $(pidof haproxy)
This will log the 1000 first bytes of all send/recv calls. Simply
look for a 502 on a send there and look at a previous recv() call
which might be huge.
Or you can also modify haproxy to capture too large responses in
your case. Locate these lines in src/proto_http.c (1.3.22) :
2638 /* too large response does not fit in buffer. */
2639 else if (rep->flags & BF_FULL) {
2640 goto hdr_response_bad;
2641 }
Then add this one :
/* too large response does not fit in buffer. */
else if (rep->flags & BF_FULL) {
+ http_capture_bad_message(&t->be->invalid_rep, t, rep, msg,
t->fe);
goto hdr_response_bad;
}
Regards,
Willy
