On Thu, Nov 08, 2018 at 11:26:25AM -0500, Nick Vatamaniuc wrote:
> Hello,
>
> I'd like to report an issue where sometimes PUT/POST request bodies do not
> get proxied. Headers are sent then the backend server waits to receive the
> body. It never arrives and after 5 minutes the socket times out. This
> happens in all the 1.8 releases up until 1.8.14 but it doesn't happen in
> 1.7. In fact one of the ways we mitigate the issue is by reverting back to
> 1.7.
>
> Normally the requests go through without an issue. We see about 1M+
> requests per day. However about 5 or 10 of those end up failing as
> described. Is there anything we can do to debug the issue further, or were
> there any changes that might lead to this specific bug in 1.8.x?
I'm not aware of anything which could produce this situation. The low
rate tends to indicate a difficult race condition involving several
elements. Just trying to bisect the situation, do you know if your
application uses 100-continue or not ? Could you try if
"option http-buffer-request" changes anything ? This option makes haproxy
wait for the body before sending the request to the server. Not only this
reduces the number of concurrent connections on the server, but it will
also help figure where the problem happens. Ideally if you can spot a
situation where such a connection is currently frozen, a "show sess all"
issued over the CLI could significantly help, by showing the session's
exact state. Be careful the output can be huge.
If you want to periodically filter the output to search for such a frozen
session, instead you can do this :
1) spot the candidates, those aged 1 minute or more :
$ echo "show sess" | socat - /path/to/socket | grep age='[0-9]*m'
0xb5fff600: proto=tcpv6 src=1.2.3.4:57355 fe=http-in be=<NONE> srv=<none>
ts=02 age=1m5s calls=1 rq[f=400000h,i=0,an=36h,rx=25s,wx=,ax=5s]
rp[f=80000000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=1,ex=] s1=[0,10h,fd=-1,ex=]
exp=5s
2) show the session's details :
$ echo "show sess 0xb5fff600" | socat - /path/to/socket
This can simply be scripted to do this every 30 seconds :
while sleep 30; do
sess=$(echo "show sess" | socat - /path/to/socket | grep age='[0-9]*m' |
cut -f1 -d:)
if [ -n "$sess" ]; then
echo "=========== $(date) ============="
for i in $sess; do echo -n "show sess $i;"done | socat - /path/to/socket
fi >> /path/to/faulty-sessions.log
done
> More details about our setup:
(...)
I'm not seeing anything wrong nor suspicious here.
Thanks,
Willy
