We have an issue with haproxy (1.5-dev22-1a34d57) where it is
intermittently not connecting to the backend server. However the
behavior it is exhibiting seems strange.
The reason I say strange is that in one example, it logged that the
client disconnected after ~49 seconds with a connection flags of "CC--".
However our config has "timeout connect 5000", so it should have timed
out connecting to the backend server after 5 seconds. Additionally we
have "retries 3" in the config, so upon timing out, it should have tried
another backend server, but it never did (the retries counter in the log
shows "0").
At the time of this log entry, the backend server is responding
properly. For the ~49 seconds prior to the log entry, the backend server
has taken other requests. The backend server is also another haproxy
(same version).
Here's an example of one such log entry:
198.228.211.13:60848 api~ platform-push/i-84d931a5 49562/0/-1/-1/49563
0/0/0/0/0 0/0 691/212 <
span class="t" style="border-color: rgb(204, 204, 204); font-style: normal;
cursor: pointer;">503 CC-- 4F8E-4624 + GET
/1/sync/notifications/subscribe?sync_box_id=12345&sender=27B9A93C-F473-4385-A662-352AD34A2453
HTTP/1.1
The log format is defined as:
%ci:%cp\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\
%U/%B\ %ST\ %tsc\ %ID\ +\ %r
Running a "show errors" on the stats socket did not return any relevant
results.
Here's the relevant portions of the haproxy config. It is not the entire
thing as the whole config is 1,513 lines long.
global
log 127.0.0.1 local0
maxconn 20480
user haproxy
group haproxy
daemon
stats socket /var/run/hapi/haproxy/haproxy.sock level admin
defaults
log global
mode http
option httplog
option dontlognull
option log-separate-errors
retries 3
option redispatch
timeout connect 5000
timeout client 60000
timeout server 170000
option clitcpka
option srvtcpka
option abortonclose
option splice-auto
monitor-uri /haproxy/ping
stats enable
stats uri /haproxy/stats
stats refresh 15
stats auth user:pass
frontend api
bind *:80
bind *:443 ssl crt /etc/haproxy/server.pem
maxconn 20000
option httpclose
option forwardfor
acl internal src 10.0.0.0/8
acl have_request_id req.fhdr(X-Request-Id) -m found
http-request set-nice -100 if internal
http-request add-header X-API-URL %[path] if !internal
http-request add-header X-Request-Timestamp %Ts.%ms
http-request add-header X-Request-Id %[req.fhdr(X-Request-Id)] if
internal have_request_id
http-request set-header X-Request-Id %{+X}o%pid-%rt if !internal ||
!have_request_id
http-request add-header X-API-Host i-4a3b1c6a
unique-id-format %{+X}o%pid-%rt
log-format %ci:%cp\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\
%ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %U/%B\ %ST\ %tsc\ %ID\ +\ %r
default_backend DEFAULT_404
acl rewrite-found req.hdr(X-Rewrite-ID,1) -m found
acl nqXn_path path_reg ^/1/sync/notifications/subscribe/([^\ ?]*)$
acl nqXn_method method OPTIONS GET HEAD POST PUT DELETE TRACE CONNECT
PATCH
http-request set-header X-Rewrite-Id nqXn if !rewrite-found nqXn_path
nqXn_method
acl rewrite-nqXn req.hdr(X-Rewrite-Id) -m str nqXn
use_backend platform-push if rewrite-nqXn
reqrep ^(OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT|PATCH)\
/1/sync/notifications/subscribe/([^\ ?]*)([\ ?].*|$) \1\
/1/sync/subscribe/\2\3 if rewrite-nqXn
backend platform-push
option httpchk GET /ping
default-server inter 15s fastinter 1s
server i-6eaf724d 10.230.23.64:80 check observe layer4
server i-84d931a5 10.230.42.8:80 check observe layer4
