Hi

On 12/04/2018 19:16, Ayush Goyal wrote:
> Hi,
>
> I have a question regarding haproxy backend connection behaviour. We
> have following setup:
>
>   +---------+     +-------+
>   | haproxy |---->| nginx |
>   +---------+     +-------+
>
> We use a haproxy cluster for ssl off-loading and then load balance
> request to
> nginx cluster. We are currently benchmarking this setup with 3 nodes
> for haproxy
> cluster and 1 nginx node. Each haproxy node has two frontend/backend
> pair. First
> frontend is a router for ssl connection which redistributes request to
> the second 
> frontend in the haproxy cluster. The second frontend is for ssl
> handshake and 
> routing requests to nginx servers. Our configuration is as follows:
>
> ```
> global
>     maxconn 100000
>     user haproxy
>     group haproxy
>     nbproc 2
>     cpu-map 1 1
>     cpu-map 2 2
>
> defaults
>     mode http
>     option forwardfor
>     timeout connect 5s
>     timeout client 30s
>     timeout server 30s
>     timeout tunnel 30m
>     timeout client-fin 5s
>
> frontend ssl_sess_id_router
>         bind *:443
>         bind-process 1
>         mode tcp
>         maxconn 100000
>         log global
>         option tcp-smart-accept
>         option splice-request
>         option splice-response
>         default_backend ssl_sess_id_router_backend
>
> backend ssl_sess_id_router_backend
>         bind-process 1
>         mode tcp
>         fullconn 50000
>         balance roundrobin
>         ...<ssl_stickiness_config>...
>         option tcp-smart-connect
>         server lbtest01 <ip1>:8443 weight 1 check send-proxy
>         server lbtest02 <ip2>:8443 weight 1 check send-proxy
>         server lbtest03 <ip3>:8443 weight 1 check send-proxy
>
> frontend nginx_ssl_fe
>         bind *:8443 ssl <ssl_options>
>         maxconn 100000
>         bind-process 2
>         option tcp-smart-accept
>         option splice-request
>         option splice-response
>         option forwardfor
>         reqadd X-Forwarded-Proto:\ https
>         timeout client-fin 5s
>         timeout http-request 8s
>         timeout http-keep-alive 30s
>         default_backend nginx_backend
>
> backend nginx_backend
>         bind-process 2
>         balance roundrobin
>         http-reuse safe
>         option tcp-smart-connect
>         option splice-request
>         option splice-response
>         timeout tunnel 30m
>         timeout http-request 8s
>         timeout http-keep-alive 30s
>         server testnginx <ip>:80  weight 1 check
> ```
>
> The nginx node has nginx with 4 workers and 8192 max clients,
> therefore the max
> number of connection it can accept is 32768.
>
> For benchmark, we are generating ~3k new connections per second where each
> connection makes 1 http request and then holds the connection for next 30
> seconds. This results in a high established connection on the first
> frontend,
> ssl_sess_id_router, ~25k per haproxy node (Total ~77k connections on 3
> haproxy
> nodes). The second frontend (nginx_ssl_fe) receives the same number of
> connection on the frontend. On nginx node, we see that active connections
> increase to ~32k.
>
> Our understanding is that haproxy should keep a 1:1 connection mapping
> for each
> new connection in frontend/backend. But there is a connection count
> mismatch
> between haproxy and nginx (Total 77k connections in all 3 haproxy for both
> frontends vs 32k connections in nginx made by nginx_backend), We are
> still not
> facing any major 5xx or connection errors. We are assuming that this is
> happening because haproxy is terminating old idle ssl connections to
> serve the
> new ones. We have following questions:
>
> 1. How the nginx_backend connections are being terminated to serve the new
> connections?
Connections are usually terminated when the client receives the whole
response. Closing the connection can be initiated by the client, server
of HAProxy (timeouts, etc..)

> 2. Why haproxy is not terminating connections on the frontend to keep
> it them at 32k
> for 1:1 mapping?
I think there is no 1:1 mapping between the number of connections in
haproxy and nginx. This is because you are chaining the two fron/back
pairs in haproxy, so when the client establishes 1 connctions with
haproxy you will see 2 established connections in haproxy stats. This
explains why the number of connections in haproxy is the double of the
ones in nginx.

> Thanks
> Ayush Goyal

-- 
Moemen MHEDHBI

Reply via email to