On Mon, 16 Apr 2018 6:09 pm Ayush Goyal <ay...@helpshift.com> wrote:

> Hi Moemen,
>
> Thanks for your response. But I think I need to clarify a few things here.
>
> On Mon, Apr 16, 2018 at 4:33 AM Moemen MHEDHBI <mmhed...@haproxy.com>
> wrote:
>
>> Hi
>>
>> On 12/04/2018 19:16, Ayush Goyal wrote:
>>
>> Hi,
>>
>> I have a question regarding haproxy backend connection behaviour. We have
>> following setup:
>>
>>   +---------+     +-------+
>>   | haproxy |---->| nginx |
>>   +---------+     +-------+
>>
>> We use a haproxy cluster for ssl off-loading and then load balance
>> request to
>> nginx cluster. We are currently benchmarking this setup with 3 nodes for
>> haproxy
>> cluster and 1 nginx node. Each haproxy node has two frontend/backend
>> pair. First
>> frontend is a router for ssl connection which redistributes request to the
>>  second
>> frontend in the haproxy cluster. The second frontend is for ssl
>> handshake and
>> routing requests to nginx servers. Our configuration is as follows:
>>
>> ```
>> global
>>     maxconn 100000
>>     user haproxy
>>     group haproxy
>>     nbproc 2
>>     cpu-map 1 1
>>     cpu-map 2 2
>>
>> defaults
>>     mode http
>>     option forwardfor
>>     timeout connect 5s
>>     timeout client 30s
>>     timeout server 30s
>>     timeout tunnel 30m
>>     timeout client-fin 5s
>>
>> frontend ssl_sess_id_router
>>         bind *:443
>>         bind-process 1
>>         mode tcp
>>         maxconn 100000
>>         log global
>>         option tcp-smart-accept
>>         option splice-request
>>         option splice-response
>>         default_backend ssl_sess_id_router_backend
>>
>> backend ssl_sess_id_router_backend
>>         bind-process 1
>>         mode tcp
>>         fullconn 50000
>>         balance roundrobin
>>         ...<ssl_stickiness_config>...
>>         option tcp-smart-connect
>>         server lbtest01 <ip1>:8443 weight 1 check send-proxy
>>         server lbtest02 <ip2>:8443 weight 1 check send-proxy
>>         server lbtest03 <ip3>:8443 weight 1 check send-proxy
>>
>> frontend nginx_ssl_fe
>>         bind *:8443 ssl <ssl_options>
>>         maxconn 100000
>>         bind-process 2
>>         option tcp-smart-accept
>>         option splice-request
>>         option splice-response
>>         option forwardfor
>>         reqadd X-Forwarded-Proto:\ https
>>         timeout client-fin 5s
>>         timeout http-request 8s
>>         timeout http-keep-alive 30s
>>         default_backend nginx_backend
>>
>> backend nginx_backend
>>         bind-process 2
>>         balance roundrobin
>>         http-reuse safe
>>         option tcp-smart-connect
>>         option splice-request
>>         option splice-response
>>         timeout tunnel 30m
>>         timeout http-request 8s
>>         timeout http-keep-alive 30s
>>         server testnginx <ip>:80  weight 1 check
>> ```
>>
>> The nginx node has nginx with 4 workers and 8192 max clients, therefore
>> the max
>> number of connection it can accept is 32768.
>>
>> For benchmark, we are generating ~3k new connections per second where each
>> connection makes 1 http request and then holds the connection for next 30
>> seconds. This results in a high established connection on the first
>> frontend,
>> ssl_sess_id_router, ~25k per haproxy node (Total ~77k connections on 3
>> haproxy
>> nodes). The second frontend (nginx_ssl_fe) receives the same number of
>> connection on the frontend. On nginx node, we see that active connections
>> increase to ~32k.
>>
>> Our understanding is that haproxy should keep a 1:1 connection mapping
>> for each
>> new connection in frontend/backend. But there is a connection count
>> mismatch
>> between haproxy and nginx (Total 77k connections in all 3 haproxy for both
>> frontends vs 32k connections in nginx made by nginx_backend), We are
>> still not
>> facing any major 5xx or connection errors. We are assuming that this is
>> happening because haproxy is terminating old idle ssl connections to
>> serve the
>> new ones. We have following questions:
>>
>> 1. How the nginx_backend connections are being terminated to serve the new
>> connections?
>>
>> Connections are usually terminated when the client receives the whole
>> response. Closing the connection can be initiated by the client, server of
>> HAProxy (timeouts, etc..)
>>
>
> Client connections are keep-alive here for 30 seconds from client side.
> Various timeout values in both nginx and haproxy are sufficiently high of
> the order of 60 seconds. Still what we are observing here is that nginx is
> closing the connection after 7-14 seconds to serve new client requests. Not
> sure why nginx or haproxy will close existing keep-alive connections to
> serve new requests when timeouts are sufficiently high?
>
>> 2. Why haproxy is not terminating connections on the frontend to keep it
>> them at 32k
>> for 1:1 mapping?
>>
>> I think there is no 1:1 mapping between the number of connections in
>> haproxy and nginx. This is because you are chaining the two fron/back pairs
>> in haproxy, so when the client establishes 1 connctions with haproxy you
>> will see 2 established connections in haproxy stats. This explains why the
>> number of connections in haproxy is the double of the ones in nginx.
>>
>
> I want to clarify about the number of connections here in each frontend.
> We are observing 77k connections in the first frontend stats i.e
> ssl_sess_id_router, initiated by client. Then we are observing another set
> of 77k connections in the nginx_ssl_fe frontend stats initiated by the
> backend of the first frontend. But corresponding connections in the backend
> for second frontend are much fewer, around 32k. This is not same as your
> explanation. The connection in nginx_ssl_fe frontend stats are more than
> double of what nginx can handle. Question is when nginx can just do 32k
> connections, how can the nginx_ssl_fe frontend accept 77k connections?
>

Seems you are forgeting haproxy does queuing instead of dropping the
frontend connections. Maybe samples of your log can be useful to see the
requests timings.

>
> --
> Ayush Goyal
>

Reply via email to