Hello! My name is Zack, and I have been in the middle of an on going HAProxy issue that has me scratching my head.
Here is the setup: Our setup is hosted by amazon, and our HAProxy (1.6.3) boxes are in each region in 3 regions. We have 2 HAProxy boxes per region for a total of 6 proxy boxes. These boxes are routed information through route 53. Their entire job is to forward data from one of our clients to our database backend. It handles this absolutely fine, except between the hours of 7pm PST and 7am PST. During these hours, our route53 health checks time out thus causing the traffic to switch to the other HAProxy box inside of the same region. During the other 12 hours of the day, we receive 0 alerts from our health checks. I have noticed that we get a series of SSL handshake failures (though this happens throughout the entire day) that causes the server to hang for a second, thus causing the health checks to fail. During the day our SSL failures do not cause the server to hang long enough to go fail the checks, they only fail at night. I have attached my HAProxy config hoping that you guys have an answer for me. Lemme know if you need any more info. I have done a few tcpdump captures during the SSL handshake failures (not at night during it failing, but during the day when it still gets the SSL handshake failures, but doesn’t fail the health check) and it seems there is a d/c and a reconnect during the handshake. Here is my config, I will be running a tcpdump tonight to capture the packets during the failure and will attach it if you guys need more info. #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log 127.0.0.1 local2 pidfile /var/run/haproxy.pid maxconn 30000 user haproxy group haproxy daemon ssl-default-bind-options no-sslv3 no-tls-tickets tune.ssl.default-dh-param 2048 # turn on stats unix socket # stats socket /var/lib/haproxy/stats` #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog retries 3 timeout http-request 5s timeout queue 1m timeout connect 31s timeout client 31s timeout server 31s maxconn 15000 # Stats stats enable stats uri /haproxy?stats stats realm Strictly\ Private stats auth $StatsUser:$StatsPass #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend shared_incoming maxconn 15000 timeout http-request 5s # Bind ports of incoming traffic bind *:1025 accept-proxy # http bind *:1026 accept-proxy ssl crt /path/to/default/ssl/cert.pem ssl crt /path/to/cert/folder/ # https bind *:1027 # Health checking port acl gs_texthtml url_reg \/gstext\.html ## allow gs to do meta tag verififcation acl gs_user_agent hdr_sub(User-Agent) -i globalsign ## allow gs to do meta tag verififcation # Add headers http-request set-header $Proxy-Header-Ip %[src] http-request set-header $Proxy-Header-Proto http if !{ ssl_fc } http-request set-header $Proxy-Header-Proto https if { ssl_fc } # Route traffic based on domain use_backend gs_verify if gs_texthtml or gs_user_agent ## allow gs meta tag verification use_backend %[req.hdr(host),lower,map_dom(/path/to/map/file.map,unknown_domain)] # Drop unrecognized traffic default_backend unknown_domain #--------------------------------------------------------------------- # Backends #--------------------------------------------------------------------- backend server0 ## added to allow gs ssl meta tag verification reqrep ^GET\ /.*\ (HTTP/.*) GET\ /GlobalSignVerification\ \1 server server0_http server0.domain.com:80/GlobalSignVerification/ backend server1 server server1_http server1.domain.net:80 backend server2 server server2_http server2.domain.net:80 backend server3 server server3_http server3.domain.net:80 backend server4 server server4_http server4.domain.net:80 backend server5 server server5_http server5.domain.net:80 backend server6 server server6_http server6.domain.net:80 backend server7 server server7_http server7.domain.net:80 backend server8 server server8_http server8.domain.net:80 backend server9 server server9_http server9.domain.net:80 backend unknown_domain timeout connect 4s timeout server 4s errorfile 503 /etc/haproxy-shared/errors/404.html
haproxy.cfg
Description: haproxy.cfg
