Sounds like ssl connections are not being reused between haproxy and tomcat. Can you send some netstat monitoring metrics showing tcp handshakes and time or close waits over time?
-- Benjamin Lee +61 4 16 BEN LEE > El 21 jun 2017, a las 17:15, Daniel Heitepriem <[email protected]> > escribió: > > Hi everyone, > > we got a problem recently which we can't explain to ourself. We got a java > application (Tomcat WAR-File) which has to handle several million of requests > per day and several thousand requests per second during peak times. Due to > this high amount we are splitting traffic using an ACL in "booking traffic" > and "availability traffic". Booking traffic is negligible but the > Availability traffic is load-balanced over several application servers. The > problem that occurs is that our external partner "floods" the > Availability-Frontend with several thousand requests per second and the > backend becomes unresponsive. If we redirect them directly to our > Tomcat-Instance via Firewall-Rules without passing through HAProxy everything > is fine. The Tomcat instances have "maxThreads=1024" and "acceptCount=500" as > their main connector settings so this shouldn't interfere with the HAProxy > configuration. > > Our HAProxy configuration running on Solaris 11 64-bit: > > HA-Proxy version 1.7.5 2017/04/03 > Copyright 2000-2017 Willy Tarreau <[email protected]> > > Build options : > TARGET = solaris > CPU = generic > CC = gcc > CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing > -Wdeclaration-after-statement -fomit-frame-pointer -DFD_SETSIZE=65536 > -D_REENTRANT > OPTIONS = USE_TPROXY=1 USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > Encrypted password support via crypt(3): yes > Built with zlib version : 1.2.8-T4mods > Running on zlib version : 1.2.11 > Compression algorithms supported : identity("identity"), deflate("deflate"), > raw-deflate("deflate"), gzip("gzip") > Running on OpenSSL version : OpenSSL 1.0.2k 26 Jan 2017 > Running on OpenSSL version : OpenSSL 1.0.2k 26 Jan 2017 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports prefer-server-ciphers : yes > Built with PCRE version : 8.39 2016-06-14 > Running on PCRE version : 8.39 2016-06-14 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built without Lua support > > Available polling systems : > poll : pref=200, test result OK > select : pref=150, test result OK > Total: 2 (2 usable), will use poll. > > Available filters : > [SPOE] spoe > [TRACE] trace > [COMP] compression > --- > global > log 127.0.0.1:514 local0 debug > daemon > maxconn 50000 > stats socket /opt/etc/haproxy/haproxy.sock mode 600 level admin > stats timeout 2m #Wait up to 2 minutes for input > tune.ssl.default-dh-param 2048 > ulimit-n 20000 > > > ssl-default-server-options no-sslv3 no-tls-tickets > ssl-default-bind-ciphers > EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:AES128+EECDH:AES128+EDH > > defaults > log global > option tcplog > log-format %f\ %b/%s\ client_ip:%ci\ client_port:%cp\ SSL_version:%sslv\ > SSL_cypher:%sslc\ %ts\ Tt:%Tt\ Tq:%Tq\ Tw:%Tw\ Tc:%Tc\ Tr:%Tr > mode http > timeout connect 5000 > timeout check 5000 > timeout client 30000 > timeout server 30000 > retries 3 > > frontend ndc > http-response set-header Strict-Transport-Security max-age=31536000;\ > includeSubdomains;\ preload > http-response set-header X-Content-Type-Options nosniff > > bind *:443 ssl crt /opt/etc/haproxy/domain_com.pem force-tlsv12 no-sslv3 > maxconn 20000 > > acl fare_availability path_beg /ndc/fare/v1/availability > acl flight_availability path_beg /ndc/flight/v1/availability > use_backend vakanz-backend if flight_availability or fare_availability > default_backend booking-backend > > backend booking-backend > server 10.2.8.28 10.2.8.23:8443 check ssl verify none minconn 500 maxconn > 500 > > backend vakanz-backend > server 10.2.8.28 10.2.8.28:8443 check ssl verify none minconn 500 maxconn > 500 > server 10.2.8.40 10.2.8.40:8443 check ssl verify none minconn 500 maxconn > 500 > server 10.2.8.41 10.2.8.41:8443 check ssl verify none minconn 500 maxconn > 500 > > Hopefully somebody can shed some light if we got a bad configuration and how > we could troubleshoot this issue. > > Thanks and regards, > Daniel
