Sorry, replace httpclose with http-server-close On 21 Jun 2017 7:55 pm, "Igor Cicimov" <[email protected]> wrote:
> Yes saw it but too late. Anyway according to the timers the Tr:26040 means > it took 26 seconds for the server to send the response. Any errors in the > backend logs? > > client_ip:193.XX.XX.XXX client_port:18935 SSL_version:TLSv1.2 > SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:26150 Tq:106 Tw:0 Tc:3 Tr:26040 > > > Try adding: > > option httpclose > > in the backend and see if that helps. > > On 21 Jun 2017 7:48 pm, "Daniel Heitepriem" <[email protected]> > wrote: > > Hi Igor, > > the config is set to "mode http" (see below) only the log output is set to > "tcplog" to be able to get a more detailed log output. Please correct me if > I'm wrong but regarding to the config HTTP-mode is (or at least should be) > used. > > > defaults > log global > option tcplog > log-format %f\ %b/%s\ client_ip:%ci\ client_port:%cp\ > SSL_version:%sslv\ SSL_cypher:%sslc\ %ts\ Tt:%Tt\ Tq:%Tq\ Tw:%Tw\ Tc:%Tc\ > Tr:%Tr > mode http > timeout connect 5000 > timeout check 5000 > timeout client 30000 > timeout server 30000 > retries 3 > > frontend ndc > http-response set-header Strict-Transport-Security max-age=31536000;\ > includeSubdomains;\ preload > http-response set-header X-Content-Type-Options nosniff > > bind *:443 ssl crt /opt/etc/haproxy/domain_com.pem force-tlsv12 > no-sslv3 > maxconn 20000 > > acl fare_availability path_beg /ndc/fare/v1/availability > acl flight_availability path_beg /ndc/flight/v1/availability > use_backend vakanz-backend if flight_availability or fare_availability > default_backend booking-backend > > backend booking-backend > server 10.2.8.28 10.2.8.23:8443 check ssl verify none minconn 500 > maxconn 500 > > backend vakanz-backend > server 10.2.8.28 10.2.8.28:8443 check ssl verify none minconn 500 > maxconn 500 > server 10.2.8.40 10.2.8.40:8443 check ssl verify none minconn 500 > maxconn 500 > server 10.2.8.41 10.2.8.41:8443 check ssl verify none minconn 500 > maxconn 500 > > Regards, > Daniel > > Am 21.06.17 um 11:37 schrieb Igor Cicimov: > > > > On 21 Jun 2017 6:34 pm, "Daniel Heitepriem" <[email protected]> > wrote: > > Nothing special. No errors, no dropped connections just an increased > server response time (Tr). An excerpt from low and high traffic times is > below: > > Jun 20 18:05:29 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.28 > client_ip:193.XX.XX.XXX client_port:50876 SSL_version:TLSv1.2 > SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:157 Tq:95 Tw:0 Tc:2 Tr:60 > Jun 20 18:05:29 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.41 > client_ip:193.XX.XX.XXX client_port:32910 SSL_version:TLSv1.2 > SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:148 Tq:82 Tw:0 Tc:1 Tr:65 > Jun 20 18:05:30 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.40 > client_ip:193.XX.XX.XXX client_port:51077 SSL_version:TLSv1.2 > SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:525 Tq:312 Tw:0 Tc:2 Tr:211 > > Jun 20 22:05:36 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.28 > client_ip:193.XX.XX.XXX client_port:48936 SSL_version:TLSv1.2 > SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:25368 Tq:101 Tw:0 Tc:3 Tr:25264 > Jun 20 22:05:36 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.41 > client_ip:193.XX.XX.XXX client_port:43030 SSL_version:TLSv1.2 > SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:23474 Tq:88 Tw:0 Tc:2 Tr:23383 > Jun 20 22:05:36 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.40 > client_ip:193.XX.XX.XXX client_port:18935 SSL_version:TLSv1.2 > SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:26150 Tq:106 Tw:0 Tc:3 Tr:26040 > > > Am 21.06.17 um 10:21 schrieb Igor Cicimov: > > > > On 21 Jun 2017 6:11 pm, "Daniel Heitepriem" <[email protected]> > wrote: > > Hi Jarno, > > yes we are decrypting TLS on the frontend (official SSL-certificate) and > re-encrypt it before sending it to the backend (company policy so not that > easy to change it to an unencrypted connection). The CPU usage is not > higher than 15-20% even during peak times and the memory usage is also > quite low (200-800MB). > > Regards, > Daniel > > Am 21.06.17 um 10:00 schrieb Jarno Huuskonen: > > Hi, >> >> On Wed, Jun 21, Daniel Heitepriem wrote: >> >>> we got a problem recently which we can't explain to ourself. We got >>> a java application (Tomcat WAR-File) which has to handle several >>> million of requests per day and several thousand requests per second >>> during peak times. Due to this high amount we are splitting traffic >>> using an ACL in "booking traffic" and "availability traffic". >>> Booking traffic is negligible but the Availability traffic is >>> load-balanced over several application servers. The problem that >>> occurs is that our external partner "floods" the >>> Availability-Frontend with several thousand requests per second and >>> the backend becomes unresponsive. If we redirect them directly to >>> >> Looks like you're decrypting tls/ssl on frontend and then >> re-encrypting on backend/server. Is one core(you're not using nbproc?) >> able to handle thousand ssl requests coming in and going out ? >> (is haproxy process using 100% cpu). >> >> -Jarno >> >> > What do you see in the haproxy log when the problem happens? > > Daniel, if using ssl to the backends shouldn't you use http mode? Per your > config you are using tcp which is default one. Afaik tcp is for ssl > passthrough. > > >
