For nbproc= 1, pinned to cpu 0, the drop in RPS/BW is both 45% for 16K I/O size. aes-ni is enabled for this processor (ssl speed test shows performance is about 5x).
Pidstat shows that cpu is heavily used by haproxy during SSL test: Average UID PID %usr %system %guest %CPU CPU Command 80: 110 6237 12.95 86.41 0.00 99.36 - haproxy 443: 110 6237 43.76 56.07 0.00 99.83 - haproxy mpstat: Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 80: 0 13.09 0.00 43.77 0.00 0.00 43.14 0.00 0.00 0.00 0.00 443: 0 43.85 0.00 30.17 0.00 0.00 25.98 0.00 0.00 0.00 0.00 Thanks, - Krishna Kumar On Thu, May 21, 2015 at 11:31 AM, Krishna Kumar (Engineering) < [email protected]> wrote: > Hi all, > > I am getting a big performance hit with SSL termination for small I/O, and > errors > when testing with bigger I/O sizes (ab version is 2.3): > > 1. Non-SSL vs SSL for small I/O (128 bytes): > ab -k -n 1000000 -c 500 http://<HAPROXY>/128 > > RPS: 181763.65 vs 133611.69 - 27% drop > BW: 63546.28 vs 46711.90 - 27% drop > > 2. Non-SSL vs SSL for medium I/O (16 KB): > ab -k -n 1000000 -c 500 http://<HAPROXY>/16K > > RPS: 62646.13 vs 21876.33 (fails mostly with 70007 error as > below) - 65% drop > BW: 1016531.41 vs 354977.59 (fails mostly with 70007 error) > - 65% drop > > 3. Non-SSL vs SSL for large I/O (128 KB): > ab -k -n 100000 -c 500 http://<HAPROXY>/128K > > RPS: 8476.99 vs "apr_poll: The timeout specified has expired > (70007)" > BW: 1086983.11 vs same error, this happens after 90000 requests > (always reproducible). > > ----------------------------------- HAProxy Build info > ------------------------------------- > HA-Proxy version 1.5.12 2015/05/02 > Copyright 2000-2015 Willy Tarreau <[email protected]> > > Build options : > TARGET = linux2628 > CPU = native > CC = gcc > CFLAGS = -O3 -march=native -g -fno-strict-aliasing > OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_TFO=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 > > Encrypted password support via crypt(3): yes > Built with zlib version : 1.2.8 > Compression algorithms supported : identity, deflate, gzip > Built with OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015 > Running on OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports prefer-server-ciphers : yes > Built with PCRE version : 8.35 2014-04-04 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built with transparent proxy support using: IP_TRANSPARENT > IPV6_TRANSPARENT IP_FREEBIND > > Available polling systems : > epoll : pref=300, test result OK > poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. > ------- Config file - even cpu cores are on 1st socket on the mb, odd cpus > are on 2nd -------- > global > daemon > maxconn 50000 > quiet > nbproc 6 > cpu-map 1 0 > cpu-map 2 2 > cpu-map 3 4 > cpu-map 4 6 > cpu-map 5 8 > cpu-map 6 10 > user haproxy > group haproxy > stats socket /var/run/haproxy.sock mode 600 level admin > stats timeout 2m > tune.bufsize 32768 > > userlist stats-auth > group admin users admin > user admin insecure-password admin > > defaults > mode http > maxconn 50000 > retries 3 > option forwardfor > option redispatch > option prefer-last-server > option splice-auto > > frontend www-http > bind-process 1 2 3 > bind *:80 > stats uri /stats > stats enable > acl AUTH http_auth(stats-auth) > acl AUTH_ADMIN http_auth(stats-auth) admin > stats http-request auth unless AUTH > default_backend www-backend > > frontend www-https > bind-process 4 5 6 > bind *:443 ssl crt /etc/ssl/private/haproxy.pem > reqadd X-Forwarded-Proto:\ https > default_backend www-backend-ssl > > backend www-backend > bind-process 1 2 3 > mode http > balance roundrobin > cookie FKSID prefix indirect nocache > server nginx-1 172.20.232.122:80 maxconn 25000 check > server nginx-2 172.20.232.125:80 maxconn 25000 check > > backend www-backend-ssl > bind-process 4 5 6 > mode http > balance roundrobin > cookie FKSID prefix indirect nocache > server nginx-1 172.20.232.122:80 maxconn 25000 check > server nginx-2 172.20.232.125:80 maxconn 25000 check > > --------------------------------------------------------------------------------------------------------------- > CPU is E5-2670, 48 core system, nic interrupts are pinned to correct > cpu's, etc. > Can someone suggest what change is required to get better results as well > as > fix the 70007 error, or share their config settings? The stats are also > captured. > For 128 byte, all 3 haproxy's are running, but for 16K, and for 128K, only > the last > haproxy is being used (and seen consistently): > > ---------------------- MPSTAT and PIDSTAT > --------------------------------- > 128 byte, port 80 > Average: CPU %usr %nice %sys %iowait %irq %soft %steal > %guest %gnice %idle > Average: 0 22.33 0.00 39.43 0.00 0.00 9.98 > 0.00 0.00 0.00 28.27 > Average: 2 22.00 0.00 33.56 0.00 0.00 15.11 > 0.00 0.00 0.00 29.33 > Average: 4 23.39 0.00 36.99 0.00 0.00 10.50 > 0.00 0.00 0.00 29.12 > > (First 3 haproxy's are used, last 3 are zero and not shown): > Average: UID PID %usr %system %guest %CPU CPU Command > Average: 110 5728 22.80 50.00 0.00 72.80 - haproxy > Average: 110 5729 22.20 48.60 0.00 70.80 - haproxy > Average: 110 5730 24.20 48.00 0.00 72.20 - haproxy > > 128 byte, port 443 > Average: CPU %usr %nice %sys %iowait %irq %soft %steal > %guest %gnice %idle > Average: 6 27.35 0.00 30.54 0.00 0.00 8.89 > 0.00 0.00 0.00 33.22 > Average: 8 30.16 0.00 31.43 0.00 0.00 11.27 > 0.00 0.00 0.00 27.14 > Average: 10 30.03 0.00 32.66 0.00 0.00 13.93 > 0.00 0.00 0.00 23.37 > > (First 3 haproxy's are not used, last 3 are shown): > Average: UID PID %usr %system %guest %CPU CPU Command > Average: 110 5731 28.29 39.86 0.00 68.14 - haproxy > Average: 110 5732 30.29 42.43 0.00 72.71 - haproxy > Average: 110 5733 29.71 45.86 0.00 75.57 - haproxy > > 16K, port 80 > Average: CPU %usr %nice %sys %iowait %irq %soft %steal > %guest %gnice %idle > Average: 0 9.69 0.00 25.37 0.00 0.00 9.62 > 0.00 0.00 0.00 55.31 > Average: 2 11.21 0.00 33.04 0.00 0.00 15.94 > 0.00 0.00 0.00 39.81 > Average: 4 12.16 0.00 35.48 0.00 0.00 21.10 > 0.00 0.00 0.00 31.26 > > (First 3 haproxy's are used, last 3 are zero and not shown): > Average: UID PID %usr %system %guest %CPU CPU Command > Average: 110 5728 9.37 31.79 0.00 41.16 - haproxy > Average: 110 5729 10.43 42.60 0.00 53.03 - haproxy > Average: 110 5730 11.12 49.72 0.00 60.84 - haproxy > > 16K, port 443 > Average: CPU %usr %nice %sys %iowait %irq %soft %steal > %guest %gnice %idle > Average: 10 43.75 0.00 30.39 0.00 0.00 25.50 > 0.00 0.00 0.00 0.36 > > (First 5 haproxy's have zero values, only last is shown) > Average: UID PID %usr %system %guest %CPU CPU Command > Average: 110 5733 43.63 55.78 0.00 99.41 - haproxy > > Thanks, > - Krishna Kumar >
