Hi all,
I am getting a big performance hit with SSL termination for small I/O, and
errors
when testing with bigger I/O sizes (ab version is 2.3):
1. Non-SSL vs SSL for small I/O (128 bytes):
ab -k -n 1000000 -c 500 http://<HAPROXY>/128
RPS: 181763.65 vs 133611.69 - 27% drop
BW: 63546.28 vs 46711.90 - 27% drop
2. Non-SSL vs SSL for medium I/O (16 KB):
ab -k -n 1000000 -c 500 http://<HAPROXY>/16K
RPS: 62646.13 vs 21876.33 (fails mostly with 70007 error as below)
- 65% drop
BW: 1016531.41 vs 354977.59 (fails mostly with 70007 error)
- 65% drop
3. Non-SSL vs SSL for large I/O (128 KB):
ab -k -n 100000 -c 500 http://<HAPROXY>/128K
RPS: 8476.99 vs "apr_poll: The timeout specified has expired
(70007)"
BW: 1086983.11 vs same error, this happens after 90000 requests
(always reproducible).
----------------------------------- HAProxy Build info
-------------------------------------
HA-Proxy version 1.5.12 2015/05/02
Copyright 2000-2015 Willy Tarreau <[email protected]>
Build options :
TARGET = linux2628
CPU = native
CC = gcc
CFLAGS = -O3 -march=native -g -fno-strict-aliasing
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 USE_TFO=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015
Running on OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
------- Config file - even cpu cores are on 1st socket on the mb, odd cpus
are on 2nd --------
global
daemon
maxconn 50000
quiet
nbproc 6
cpu-map 1 0
cpu-map 2 2
cpu-map 3 4
cpu-map 4 6
cpu-map 5 8
cpu-map 6 10
user haproxy
group haproxy
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 2m
tune.bufsize 32768
userlist stats-auth
group admin users admin
user admin insecure-password admin
defaults
mode http
maxconn 50000
retries 3
option forwardfor
option redispatch
option prefer-last-server
option splice-auto
frontend www-http
bind-process 1 2 3
bind *:80
stats uri /stats
stats enable
acl AUTH http_auth(stats-auth)
acl AUTH_ADMIN http_auth(stats-auth) admin
stats http-request auth unless AUTH
default_backend www-backend
frontend www-https
bind-process 4 5 6
bind *:443 ssl crt /etc/ssl/private/haproxy.pem
reqadd X-Forwarded-Proto:\ https
default_backend www-backend-ssl
backend www-backend
bind-process 1 2 3
mode http
balance roundrobin
cookie FKSID prefix indirect nocache
server nginx-1 172.20.232.122:80 maxconn 25000 check
server nginx-2 172.20.232.125:80 maxconn 25000 check
backend www-backend-ssl
bind-process 4 5 6
mode http
balance roundrobin
cookie FKSID prefix indirect nocache
server nginx-1 172.20.232.122:80 maxconn 25000 check
server nginx-2 172.20.232.125:80 maxconn 25000 check
---------------------------------------------------------------------------------------------------------------
CPU is E5-2670, 48 core system, nic interrupts are pinned to correct cpu's,
etc.
Can someone suggest what change is required to get better results as well
as
fix the 70007 error, or share their config settings? The stats are also
captured.
For 128 byte, all 3 haproxy's are running, but for 16K, and for 128K, only
the last
haproxy is being used (and seen consistently):
---------------------- MPSTAT and PIDSTAT
---------------------------------
128 byte, port 80
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: 0 22.33 0.00 39.43 0.00 0.00 9.98 0.00
0.00 0.00 28.27
Average: 2 22.00 0.00 33.56 0.00 0.00 15.11 0.00
0.00 0.00 29.33
Average: 4 23.39 0.00 36.99 0.00 0.00 10.50 0.00
0.00 0.00 29.12
(First 3 haproxy's are used, last 3 are zero and not shown):
Average: UID PID %usr %system %guest %CPU CPU Command
Average: 110 5728 22.80 50.00 0.00 72.80 - haproxy
Average: 110 5729 22.20 48.60 0.00 70.80 - haproxy
Average: 110 5730 24.20 48.00 0.00 72.20 - haproxy
128 byte, port 443
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: 6 27.35 0.00 30.54 0.00 0.00 8.89 0.00
0.00 0.00 33.22
Average: 8 30.16 0.00 31.43 0.00 0.00 11.27 0.00
0.00 0.00 27.14
Average: 10 30.03 0.00 32.66 0.00 0.00 13.93 0.00
0.00 0.00 23.37
(First 3 haproxy's are not used, last 3 are shown):
Average: UID PID %usr %system %guest %CPU CPU Command
Average: 110 5731 28.29 39.86 0.00 68.14 - haproxy
Average: 110 5732 30.29 42.43 0.00 72.71 - haproxy
Average: 110 5733 29.71 45.86 0.00 75.57 - haproxy
16K, port 80
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: 0 9.69 0.00 25.37 0.00 0.00 9.62 0.00
0.00 0.00 55.31
Average: 2 11.21 0.00 33.04 0.00 0.00 15.94 0.00
0.00 0.00 39.81
Average: 4 12.16 0.00 35.48 0.00 0.00 21.10 0.00
0.00 0.00 31.26
(First 3 haproxy's are used, last 3 are zero and not shown):
Average: UID PID %usr %system %guest %CPU CPU Command
Average: 110 5728 9.37 31.79 0.00 41.16 - haproxy
Average: 110 5729 10.43 42.60 0.00 53.03 - haproxy
Average: 110 5730 11.12 49.72 0.00 60.84 - haproxy
16K, port 443
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: 10 43.75 0.00 30.39 0.00 0.00 25.50 0.00
0.00 0.00 0.36
(First 5 haproxy's have zero values, only last is shown)
Average: UID PID %usr %system %guest %CPU CPU Command
Average: 110 5733 43.63 55.78 0.00 99.41 - haproxy
Thanks,
- Krishna Kumar
