Hi all,
haproxy is used for http and https load balancing with TLS termination
on haproxy side.
I'm using openbsd -stable on this box. I got CPU saturated with
250Mbps traffic in/out summary on frontend NICs and 3000 ESTABLISHED
connections on frontent interface to haproxy.
# all connections to haproxy
lsof -ni | grep -i haproxy | wc -l
3683
# established connections on frontend bind IP address
lsof -ni | grep -i haproxy | grep ESTABLISHED | grep xxx.xxx.xxx | wc -l
3041
It was 99% cpu usage when I've used SP kernel (single CPU) and no
nbproc in config, so I've switched to MP (multiprocessor) kernel and
enabled nbproc 4.
>From top output memory usage is quite low, but there is 57% CPU
interrupt on CPU0:
$ top
load averages: 4.01, 3.95, 3.82
29 processes: 1 running, 24 idle, 4 on processor
CPU0 states: 16.8% user, 0.0% nice, 8.4% system, 57.3% interrupt, 17.6% idle
CPU1 states: 29.0% user, 0.0% nice, 35.6% system, 0.6% interrupt, 34.8% idle
CPU2 states: 30.9% user, 0.0% nice, 32.9% system, 0.6% interrupt, 35.5% idle
CPU3 states: 23.8% user, 0.0% nice, 36.3% system, 0.6% interrupt, 39.3% idle
Memory: Real: 773M/1389M act/tot Free: 14G Cache: 81M Swap: 0K/16G
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
8147 _haproxy 61 0 240M 83M run - 8:12 69.38% haproxy
19935 _haproxy 61 0 241M 85M onproc - 7:53 69.19% haproxy
22974 _haproxy 60 0 235M 78M onproc - 6:38 64.55% haproxy
10729 _haproxy 61 0 228M 71M onproc - 6:50 61.67% haproxy
Current CPU - E5-2609 v2 - 4 core.
cpu0: Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz, 2500.38 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS
I've built haproxy from source as in openbsd's packages there is only
1.4 available.
As you can see haproxy -vv show it's built with no PCRE JIT support,
but I've also tried to use complied version with PCRE JIT support
enabled - I didn't noticed any imporvements during testing.
$ haproxy -vv
HA-Proxy version 1.5.11-7 2015/03/17
Copyright 2000-2015 Willy Tarreau <[email protected]>
Build options :
TARGET = openbsd
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): no
Built with zlib version : 1.2.3
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : LibreSSL 2.0
Running on OpenSSL version : LibreSSL 2.0
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : no (libpcre build without JIT?)
Built with transparent proxy support using: SO_BINDANY
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
$ cat /etc/haproxy/haproxy.cfg
global
log loghost
maxconn 600000
chroot /var/haproxy
uid 604
gid 604
daemon
#debug
nbproc 4
#quiet
pidfile /var/run/haproxy.pid
ssl-default-bind-options no-sslv3 no-tls-tickets
ssl-default-bind-ciphers
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
tune.ssl.cachesize 200000
tune.ssl.lifetime 7200
tune.ssl.default-dh-param 1024
tune.bufsize 16384
tune.maxrewrite 1024
defaults
log global
mode http
option httplog
option forwardfor
option http-server-close
option dontlognull
option redispatch
retries 2
maxconn 300000
stats enable
stats uri /haproxy?stats
stats auth admin:mysecurepassword
timeout connect 5s
timeout client 5s
timeout server 120s
timeout queue 60s
timeout http-request 10s
timeout http-keep-alive 1s
default-server inter 5s fall 8 rise 2 slowstart 5s
frontend ft-https
bind xxxxxx:443 ssl crt /etc/ssl/haproxy/domain1.com crt
/etc/ssl/haproxy/domain2.com crt /etc/ssl/haproxy/domain3.com
reqadd X-Forwarded-Proto:\ https
compression algo gzip
compression type text/html text/plain
acl static_content path_end .jpg .gif .png .ico .otf .woff
.eot .ttf .css .js
acl varnish_available nbsrv(bk-varnish) ge 1
acl acl-domain1 hdr(Host) -i domain1.com
acl acl-domain2 hdr(Host) -i domain2.com
acl acl-domain2 hdr(Host) -i domain3.com
use_backend bk-varnish if varnish_available static_content
use_backend bk-domain1-https if acl-domain1
use_backend bk-domain2-https if acl-domain2
default_backend bk-domain3-https
frontend ft-http
bind xxxxxx:80
reqadd X-Forwarded-Proto:\ http
compression algo gzip
compression type text/html text/plain
acl static_content path_end .jpg .gif .png .ico .otf .woff
.eot .ttf .css .js
acl varnish_available nbsrv(bk-varnish) ge 1
acl acl-domain1 hdr(Host) -i domain1.com
acl acl-domain2 hdr(Host) -i domain2.com
acl acl-domain2 hdr(Host) -i domain3.com
use_backend bk-varnish if varnish_available static_content
use_backend bk-domain1-http if acl-domain1
use_backend bk-domain2-http if acl-domain2
default_backend bk-domain3-http
backend bk-varnish
balance roundrobin
option httpchk HEAD /varnishcheck
http-check expect status 200
server cache0 192.168.0.101:6081 check inter 2s fall 2 rise 2
slowstart 1s
server cache1 192.168.0.102:6081 check inter 2s fall 2 rise 2
slowstart 1s
backend bk-domain1-http
option httpchk HEAD /
cookie MYSESSID prefix
option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain1.com
http-check expect status 200
balance source
hash-type consistent
server server1 192.168.0.71:80 check
server server2 192.168.0.72:80 check
server server3 192.168.0.73:80 check
backend bk-domain1-https
option httpchk HEAD /
cookie MYSESSID prefix
option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain1.com
http-check expect status 200
balance source
hash-type consistent
server server1 192.168.0.71:443 check ssl verify none
server server2 192.168.0.72:443 check ssl verify none
server server3 192.168.0.73:443 check ssl verify none
backend bk-domain2-http
option httpchk HEAD /
cookie MYSESSID prefix
option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain2.com
http-check expect status 200
balance source
hash-type consistent
server server4 192.168.0.74:80 check
server server5 192.168.0.75:80 check
server server6 192.168.0.76:80 check
backend bk-domain2-https
option httpchk HEAD /
cookie MYSESSID prefix
option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain2.com
http-check expect status 200
balance source
hash-type consistent
server server4 192.168.0.74:443 check ssl verify none
server server5 192.168.0.75:443 check ssl verify none
server server6 192.168.0.76:443 check ssl verify none
backend bk-domain3-http
option httpchk HEAD /
cookie MYSESSID prefix
option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain3.com
http-check expect status 200
balance source
hash-type consistent
server server4 192.168.0.74:80 check
server server5 192.168.0.75:80 check
server server6 192.168.0.76:80 check
backend bk-domain3-https
option httpchk HEAD /
cookie MYSESSID prefix
option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain3.com
http-check expect status 200
balance source
hash-type consistent
server server4 192.168.0.74:443 check ssl verify none
server server5 192.168.0.75:443 check ssl verify none
server server6 192.168.0.76:443 check ssl verify none
Let me kindl ask how to find why haproxy causing, so high CPU usage.
--
--
With regards,
Eugene Sudyr
