Hi all, haproxy is used for http and https load balancing with TLS termination on haproxy side.
I'm using openbsd -stable on this box. I got CPU saturated with 250Mbps traffic in/out summary on frontend NICs and 3000 ESTABLISHED connections on frontent interface to haproxy. # all connections to haproxy lsof -ni | grep -i haproxy | wc -l 3683 # established connections on frontend bind IP address lsof -ni | grep -i haproxy | grep ESTABLISHED | grep xxx.xxx.xxx | wc -l 3041 It was 99% cpu usage when I've used SP kernel (single CPU) and no nbproc in config, so I've switched to MP (multiprocessor) kernel and enabled nbproc 4. >From top output memory usage is quite low, but there is 57% CPU interrupt on CPU0: $ top load averages: 4.01, 3.95, 3.82 29 processes: 1 running, 24 idle, 4 on processor CPU0 states: 16.8% user, 0.0% nice, 8.4% system, 57.3% interrupt, 17.6% idle CPU1 states: 29.0% user, 0.0% nice, 35.6% system, 0.6% interrupt, 34.8% idle CPU2 states: 30.9% user, 0.0% nice, 32.9% system, 0.6% interrupt, 35.5% idle CPU3 states: 23.8% user, 0.0% nice, 36.3% system, 0.6% interrupt, 39.3% idle Memory: Real: 773M/1389M act/tot Free: 14G Cache: 81M Swap: 0K/16G PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND 8147 _haproxy 61 0 240M 83M run - 8:12 69.38% haproxy 19935 _haproxy 61 0 241M 85M onproc - 7:53 69.19% haproxy 22974 _haproxy 60 0 235M 78M onproc - 6:38 64.55% haproxy 10729 _haproxy 61 0 228M 71M onproc - 6:50 61.67% haproxy Current CPU - E5-2609 v2 - 4 core. cpu0: Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz, 2500.38 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS I've built haproxy from source as in openbsd's packages there is only 1.4 available. As you can see haproxy -vv show it's built with no PCRE JIT support, but I've also tried to use complied version with PCRE JIT support enabled - I didn't noticed any imporvements during testing. $ haproxy -vv HA-Proxy version 1.5.11-7 2015/03/17 Copyright 2000-2015 Willy Tarreau <w...@1wt.eu> Build options : TARGET = openbsd CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): no Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : LibreSSL 2.0 Running on OpenSSL version : LibreSSL 2.0 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.35 2014-04-04 PCRE library supports JIT : no (libpcre build without JIT?) Built with transparent proxy support using: SO_BINDANY Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. $ cat /etc/haproxy/haproxy.cfg global log loghost maxconn 600000 chroot /var/haproxy uid 604 gid 604 daemon #debug nbproc 4 #quiet pidfile /var/run/haproxy.pid ssl-default-bind-options no-sslv3 no-tls-tickets ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA tune.ssl.cachesize 200000 tune.ssl.lifetime 7200 tune.ssl.default-dh-param 1024 tune.bufsize 16384 tune.maxrewrite 1024 defaults log global mode http option httplog option forwardfor option http-server-close option dontlognull option redispatch retries 2 maxconn 300000 stats enable stats uri /haproxy?stats stats auth admin:mysecurepassword timeout connect 5s timeout client 5s timeout server 120s timeout queue 60s timeout http-request 10s timeout http-keep-alive 1s default-server inter 5s fall 8 rise 2 slowstart 5s frontend ft-https bind xxxxxx:443 ssl crt /etc/ssl/haproxy/domain1.com crt /etc/ssl/haproxy/domain2.com crt /etc/ssl/haproxy/domain3.com reqadd X-Forwarded-Proto:\ https compression algo gzip compression type text/html text/plain acl static_content path_end .jpg .gif .png .ico .otf .woff .eot .ttf .css .js acl varnish_available nbsrv(bk-varnish) ge 1 acl acl-domain1 hdr(Host) -i domain1.com acl acl-domain2 hdr(Host) -i domain2.com acl acl-domain2 hdr(Host) -i domain3.com use_backend bk-varnish if varnish_available static_content use_backend bk-domain1-https if acl-domain1 use_backend bk-domain2-https if acl-domain2 default_backend bk-domain3-https frontend ft-http bind xxxxxx:80 reqadd X-Forwarded-Proto:\ http compression algo gzip compression type text/html text/plain acl static_content path_end .jpg .gif .png .ico .otf .woff .eot .ttf .css .js acl varnish_available nbsrv(bk-varnish) ge 1 acl acl-domain1 hdr(Host) -i domain1.com acl acl-domain2 hdr(Host) -i domain2.com acl acl-domain2 hdr(Host) -i domain3.com use_backend bk-varnish if varnish_available static_content use_backend bk-domain1-http if acl-domain1 use_backend bk-domain2-http if acl-domain2 default_backend bk-domain3-http backend bk-varnish balance roundrobin option httpchk HEAD /varnishcheck http-check expect status 200 server cache0 192.168.0.101:6081 check inter 2s fall 2 rise 2 slowstart 1s server cache1 192.168.0.102:6081 check inter 2s fall 2 rise 2 slowstart 1s backend bk-domain1-http option httpchk HEAD / cookie MYSESSID prefix option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain1.com http-check expect status 200 balance source hash-type consistent server server1 192.168.0.71:80 check server server2 192.168.0.72:80 check server server3 192.168.0.73:80 check backend bk-domain1-https option httpchk HEAD / cookie MYSESSID prefix option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain1.com http-check expect status 200 balance source hash-type consistent server server1 192.168.0.71:443 check ssl verify none server server2 192.168.0.72:443 check ssl verify none server server3 192.168.0.73:443 check ssl verify none backend bk-domain2-http option httpchk HEAD / cookie MYSESSID prefix option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain2.com http-check expect status 200 balance source hash-type consistent server server4 192.168.0.74:80 check server server5 192.168.0.75:80 check server server6 192.168.0.76:80 check backend bk-domain2-https option httpchk HEAD / cookie MYSESSID prefix option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain2.com http-check expect status 200 balance source hash-type consistent server server4 192.168.0.74:443 check ssl verify none server server5 192.168.0.75:443 check ssl verify none server server6 192.168.0.76:443 check ssl verify none backend bk-domain3-http option httpchk HEAD / cookie MYSESSID prefix option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain3.com http-check expect status 200 balance source hash-type consistent server server4 192.168.0.74:80 check server server5 192.168.0.75:80 check server server6 192.168.0.76:80 check backend bk-domain3-https option httpchk HEAD / cookie MYSESSID prefix option httpchk HEAD / HTTP/1.1\r\nHost:\ www.domain3.com http-check expect status 200 balance source hash-type consistent server server4 192.168.0.74:443 check ssl verify none server server5 192.168.0.75:443 check ssl verify none server server6 192.168.0.76:443 check ssl verify none Let me kindl ask how to find why haproxy causing, so high CPU usage. -- -- With regards, Eugene Sudyr