Problem: The haproxy is saturated with just a few k of requests while ksoftirqd is at 100%
Hardware details: Product Name: PowerEdge R640 2XCPU: Version: Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz OS: Ubuntu 18.04.2 Kernel: 4.15.0-46-generic Kernel: nproc 40 Haproxy: haproxy -vv HA-Proxy version 1.9.6 2019/03/29 - https://haproxy.org/ Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019 Running on OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.5 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with PCRE version : 8.42 2018-03-20 Running on PCRE version : 8.42 2018-03-20 PCRE library supports JIT : no (USE_PCRE_JIT not set) Encrypted password support via crypt(3): yes Built with multi-threading support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTX side=FE|BE h2 : mode=HTTP side=FE <default> : mode=HTX side=FE|BE <default> : mode=TCP|HTTP side=FE|BE Available filters : [SPOE] spoe [COMP] compression [CACHE] cache [TRACE] trace CPU Cores: :~# cat /sys/bus/cpu/devices/cpu0/topology/core_siblings_list 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 :~# cat /sys/bus/cpu/devices/cpu1/topology/core_siblings_list 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 nproc 40 According to haproxy docs, it is recommended to set the network interfaces IRQ on the same CPU that haproxy is mapped to run on, but on different to cores. https://www.haproxy.com/documentation/hapee/1-5r2/configuration/system-tuning/ haproxy.cfg global nbproc 1 nbthread 10 # cpu-map auto:1/1-4 0-3 cpu-map odd 1-20 nproc 40 [cid:4396add8-c72d-4869-9953-1c10f3fad856] ab -k -r -c 100 -n 1000000 {} This is ApacheBench, Version 2.3 <$Revision: 1807734 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 192.168.226.104 (be patient) Server Software: nginx/1.15.6 Server Hostname: 192.168.226.104 Server Port: 8888 Document Path: / Document Length: 0 bytes Concurrency Level: 100 Time taken for tests: 41.252 seconds Complete requests: 1000000 Failed requests: 0 Keep-Alive requests: 992009 Total transferred: 160960045 bytes HTML transferred: 0 bytes Requests per second: 24241.00 [#/sec] (mean) Time per request: 4.125 [ms] (mean) Time per request: 0.041 [ms] (mean, across all concurrent requests) Transfer rate: 3810.38 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 5 Processing: 0 4 27.1 2 1042 Waiting: 0 4 27.1 2 1042 Total: 0 4 27.1 2 1042 Percentage of the requests served within a certain time (ms) 50% 2 66% 5 75% 6 80% 6 90% 8 95% 8 98% 9 99% 9 100% 1042 (longest request) Completed 100000 requests Completed 200000 requests Completed 300000 requests Completed 400000 requests Completed 500000 requests Completed 600000 requests Completed 700000 requests Completed 800000 requests Completed 900000 requests Completed 1000000 requests Finished 1000000 requests Thanks
