Hi, On Tue, Jul 07, 2015 at 01:24:28PM +0530, Krishna Kumar (Engineering) wrote: > Hi all, > > This is not related to haproxy, but I am having a performance issue with > number of > packets processed. I am running haproxy on a 48 core system (we have 64 > such servers > at present, which is going to increase for production tessting), where cpus > 0,2,4,6,..46 > are part of NUMA node 1, and cpus 1,3,5,7,.. 47 are part of NUMA node 2. > The systems > are running Debian 7, with 3.16.0-23 (kernel has both CONFIG_XPS and > CONFIG_RPS > enabled). nbproc is set to 12, and each haproxy is bound to cpus 0,2,4, ... > 22, so that > they are on the same socket, as seen here: > > # ps -efF | egrep "hap|PID" | cut -c1-80 > UID PID PPID C SZ RSS PSR STIME TTY TIME CMD > haproxy 3099 1 17 89697 324024 0 18:37 ? 00:11:19 haproxy > -f hap > haproxy 3100 1 18 87171 314324 2 18:37 ? 00:12:00 haproxy > -f hap > haproxy 3101 1 18 87214 305328 4 18:37 ? 00:12:00 haproxy > -f hap > haproxy 3102 1 19 89215 322676 6 18:37 ? 00:12:02 haproxy > -f hap > haproxy 3103 1 18 86788 310976 8 18:37 ? 00:11:59 haproxy > -f hap > haproxy 3104 1 18 87197 314888 10 18:37 ? 00:12:00 haproxy > -f hap > haproxy 3105 1 18 91311 319784 12 18:37 ? 00:11:59 haproxy > -f hap > haproxy 3106 1 18 88785 305576 14 18:37 ? 00:12:00 haproxy > -f hap > haproxy 3107 1 19 90366 326428 16 18:37 ? 00:12:09 haproxy > -f hap > haproxy 3108 1 19 89758 320780 18 18:37 ? 00:12:09 haproxy > -f hap > haproxy 3109 1 19 87670 314752 20 18:37 ? 00:12:07 haproxy > -f hap > haproxy 3110 1 19 87763 316672 22 18:37 ? 00:12:10 haproxy > -f hap > > set_irq_affinity.sh was run on the ixgbe card, and /proc/irq/*/smp_affinity > shows that each > irq is bound to cpus 0-47 correctly. However, I see that packets are being > processed on > cpus of the 2nd socket too, though user/system usage is zero on those as > haproxy does > not run on those cores.
That's normal if you pinned the IRQs to cpus 0-47, you'd like to pin IRQs only to the CPUs of the first socket (ie: 0, 2, 4, 6, ..., 46 if I understand it right). Also, double-check that you don't have irqbalance. It can change the settings in your back, that's really unpleasant. Willy
