Lets start step-by-step
On Wed, 2019-04-03 at 16:56 +0200, Petr Špaček wrote:
> Hello,
>
> as you already found out it is complicated ;-)
>
> Linux kernel has its own magic algorithms to schedule work on
> multi-core/multi-socket/NUMA machines and it DNS benchmarking also very
> much depends on network card, its drivers etc.
>
> If we were going to fine-tune your setup we would have to go into details:
>
> What is your CPU architecture? Number of sockets, CPU in them etc.?
It's russian Elbrus CPU, i have a little info about it architecture. 4
socket motherboard with 8-core CPU at each socket.
elbrus01 ~/src/dnsperf # uname -a
Linux elbrus01 4.9.0-2.2-e8c #1 SMP Mon Nov 12 10:52:48 GMT 2018 e2k E8C
E8C-SWTX GNU/Linux
elbrus01 ~/src/dnsperf # cat /etc/mcst_version
4.0-rc2
elbrus01 ~/src/dnsperf #
> How is operating memory connected to CPUs?
> Is it NUMA?
I think it is NUMA. I can see some memory scew across numa nodes
elbrus01 ~/src/dnsperf # numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 32 33 34 35 36 37
38 39 48 49 50 51 52 53 54 55
cpubind: 0 1 2 3
nodebind: 0 1 2 3
membind: 0 1 2 3
elbrus01 ~/src/dnsperf # numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64433 MB
node 0 free: 63261 MB
node 1 cpus: 16 17 18 19 20 21 22 23
node 1 size: 64467 MB
node 1 free: 62363 MB
node 2 cpus: 32 33 34 35 36 37 38 39
node 2 size: 64467 MB
node 2 free: 63768 MB
node 3 cpus: 48 49 50 51 52 53 54 55
node 3 size: 64467 MB
node 3 free: 63811 MB
node distances:
node 0 1 2 3
0: 10 20 20 20
1: 20 10 20 20
2: 20 20 10 20
3: 20 20 20 10
elbrus01 ~/src/dnsperf #
> Do you have irqbalance enabled?
I try to use irqbalance on previous OperatinSystem based on 3.11 Linux
kernel. New one OS based on 4.9 Linux kernel has no irqbalance binary at
all.
> Have you somehow configured IRQ affinity?
> What is your network card (how many IO queues it has)?
Network card is Intel 540 10Gbe PCI card. PIC lines are connected to
CPU0 socket. NSD prefer all the IRQ bounded to CPU0, memory allocated at
numa-node0 AND workers bound to CPU0/CPU1. KNOT prefer IRQ spreaded
across all 4 CPUs and workers using all CPUs.
> Did you configure network card queues and other driver settings explicitly?
> etc.
I spread incoming UDP across all 32 RX-queues
elbrus01 ~/src/dnsperf # ethtool -N eth4 rx-flow-hash udp4 sdfn
And affine each net4 IRQ to its personal CPU core
elbrus01 ~/src/dnsperf # echo 00800000,00000000
> /proc/irq/120/smp_affinity
elbrus01 ~/src/dnsperf # echo 00400000,00000000
> /proc/irq/119/smp_affinity
elbrus01 ~/src/dnsperf # echo 00200000,00000000
> /proc/irq/118/smp_affinity
elbrus01 ~/src/dnsperf # echo 00100000,00000000
> /proc/irq/117/smp_affinity
elbrus01 ~/src/dnsperf # echo 00080000,00000000
> /proc/irq/116/smp_affinity
elbrus01 ~/src/dnsperf # echo 00040000,00000000
> /proc/irq/115/smp_affinity
elbrus01 ~/src/dnsperf # echo 00020000,00000000
> /proc/irq/114/smp_affinity
elbrus01 ~/src/dnsperf # echo 00010000,00000000
> /proc/irq/113/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000080,00000000
> /proc/irq/112/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000040,00000000
> /proc/irq/111/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000020,00000000
> /proc/irq/110/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000010,00000000
> /proc/irq/109/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000008,00000000
> /proc/irq/108/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000004,00000000
> /proc/irq/107/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000002,00000000
> /proc/irq/106/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000001,00000000
> /proc/irq/105/smp_affinity
elbrus01 ~/src/dnsperf # echo 00800000 > /proc/irq/104/smp_affinity
elbrus01 ~/src/dnsperf # echo 00400000 > /proc/irq/103/smp_affinity
elbrus01 ~/src/dnsperf # echo 00200000 > /proc/irq/102/smp_affinity
elbrus01 ~/src/dnsperf # echo 00100000 > /proc/irq/101/smp_affinity
elbrus01 ~/src/dnsperf # echo 00080000 > /proc/irq/100/smp_affinity
elbrus01 ~/src/dnsperf # echo 00040000 > /proc/irq/99/smp_affinity
elbrus01 ~/src/dnsperf # echo 00020000 > /proc/irq/98/smp_affinity
elbrus01 ~/src/dnsperf # echo 00010000 > /proc/irq/97/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000080 > /proc/irq/96/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000040 > /proc/irq/95/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000020 > /proc/irq/94/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000010 > /proc/irq/93/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000008 > /proc/irq/92/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000004 > /proc/irq/91/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000002 > /proc/irq/90/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000001 > /proc/irq/89/smp_affinity
A /proc/interrupt output is attached
The load are starts with
[nikor@kaa5 dnsperf]$ ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T72 -t
500 -S 1 -q 1000 -D
Different runs shows random numer of unused cores from 4 to 1. The
request-per-seconds changes accordingly. Less unused cores get more
performance.
>
>
> Fine tunning always has to take into account your specific environment
> and it is hard to provide general advice.
>
> If you find specific reproducible problem please report it to our Gitlab:
> https://gitlab.labs.nic.cz/knot/knot-dns/issues/
>
> Please understand that amount of time and hardware we can allocate for
> free support is limited. In case you require fine-tunning for your
> specific deployment please consider byuing professional support:
> https://www.knot-dns.cz/support/
>
It is reproducible case but in very specific environment. The overall
result may be good enought for production usage. At this case
professional support will be good option.
Myself interested in such strange behavior. Hope this case may be
usefull for you to. I will resend this thread to OS developers. May be
they can clear this issue.
Thank you for you attention.
> Thank you for understanding.
> Petr Špaček @ CZ.NIC
>
>
> On 03. 04. 19 16:37, Sergey Petrov wrote:
> > I reverse the client and the server. So the server now is 36-cores intel
> > box (72 HT-core)
> >
> > Starting with small loads i see knot use lower cores except core-0.
> > When adding more load i see cores 0-17 AND 37-54 are used but not to
> > 100% level. At maximum load i see all cores are about 100% used.
> >
> > It seems to me as system scheduler feature. First it starts with lower
> > number cores, then add cores from second CPU socket and after all
> > HT-cores.
> >
> > On Wed, 2019-04-03 at 12:53 +0300, Sergey Petrov wrote:
> >> On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
> >>> On 03. 04. 19 10:45, Sergey Petrov wrote:
> >>>> I perfoms benchmarks with knot-dns as a authoritative server and dnsperf
> >>>> as a workload client. Knot server has 32 cores. Interrupts from 10Gb
> >>>> network card are spreaded across all 32 cores. Knot configured with
> >>>> 64 udp-workers. Each knot thread assigned to one core. So there are at
> >>>> least two knot threads assigned to one core. Then i start dnsperf with
> >>>> command
> >>>>
> >>>> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
> >>>>
> >>>> htop on knot server shows 3-4 cores completly unused. Then i restart
> >>>> dnsperf unused cores are changes.
> >>>>
> >>>> That is the reason for unused core?
> >>>
> >>> Well, sometimes dnsperf is too slow :-)
> >>>
> >>> I recommend to check this:
> >>> - Make sure dnsperf ("source machine") is not 100 % utilized.
> >>> - Try to increase number of sockets used by dnsperf, i.e. -c parameter.
> >>> I would try also values like 500 and 1000 to see if it makes any
> >>> difference. It might change results significantly because Linux kernel
> >>> is using hashes over some packet fields and low number of sockets might
> >>> result in uneven query distribution.
> >>>
> >>> Please let us know what are your new results.
> >>>
> >>
> >> The source machne is about 15% utilized.
> >>
> >> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
> >>
> >> get us some performance penalty (260000 rps VS 310000 rps) and more even
> >> distribution across all cores with 100% usages of all eight cores on
> >> last CPU socket. While other CPU socket cores are aproximately 60%
> >> loaded.
> >>
> >> Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
> >> core load distribution and even more performance penalty.
> >>
> >> Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
> >> utilized cores and 2 50% utilized cores with about 300000 rps
> >>
> >> The question is that prevents knot thread on 0% used core to serve a
> >> packet arrived with IRQ bounded to another core? May be you have some
> >> developer guide can answer this question?
> >>
>
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7 CPU16 CPU17 CPU18 CPU19 CPU20
CPU21 CPU22 CPU23 CPU32 CPU33 CPU34 CPU35
CPU36 CPU37 CPU38 CPU39 CPU48 CPU49 CPU50
CPU51 CPU52 CPU53 CPU54 CPU55
0: 31 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-edge timer
1: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi SCI
3: 2700517 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi Elbrus arch Zilog
4: 213888 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi eth0
5: 788 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi snd_hda_intel:card0
6: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi mpv
9: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi mpv
11: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi ide0, ide1
12: 161 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1,
ohci_hcd:usb3
13: 34 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2,
ohci_hcd:usb4
20: 195908 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi ahci[0000:01:03.0]
21: 1787 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 IO-APIC-fasteoi ahci[0000:01:03.1]
27: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 l-gpio-irqchip ltc2954
89: 14395748 0 0 30118272 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 8952138 0 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-0
90: 1 13286118 26416897 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 9171734 0 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-1
91: 1 27871059 13117870 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 8171681 0 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-2
92: 26398861 0 0 11740709 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 8323524 0 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-3
93: 1 0 0 30236373 12497843 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 5904271 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-4
94: 1 0 30140677 0 0 12551876
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 5300404 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-5
95: 1 30074331 0 0 0 0
13605595 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 4739069 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-6
96: 29432217 0 0 0 0 0
0 13687641 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 7078862 0 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-7
97: 17 0 0 28751382 0 0
0 0 11090059 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 12707868 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-8
98: 17 0 27874284 0 0 0
0 0 0 13083807 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 13687042 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-9
99: 17 26923754 0 0 0 0
0 0 0 0 9481528 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 12285021 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-10
100: 29042746 0 0 0 0 0
0 0 0 0 0 11095915 0 0
0 0 0 0 0 0 0
0 0 0 0 0 12293319 0
0 0 0 0 PCI-MSI-edge eth4-TxRx-11
101: 17 0 0 25933504 0 0
0 0 0 0 0 0 9812438 0
0 0 0 0 0 0 0
0 0 0 0 0 0 14921381
0 0 0 0 PCI-MSI-edge eth4-TxRx-12
102: 17 0 25580205 0 0 0
0 0 0 0 0 0 0 10976437
0 0 0 0 0 0 0
0 0 0 0 0 0 15278564
0 0 0 0 PCI-MSI-edge eth4-TxRx-13
103: 17 24988997 0 0 0 0
0 0 0 0 0 0 0 0
12340097 0 0 0 0 0 0
0 0 0 0 0 0 17195659
0 0 0 0 PCI-MSI-edge eth4-TxRx-14
104: 26310501 0 0 0 0 0
0 0 0 0 0 0 0 0
0 13544016 0 0 0 0 0
0 0 0 0 0 0 18526995
0 0 0 0 PCI-MSI-edge eth4-TxRx-15
105: 1 0 0 2183339 0 0
0 16736991 16 0 0 0 0 0
0 0 8598342 0 0 0 0
0 0 0 0 0 0 0
32789328 0 0 0 PCI-MSI-edge eth4-TxRx-16
106: 1 0 2715688 0 0 0
17557390 0 0 16 0 0 0
0 0 0 0 8136211 0 0
0 0 0 0 0 0 0 0
37183179 0 0 0 PCI-MSI-edge eth4-TxRx-17
107: 1 2310137 0 0 0 16969845
0 0 0 0 16 0 0 0
0 0 0 0 9446285 0 0
0 0 0 0 0 0 0
37893681 0 0 0 PCI-MSI-edge eth4-TxRx-18
108: 1893123 0 0 0 18352458 0
0 0 0 0 0 16 0 0
0 0 0 0 0 6093145 0
0 0 0 0 0 0 0
31019388 0 0 0 PCI-MSI-edge eth4-TxRx-19
109: 1 0 0 2353333 0 0
0 19522164 0 0 0 0 16 0
0 0 0 0 0 0 8286390
0 0 0 0 0 0 0
0 31964719 0 0 PCI-MSI-edge eth4-TxRx-20
110: 1 0 2420839 0 0 0
16115261 0 0 0 0 0 0
16 0 0 0 0 0 0
0 7473971 0 0 0 0 0 0
0 33883922 0 0 PCI-MSI-edge eth4-TxRx-21
111: 1 2176816 0 0 0 18220845
0 0 0 0 0 0 0 0
16 0 0 0 0 0 0
0 9316648 0 0 0 0 0
0 36199326 0 0 PCI-MSI-edge eth4-TxRx-22
112: 3144495 0 0 0 18550231 0
0 0 0 0 0 0 0 0
0 16 0 0 0 0 0
0 0 9945592 0 0 0 0
0 31478678 0 0 PCI-MSI-edge eth4-TxRx-23
113: 17 0 0 394098 0 0
0 5504068 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 3902580 0 0 0
0 0 13347515 0 PCI-MSI-edge eth4-TxRx-24
114: 17 0 712281 0 0 0
5822978 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 5279164 0 0
0 0 11420128 0 PCI-MSI-edge eth4-TxRx-25
115: 17 1455063 0 0 0 6673171
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 4423364 0
0 0 15162524 0 PCI-MSI-edge eth4-TxRx-26
116: 1357335 0 0 0 6397831 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 4416212
0 0 7470693 0 PCI-MSI-edge eth4-TxRx-27
117: 17 0 0 1294331 0 0
0 7202345 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
4255754 0 0 9813127 PCI-MSI-edge eth4-TxRx-28
118: 17 0 458608 0 0 0
5554231 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 3459655 0 15054055 PCI-MSI-edge eth4-TxRx-29
119: 17 1074690 0 0 0 6492114
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 4024457 15067816 PCI-MSI-edge eth4-TxRx-30
120: 750590 0 0 0 6853673 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 14685946 PCI-MSI-edge eth4-TxRx-31
121: 2 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 PCI-MSI-edge eth4
NMI: 179 211 211 211 211 211
197 203 211 210 211 195 211
195 211 203 211 203 211 173
149 211 211 211 211 211 211
211 203 211 211 211 Non-maskable interrupts
LOC: 2690702 3577631 927608 3012448 836974 704647
812497 774994 1609967 982372 770319 731602 813070
967545 835225 752788 896413 848826 771334 855783
776418 734642 646440 1758086 900939 934732 794667
825147 775477 786385 757958 679243 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 Spurious interrupts
RTR: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 read retries
RES: 9579545 10116787 9048573 8935907 9537298 8470927
9759928 9262508 11704222 12630459 12087708 13072945 11166786
12615392 11867499 11799931 6984982 6250877 6944773 6048963
5924437 6658413 5835097 5803891 3867989 4046537 3793608
3170904 5306966 4504016 9682562 8266384 Rescheduling interrupts
CAL: 4294967183 31728 3315 2835 285 416
115 577 751 31 967 138 882
705 30 605 12035 1599 814 2225
6571 1401 713 1474 4184 4105 1776
159 376 291 893 503 Function call interrupts
TLB: 205797 621285 891373 900026 582659 525263
287189 83956 1406117 841357 557256 1478443 1295531
1960996 128262 1493340 294275 709719 478513 91296
119013 306136 308558 133126 80618 90392 1410058
557393 84746 83927 122797 90918 TLB shootdowns
v249: 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 RDMA interrupts
ERR: 0
MIS: 0
--
https://lists.nic.cz/cgi-bin/mailman/listinfo/knot-dns-users
