Hi Jonathan
Try increasing the number or transmit channels:
ethtool -L eno1 rx 4 tx 4
And reduce the number of udp-workers
Daniel
On 10/8/20 2:12 AM, Jonathan Hewlett wrote:
Hi Danel,
I'm sorry for the late reply; I've been on holiday/vacation and
playing catch-up! ;)
ethtool:
Channel parameters for eno1:
Pre-set maximums:
RX: 4
TX: 4
Other: 0
Combined: 0
Current hardware settings:
RX: 4
TX: 1
Other: 0
Combined: 0
knotc output:
Knot DNS 2.9.5
Target: linux-gnu x86_64
Compiler: gcc
CFLAGS: -O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection -DNDEBUG
-Wno-unused -Wall -Wshadow -Werror=format-security
-Werror=implicit -Wstrict-prototypes
LIBS: -Wl,-z,relro -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld
LibURCU: -lurcu
GnuTLS: -lgnutls -I/usr/include/p11-kit-1
Libedit: -ledit -I/usr/include/editline
LMDB: shared -llmdb
Config: 500 MiB default mapsize
Prefix: /usr
Run dir: /run/knot
Storage dir: /var/lib/knot
Config dir: /etc/knot
Module dir:
Static modules: cookies dnsproxy dnstap geoip noudp onlinesign
queryacl rrl stats synthrecordwhoami
Shared modules:
Knot DNS libraries: yes
Knot DNS daemon: yes
Knot DNS utilities: yes
Knot DNS documentation: yes
Use recvmmsg: yes
Use SO_REUSEPORT(_LB): yes
Memory allocator: auto
Fast zone parser: yes
Utilities with IDN: libidn2
Utilities with Dnstap: yes
MaxMind DB support: yes
Systemd integration: yes
POSIX capabilities: yes
PKCS #11 support: yes
Ed25519 support: yes
Ed448 support: no
Code coverage: no
Sanitizer: no
LibFuzzer: no
OSS-Fuzz: no
Continued thanks! ;)
Jonathan
On Thu, 24 Sep 2020 at 11:14, Daniel Salzman
<[email protected] <mailto:[email protected]>> wrote:
Hi Jonathan,
If you have just one 1GbE port, I would recommend reducing the
number of udp-workers. What about 12 or less?
How do you install Knot DNS (repository or source code)?
What do say these commands?
# sudo ethtool -l eno1
# knotc status configure
Daniel
On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
Hi Daniel
* Server listens: 967 (multiple v4 and v6 networks)
* Physical server
* NIC details
*-network:0
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet
PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:02:00.0
logical name: eno1
version: 00
serial: b0:83:fe:c8:f3:ec
size: 1Gbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master
cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt
1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes
driver=tg3 driverversion=3.137 duplex=full firmware=FFV21.60.2 bc
5720-v1.39 ip=38.88.50.50 latency=0 link=yes multicast=yes
port=twisted pair
speed=1Gbit/s
resources: irq:16 memory:d80a0000-d80affff
memory:d80b0000-d80bffff memory:d80c0000-d80cffff
memory:dd800000-dd83ffff
Thanks for the continuing support
Jonathan
On Wed, 23 Sep 2020 at 12:47, Daniel Salzman
<[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
wrote:
Jonathan,
there must be something wrong with the networking. How
many `server.listen` configuration values
do yo have? Is it a physical server or a virtual one?
What is the network card used?
Daneil
On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
> Daniel
> Thank you for your time
>
> # knotc status workers
> UDP workers: 24, TCP workers 24, background workers:
10 (running: 0, pending: 0)
>
> image.png
>
> Kind regards
> Jonathan
>
>
> On Wed, 23 Sep 2020 at 07:59, <[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>> <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>>> wrote:
>
> Hi Jonathan,
>
> On 2020-09-22 23:11, Jonathan Hewlett wrote:
> > Hi,
> >
> > We run around 50 knot servers around the world,
with great success -
> > thanks for the software!
>
> Thanks for this feedback :-)
>
> >
> > We are just a little curious about seemingly
high CPU usage, and how
> > best to interpret the results, if someone could
advise? Specifically,
> > we are seeing the majority of the cpu time in
'system' and not 'user',
> > what would that suggest is happening perhaps?
>
> Could you provide me the output from `knotc status
workers`?
>
> I like htop. Are you able to send me a screenshot
when running htop? :-)
> [Don't forget to configure:
> F2
> Display options->Detailed CPU time
> (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
> F10
> ]
>
> Best,
> Daniel
>
> >
> > Any tuning help gratefully received!
> >
> > Our platform (sample)
> >
> > 24 Core CPU E5-2440 0 @ 2.40
> > 64Gb RAM
> > SSD
> >
> > knot-2.9.5, running on Centos 8.2
> >
> > Currently receiving around 4k pps
> >
> > Server is authoritative for a 278 zones, the
largest zone file being
> > around 600Mb
> >
> > Output of top
> >
> > top - 20:50:38 up 6 days, 4:49, 2 users, load
average: 13.47,
> > 12.34, 11.83
> > Tasks: 321 total, 2 running, 318 sleeping, 0
stopped, 1 zombie
> > %Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id,
0.3 wa, 0.7 hi, 0.7
> > si, 0.0 st
> > %Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id,
0.0 wa, 0.3 hi, 0.7
> > si, 0.0 st
> > %Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id,
0.0 wa, 0.7 hi, 0.3
> > si, 0.0 st
> > %Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id,
0.0 wa, 0.3 hi, 0.3
> > si, 0.0 st
> > %Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > %Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id,
0.0 wa, 0.3 hi, 0.3
> > si, 0.0 st
> > %Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > %Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > %Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id,
0.0 wa, 0.7 hi, 0.0
> > si, 0.0 st
> > %Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > %Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id,
0.0 wa, 0.7 hi, 0.0
> > si, 0.0 st
> > %Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > %Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id,
0.0 wa, 0.7 hi, 0.3
> > si, 0.0 st
> > %Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > %Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id,
0.0 wa, 0.7 hi, 3.7
> > si, 0.0 st
> > %Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id,
0.0 wa, 0.3 hi, 0.3
> > si, 0.0 st
> > %Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id,
0.0 wa, 1.0 hi, 4.0
> > si, 0.0 st
> > %Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > %Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id,
0.0 wa, 0.7 hi, 4.0
> > si, 0.0 st
> > %Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id,
0.0 wa, 0.3 hi, 0.3
> > si, 0.0 st
> > %Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id,
0.0 wa, 0.7 hi, 1.3
> > si, 0.0 st
> > %Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id,
0.0 wa, 0.3 hi, 0.3
> > si, 0.0 st
> > %Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id,
0.0 wa, 0.7 hi, 3.7
> > si, 0.0 st
> > %Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id,
0.0 wa, 0.3 hi, 0.0
> > si, 0.0 st
> > MiB Mem : 64107.4 total, 46764.9 free,
11733.6 used, 5608.8
> > buff/cache
> > MiB Swap: 32248.0 total, 32248.0 free,
0.0 used. 51670.0 avail
> > Mem
> >
> > PID USER PR NI VIRT RES SHR S
%CPU %MEM TIME+
> > COMMAND
> > 213370 knot 20 0 34.0g 9.6g 41784 S
644.9 15.3 1319:12
> > /usr/sbin/knotd
> >
> > Thanks again for this great product.
> >
> > Jonathan
>
>
>
> --
> --\--
> Jonathan Hewlett
> ^Senior\ Systems\ Engineer\n$
> CentralNic Group plc (LSE:CNIC)
> https://www.centralnic.com
<https://www.centralnic.com/>
> Tel: +44.7548243020
> CentralNic Group plc is a company registered in
England and Wales with company number 8576358. Registered Offices:
Saddlers House, Gutter Lane, London EC2V 6AE.
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and
Wales with company number 8576358. Registered Offices: Saddlers
House, Gutter Lane, London EC2V 6AE.
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales
with company number 8576358. Registered Offices: Saddlers House,
Gutter Lane, London EC2V 6AE.
