Hi Jonathan Try increasing the number or transmit channels: ethtool -L eno1 rx 4 tx 4
And reduce the number of udp-workers Daniel On 10/8/20 2:12 AM, Jonathan Hewlett wrote: > Hi Danel, > I'm sorry for the late reply; I've been on holiday/vacation and playing > catch-up! ;) > > ethtool: > Channel parameters for eno1: > Pre-set maximums: > RX: 4 > TX: 4 > Other: 0 > Combined: 0 > Current hardware settings: > RX: 4 > TX: 1 > Other: 0 > Combined: 0 > > knotc output: > > Knot DNS 2.9.5 > > Target: linux-gnu x86_64 > Compiler: gcc > CFLAGS: -O2 -g -pipe -Wall -Werror=format-security > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions > -fstack-protector-strong -grecord-gcc-switches > -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 > -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic > -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -DNDEBUG > -Wno-unused -Wall -Wshadow -Werror=format-security -Werror=implicit > -Wstrict-prototypes > LIBS: -Wl,-z,relro -Wl,-z,now > -specs=/usr/lib/rpm/redhat/redhat-hardened-ld > LibURCU: -lurcu > GnuTLS: -lgnutls -I/usr/include/p11-kit-1 > Libedit: -ledit -I/usr/include/editline > LMDB: shared -llmdb > Config: 500 MiB default mapsize > > Prefix: /usr > Run dir: /run/knot > Storage dir: /var/lib/knot > Config dir: /etc/knot > Module dir: > > Static modules: cookies dnsproxy dnstap geoip noudp onlinesign queryacl > rrl stats synthrecordwhoami > Shared modules: > > Knot DNS libraries: yes > Knot DNS daemon: yes > Knot DNS utilities: yes > Knot DNS documentation: yes > > Use recvmmsg: yes > Use SO_REUSEPORT(_LB): yes > Memory allocator: auto > Fast zone parser: yes > Utilities with IDN: libidn2 > Utilities with Dnstap: yes > MaxMind DB support: yes > Systemd integration: yes > POSIX capabilities: yes > PKCS #11 support: yes > Ed25519 support: yes > Ed448 support: no > Code coverage: no > Sanitizer: no > LibFuzzer: no > OSS-Fuzz: no > > Continued thanks! ;) > > Jonathan > > On Thu, 24 Sep 2020 at 11:14, Daniel Salzman <[email protected] > <mailto:[email protected]>> wrote: > > Hi Jonathan, > > If you have just one 1GbE port, I would recommend reducing the number of > udp-workers. What about 12 or less? > > How do you install Knot DNS (repository or source code)? > > What do say these commands? > # sudo ethtool -l eno1 > # knotc status configure > > Daniel > > On 9/23/20 5:45 PM, Jonathan Hewlett wrote: > > Hi Daniel > > > > * Server listens: 967 (multiple v4 and v6 networks) > > * Physical server > > * NIC details > > *-network:0 > > description: Ethernet interface > > product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe > > vendor: Broadcom Inc. and subsidiaries > > physical id: 0 > > bus info: pci@0000:02:00.0 > > logical name: eno1 > > version: 00 > > serial: b0:83:fe:c8:f3:ec > > size: 1Gbit/s > > capacity: 1Gbit/s > > width: 64 bits > > clock: 33MHz > > capabilities: pm vpd msi msix pciexpress bus_master cap_list rom > ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd > autonegotiation > > configuration: autonegotiation=on broadcast=yes driver=tg3 > driverversion=3.137 duplex=full firmware=FFV21.60.2 bc 5720-v1.39 > ip=38.88.50.50 latency=0 link=yes multicast=yes port=twisted pair > > speed=1Gbit/s > > resources: irq:16 memory:d80a0000-d80affff > memory:d80b0000-d80bffff memory:d80c0000-d80cffff memory:dd800000-dd83ffff > > > > Thanks for the continuing support > > Jonathan > > > > > > On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > Jonathan, > > > > there must be something wrong with the networking. How many > `server.listen` configuration values > > do yo have? Is it a physical server or a virtual one? What is the > network card used? > > > > Daneil > > > > On 9/23/20 1:18 PM, Jonathan Hewlett wrote: > > > Daniel > > > Thank you for your time > > > > > > # knotc status workers > > > UDP workers: 24, TCP workers 24, background workers: 10 (running: > 0, pending: 0) > > > > > > image.png > > > > > > Kind regards > > > Jonathan > > > > > > > > > On Wed, 23 Sep 2020 at 07:59, <[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> <mailto:[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>>>> wrote: > > > > > > Hi Jonathan, > > > > > > On 2020-09-22 23:11, Jonathan Hewlett wrote: > > > > Hi, > > > > > > > > We run around 50 knot servers around the world, with great > success - > > > > thanks for the software! > > > > > > Thanks for this feedback :-) > > > > > > > > > > > We are just a little curious about seemingly high CPU > usage, and how > > > > best to interpret the results, if someone could advise? > Specifically, > > > > we are seeing the majority of the cpu time in 'system' and > not 'user', > > > > what would that suggest is happening perhaps? > > > > > > Could you provide me the output from `knotc status workers`? > > > > > > I like htop. Are you able to send me a screenshot when > running htop? :-) > > > [Don't forget to configure: > > > F2 > > > Display options->Detailed CPU time > > > (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest) > > > F10 > > > ] > > > > > > Best, > > > Daniel > > > > > > > > > > > Any tuning help gratefully received! > > > > > > > > Our platform (sample) > > > > > > > > 24 Core CPU E5-2440 0 @ 2.40 > > > > 64Gb RAM > > > > SSD > > > > > > > > knot-2.9.5, running on Centos 8.2 > > > > > > > > Currently receiving around 4k pps > > > > > > > > Server is authoritative for a 278 zones, the largest zone > file being > > > > around 600Mb > > > > > > > > Output of top > > > > > > > > top - 20:50:38 up 6 days, 4:49, 2 users, load average: > 13.47, > > > > 12.34, 11.83 > > > > Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, > 1 zombie > > > > %Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 > hi, 0.7 > > > > si, 0.0 st > > > > %Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 > hi, 0.7 > > > > si, 0.0 st > > > > %Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 > hi, 0.3 > > > > si, 0.0 st > > > > %Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 > hi, 0.3 > > > > si, 0.0 st > > > > %Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 > hi, 0.3 > > > > si, 0.0 st > > > > %Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 > hi, 0.3 > > > > si, 0.0 st > > > > %Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 > hi, 3.7 > > > > si, 0.0 st > > > > %Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 > hi, 0.3 > > > > si, 0.0 st > > > > %Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 > hi, 4.0 > > > > si, 0.0 st > > > > %Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > %Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 > hi, 4.0 > > > > si, 0.0 st > > > > %Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 > hi, 0.3 > > > > si, 0.0 st > > > > %Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 > hi, 1.3 > > > > si, 0.0 st > > > > %Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 > hi, 0.3 > > > > si, 0.0 st > > > > %Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 > hi, 3.7 > > > > si, 0.0 st > > > > %Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 > hi, 0.0 > > > > si, 0.0 st > > > > MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, > 5608.8 > > > > buff/cache > > > > MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. > 51670.0 avail > > > > Mem > > > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM > TIME+ > > > > COMMAND > > > > 213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 > 1319:12 > > > > /usr/sbin/knotd > > > > > > > > Thanks again for this great product. > > > > > > > > Jonathan > > > > > > > > > > > > -- > > > --\-- > > > Jonathan Hewlett > > > ^Senior\ Systems\ Engineer\n$ > > > CentralNic Group plc (LSE:CNIC) > > > https://www.centralnic.com <https://www.centralnic.com/> > > > Tel: +44.7548243020 > > > CentralNic Group plc is a company registered in England and Wales > with company number 8576358. Registered Offices: Saddlers House, Gutter Lane, > London EC2V 6AE. > > > > > > > > -- > > --\-- > > Jonathan Hewlett > > ^Senior\ Systems\ Engineer\n$ > > CentralNic Group plc (LSE:CNIC) > > https://www.centralnic.com <https://www.centralnic.com/> > > Tel: +44.7548243020 > > CentralNic Group plc is a company registered in England and Wales with > company number 8576358. Registered Offices: Saddlers House, Gutter Lane, > London EC2V 6AE. > > > > -- > --\-- > Jonathan Hewlett > ^Senior\ Systems\ Engineer\n$ > CentralNic Group plc (LSE:CNIC) > https://www.centralnic.com <https://www.centralnic.com/> > Tel: +44.7548243020 > CentralNic Group plc is a company registered in England and Wales with > company number 8576358. Registered Offices: Saddlers House, Gutter Lane, > London EC2V 6AE. -- https://lists.nic.cz/mailman/listinfo/knot-dns-users
