Bug#987927: bind9: unreasonable resource use and slow startup with lots of IP addresses
On 11/05/21 02:30 PM, Ondřej Surý wrote: > Control: forwarded -1 > https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5012 > > Hi, > > coincidentally, I’ve been working (well, experimenting would be better word) > with > reducing the contention in the memory allocator and the first patch in the > branch > might help with the initialization time. Not so much with contention, for > that the > work on the branch will have to be complete (e.g. this will go into upstream > 9.18, > not 9.16), but I thought you might be interested in the work in progress. > > This particular branch is very fresh, but I have at least 3 or 4 different > approaches > with different experiments. As far as I understand part of this work has been merged in 9.16.25, see https://kb.isc.org/docs/bind-memory-consumption-explained Additionally, the memory allocator has been switched to jmealloc in 9.17.19 bind9 (1:9.17.19-2) unstable; urgency=medium * Add libjemalloc-dev to Build-Depends Russell, could you test again? Bernhard
Bug#987927: bind9: unreasonable resource use and slow startup with lots of IP addresses
Control: forwarded -1
https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5012
Hi,
coincidentally, I’ve been working (well, experimenting would be better word)
with
reducing the contention in the memory allocator and the first patch in the
branch
might help with the initialization time. Not so much with contention, for that
the
work on the branch will have to be complete (e.g. this will go into upstream
9.18,
not 9.16), but I thought you might be interested in the work in progress.
This particular branch is very fresh, but I have at least 3 or 4 different
approaches
with different experiments.
Ondrej
--
Ondřej Surý (He/Him)
ond...@sury.org
> On 2. 5. 2021, at 9:16, root wrote:
>
> Package: bind9
> Version: 1:9.16.13-1
> Severity: normal
>
> May 2 16:38:37 sjl named[7372]: listening on IPv4 interface lo, 127.0.0.1#53
> May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4,
> 10.0.2.45#53
> May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4,
> 10.0.40.1#53
> May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4,
> 10.0.40.2#53
> May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4,
> 10.0.40.3#53
> [...]
> May 2 16:39:33 sjl named[7372]: listening on IPv4 interface eno4,
> 10.0.47.0#53
> May 2 16:39:33 sjl named[7372]: listening on IPv4 interface eno4,
> 10.0.48.0#53
> May 2 16:39:33 sjl named[7372]: listening on IPv4 interface eno4,
> 10.0.49.0#53
> May 2 16:39:33 sjl named[7372]: listening on IPv6 interface lo, ::1#53
>
> On a system with 2560 extra IPv4 addresses for test purposes a default
> configuration of bind9 takes one minute on a reasonably fast 64bit system (two
> E5-2620 CPUs). See the above for example startup log entries.
>
> May 2 16:39:36 sjl named[7372]: zone localhost/IN: loaded serial 2
> May 2 16:39:36 sjl named[7372]: all zones loaded
> May 2 16:39:36 sjl named[7372]: running
> May 2 16:39:36 sjl named[7372]: socket: file descriptor exceeds limit
> (123273/21000)
> May 2 16:39:36 sjl named[7372]: managed-keys-zone: Unable to fetch DNSKEY
> set '.': not enough free resources
> May 2 16:39:36 sjl named[7372]: socket: file descriptor exceeds limit
> (123273/21000)
>
> Then the startup doesn't complete properly with errors like the above.
>
> OPTIONS="-u bind -S 15"
>
> Putting something like the above in /etc/default/named fixes the errors, but
> it still takes a long time and really 150,000 file handles shouldn't be
> required for 2560 IP addresses.
>
>listen-on { 10.0.2.45; };
>
> Putting the above in named.conf.options got it to work correctly in this
> regard. But I expect it to not use unreasonable amounts of resources without
> that configuration.
>
> -- System Information:
> Debian Release: bullseye/sid
> APT prefers testing
> APT policy: (500, 'testing')
> Architecture: amd64 (x86_64)
>
> Kernel: Linux 5.10.0-6-amd64 (SMP w/24 CPU threads)
> Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
> Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8),
> LANGUAGE=en_AU:en
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
> LSM: SELinux: enabled - Mode: Permissive - Policy name: default
>
> Versions of packages bind9 depends on:
> ii adduser3.118
> ii bind9-libs 1:9.16.13-1
> ii bind9-utils1:9.16.13-1
> ii debconf [debconf-2.0] 1.5.75
> ii dns-root-data 2021011101
> ii init-system-helpers1.60
> ii iproute2 5.10.0-4
> ii libc6 2.31-11
> ii libcap21:2.44-1
> ii libfstrm0 0.6.0-1+b1
> ii libjson-c5 0.15-2
> ii liblmdb0 0.9.24-1
> ii libmaxminddb0 1.5.2-1
> ii libprotobuf-c1 1.3.3-1+b2
> ii libssl1.1 1.1.1k-1
> ii libuv1 1.40.0-1
> ii libxml22.9.10+dfsg-6.3+b1
> ii lsb-base 11.1.0
> ii netbase6.3
> ii zlib1g 1:1.2.11.dfsg-2
>
> bind9 recommends no packages.
>
> Versions of packages bind9 suggests:
> pn bind-doc
> ii bind9-dnsutils [dnsutils] 1:9.16.13-1
> ii dnsutils 1:9.16.13-1
> pn resolvconf
> pn ufw
>
> -- Configuration Files:
> /etc/bind/named.conf.local changed:
> //
> // Do any local configuration here
> //
> // Consider adding the 1918 zones here, if they are not used in your
> // organization
> //include "/etc/bind/zones.rfc1918";
> //include "/etc/bind/named.conf.postal";
>
> /etc/bind/named.conf.options changed:
> options {
> directory "/var/cache/bind";
> // If there is a firewall between you and nameservers you want
> // to talk to, you may need to fix the firewall to allow multiple
> // ports to talk. See http://www.kb.cert.org/vuls/id/800113
> // If your ISP provided one or more IP addresses for stable
> // nameservers, you probably want
Bug#987927: bind9: unreasonable resource use and slow startup with lots of IP addresses
Package: bind9
Version: 1:9.16.13-1
Severity: normal
May 2 16:38:37 sjl named[7372]: listening on IPv4 interface lo, 127.0.0.1#53
May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4, 10.0.2.45#53
May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4, 10.0.40.1#53
May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4, 10.0.40.2#53
May 2 16:38:37 sjl named[7372]: listening on IPv4 interface eno4, 10.0.40.3#53
[...]
May 2 16:39:33 sjl named[7372]: listening on IPv4 interface eno4, 10.0.47.0#53
May 2 16:39:33 sjl named[7372]: listening on IPv4 interface eno4, 10.0.48.0#53
May 2 16:39:33 sjl named[7372]: listening on IPv4 interface eno4, 10.0.49.0#53
May 2 16:39:33 sjl named[7372]: listening on IPv6 interface lo, ::1#53
On a system with 2560 extra IPv4 addresses for test purposes a default
configuration of bind9 takes one minute on a reasonably fast 64bit system (two
E5-2620 CPUs). See the above for example startup log entries.
May 2 16:39:36 sjl named[7372]: zone localhost/IN: loaded serial 2
May 2 16:39:36 sjl named[7372]: all zones loaded
May 2 16:39:36 sjl named[7372]: running
May 2 16:39:36 sjl named[7372]: socket: file descriptor exceeds limit
(123273/21000)
May 2 16:39:36 sjl named[7372]: managed-keys-zone: Unable to fetch DNSKEY set
'.': not enough free resources
May 2 16:39:36 sjl named[7372]: socket: file descriptor exceeds limit
(123273/21000)
Then the startup doesn't complete properly with errors like the above.
OPTIONS="-u bind -S 15"
Putting something like the above in /etc/default/named fixes the errors, but
it still takes a long time and really 150,000 file handles shouldn't be
required for 2560 IP addresses.
listen-on { 10.0.2.45; };
Putting the above in named.conf.options got it to work correctly in this
regard. But I expect it to not use unreasonable amounts of resources without
that configuration.
-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-6-amd64 (SMP w/24 CPU threads)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8),
LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: SELinux: enabled - Mode: Permissive - Policy name: default
Versions of packages bind9 depends on:
ii adduser3.118
ii bind9-libs 1:9.16.13-1
ii bind9-utils1:9.16.13-1
ii debconf [debconf-2.0] 1.5.75
ii dns-root-data 2021011101
ii init-system-helpers1.60
ii iproute2 5.10.0-4
ii libc6 2.31-11
ii libcap21:2.44-1
ii libfstrm0 0.6.0-1+b1
ii libjson-c5 0.15-2
ii liblmdb0 0.9.24-1
ii libmaxminddb0 1.5.2-1
ii libprotobuf-c1 1.3.3-1+b2
ii libssl1.1 1.1.1k-1
ii libuv1 1.40.0-1
ii libxml22.9.10+dfsg-6.3+b1
ii lsb-base 11.1.0
ii netbase6.3
ii zlib1g 1:1.2.11.dfsg-2
bind9 recommends no packages.
Versions of packages bind9 suggests:
pn bind-doc
ii bind9-dnsutils [dnsutils] 1:9.16.13-1
ii dnsutils 1:9.16.13-1
pn resolvconf
pn ufw
-- Configuration Files:
/etc/bind/named.conf.local changed:
//
// Do any local configuration here
//
// Consider adding the 1918 zones here, if they are not used in your
// organization
//include "/etc/bind/zones.rfc1918";
//include "/etc/bind/named.conf.postal";
/etc/bind/named.conf.options changed:
options {
directory "/var/cache/bind";
// If there is a firewall between you and nameservers you want
// to talk to, you may need to fix the firewall to allow multiple
// ports to talk. See http://www.kb.cert.org/vuls/id/800113
// If your ISP provided one or more IP addresses for stable
// nameservers, you probably want to use them as forwarders.
// Uncomment the following block, and insert the addresses replacing
// the all-0's placeholder.
// forwarders {
// 0.0.0.0;
// };
//
// If BIND logs error messages about the root key being expired,
// you will need to update your keys. See https://www.isc.org/bind-keys
//
dnssec-validation auto;
listen-on { 10.0.2.45; };
listen-on-v6 { any; };
};
/etc/default/named changed:
RESOLVCONF=no
OPTIONS="-u bind"
-- debconf information:
bind9/start-as-user: bind
bind9/different-configuration-file:
bind9/run-resolvconf: false
