Hi, Resending ... bad format.
dnf install wget bzip2 gcc make -y wget https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2 bzip2 -d jemalloc-5.3.0.tar.bz2 && tar -xf jemalloc-5.3.0.tar && cd jemalloc-5.3.0 ./configure make make install reboot -f Dmitri. -----Original Message----- From: Dmitri Pavlov Sent: Tuesday, August 2, 2022 6:14 PM To: Ondřej Surý <ond...@isc.org> Cc: bind-users@lists.isc.org Subject: RE: High memory consumption in bind 9.18.2 Hi, Thank you very much for your feedback, Ondrej. Sharing the steps. Very simple: configure -> make -> make install. Very simple configuration. Just the zone file is big. Please, see the attached. 1. I followed the instructions from here https://bind9.readthedocs.io/en/v9_18_4/chapter10.html No git. 2. Rocky 9 uname -a Linux server.test.zone 5.14.0-70.17.1.el9_0.x86_64 #1 SMP PREEMPT Wed Jul 13 18:23:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Package gcc-11.2.1-9.4.el9.x86_64 9.18.5 ============= OUTPUT ============= pmap -x 27821 | tail -n 1 total kB 34853832 29425312 29416180 smemstat -p named PID Swap USS PSS RSS User Command 27821 0.0 B 28.1 G 28.1 G 28.1 G root /usr/local/sbin/named Total: 0.0 B 28.1 G 28.1 G 28.1 ============= ============= ============= 9.16.21 ============= OUTPUT ============= pmap -x 18026 | tail -n 1 total kB 28161152 27414164 27406208 smemstat -p named PID Swap USS PSS RSS User Command 18026 0.0 B 26.1 G 26.1 G 26.1 G root /usr/local/sbin/named Total: 0.0 B 26.1 G 26.1 G 26.1 G ============= ============= ============= 2 GB diff is still there. Steps have been attached. Will be very grateful, could you please, have a look at the steps provided? They are quite straightforward and self-explanatory. What is that "magic souse" I should apply, when there is Rocky 9 or RHEL 9 machine in front of me. I believe if I replicate your setup/steps, most likely I'll get the same output. However, if looking at the articles available to the wide audience (like this https://bind9.readthedocs.io/en/v9_18_4/chapter10.html or any other(?)), what should my steps be to get the expected smaller memory footprint. Or maybe there are obvious flaws in the experiment? Thanking you very much for patience and cooperation, Dmitri. -----Original Message----- From: Ondřej Surý <ond...@isc.org> Sent: Monday, August 1, 2022 8:18 PM To: Dmitri Pavlov <dpav...@perforce.com> Cc: bind-users@lists.isc.org Subject: Re: High memory consumption in bind 9.18.2 $ wc -l gen.db 100000000 gen.db generated with: #!/bin/env python3 for x in range(0, 10000): for y in range(0, 2500): print(f"az{x}-{y} IN A 10.53.0.1") print(f"bz{x}-{y} IN A 10.53.0.2") print(f"ca{x}-{y} IN A 10.53.0.3") print(f"xh{x}-{y} IN CNAME az{x}-{y}”) ## BIND 9.16(git) 01-Aug-2022 19:06:27.151 starting BIND 9.16.32-dev (Extended Support Version) <id:c863061a13> 01-Aug-2022 19:06:27.151 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) 01-Aug-2022 19:06:27.151 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' 01-Aug-2022 19:06:27.151 running as: named -c named.conf -g -p 12345 01-Aug-2022 19:06:27.151 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f] 01-Aug-2022 19:06:27.151 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 01-Aug-2022 19:06:27.151 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 01-Aug-2022 19:06:27.151 compiled with libxml2 version: 2.9.10 01-Aug-2022 19:06:27.151 linked to libxml2 version: 20910 01-Aug-2022 19:06:27.151 compiled with json-c version: 0.15 01-Aug-2022 19:06:27.151 linked to json-c version: 0.15 01-Aug-2022 19:06:27.151 compiled with zlib version: 1.2.11 01-Aug-2022 19:06:27.151 linked to zlib version: 1.2.11 $ smem -P name[d] -a SWAP:193304 USS:30451812 PSS:30452272 RSS:30454872 ## BIND 9.18(git) 01-Aug-2022 18:12:57.227 starting BIND 9.18.6-dev (Stable Release) <id:097a57e> 01-Aug-2022 18:12:57.227 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) 01-Aug-2022 18:12:57.227 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' 01-Aug-2022 18:12:57.227 running as: named -c named.conf -g -p 12345 01-Aug-2022 18:12:57.227 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f] 01-Aug-2022 18:12:57.227 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 01-Aug-2022 18:12:57.227 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 01-Aug-2022 18:12:57.227 compiled with libxml2 version: 2.9.10 01-Aug-2022 18:12:57.227 linked to libxml2 version: 20910 01-Aug-2022 18:12:57.227 compiled with json-c version: 0.15 01-Aug-2022 18:12:57.227 linked to json-c version: 0.15 01-Aug-2022 18:12:57.227 compiled with zlib version: 1.2.11 01-Aug-2022 18:12:57.227 linked to zlib version: 1.2.11 $ smem -P name[d] -a SWAP:645960 USS:29446788 PSS:29447529 RSS:29451056 ## BIND 9.19(git) 01-Aug-2022 18:18:20.127 starting BIND 9.19.4-dev (Development Release) <id:fa1f35b> 01-Aug-2022 18:18:20.127 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) 01-Aug-2022 18:18:20.127 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' 01-Aug-2022 18:18:20.127 running as: named -c named.conf -g -p 12345 01-Aug-2022 18:18:20.127 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f] 01-Aug-2022 18:18:20.127 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 01-Aug-2022 18:18:20.127 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 01-Aug-2022 18:18:20.127 compiled with libxml2 version: 2.9.10 01-Aug-2022 18:18:20.127 linked to libxml2 version: 20910 01-Aug-2022 18:18:20.127 compiled with json-c version: 0.15 01-Aug-2022 18:18:20.127 linked to json-c version: 0.15 01-Aug-2022 18:18:20.127 compiled with zlib version: 1.2.11 01-Aug-2022 18:18:20.127 linked to zlib version: 1.2.11 $ smem -P name[d] -a SWAP:1032108 USS:29062164 PSS:29062968 RSS:29066580 So, it’s (USS+Swap): 9.16(git): 30645116 9.18(git): 30092748 9.19(git): 30094272 So, no, I am unable to reproduce the results and yet again, I see a lower memory usage with 9.18+. But I see a common pattern here. I think both you and the OP were using CentOS/RHEL 7 which is using GCC 4.8. GCC was improved significantly since then. I would suggest to repeat the experiment on RHEL 9 if you can reproduce the same results. Ondrej -- Ondřej Surý (He/Him) ond...@isc.org My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours. > On 1. 8. 2022, at 17:46, Ondřej Surý <ond...@isc.org> wrote: > >> On 1. 8. 2022, at 17:19, Dmitri Pavlov <dpav...@perforce.com> wrote: >> >> I’m pretty much sure you will get the same results in your lab. > > I don’t want to delve into vague description of your experiment. > You’ll have to share the exact scripts. > > Even this: > >> just the time-consuming task is to generate 100 MIL A + CNAME dummy records >> into the zone file. > > could be violently different. Are those random names? Or > deterministically generated? What are the CNAME targets? > > What is the exact `named.conf`? > > What are the exact library versions? What is the exact ./configure invocation? > > How do you execute `named`? > > And yet again - the memory statistics from the statschannel could show > the breakdown of the internal memory contexts and also show if there’s > a difference between the internally reported memory usage and > externally reported memory usage, e.g. is this something allocated in named, > or is this allocated in the libraries? > >> Should the memory reduction apply to our experiment? > > The question doesn’t really make sense. We have not measured any > increase in our test scenarios, which doesn’t mean you can’t find different > scenarios with a memory increase. > > Ondrej > -- > Ondřej Surý (He/Him) > ond...@isc.org > > My working hours and your working hours may be different. Please do not feel > obligated to reply outside your normal working hours. > CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately. -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
