Hi Ondřej, Sorry to bother you one more time regarding the same topic. I have looked through your shared logs one more time. This is what you have shared
YOUR LAB RESULTS ARE: BIND 9.16.32 / BIND 9.18.6 / BIND 9.19.4 RSS:30454872 / RSS:29451056 / RSS:29066580 OUR LAB RESULTS ARE: BIND 9.16.21 / BIND 9.18.5 RSS:27406208 / RSS:29416180 Therefore, the results 9.18.6: RSS 29451056 (your lab) vs 9.18.5: RSS 29416180 (our lab) are very close. However the KB https://kb.isc.org/docs/bind-memory-consumption-explained -> BIND Memory Consumption Explained, it is about "increased memory consumption in BIND 9.16 up to and including 9.16.24" vs 9.18.x. As you see the results provided by you from 9.16.32, they neither prove nor disapprove what the article describes. Therefore, a very small request. Would it be possible on your side to run the same experiment as with (BIND 9.16.32 / BIND 9.18.6 / BIND 9.19.4) one more time but with BIND 9.16.21 (or any other version in 9.16.x <25 range )? Thank you very much in advance, Kind regards, Dmitri. -----Original Message----- From: Ondřej Surý <ond...@isc.org> Sent: Tuesday, August 2, 2022 6:30 PM To: Dmitri Pavlov <dpav...@perforce.com> Cc: bind-users@lists.isc.org Subject: Re: High memory consumption in bind 9.18.2 Well, then I don’t know the reason for the difference in your case. And I don’t personally see a compelling reason to investigate a 10% increase in artificial scenario like this since it apparently doesn’t apply to all scenarios. However, you are free to do the further investigation yourself. We are refactoring the database for storing the resource records in 9.20 and it's probably better spent time to work on the refactoring than look at this. As usual, we would accept any well commented and well thought patches. Ondřej -- Ondřej Surý — ISC (He/Him) My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours. > On 2. 8. 2022, at 17:23, Dmitri Pavlov <dpav...@perforce.com> wrote: > > Hi, > > Resending ... bad format. > > dnf install wget bzip2 gcc make -y > wget > https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc- > 5.3.0.tar.bz2 > bzip2 -d jemalloc-5.3.0.tar.bz2 && tar -xf jemalloc-5.3.0.tar && cd > jemalloc-5.3.0 ./configure make make install reboot -f > > Dmitri. > > > > -----Original Message----- > From: Dmitri Pavlov > Sent: Tuesday, August 2, 2022 6:14 PM > To: Ondřej Surý <ond...@isc.org> > Cc: bind-users@lists.isc.org > Subject: RE: High memory consumption in bind 9.18.2 > > Hi, > > Thank you very much for your feedback, Ondrej. > > Sharing the steps. Very simple: configure -> make -> make install. Very > simple configuration. Just the zone file is big. Please, see the attached. > > 1. I followed the instructions from here > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbind > 9.readthedocs.io%2Fen%2Fv9_18_4%2Fchapter10.html&data=05%7C01%7Cdp > avlov%40perforce.com%7Cbefcaa04b0ed465b614908da749be1b0%7C95b666d19a75 > 49ab95a38969fbcdc08c%7C0%7C0%7C637950510129140945%7CUnknown%7CTWFpbGZs > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > %7C3000%7C%7C%7C&sdata=fY1tDUrmD8F7a2dmMCxKjzoibMjq6oKQr7%2FBF7tvi > 6E%3D&reserved=0 > No git. > > 2. Rocky 9 > uname -a > Linux server.test.zone 5.14.0-70.17.1.el9_0.x86_64 #1 SMP PREEMPT Wed > Jul 13 18:23:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Package > gcc-11.2.1-9.4.el9.x86_64 > > 9.18.5 > ============= OUTPUT ============= > pmap -x 27821 | tail -n 1 > total kB 34853832 29425312 29416180 > > smemstat -p named > PID Swap USS PSS RSS User Command > 27821 0.0 B 28.1 G 28.1 G 28.1 G root > /usr/local/sbin/named > Total: 0.0 B 28.1 G 28.1 G 28.1 > ============= ============= ============= > > 9.16.21 > ============= OUTPUT ============= > pmap -x 18026 | tail -n 1 > total kB 28161152 27414164 27406208 > > smemstat -p named > PID Swap USS PSS RSS User Command > 18026 0.0 B 26.1 G 26.1 G 26.1 G root > /usr/local/sbin/named > Total: 0.0 B 26.1 G 26.1 G 26.1 G > ============= ============= ============= > > 2 GB diff is still there. > > Steps have been attached. > > Will be very grateful, could you please, have a look at the steps provided? > They are quite straightforward and self-explanatory. What is that "magic > souse" I should apply, when there is Rocky 9 or RHEL 9 machine in front of > me. I believe if I replicate your setup/steps, most likely I'll get the same > output. However, if looking at the articles available to the wide audience > (like this > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbind9.readthedocs.io%2Fen%2Fv9_18_4%2Fchapter10.html&data=05%7C01%7Cdpavlov%40perforce.com%7Cbefcaa04b0ed465b614908da749be1b0%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637950510129140945%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fY1tDUrmD8F7a2dmMCxKjzoibMjq6oKQr7%2FBF7tvi6E%3D&reserved=0 > or any other(?)), what should my steps be to get the expected smaller memory > footprint. Or maybe there are obvious flaws in the experiment? > > Thanking you very much for patience and cooperation, Dmitri. > > > > > -----Original Message----- > From: Ondřej Surý <ond...@isc.org> > Sent: Monday, August 1, 2022 8:18 PM > To: Dmitri Pavlov <dpav...@perforce.com> > Cc: bind-users@lists.isc.org > Subject: Re: High memory consumption in bind 9.18.2 > > $ wc -l gen.db > 100000000 gen.db > > generated with: > > #!/bin/env python3 > > for x in range(0, 10000): > for y in range(0, 2500): > print(f"az{x}-{y} IN A 10.53.0.1") > print(f"bz{x}-{y} IN A 10.53.0.2") > print(f"ca{x}-{y} IN A 10.53.0.3") > print(f"xh{x}-{y} IN CNAME az{x}-{y}”) > > ## BIND 9.16(git) > > 01-Aug-2022 19:06:27.151 starting BIND 9.16.32-dev (Extended Support > Version) <id:c863061a13> > 01-Aug-2022 19:06:27.151 running on Linux x86_64 5.10.0-16-amd64 #1 > SMP Debian 5.10.127-1 (2022-06-30) > 01-Aug-2022 19:06:27.151 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb > -Wno-deprecated-declarations -fno-omit-frame-pointer > -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 > -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' > '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' > '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' > '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' > 01-Aug-2022 19:06:27.151 running as: named -c named.conf -g -p 12345 > 01-Aug-2022 19:06:27.151 compiled by GCC 12.0.1 20220319 > (experimental) [master r12-7719-g8ca61ad148f] > 01-Aug-2022 19:06:27.151 compiled with OpenSSL version: OpenSSL 3.0.5 > 5 Jul 2022 > 01-Aug-2022 19:06:27.151 linked to OpenSSL version: OpenSSL 3.0.5 5 > Jul 2022 > 01-Aug-2022 19:06:27.151 compiled with libxml2 version: 2.9.10 > 01-Aug-2022 19:06:27.151 linked to libxml2 version: 20910 > 01-Aug-2022 19:06:27.151 compiled with json-c version: 0.15 > 01-Aug-2022 19:06:27.151 linked to json-c version: 0.15 > 01-Aug-2022 19:06:27.151 compiled with zlib version: 1.2.11 > 01-Aug-2022 19:06:27.151 linked to zlib version: 1.2.11 > > $ smem -P name[d] -a > SWAP:193304 USS:30451812 PSS:30452272 RSS:30454872 > > ## BIND 9.18(git) > > 01-Aug-2022 18:12:57.227 starting BIND 9.18.6-dev (Stable Release) > <id:097a57e> > 01-Aug-2022 18:12:57.227 running on Linux x86_64 5.10.0-16-amd64 #1 > SMP Debian 5.10.127-1 (2022-06-30) > 01-Aug-2022 18:12:57.227 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb > -Wno-deprecated-declarations -fno-omit-frame-pointer > -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 > -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' > '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' > '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' > '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' > 01-Aug-2022 18:12:57.227 running as: named -c named.conf -g -p 12345 > 01-Aug-2022 18:12:57.227 compiled by GCC 12.0.1 20220319 > (experimental) [master r12-7719-g8ca61ad148f] > 01-Aug-2022 18:12:57.227 compiled with OpenSSL version: OpenSSL 3.0.5 > 5 Jul 2022 > 01-Aug-2022 18:12:57.227 linked to OpenSSL version: OpenSSL 3.0.5 5 > Jul 2022 > 01-Aug-2022 18:12:57.227 compiled with libxml2 version: 2.9.10 > 01-Aug-2022 18:12:57.227 linked to libxml2 version: 20910 > 01-Aug-2022 18:12:57.227 compiled with json-c version: 0.15 > 01-Aug-2022 18:12:57.227 linked to json-c version: 0.15 > 01-Aug-2022 18:12:57.227 compiled with zlib version: 1.2.11 > 01-Aug-2022 18:12:57.227 linked to zlib version: 1.2.11 > > $ smem -P name[d] -a > SWAP:645960 USS:29446788 PSS:29447529 RSS:29451056 > > ## BIND 9.19(git) > > 01-Aug-2022 18:18:20.127 starting BIND 9.19.4-dev (Development > Release) <id:fa1f35b> > 01-Aug-2022 18:18:20.127 running on Linux x86_64 5.10.0-16-amd64 #1 > SMP Debian 5.10.127-1 (2022-06-30) > 01-Aug-2022 18:18:20.127 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb > -Wno-deprecated-declarations -fno-omit-frame-pointer > -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 > -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' > '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' > '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' > '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' > 01-Aug-2022 18:18:20.127 running as: named -c named.conf -g -p 12345 > 01-Aug-2022 18:18:20.127 compiled by GCC 12.0.1 20220319 > (experimental) [master r12-7719-g8ca61ad148f] > 01-Aug-2022 18:18:20.127 compiled with OpenSSL version: OpenSSL 3.0.5 > 5 Jul 2022 > 01-Aug-2022 18:18:20.127 linked to OpenSSL version: OpenSSL 3.0.5 5 > Jul 2022 > 01-Aug-2022 18:18:20.127 compiled with libxml2 version: 2.9.10 > 01-Aug-2022 18:18:20.127 linked to libxml2 version: 20910 > 01-Aug-2022 18:18:20.127 compiled with json-c version: 0.15 > 01-Aug-2022 18:18:20.127 linked to json-c version: 0.15 > 01-Aug-2022 18:18:20.127 compiled with zlib version: 1.2.11 > 01-Aug-2022 18:18:20.127 linked to zlib version: 1.2.11 > > $ smem -P name[d] -a > SWAP:1032108 USS:29062164 PSS:29062968 RSS:29066580 > > So, it’s (USS+Swap): > 9.16(git): 30645116 > 9.18(git): 30092748 > 9.19(git): 30094272 > > So, no, I am unable to reproduce the results and yet again, I see a lower > memory usage with 9.18+. > > But I see a common pattern here. I think both you and the OP were using > CentOS/RHEL 7 which is using GCC 4.8. > > GCC was improved significantly since then. I would suggest to repeat the > experiment on RHEL 9 if you can reproduce the same results. > > Ondrej > -- > Ondřej Surý (He/Him) > ond...@isc.org > > My working hours and your working hours may be different. Please do not feel > obligated to reply outside your normal working hours. > >>> On 1. 8. 2022, at 17:46, Ondřej Surý <ond...@isc.org> wrote: >>> >>>> On 1. 8. 2022, at 17:19, Dmitri Pavlov <dpav...@perforce.com> wrote: >>> >>> I’m pretty much sure you will get the same results in your lab. >> >> I don’t want to delve into vague description of your experiment. >> You’ll have to share the exact scripts. >> >> Even this: >> >>> just the time-consuming task is to generate 100 MIL A + CNAME dummy records >>> into the zone file. >> >> could be violently different. Are those random names? Or >> deterministically generated? What are the CNAME targets? >> >> What is the exact `named.conf`? >> >> What are the exact library versions? What is the exact ./configure >> invocation? >> >> How do you execute `named`? >> >> And yet again - the memory statistics from the statschannel could >> show the breakdown of the internal memory contexts and also show if >> there’s a difference between the internally reported memory usage and >> externally reported memory usage, e.g. is this something allocated in named, >> or is this allocated in the libraries? >> >>> Should the memory reduction apply to our experiment? >> >> The question doesn’t really make sense. We have not measured any >> increase in our test scenarios, which doesn’t mean you can’t find different >> scenarios with a memory increase. >> >> Ondrej >> -- >> Ondřej Surý (He/Him) >> ond...@isc.org >> >> My working hours and your working hours may be different. Please do not feel >> obligated to reply outside your normal working hours. >> > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately. -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
