Hey, I did a measurement with 1M small generated zones that we are using internally for the performance testing and here are some numbers:
The measured values are USS/PSS/RSS using `smem -P named -k` BIND 9.16 w/o jemalloc: 10.9G/10.9G/10.9G (default configuration) BIND 9.16 with jemalloc: 10.1G/10.2G/10.2G [1] BIND 9.18 w/o jemalloc: 10.7G/10.7G/10.7G (not recommended) BIND 9.18 with jemalloc: 9.9G/9.9G/9.9G (default configuration) BIND 9.19 w/o jemalloc: 10.5G/10.5G/10.6G BIND 9.19 with jemalloc: 9.8G/9.8G/9.8G [2] This is consistent with our other measurements that the memory usage is slightly lower with 9.18 compared to 9.16. As you hadn’t shared any other details, there’s not much we can do here, so you are pretty much on your own. But I would say that 1GB extra of memory in the context of loading 1M zones is not worth too much effort. 1. just preloading jemalloc saves some memory as compared to the default system allocator 2. our expectations are to go even lower during the 9.19/9.20 development cycle, but no promises yet Cheers, -- Ondřej Surý (He/Him) ond...@isc.org My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours. > On 11. 7. 2022, at 6:25, Ondřej Surý <ond...@isc.org> wrote: > > Hi, > > yes, I did. And I see no problem here. The software changes between the > versions and roughly 10% increase doesn’t seem like something that should be > worrying or worth any deep investigation. You simply cannot expect “faster”, > “better”, “contains new features” and *“same”* together. > > Ondřej > -- > Ondřej Surý — ISC (He/Him) > > My working hours and your working hours may be different. Please do not feel > obligated to reply outside your normal working hours. > >> On 11. 7. 2022, at 6:09, Raman kumar <kumarraman....@gmail.com> wrote: >> >> >> Hello, >> >> Did you get a chance to look into the data that I shared above? The >> tools(ps_mem, pmap ) that you recommended to compare the RAM consumption >> between 9.18.x and 9.16.x are also showing that memory consumption is more >> in 9.18.x compared to 9.16.x. >> >> Thanks in advance. >> >> Regards, >> Raman >> >> >> >> >> On Fri, Jul 1, 2022 at 9:08 PM Raman kumar <kumarraman....@gmail.com> wrote: >> Hello, >> >> In continuation to my previous mails, sharing memory usage by bind 9.16.21 >> and 9.18.3 using pmap and ps_mem tools. >> >> We have observed memory usage in 9.18.3 is higher by approximately 1.10 GB >> with the same amount of data loaded on both bind versions. >> Attaching pmap output for both bind versions in this mail which also shows >> memory used by heap/mmap on 9.18.3 is ~10.16GB whereas 9.16.21 was consuming >> ~9.05GB >> >> 9.16.21 >> ====== >> ps_mem 7543 >> Private + Shared = RAM used Program >> 9.1 GiB + 262.5 KiB = 9.1 GiB named >> >> 9.18.3 >> ===== >> ps_mem 8456 >> Private + Shared = RAM used Program >> 10.2 GiB + 162.0 KiB = 10.2 GiB named >> >> Regards, >> Raman >> >> On Fri, Jun 24, 2022 at 10:06 AM Ondřej Surý <ond...@isc.org> wrote: >> >>> On 24. 6. 2022, at 5:28, Nagaraju Amarana <amarana.nagar...@gmail.com> >>> wrote: >>> >>> >>> Hi Ondrej, >>> >>> As per the BIND stats usage is lower than the 9.16 however it is in the >>> other way when compared with the system stats. Any idea on this difference? >> >> Hi, >> >> I already mentioned some below - memory fragmentation plays a role. Using >> jemalloc helps with that. >> >> Other factors are mentioned in the resources I’ve sent - have you read it? - >> shared libraries are also accounted in this space. Using the numbers from >> statm (or even ps/top) is only informative and makes sense only as hint - >> f.e. when memory leaks. Or comparing the exactly same builds with same >> libraries linked etc. >> >>> I think user need to consider the memory allocation based on the system >>> usage otherwise OS may crash the process due to out of memory. >> >> No, not really. First of all, OOM killer makes sure it doesn’t crash the >> operating system, but kills a selected process to free up the memory. >> >>> Do you agree? With this can i say 9.18.x memory usage is higher than 9.16.x? >> >> No, not really. >> >> Ondrej >> -- >> Ondřej Surý — ISC (He/Him) >> >> My working hours and your working hours may be different. Please do not feel >> obligated to reply outside your normal working hours. >> >>> Regards, >>> Nagaraju A. >>> >>> >>> On Fri, Jun 24, 2022 at 12:08 AM Ondřej Surý <ond...@isc.org> wrote: >>> The stats that you sent does show that 9.18 is internally using little bit >>> less memory than 9.16. Also please note that your 9.16 version is outdated. >>> >>> As for the differences - are you using jemalloc to compile BIND 9.18? This >>> is the recommended memory allocator for production as it does greatly >>> reduce memory fragmentation and increase performance. The statm output is >>> not 100% accurate, you need to analyze the output of smaps file to get an >>> accurate picture, I’ve listed some tools that can do that below. >>> >>> Measuring actual memory usage is difficult, there are some good hints here: >>> https://stackoverflow.com/questions/131303/how-can-i-measure-the-actual-memory-usage-of-an-application-or-process >>> >>> That said, I see nothing wrong in the output you posted here. In fact, what >>> Petr said earlier is true - the memory usage went down. (Just disregard the >>> “total” from stats - it’s misleading it never goes down, just up, it only >>> accounts allocations, but deallocations are not counted - e.g. it's counter >>> only for developers.) >>> >>> Use pmap, ps_mem, smem or valgrind massif tool. You can also make jemalloc >>> to print various memory statistics at the program exit by setting >>> environment variable (see the man page for jemalloc for more details). >>> >>> Ondřej >>> -- >>> Ondřej Surý — ISC (He/Him) >>> >>> My working hours and your working hours may be different. Please do not >>> feel obligated to reply outside your normal working hours. >>> >>>> On 23. 6. 2022, at 19:07, Raman kumar <kumarraman....@gmail.com> wrote: >>>> >>>> >>>> Hello, >>>> >>>> Thanks for your reply! >>>> >>>> I did the analysis of stats generated, top command output and statm file. >>>> Memory consumption(inUse) in 9.16 is matching in all places. >>>> >>>> In 9.18 memory consumption(inUse) is matching in statm file and top >>>> command output, but it is not matching with stats generated (please see >>>> images below for stats, top command output and statm of bind version 9.16 >>>> and 9.18). >>>> >>>> Can you please guide us to debug high memory uses in bind 9.18 based upon >>>> the data shared below? >>>> Also need your inputs why stats are not matching with memory(InUse) in >>>> bind version 9.18? >>>> >>>> <image.png> >>>> >>>> <image.png> >>>> <image.png> >>>> Regards, >>>> Raman >>>> >>>> On Wed, Jun 22, 2022 at 10:54 AM Ondřej Surý <ond...@isc.org> wrote: >>>> And what did you find looking at the new data? What are the differences? >>>> And by how much? >>>> >>>> You should not expect other people doing the analysis for yourself. >>>> >>>> Ondrej >>>> -- >>>> Ondřej Surý — ISC (He/Him) >>>> >>>> My working hours and your working hours may be different. Please do not >>>> feel obligated to reply outside your normal working hours. >>>> >>>>> On 22. 6. 2022, at 6:30, Raman kumar <kumarraman....@gmail.com> wrote: >>>>> >>>>> >>>>> Hello, >>>>> >>>>> As suggested, please find the attached artifacts which includes stats by >>>>> configuring statistics channel in named.conf, content at >>>>> /proc/<PID>/statm, top command output for both 9.16.21 and 9.18.3. >>>>> >>>>> Regards, >>>>> Raman >>>>> >>>>> On Thu, Jun 16, 2022 at 12:55 PM Ondřej Surý <ond...@isc.org> wrote: >>>>> Hi, >>>>> >>>>> we cannot really help you if you ignore everything that was said to you >>>>> regarding the memory measurements. >>>>> >>>>> Ondrej. >>>>> -- >>>>> Ondřej Surý (He/Him) >>>>> ond...@isc.org >>>>> >>>>> My working hours and your working hours may be different. Please do not >>>>> feel obligated to reply outside your normal working hours. >>>>> >>>>> > On 16. 6. 2022, at 9:02, Raman kumar <kumarraman....@gmail.com> wrote: >>>>> > >>>>> > Hello All, >>>>> > >>>>> > We configured bind 9.18, using jemalloc but still memory consumption is >>>>> > high in 9.18 as compared to 9.16. >>>>> > >>>>> > On version 9.16.21, RAM consumption was 3.8 GB without jemalloc. And on >>>>> > 9.18.2, RAM consumption is 4.2 GB with jemalloc with the same data. >>>>> > >>>>> > Is this the expected behaviour or any more tuning is needed? >>>>> > >>>>> > One more thing: does CNAME record length also impact the memory used? >>>>> > >>>>> > Regards, >>>>> > Raman >>>>> > >>>>> > >>>>> > On Thu, May 19, 2022 at 3:52 PM Petr Špaček <pspa...@isc.org> wrote: >>>>> > On 18. 05. 22 22:39, Ondřej Surý wrote: >>>>> > > Hi Klarstein, >>>>> > > >>>>> > > Gathering the output of named statschannel should be good enough for >>>>> > > initial assessment (json please). >>>>> > > >>>>> > > For 9.18, make sure the jemalloc is being used at runtime. >>>>> > >>>>> > Here are commands you asked for: >>>>> > >>>>> > 1. when running ./configure, make sure the output at the end has this: >>>>> > >>>>> > Configuration summary: >>>>> > ------------------------------------------------------------------------------- >>>>> > Optional features enabled: >>>>> > Memory allocator: jemalloc >>>>> > >>>>> > >>>>> > 2. Then, configure statistics channel in named.conf like this: >>>>> > >>>>> > statistics-channels { >>>>> > inet 127.0.0.1 port 8080; >>>>> > }; >>>>> > >>>>> > >>>>> > 3. With that in place you can grab stats from this URL: >>>>> > http://127.0.0.1:8080/json/v1 >>>>> > >>>>> > Configuration for v9.16 is the same, just skip the jemalloc part. >>>>> > >>>>> > 4. Bonus points for grabbing /proc/<PID>/statm content at the same time >>>>> > as content of the JSON stats endpoint (if you are on Linux). >>>>> > >>>>> > I hope it helps. >>>>> > Petr Špaček >>>>> > >>>>> > >>>>> > >>>>> > > >>>>> > > Ondrej >>>>> > > -- >>>>> > > Ondřej Surý — ISC (He/Him) >>>>> > > >>>>> > > My working hours and your working hours may be different. Please do >>>>> > > not feel obligated to reply outside your normal working hours. >>>>> > > >>>>> > >> On 18. 5. 2022, at 22:32, Klaus Darilion via bind-users >>>>> > >> <bind-users@lists.isc.org> wrote: >>>>> > >> >>>>> > >> Can you please provide some commands whose output you are >>>>> > >> interested? I want to collect the statistics for 9.16 before >>>>> > >> updating to 9.18. >>>>> > >> Thanks >>>>> > >> Klaus >>>>> > >> >>>>> > >>> -----Ursprüngliche Nachricht----- >>>>> > >>> Von: bind-users <bind-users-boun...@lists.isc.org> Im Auftrag von >>>>> > >>> Petr >>>>> > >>> Špacek >>>>> > >>> Gesendet: Mittwoch, 18. Mai 2022 18:20 >>>>> > >>> An: bind-users@lists.isc.org >>>>> > >>> Betreff: Re: AW: High memory consumption in bind 9.18.2 >>>>> > >>> >>>>> > >>> I would be very interested in hearing more! >>>>> > >>> >>>>> > >>> In majority of our internal testing 9.16 has higher memory >>>>> > >>> consumption >>>>> > >>> than 9.18, especially when 9.18 is compiled with libjemalloc. And >>>>> > >>> the >>>>> > >>> differences are not small, for some configurations it can be even >>>>> > >>> 2x or >>>>> > >>> 3x more on 9.16 than it is on 9.18. >>>>> > >>> >>>>> > >>> If you encounter it again please get back to us so we can diagnose >>>>> > >>> it. >>>>> > >>> >>>>> > >>> Thank you! >>>>> > >>> Petr Špaček >>>>> > >>> >>>>> > >>> >>>>> > >>>> On 18. 05. 22 8:56, Klaus Darilion via bind-users wrote: >>>>> > >>>> I remember we had similar issues with 9.18 (isc ppa packages) and >>>>> > >>>> hence >>>>> > >>> wen't back to 9.16. But I can not remember the details. >>>>> > >>>> >>>>> > >>>> regards >>>>> > >>>> Klaus >>>>> > >>>> >>>>> > >>>>> -----Ursprüngliche Nachricht----- >>>>> > >>>>> Von: bind-users <bind-users-boun...@lists.isc.org> Im Auftrag von >>>>> > >>> Ondrej >>>>> > >>>>> Surý101 71 l t1h, 18. Mai 2022 08:37 >>>>> > >>>>> An: Raman kumar <kumarraman....@gmail.com> >>>>> > >>>>> Cc: bind-users@lists.isc.org >>>>> > >>>>> Betreff: Re: High memory consumption in bind 9.18.2 >>>>> > >>>>> >>>>> > >>>>> You did not provided any details, so we can’t really help you. >>>>> > >>>>> >>>>> > >>>>> What is “RAM consumption” anyway? VSZ, RSS, numbers pulled from >>>>> > >>> stats >>>>> > >>>>> channel from named? >>>>> > >>>>> >>>>> > >>>>> What’s the hardware, what is the configuration, how was BIND 9 >>>>> > >>>>> compiled >>>>> > >>>>> (or packaged)? >>>>> > >>>>> >>>>> > >>>>> The more details, the better >>>>> > >>>>> >>>>> > >>>>> Ondrej >>>>> > >>>>> -- >>>>> > >>>>> Ondřej Surý (He/Him) >>>>> > >>>>> ond...@isc.org >>>>> > >>>>> >>>>> > >>>>> My working hours and your working hours may be different. Please >>>>> > >>>>> do >>>>> > >>> not >>>>> > >>>>> feel obligated to reply outside your normal working hours. >>>>> > >>>>> >>>>> > >>>>>> On 18. 5. 2022, at 8:32, Raman kumar <kumarraman....@gmail.com> >>>>> > >>>>> wrote: >>>>> > >>>>>> >>>>> > >>>>>> Hello Team, >>>>> > >>>>>> >>>>> > >>>>>> While upgrading from BIND 9.16.10 to 9.18.2, we have observed >>>>> > >>>>>> high >>>>> > >>>>> memory consumption. >>>>> > >>>>>> >>>>> > >>>>>> On version 9.16.2, RAM consumption was 3.8 GB. And on 9.18.2, RAM >>>>> > >>>>> consumption is 4.5 GB. Due to this an increase of approximately >>>>> > >>>>> 20 % >>>>> > >>>>> memory is observed. >>>>> > >>>>>> >>>>> > >>>>>> Is this the expected behaviour or any tuning is needed? >>>>> > >>>>>> >>>>> > >>>>>> Thanks in advance. >>>>> > >>>>>> >>>>> > >>>>>> Regards, >>>>> > >>>>>> Raman >>>>> > >>>>>> -- >>>>> > >>>>>> Visit https://lists.isc.org/mailman/listinfo/bind-users to >>>>> > >>>>>> unsubscribe >>>>> > >>> from >>>>> > >>>>> this list >>>>> > >>>>>> >>>>> > >>>>>> ISC funds the development of this software with paid support >>>>> > >>>>> subscriptions. Contact us at https://www.isc.org/contact/ for more >>>>> > >>>>> information. >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> bind-users mailing list >>>>> > >>>>>> bind-users@lists.isc.org >>>>> > >>>>>> https://lists.isc.org/mailman/listinfo/bind-users >>>>> > >>>> >>>>> > >>> >>>>> > >>> >>>>> > >>> -- >>>>> > >>> Petr Špaček >>>>> > >>> -- >>>>> > >>> Visit https://lists.isc.org/mailman/listinfo/bind-users to >>>>> > >>> unsubscribe from this >>>>> > >>> list >>>>> > >>> >>>>> > >>> ISC funds the development of this software with paid support >>>>> > >>> subscriptions. >>>>> > >>> Contact us at https://www.isc.org/contact/ for more information. >>>>> > >>> >>>>> > >>> >>>>> > >>> bind-users mailing list >>>>> > >>> bind-users@lists.isc.org >>>>> > >>> https://lists.isc.org/mailman/listinfo/bind-users >>>>> > >> -- >>>>> > >> Visit https://lists.isc.org/mailman/listinfo/bind-users to >>>>> > >> unsubscribe from this list >>>>> > >> >>>>> > >> ISC funds the development of this software with paid support >>>>> > >> subscriptions. Contact us at https://www.isc.org/contact/ for more >>>>> > >> information. >>>>> > >> >>>>> > >> >>>>> > >> bind-users mailing list >>>>> > >> bind-users@lists.isc.org >>>>> > >> https://lists.isc.org/mailman/listinfo/bind-users >>>>> > >>>>> > >>>>> > -- >>>>> > Petr Špaček >>>>> > -- >>>>> > Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe >>>>> > from this list >>>>> > >>>>> > ISC funds the development of this software with paid support >>>>> > subscriptions. Contact us at https://www.isc.org/contact/ for more >>>>> > information. >>>>> > >>>>> > >>>>> > bind-users mailing list >>>>> > bind-users@lists.isc.org >>>>> > https://lists.isc.org/mailman/listinfo/bind-users >>>>> > -- >>>>> > Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe >>>>> > from this list >>>>> > >>>>> > ISC funds the development of this software with paid support >>>>> > subscriptions. Contact us at https://www.isc.org/contact/ for more >>>>> > information. >>>>> > >>>>> > >>>>> > bind-users mailing list >>>>> > bind-users@lists.isc.org >>>>> > https://lists.isc.org/mailman/listinfo/bind-users >>>>> >>> -- >>> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from >>> this list >>> >>> ISC funds the development of this software with paid support subscriptions. >>> Contact us at https://www.isc.org/contact/ for more information. >>> >>> >>> bind-users mailing list >>> bind-users@lists.isc.org >>> https://lists.isc.org/mailman/listinfo/bind-users >>> >>> >>> -- >>> Regards, >>> Nagaraju A. >> -- >> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from >> this list >> >> ISC funds the development of this software with paid support subscriptions. >> Contact us at https://www.isc.org/contact/ for more information. >> >> >> bind-users mailing list >> bind-users@lists.isc.org >> https://lists.isc.org/mailman/listinfo/bind-users -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
