Re: [Dnsmasq-discuss] Memory usage growing over time — possibly due to logging or caching (v2.90 / OpenEmbedded kirkstone)
On Mon, Apr 14, 2025 at 10:58:25PM +0100, Simon Kelley wrote: > Instead of restarting dnsmasq, try deleting the log file and telling dnsmasq > to open a new one. > > > rm /var/log/dnsmasq.log > killall -USR2 dnsmasq > > > If that also causes the memory usage to fall, then you are being led astray > by a the log file being memory mapped. Which matches design choice "Used memory is wasted memory". > If the memory usage doesn't fall, there might be a real memory leak. Acknowledge on **there might be** a real memory leak. Groeten Geert Stappers -- Silence is hard to parse ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Memory usage growing over time — possibly due to logging or caching (v2.90 / OpenEmbedded kirkstone)
Instead of restarting dnsmasq, try deleting the log file and telling dnsmasq to open a new one. rm /var/log/dnsmasq.log killall -USR2 dnsmasq If that also causes the memory usage to fall, then you are being led astray by a the lg file being memory mapped. If the memeory usage doesn't fall, there might be a real memory leak. Cheers, Simon. On 14/04/2025 17:55, Nitesh Divecha wrote: Hello Simon, Thank you for your feedback. Yes, when I turn on --log-facility=/var/log/dnsmasq.log, I see memory-size growing. # systemctl status dnsmasq * dnsmasq.service - DNS forwarder and DHCP server Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/dnsmasq.service.d `-override.conf Active: active (running) since Sat 2025-04-12 21:27:30 UTC; 15h ago Process: 427 ExecStartPre=/usr/bin/dnsmasq --test (code=exited, status=0/SUCCESS) Process: 434 ExecStart=/usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --local-service (code=exited, status=0/SUCCESS) Main PID: 437 (dnsmasq) Tasks: 1 (limit: 1042) Memory: 16.3M CGroup: /system.slice/dnsmasq.service `- 437 /usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --local-service # ls -lah /var/log/dnsmasq.log -rw-rw 1 nobody root 16M Apr 13 13:05 /var/log/dnsmasq.log And if I restart the service it will reset the log file and memory-size will go down. # systemctl status dnsmasq --no-pager ● dnsmasq.service - DNS forwarder and DHCP server Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/dnsmasq.service.d └─override.conf Active: active (running) since Mon 2025-04-14 16:08:27 UTC; 43min ago Process: 2238 ExecStartPre=/usr/bin/dnsmasq --test (code=exited, status=0/SUCCESS) Process: 2239 ExecStart=/usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --bind-dynamic --local-service --listen-address=127.0.0.1 --listen-address=192.0.2.2 (code=exited, status=0/SUCCESS) Main PID: 2242 (dnsmasq) Tasks: 1 (limit: 1042) Memory: 1.3M CGroup: /system.slice/dnsmasq.service └─ 2242 /usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --bind-dynamic --local-service --listen-address=127.0.0.1 --listen-address=192.0.2.2 Apr 14 16:08:27 2026240055 systemd[1]: Starting DNS forwarder and DHCP server... Apr 14 16:08:27 2026240055 dnsmasq[2238]: dnsmasq: syntax check OK. Apr 14 16:08:27 2026240055 systemd[1]: Started DNS forwarder and DHCP server. # ls -lah /var/log/dnsmasq.log -rw-rw 1 nobody root 1.2M Apr 14 16:52 /var/log/dnsmasq.log Cheers, Nitesh On Apr 14, 2025, at 10:48 AM, Simon Kelley wrote: On 4/13/25 13:45, Nitesh Divecha via Dnsmasq-discuss wrote: Hi all, I'm using dnsmasq version 2.90 in an embedded Linux project based on the Yocto Project (kirkstone branch). The recipe in use is from the OpenEmbedded meta-networking layer: https://layers.openembedded.org/layerindex/recipe/304520/ I've observed that dnsmasq memory usage steadily increases over time, even under light DNS query loads. For example: - On boot: ~1.1 MB - After 1 hour: ~3.3 MB - After ~14 hours: ~15.3 MB Relevant configuration (via NetworkManager): # cat /etc/NetworkManager/dnsmasq.d/dnsmasq-dns.conf no-negcache stop-dns-rebind server=8.8.8.8 server=1.1.1.1 all-servers log-queries log-facility=/var/log/dnsmasq.log I noticed that the `/var/log/dnsmasq.log` file grows continuously (currently ~15M), and suspect that this may be contributing to memory growth — either due to internal buffering or cumulative logging state. My questions: 1. Is this memory growth expected when `log-queries` is enabled with `log-facility`? 2. Does `dnsmasq` buffer log output internally, or are there known issues with memory not being reclaimed after prolonged logging? 3. Are there recommended best practices for logging in long-running embedded environments to avoid this kind of memory usage? Any suggestions or insights would be greatly appreciated. Thanks in advance, Cheers, Nitesh Do you only see this memory-size growth when logging to a file? It's possible that there's a memory leak elsewhere, but unlikely that one that big could have been undetected until now. It's possible that the libc in use is memory-mapping the log file. Dnsmasq uses very standard open() and write() calls, but it's up to the libc how it implements that. Memory-mapping the file and converting write() calls to memory writes would be a legitimate thing to do. In this case some measures of the process's memory usage would increase, but the actual physical RAM usage would not - long untouched pages will be written out to the filesystem and the RAM freed for reuse, just like swapping. Dnsmasq allows log-file rotation in the conventional way. The simplest thing to do i
Re: [Dnsmasq-discuss] Memory usage growing over time — possibly due to logging or caching (v2.90 / OpenEmbedded kirkstone)
On Mon, Apr 14, 2025 at 12:55:03PM -0400, Nitesh Divecha via Dnsmasq-discuss wrote: > > On Apr 14, 2025, at 10:48 AM, Simon Kelley wrote: > > > > Do you only see this memory-size growth when logging to a file? It's > > possible that there's a memory leak elsewhere, but unlikely that > > one that big could have been undetected until now. > > > > It's possible that the libc in use is memory-mapping the log > > file. Dnsmasq uses very standard open() and write() calls, but it's > > up to the libc how it implements that. Memory-mapping the file and > > converting write() calls to memory writes would be a legitimate > > thing to do. In this case some measures of the process's memory > > usage would increase, but the actual physical RAM usage would not - > > long untouched pages will be written out to the filesystem and the > > RAM freed for reuse, just like swapping. > > > > Dnsmasq allows log-file rotation in the conventional way. The > > simplest thing to do is to delete the log file and then send the > > dnsmasq process SIGUSR2. That makes dnsmasq close and reopen the > > file. The close removes the last reference to the deleted file and > > its disk storage is freed-up. The re-open creates a new log file > > which begins empty. Real logfile rotation renames the live file > > instead of deleting it, and only deletes old logfiles after a few > > days: see man logrotate for details on that. > > > > Yes, when I turn on --log-facility=/var/log/dnsmasq.log, I see memory-size > growing. :-) > # systemctl status dnsmasq > * dnsmasq.service - DNS forwarder and DHCP server > Active: active (running) since Sat 2025-04-12 21:27:30 UTC; 15h ago > Memory: 16.3M > > > # ls -lah /var/log/dnsmasq.log > -rw-rw 1 nobody root 16M Apr 13 13:05 /var/log/dnsmasq.log > > > And if I restart the service it will reset the log file and memory-size will > go down. > > # systemctl status dnsmasq --no-pager > ● dnsmasq.service - DNS forwarder and DHCP server > Active: active (running) since Mon 2025-04-14 16:08:27 UTC; 43min ago > Memory: 1.3M > > # ls -lah /var/log/dnsmasq.log > -rw-rw 1 nobody root 1.2M Apr 14 16:52 /var/log/dnsmasq.log > Groeten Geert Stappers -- Silence is hard to parse ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Memory usage growing over time — possibly due to logging or caching (v2.90 / OpenEmbedded kirkstone)
On Mon, 14 Apr 2025, Nitesh Divecha via Dnsmasq-discuss wrote: I've observed that dnsmasq memory usage steadily increases over time, It's possible that the libc in use is memory-mapping the log file. [...] In this case some measures of the process's memory usage would increase, but the actual physical RAM usage would not - long untouched pages will be written out to the filesystem and the RAM freed for reuse, just like swapping. # systemctl status dnsmasq [...] Memory: 16.3M This is a very coarse view of memory usage and doesn't tell you much about what's actually going on. For a much more detailed view of how the dnsmasq process uses memory, look in /proc/$(pidof dnsmasq)/smaps or in the smaps_rollup file for a summary, or you can run `pmap -X $(pidof dnsmasq)` for a slightly more compact, columnar view. If you monitor the contents of this file or the pmap output over time, you should be able to see where the memory is being used. https://docs.kernel.org/filesystems/proc.html has information about what the different fields mean. Memory management gets a little hairy, but the most important parts (IMO and as I understand it) is "RSS", which shows how much of that mapping is using up actual, physical RAM; "Shared", which shows how much memory is shared with at least one other process (thus only taking up memory once even though it's mapped into multiple processes) vs "Private", which is unique to this process; and "Clean" which has not been modified compared to the underlying storage and could thus be evicted from memory if space is needed vs "Dirty", which must first be written to storage before it can be reclaimed. -- Peter Tirsek ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Memory usage growing over time — possibly due to logging or caching (v2.90 / OpenEmbedded kirkstone)
Hello Simon, Thank you for your feedback. Yes, when I turn on --log-facility=/var/log/dnsmasq.log, I see memory-size growing. # systemctl status dnsmasq * dnsmasq.service - DNS forwarder and DHCP server Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/dnsmasq.service.d `-override.conf Active: active (running) since Sat 2025-04-12 21:27:30 UTC; 15h ago Process: 427 ExecStartPre=/usr/bin/dnsmasq --test (code=exited, status=0/SUCCESS) Process: 434 ExecStart=/usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --local-service (code=exited, status=0/SUCCESS) Main PID: 437 (dnsmasq) Tasks: 1 (limit: 1042) Memory: 16.3M CGroup: /system.slice/dnsmasq.service `- 437 /usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --local-service # ls -lah /var/log/dnsmasq.log -rw-rw 1 nobody root 16M Apr 13 13:05 /var/log/dnsmasq.log And if I restart the service it will reset the log file and memory-size will go down. # systemctl status dnsmasq --no-pager ● dnsmasq.service - DNS forwarder and DHCP server Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/dnsmasq.service.d └─override.conf Active: active (running) since Mon 2025-04-14 16:08:27 UTC; 43min ago Process: 2238 ExecStartPre=/usr/bin/dnsmasq --test (code=exited, status=0/SUCCESS) Process: 2239 ExecStart=/usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --bind-dynamic --local-service --listen-address=127.0.0.1 --listen-address=192.0.2.2 (code=exited, status=0/SUCCESS) Main PID: 2242 (dnsmasq) Tasks: 1 (limit: 1042) Memory: 1.3M CGroup: /system.slice/dnsmasq.service └─ 2242 /usr/bin/dnsmasq -x /run/dnsmasq.pid -7 /etc/NetworkManager/dnsmasq.d --bind-dynamic --local-service --listen-address=127.0.0.1 --listen-address=192.0.2.2 Apr 14 16:08:27 2026240055 systemd[1]: Starting DNS forwarder and DHCP server... Apr 14 16:08:27 2026240055 dnsmasq[2238]: dnsmasq: syntax check OK. Apr 14 16:08:27 2026240055 systemd[1]: Started DNS forwarder and DHCP server. # ls -lah /var/log/dnsmasq.log -rw-rw 1 nobody root 1.2M Apr 14 16:52 /var/log/dnsmasq.log Cheers, Nitesh > On Apr 14, 2025, at 10:48 AM, Simon Kelley wrote: > > > > On 4/13/25 13:45, Nitesh Divecha via Dnsmasq-discuss wrote: >> Hi all, >> I'm using dnsmasq version 2.90 in an embedded Linux project based on the >> Yocto Project (kirkstone branch). The recipe in use is from the OpenEmbedded >> meta-networking layer: >> https://layers.openembedded.org/layerindex/recipe/304520/ >> I've observed that dnsmasq memory usage steadily increases over time, even >> under light DNS query loads. For example: >> - On boot: ~1.1 MB >> - After 1 hour: ~3.3 MB >> - After ~14 hours: ~15.3 MB >> Relevant configuration (via NetworkManager): >> # cat /etc/NetworkManager/dnsmasq.d/dnsmasq-dns.conf >> no-negcache >> stop-dns-rebind >> server=8.8.8.8 >> server=1.1.1.1 >> all-servers >> log-queries >> log-facility=/var/log/dnsmasq.log >> I noticed that the `/var/log/dnsmasq.log` file grows continuously (currently >> ~15M), and suspect that this may be contributing to memory growth — either >> due to internal buffering or cumulative logging state. >> My questions: >> 1. Is this memory growth expected when `log-queries` is enabled with >> `log-facility`? >> 2. Does `dnsmasq` buffer log output internally, or are there known issues >> with memory not being reclaimed after prolonged logging? >> 3. Are there recommended best practices for logging in long-running embedded >> environments to avoid this kind of memory usage? >> Any suggestions or insights would be greatly appreciated. >> Thanks in advance, >> Cheers, >> Nitesh > > Do you only see this memory-size growth when logging to a file? It's possible > that there's a memory leak elsewhere, but unlikely that one that big could > have been undetected until now. > > It's possible that the libc in use is memory-mapping the log file. Dnsmasq > uses very standard open() and write() calls, but it's up to the libc how it > implements that. Memory-mapping the file and converting write() calls to > memory writes would be a legitimate thing to do. In this case some measures > of the process's memory usage would increase, but the actual physical RAM > usage would not - long untouched pages will be written out to the filesystem > and the RAM freed for reuse, just like swapping. > > Dnsmasq allows log-file rotation in the conventional way. The simplest thing > to do is to delete the log file and then send the dnsmasq process SIGUSR2. > That makes dnsmasq close and reopen the file. The close removes the last > reference to the deleted file and its disk storage is freed-up. The re-open > creates a new log file which begins empty. Real logfil
Re: [Dnsmasq-discuss] Memory usage growing over time — possibly due to logging or caching (v2.90 / OpenEmbedded kirkstone)
On 4/13/25 13:45, Nitesh Divecha via Dnsmasq-discuss wrote: Hi all, I'm using dnsmasq version 2.90 in an embedded Linux project based on the Yocto Project (kirkstone branch). The recipe in use is from the OpenEmbedded meta-networking layer: https://layers.openembedded.org/layerindex/recipe/304520/ I've observed that dnsmasq memory usage steadily increases over time, even under light DNS query loads. For example: - On boot: ~1.1 MB - After 1 hour: ~3.3 MB - After ~14 hours: ~15.3 MB Relevant configuration (via NetworkManager): # cat /etc/NetworkManager/dnsmasq.d/dnsmasq-dns.conf no-negcache stop-dns-rebind server=8.8.8.8 server=1.1.1.1 all-servers log-queries log-facility=/var/log/dnsmasq.log I noticed that the `/var/log/dnsmasq.log` file grows continuously (currently ~15M), and suspect that this may be contributing to memory growth — either due to internal buffering or cumulative logging state. My questions: 1. Is this memory growth expected when `log-queries` is enabled with `log-facility`? 2. Does `dnsmasq` buffer log output internally, or are there known issues with memory not being reclaimed after prolonged logging? 3. Are there recommended best practices for logging in long-running embedded environments to avoid this kind of memory usage? Any suggestions or insights would be greatly appreciated. Thanks in advance, Cheers, Nitesh Do you only see this memory-size growth when logging to a file? It's possible that there's a memory leak elsewhere, but unlikely that one that big could have been undetected until now. It's possible that the libc in use is memory-mapping the log file. Dnsmasq uses very standard open() and write() calls, but it's up to the libc how it implements that. Memory-mapping the file and converting write() calls to memory writes would be a legitimate thing to do. In this case some measures of the process's memory usage would increase, but the actual physical RAM usage would not - long untouched pages will be written out to the filesystem and the RAM freed for reuse, just like swapping. Dnsmasq allows log-file rotation in the conventional way. The simplest thing to do is to delete the log file and then send the dnsmasq process SIGUSR2. That makes dnsmasq close and reopen the file. The close removes the last reference to the deleted file and its disk storage is freed-up. The re-open creates a new log file which begins empty. Real logfile rotation renames the live file instead of deleting it, and only deletes old logfiles after a few days: see man logrotate for details on that. Cheers, Simon. ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
[Dnsmasq-discuss] Memory usage growing over time — possibly due to logging or caching (v2.90 / OpenEmbedded kirkstone)
Hi all, I'm using dnsmasq version 2.90 in an embedded Linux project based on the Yocto Project (kirkstone branch). The recipe in use is from the OpenEmbedded meta-networking layer: https://layers.openembedded.org/layerindex/recipe/304520/ I've observed that dnsmasq memory usage steadily increases over time, even under light DNS query loads. For example: - On boot: ~1.1 MB - After 1 hour: ~3.3 MB - After ~14 hours: ~15.3 MB Relevant configuration (via NetworkManager): # cat /etc/NetworkManager/dnsmasq.d/dnsmasq-dns.conf no-negcache stop-dns-rebind server=8.8.8.8 server=1.1.1.1 all-servers log-queries log-facility=/var/log/dnsmasq.log I noticed that the `/var/log/dnsmasq.log` file grows continuously (currently ~15M), and suspect that this may be contributing to memory growth — either due to internal buffering or cumulative logging state. My questions: 1. Is this memory growth expected when `log-queries` is enabled with `log-facility`? 2. Does `dnsmasq` buffer log output internally, or are there known issues with memory not being reclaimed after prolonged logging? 3. Are there recommended best practices for logging in long-running embedded environments to avoid this kind of memory usage? Any suggestions or insights would be greatly appreciated. Thanks in advance, Cheers, Nitesh ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
