Re: [Dnsmasq-discuss] Remove upper limit of 10,000 for cache size
On 09/05/18 10:21, Dominik DL6ER wrote: > Dear Geert and mailinglist members, > >> Thing I wonder about is how the cache size clipping was discovered. > I recently sent a SIGUSR1 to our dnsmasq because a user said that some > queries have continuously been answered NXDOMAIN although they > shouldn't. As they have been answered within less than a millisecond, I > thought it must be an issue with the cache. Make a long story short it > was their fault (an ordinary typo). > > However, by chance, I found > > May 4 18:06:13 dnsmasq[22952]: cache size 150, 2191/57999 cache > insertions re-used unexpired cache entries. > > which seemed odd to me. I then looked at the man page and increased the > cache size limit to 100,000 in the config but the startup message told > me that the cache size is only 10,000. The way to the location where > this clipping is done wasn't long from here and I figured removing might > just be the best solution. So with the previous cache size, less than 4% of queries cause eviction of a name from the cache before the time-to-live expires and it disappears anyway. Only a small fraction of those will be re-queried so the number of extra cache-misses is in the noise. I'd say that the cache size you had before was pretty much optimal, but in response you tried to increase it to more than the total number of cache insertions ever, during that run of dnsmasq. Of course you might have been restarting dnsmasq frequently, so that's a sample over a short period, but I'd guess otherwise, those numbers say that 100,000 makes no sense at all. Whenever I've looked at this, the optimal cache size is smaller that most people think. The local copy of dnsmasq here, which is handling a network of a dozen machines, has a tiny cache, and very similar stats. It just doesn't need to be any bigger. (and that instance is doing DNSSEC, so the cache holding all sorts of DNSSEC records too.) cache size 600, 50/28526 cache insertions re-used unexpired cache entries. > >> I'm trying to tell that the performance penality that Simon warns us about, >> might by canceled by high computing power. > I agree, but you should probably not be running a caching DNS server > with hundreds of active clients on a really low-power embedded machine > like the good old Raspberry Pi in its first version. > We have a dedicated (small) server for DNS, DHCP and email. dnsmasq is > able to handle DNS blazingly fast - even with a maximum cache size of > 100,000 (I may even want to increase this further if it prevents > deletion of unexpired cache entries). Be careful, the cache might have been set to 100,000 but based on the numbers above, the average number of names in the cache will have much been lower. The absolute upper-bound is 57999, but expiry of TTL and re-querying of common names with short TTLs will make it smaller still. We can't be sure that 100,000 is OK unless we make tests to actually fill the cache with 100,000 unique names with long TTLs, and then test _reverse_ lookups on that cache. > > I'm just trying to make clear that removing this artificial limit may > improve the situation for those on beefier hardware but not impact the > others as they are responsible for what they set when they decide to > manually tweak their settings in this regard. It's a value where I think > the hand-holding dnsmasq is doing for possibly supporting embedded > devices better is just too much. In the end, Simon has to say if or not > this artificial clipping can be removed or not. I think yes, because it > doesn't affect anyone who has not changed the default value and allows > the others to use any value for cache size them deem right for their > hardware and application. > The problem is that people naturally think "more is better" but my experience is that more is not necessarily better, and too big is definitely worse, though I don't have an up-to-date idea about how big is too big on the various classes of modern hardware. > Best regards, > Dominik > > > ___ > Dnsmasq-discuss mailing list > Dnsmasq-discuss@lists.thekelleys.org.uk > http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss > ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Remove upper limit of 10,000 for cache size
On Wed, May 09, 2018 at 11:21:31AM +0200, Dominik DL6ER wrote: > Geert Stappers wrote: > > [ ... ] I'm trying to tell that the performance penality that Simon > > warns us about, might by canceled by high computing power. > > I agree, but you should probably not be running a caching DNS server > with hundreds of active clients on a really low-power embedded machine > like the good old Raspberry Pi in its first version. :-) > I'm just trying to make clear that removing this artificial limit may > improve the situation for those on beefier hardware but not impact the > others as they are responsible for what they set when they decide to > manually tweak their settings in this regard. It's a value where I think > the hand-holding dnsmasq is doing for possibly supporting embedded > devices better is just too much. In the end, Simon has to say if or not > this artificial clipping can be removed or not. I think yes, because it > doesn't affect anyone who has not changed the default value and allows > the others to use any value for cache size them deem right for their > hardware and application. I also think that the clipping should be removed. Resubmitting the patch with a new commit message would be a good thing. (The original patch commit message would result in an ugly git log when `git am`, "git apply-mail" was used.)` I'll follow up this posting with a patch on the manual page. Groeten Geert Stappers -- Leven en laten leven ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Remove upper limit of 10,000 for cache size
Dear Geert and mailinglist members, > Thing I wonder about is how the cache size clipping was discovered. I recently sent a SIGUSR1 to our dnsmasq because a user said that some queries have continuously been answered NXDOMAIN although they shouldn't. As they have been answered within less than a millisecond, I thought it must be an issue with the cache. Make a long story short it was their fault (an ordinary typo). However, by chance, I found May 4 18:06:13 dnsmasq[22952]: cache size 150, 2191/57999 cache insertions re-used unexpired cache entries. which seemed odd to me. I then looked at the man page and increased the cache size limit to 100,000 in the config but the startup message told me that the cache size is only 10,000. The way to the location where this clipping is done wasn't long from here and I figured removing might just be the best solution. > I'm trying to tell that the performance penality that Simon warns us about, > might by canceled by high computing power. I agree, but you should probably not be running a caching DNS server with hundreds of active clients on a really low-power embedded machine like the good old Raspberry Pi in its first version. We have a dedicated (small) server for DNS, DHCP and email. dnsmasq is able to handle DNS blazingly fast - even with a maximum cache size of 100,000 (I may even want to increase this further if it prevents deletion of unexpired cache entries). I'm just trying to make clear that removing this artificial limit may improve the situation for those on beefier hardware but not impact the others as they are responsible for what they set when they decide to manually tweak their settings in this regard. It's a value where I think the hand-holding dnsmasq is doing for possibly supporting embedded devices better is just too much. In the end, Simon has to say if or not this artificial clipping can be removed or not. I think yes, because it doesn't affect anyone who has not changed the default value and allows the others to use any value for cache size them deem right for their hardware and application. Best regards, Dominik ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] Remove upper limit of 10,000 for cache size
On Tue, May 08, 2018 at 11:16:51PM +0200, Dominik wrote: > On 08.05.2018 22:17, Simon Kelley wrote: > > From: Dominik Derigs , Date: Tue, 8 May 2018 18:44:41 +0200 > >> [PATCH] Remove upper limit of 10,000 for cache size. > >> > >> We should allow users to set any (maximum) cache size they like to > >> set. Even embedded devices usually ship with at least 1 GB of RAM > >> nowadays so memory shouldn't be an issue. > >> > >> Furthermore, this clipping is also not documented in the man page. > > > > The reason for the limit is actually performance: there may be plenty of > > RAM, but the larger the cache is, the slower it is. This is true for > > reverse (PTR) queries, which are less optimised than normal forward queries. > > > > I accept that the limit may now be too small, but it would be worth > > doing some measurements of cache performance before raising or removing it. > > > > > Hey Simon, Hello Mailinglist, > removing the upper limit will not change anything except for the few > users that have set this value manually to a very large number. However, > if they did so they were surely not expecting that dnsmasq could just > ignore their setting. The undocumented clipping is indeed harmfull. > Personal experience with dnsmasq as a caching DNS server for a few > hundred fairly active clients shows no notable performance impact even > when allowing the maximum cache size to be > 100,000 (query reply time > from cache is on the order of < 5 msec). It may always be that I miss > something but even when dnsmasq's cache would be 10 (or more) times > slower, it would still be much faster than when we'd periodically ask > upstream servers. Yes, the benefit of caching. Thing that you might miss is how fast the dnsmasq server for the few hunderd fairly active clients is. I'm trying to tell that the performance penality that Simon warns us about, might by canceled by high computing power. Thing I wonder about is how the cache size clipping was discovered. Groeten Geert Stappers -- Leven en laten leven ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
