Re: Unbound DNS entry pre(caching)
On 4/23/19 11:47 AM, Tihomir Loncaric via Unbound-users wrote: Thanks to all three of you for quick answers and help. All valid points, except first answer with auth-zone I don't fully understand on how to leverage. :) Spam/adblock is already implemented through web-filter. Earlier I have been pre-caching (pinning) entries with Alexa top 1000 sites list ... > ... c) b doesn't allow me to create our own actual top 1000 list based on actual resolving of user entries If you really wanted to create your own popular-query list, you could write a python script which would be called by the python module. A rather simple way would be to print query names to a text file. Let a cron job pick up the file later and do the number crunching. It might use a lot of disk space, but Unbound shouldn't be bogged down. It is just a place to start and improve from. -Eric
Re: Unbound DNS entry pre(caching)
Thanks to all three of you for quick answers and help. All valid points, except first answer with auth-zone I don't fully understand on how to leverage. :)Spam/adblock is already implemented through web-filter. Earlier I have been pre-caching (pinning) entries with Alexa top 1000 sites list, refreshed by querying every 3 hours, list filtered where TTL < 3 hours But couple issues there: a) if TTL is greater than refresh period (3 hours), request would be answered locally by DNS server without refreshing of DNS cached entry, until actually entry expired from DNS cache b) it doesn't refresh sites that guests are actually accessing, but only top 1000 sites. Unbound pre-caching should help here. c) b doesn't allow me to create our own actual top 1000 list based on actual resolving of user entries Now current plan is: - set cache-min-ttl to 1 hour (tradeoff)- use pre-caching- execute top 1000 sites list load on on unbound service startup and every 3 hours- restart unbound service after each cruise I believe all should help, but again not sure if anything else can be done. It would be nice if entries not asked for predefined amount of time (eg. 2 weeks cruise) could be automatically phased out of cache in accordance to configuration parameter, so cache doesn't get filled with stale unused entries and refreshes of such entries don't use up bandwidth.This would also eliminate need to restart unbound service and loose both cache & usage statistics. Not sure if unbound has any MRU/MFU counters for DNS entries, so I could purge entries that are not used often/recently by some script?For example after 2 weeks I execute command to purge any entry that was not used at least n times or for x minutes and keep all others in the cache.This would trim the cache and keep it maintainable. When would entries for which cache-min-ttl applies be refreshed if also pre-caching is also used?Eg. entry that is set for TTL of 60 seconds gets increased to 3600 seconds TTL by setting cache-min-ttl parameter to 3600.When would entry be refreshed due to pre-caching refresh when 90% TTL expires? In other words, would refresh of such entry happen after 54 seconds or after 3240 seconds? Also not being able to control this 90% parameter and need to get entry queried during last 10% of TTL in order to refresh it will force flush some of the entries from cache without me being able to pin them.This will not be an issue for top 1000 sites as they are known, but will be for any guest-accessed site not known to me in advance. Again thanks for all the help! Tiho On Tuesday, April 23, 2019, 3:02:55 PM GMT+2, Daisuke HIGASHI wrote: Tihomir Loncaric via Unbound-users : Is there anything else that I could use out of the box? What other existing parameters would help towards this caching goal? If you have complete list of domainames to be cached, keep making queries (forever) to your DNS server e.g. while :; do dnsperf -Q 100 -s 127.0.0.1 -d querylist; done
Re: Unbound DNS entry pre(caching)
Tihomir Loncaric via Unbound-users : > > Is there anything else that I could use out of the box? What other > existing parameters would help towards this caching goal? > If you have complete list of domainames to be cached, keep making queries (forever) to your DNS server e.g. while :; do dnsperf -Q 100 -s 127.0.0.1 -d querylist; done
Re: Unbound DNS entry pre(caching)
On 4/22/19 8:02 AM, ѽ҉ᶬḳ℠ via Unbound-users wrote: To mitigate upstream queries (save bandwidth, speed up queries, enhance privacy) it might be worthwhile to consider (pre)serving a copy of the root (.) for local usage via auth-zone as described in example.conf.in (Authority zones) in the package documentation. On 22/04/2019 13:30, Tihomir Loncaric via Unbound-users wrote: Hi all, Wanted to congratulate you on great work with unbound ! My use case of unbound is on ships using satellite uplinks, so in other words high-latency and high-bandwidth... relatively speaking, but surely enough bandwidth for DNS queries. So idea would be to cache and then preemptively re-cache DNS queries as much as possible so to speed up Internet access for users. This could cut up to 500-800 ms from every DNS query and remove lag on DNS side. This, together with WAN TCP optimization (SYN) would make satellite uplink not so laggy for users onboard. So I notice most of the DNS entries rarely change and local unbound onboard could surely cache lots of entries considering memory and CPU are available nowadays. Thus instead of expiring cached entries after TTL I would like to keep refreshing them regularly and keep them available for some pre-defined time (eg. 2-3 weeks configurable) due to cruise length. I believe this proactive approach with cache & refresh would be more appropriate for such environment. Checking out for options in Unbound I have identified couple of mechanisms to enable this but all seem to lack some features. Prefetch is great feature but seems somewhat limited for entries to be refreshed during last 10% of TTL and only if user resolve entry during that last 10% of TTL time. Furthermore that 10% seems not configurable in config. I know setting it like this increases cache hit ratio for often used entries (ones that also get hit during last 10%) but is not flexible enough. I am trying to cache much beyond that time frame (2-3 weeks - parameter 1) and cannot always guarantee users will be resolving within last 10% of TTL (eg. during night) so I would like to set automated refresh to do refresh on 90% TTL, if DNS entry was asked for more or equal to 0 ... n times after being cached (parameter 2). Of course all up to maximum number of cached entries which would be set appropriately. This would allow for preemptive caching based on number of times entry is queried during TTL and overall length of time to keep such entries in cache. So in other words we would trade-off some bandwidth used in order to reduce DNS latency. Serve-expired is another great feature, but what I am proposing above would work similarly and wouldn't break DNS in case entries are changed, though with some bandwidth trade-off for refreshes. cache-min-ttl would definitely break certain resolutions, but I would use it with 30 - 60 min TTL which is sensible trade off so refresh doesn't happen too often and any changes are still picked up with regular refreshes. Is there anything else that I could use out of the box? What other existing parameters would help towards this caching goal? Thanks, Tiho The options related to fetch or prefetch may also help. A little more sophisticated, download a favorite spam/adblock list and install those domains as "local-zone: static". This may prevent noisy nonsense from slowing down web browsing. -Eric
Re: Unbound DNS entry pre(caching)
To mitigate upstream queries (save bandwidth, speed up queries, enhance privacy) it might be worthwhile to consider (pre)serving a copy of the root (.) for local usage via auth-zone as described in example.conf.in (Authority zones) in the package documentation. On 22/04/2019 13:30, Tihomir Loncaric via Unbound-users wrote: Hi all, Wanted to congratulate you on great work with unbound ! My use case of unbound is on ships using satellite uplinks, so in other words high-latency and high-bandwidth... relatively speaking, but surely enough bandwidth for DNS queries. So idea would be to cache and then preemptively re-cache DNS queries as much as possible so to speed up Internet access for users. This could cut up to 500-800 ms from every DNS query and remove lag on DNS side. This, together with WAN TCP optimization (SYN) would make satellite uplink not so laggy for users onboard. So I notice most of the DNS entries rarely change and local unbound onboard could surely cache lots of entries considering memory and CPU are available nowadays. Thus instead of expiring cached entries after TTL I would like to keep refreshing them regularly and keep them available for some pre-defined time (eg. 2-3 weeks configurable) due to cruise length. I believe this proactive approach with cache & refresh would be more appropriate for such environment. Checking out for options in Unbound I have identified couple of mechanisms to enable this but all seem to lack some features. Prefetch is great feature but seems somewhat limited for entries to be refreshed during last 10% of TTL and only if user resolve entry during that last 10% of TTL time. Furthermore that 10% seems not configurable in config. I know setting it like this increases cache hit ratio for often used entries (ones that also get hit during last 10%) but is not flexible enough. I am trying to cache much beyond that time frame (2-3 weeks - parameter 1) and cannot always guarantee users will be resolving within last 10% of TTL (eg. during night) so I would like to set automated refresh to do refresh on 90% TTL, if DNS entry was asked for more or equal to 0 ... n times after being cached (parameter 2). Of course all up to maximum number of cached entries which would be set appropriately. This would allow for preemptive caching based on number of times entry is queried during TTL and overall length of time to keep such entries in cache. So in other words we would trade-off some bandwidth used in order to reduce DNS latency. Serve-expired is another great feature, but what I am proposing above would work similarly and wouldn't break DNS in case entries are changed, though with some bandwidth trade-off for refreshes. cache-min-ttl would definitely break certain resolutions, but I would use it with 30 - 60 min TTL which is sensible trade off so refresh doesn't happen too often and any changes are still picked up with regular refreshes. Is there anything else that I could use out of the box? What other existing parameters would help towards this caching goal? Thanks, Tiho
