Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Fab ! Thanks ! On 1 December 2016 at 16:18, Ralph Dolmans via Unbound-users wrote: > Local data is handled before the cache lookup. So the cache entries will > be ignored as soon as there is matching local data. > > Regards, > -- Ralph > > > On 01-12-16 16:53, Over Dexia via Unbound-users wrote: >> Am 30.11.2016 um 17:41 schrieb Tim Smith via Unbound-users: >>> Interesting idea, nice bit of thinking outside the box ! But in >>> relation to "should be alright after cache data times out" , does >>> that mean that the idea might not work too well if I've got "prefetch: >>> yes" in my config ? >> >> In the situation where a cached resolution has been prefetched before >> the dump, and that site is supposed to be blacklisted/nullrouted, you'd >> add >=90% of the ttl (up to cache-max-ttl) to the time until it blacks >> out for good ... >> >> ... *unless* unbound either checks local data before the cache, or the >> load_cache checks against local data and doesn't taint the cache with >> the entries loaded, I'd assume. But essentially, that's beyond my >> knowledge of the execution flow... j. >>
Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Local data is handled before the cache lookup. So the cache entries will be ignored as soon as there is matching local data. Regards, -- Ralph On 01-12-16 16:53, Over Dexia via Unbound-users wrote: > Am 30.11.2016 um 17:41 schrieb Tim Smith via Unbound-users: >> Interesting idea, nice bit of thinking outside the box ! But in >> relation to "should be alright after cache data times out" , does >> that mean that the idea might not work too well if I've got "prefetch: >> yes" in my config ? > > In the situation where a cached resolution has been prefetched before > the dump, and that site is supposed to be blacklisted/nullrouted, you'd > add >=90% of the ttl (up to cache-max-ttl) to the time until it blacks > out for good ... > > ... *unless* unbound either checks local data before the cache, or the > load_cache checks against local data and doesn't taint the cache with > the entries loaded, I'd assume. But essentially, that's beyond my > knowledge of the execution flow... j. >
Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Am 30.11.2016 um 17:41 schrieb Tim Smith via Unbound-users: > Interesting idea, nice bit of thinking outside the box ! But in > relation to "should be alright after cache data times out" , does > that mean that the idea might not work too well if I've got "prefetch: > yes" in my config ? In the situation where a cached resolution has been prefetched before the dump, and that site is supposed to be blacklisted/nullrouted, you'd add >=90% of the ttl (up to cache-max-ttl) to the time until it blacks out for good ... ... *unless* unbound either checks local data before the cache, or the load_cache checks against local data and doesn't taint the cache with the entries loaded, I'd assume. But essentially, that's beyond my knowledge of the execution flow... j.
Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Interesting idea, nice bit of thinking outside the box ! But in relation to "should be alright after cache data times out" , does that mean that the idea might not work too well if I've got "prefetch: yes" in my config ? On 30 November 2016 at 15:59, Over Dexia via Unbound-users wrote: > > Or you go back to the reload and do a dump_cache before and a > load_cache after that... > Yeah, I know, it's not entirely clean (after all, there might have been > redirects loaded which are also present in the cache. Should be alright > after the cache data times out, though.) > > regards, j. > > >
Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Am 30.11.2016 um 15:25 schrieb Tim Smith via Unbound-users: > Hi Ralph, > > Hmm... bad news ...have just looked into it and OpenBSD default config is : > remote-control: > control-enable: yes > control-use-cert: no > control-interface: /var/run/unbound.sock > > > So guess I'll have to live with it until the next version. Or you go back to the reload and do a dump_cache before and a load_cache after that... Yeah, I know, it's not entirely clean (after all, there might have been redirects loaded which are also present in the cache. Should be alright after the cache data times out, though.) regards, j.
Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Hi Ralph, Hmm... bad news ...have just looked into it and OpenBSD default config is : remote-control: control-enable: yes control-use-cert: no control-interface: /var/run/unbound.sock So guess I'll have to live with it until the next version. Thanks again. On 30 November 2016 at 13:36, Ralph Dolmans via Unbound-users wrote: > Hi, > > Unbound-control sets up a TLS connection for every command. Setting up > 30k TLS connections one after another does take some time. > > If your unbound daemon and unbound-control are on the same machine you > could try to communicate over a secured local socket and disable > control-use-cert. This might speed it up a little. > > I just added some code to Unbound to bulk add and remove local-zones and > local-data, by reading input from stdin. This gives the possibility to > do the update over a single connection using local_zones, > local_zones_remove, local_datas and local_datas_remove. This code will > be in the next Unbound release. > > Regards, > -- Ralph > > On 28-11-16 13:18, Tim Smith via Unbound-users wrote: >> Hello list, >> >> I'm running Unbound 1.5.9 on OpenBSD 6. >> >> I have the need to periodically add certain local data to unbound as >> part of a dynamic security block list (i.e. redirect to null). These >> lists can be 30k+ items in length. >> >> Originally, I had this scripted as follows : >>- Get and parse data into unbound config format >>- Reload unbound >> >> Obviously the problem with that approach is that reloading unbound >> causes cache to be flushed which is not cool. However on the >> plus side the script runs super-quick as its just a config reload ! >> >> So I looked at my options and came up with a second script : >>- Get and parse data into two groups : remove and add >>- On "remove" set run unbound-control local_zone_remove >>- On "add" set run unbound-control local_zone and unbound-control >> local_data >> >> The problem is its taking forever to process that because calls to >> unbound-control are so slow ! e.g. its been well over an hour now and >> it *still* hasn't finished processing 30k items. The server is not >> slow, the server is not heavily loaded, and my script is simple >> the problem is very much the slowness of the calls to unbound-control. >> >> Help !! >> >> Thanks >> >> Tim >>
Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Hi Ralph, Thanks for your reply. Interesting idea to try a local socket, will look into that some more. Good to hear you added bulk add/remove ... that was on my wish list ! On 30 November 2016 at 13:36, Ralph Dolmans via Unbound-users On 30 November 2016 at 13:36, Ralph Dolmans via Unbound-users wrote: > Hi, > > Unbound-control sets up a TLS connection for every command. Setting up > 30k TLS connections one after another does take some time. > > If your unbound daemon and unbound-control are on the same machine you > could try to communicate over a secured local socket and disable > control-use-cert. This might speed it up a little. > > I just added some code to Unbound to bulk add and remove local-zones and > local-data, by reading input from stdin. This gives the possibility to > do the update over a single connection using local_zones, > local_zones_remove, local_datas and local_datas_remove. This code will > be in the next Unbound release. > > Regards, > -- Ralph > > On 28-11-16 13:18, Tim Smith via Unbound-users wrote: >> Hello list, >> >> I'm running Unbound 1.5.9 on OpenBSD 6. >> >> I have the need to periodically add certain local data to unbound as >> part of a dynamic security block list (i.e. redirect to null). These >> lists can be 30k+ items in length. >> >> Originally, I had this scripted as follows : >>- Get and parse data into unbound config format >>- Reload unbound >> >> Obviously the problem with that approach is that reloading unbound >> causes cache to be flushed which is not cool. However on the >> plus side the script runs super-quick as its just a config reload ! >> >> So I looked at my options and came up with a second script : >>- Get and parse data into two groups : remove and add >>- On "remove" set run unbound-control local_zone_remove >>- On "add" set run unbound-control local_zone and unbound-control >> local_data >> >> The problem is its taking forever to process that because calls to >> unbound-control are so slow ! e.g. its been well over an hour now and >> it *still* hasn't finished processing 30k items. The server is not >> slow, the server is not heavily loaded, and my script is simple >> the problem is very much the slowness of the calls to unbound-control. >> >> Help !! >> >> Thanks >> >> Tim >>
Re: Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?
Hi, Unbound-control sets up a TLS connection for every command. Setting up 30k TLS connections one after another does take some time. If your unbound daemon and unbound-control are on the same machine you could try to communicate over a secured local socket and disable control-use-cert. This might speed it up a little. I just added some code to Unbound to bulk add and remove local-zones and local-data, by reading input from stdin. This gives the possibility to do the update over a single connection using local_zones, local_zones_remove, local_datas and local_datas_remove. This code will be in the next Unbound release. Regards, -- Ralph On 28-11-16 13:18, Tim Smith via Unbound-users wrote: > Hello list, > > I'm running Unbound 1.5.9 on OpenBSD 6. > > I have the need to periodically add certain local data to unbound as > part of a dynamic security block list (i.e. redirect to null). These > lists can be 30k+ items in length. > > Originally, I had this scripted as follows : >- Get and parse data into unbound config format >- Reload unbound > > Obviously the problem with that approach is that reloading unbound > causes cache to be flushed which is not cool. However on the > plus side the script runs super-quick as its just a config reload ! > > So I looked at my options and came up with a second script : >- Get and parse data into two groups : remove and add >- On "remove" set run unbound-control local_zone_remove >- On "add" set run unbound-control local_zone and unbound-control > local_data > > The problem is its taking forever to process that because calls to > unbound-control are so slow ! e.g. its been well over an hour now and > it *still* hasn't finished processing 30k items. The server is not > slow, the server is not heavily loaded, and my script is simple > the problem is very much the slowness of the calls to unbound-control. > > Help !! > > Thanks > > Tim >
