Re: [dnsdist] dnsdist capacity
Hi Alexey, On 11/9/18 8:02 PM, Alexey Bokhanovskiy wrote: > How many the same backend servers I can use (your reccomendation for a > lot of qps)? It's a bit complicated to answer without knowing more about your setup and the type of traffic you are seeing. I think you should really set up some metrics using carbon/grafana or prometheus, and look at what the bottleneck is (CPU, network, interrupts, context switchs..). Note that if you are using the latest version, you can display the name of the various dnsdist threads to known which one is CPU-bound for example. Best regards, Remi > > *От:* dnsdist от имени Remi > Gacogne > *Отправлено:* 9 ноября 2018 г. 19:41 > *Кому:* dnsdist@mailman.powerdns.com > *Тема:* Re: [dnsdist] dnsdist capacity > > Hi Alexey, > > On 11/9/18 2:07 PM, Alexey Bokhanovskiy wrote: >> Now we are using two DNS loadbalancers with seven servers powerdns and >> we found that the capacity loadbalancer has limit approximately 50k qps >> >> Our version of dnsdist is 1.3.3-1pdns.bionic. >> >> Is 50kqps the limit of software or hardware? Or need to additional >> configuration? > > This seems ridiculously low for your hardware, but it's hard to know for > sure without seeing your configuration. > For what it's worth, you should be able to reach easily 450k QPS without > the packet cache being enabled and much more with it. I have seen one > dnsdist handle around 1M qps at 100% cache hit ratio in an ideal setup > in a lab. You should never reach that over a single dnsdist in > production, but still :) > > You should first read [1], and tune at least setMaxUDPOutstanding(), use > several threads to handle incoming queries and consider adding several > newServer() directives (adding the same backend twice might help because > you then have more threads handling the responses) and/or use the > 'sockets' parameter on the 'newServer()' directive depending on your setup. > > Then the next step should be to get some metrics using `carbonServer()` > [2] and metronome [3], for example, so you know what's going on. > Then you might want to upgrade your kernel (recent Linux kernels are a > lot better at handling UDP packets) and tune your OS and your network cards. > > I would advise removing any rule that you have in dnsdist's > configuration first and using one of the built-in load-balancing policy > to be sure that Lua isn't your bottleneck. Please also remember that > without the cache, dnsdist won't be faster than your backends, so > perhaps benchmark them as well. > > Good luck! > > [1]: https://dnsdist.org/advanced/tuning.html > [2]: https://dnsdist.org/guides/carbon.html > [3]: https://github.com/ahupowerdns/metronome > > Best regards, > -- > Remi Gacogne > PowerDNS.COM BV - https://www.powerdns.com/ > > > IMPORTANT NOTICE > The content of this e-mail is intended for the addressee(s) only and may > contain information that is confidential and/or otherwise protected from > disclosure. If you are not the intended recipient, please note that any > copying, distribution or any other use or dissemination of the > information contained in this e-mail (and its attachments) is strictly > prohibited. If you have received this e-mail in error, kindly notify the > sender immediately by replying to this e-mail and delete the e-mail and > any copies thereof. > > Tele2 AB (publ) and its subsidiaries (“Tele2 Group”) accepts no > responsibility for the consequences of any viruses, corruption or other > interference transmitted by e-mail. -- Remi Gacogne PowerDNS.COM BV - https://www.powerdns.com/ signature.asc Description: OpenPGP digital signature ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist capacity
> How many the same backend servers I can use (your reccomendation for a lot of > qps)? > > Could you explain more information about the 'sockets' parameter on the > 'newServer()' . Is it mean 'reuseport'? > Thank you for your support! "sockets" increases the number of connections to the same backend: sockets=NUM-- Number of sockets (and thus source ports) used toward the backend server, defaults to a single one From https://dnsdist.org/reference/config.html?highlight=sockets Try 8 and see what happens. Bert ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist capacity
Hi Remi,
Thank you for your answer and recommendation.
We have basic configuration loadbalancer and seems our problem in it.
I will try to add the same backend server (now we have just one record for each
separate server):
newServer({address="127.0.0.1:5300", qps=10, name="lacalpdns10"})
newServer({address="x.x.x.x", qps=20, name="pdns12"})
newServer({address="x.x.x.x", qps=20, name="pdns13"})
etc.
How many the same backend servers I can use (your reccomendation for a lot of
qps)?
Could you explain more information about the 'sockets' parameter on the
'newServer()' . Is it mean 'reuseport'?
Thank you for your support!
BR, Alexey
От: dnsdist от имени Remi Gacogne
Отправлено: 9 ноября 2018 г. 19:41
Кому: dnsdist@mailman.powerdns.com
Тема: Re: [dnsdist] dnsdist capacity
Hi Alexey,
On 11/9/18 2:07 PM, Alexey Bokhanovskiy wrote:
> Now we are using two DNS loadbalancers with seven servers powerdns and
> we found that the capacity loadbalancer has limit approximately 50k qps
>
> Our version of dnsdist is 1.3.3-1pdns.bionic.
>
> Is 50kqps the limit of software or hardware? Or need to additional
> configuration?
This seems ridiculously low for your hardware, but it's hard to know for
sure without seeing your configuration.
For what it's worth, you should be able to reach easily 450k QPS without
the packet cache being enabled and much more with it. I have seen one
dnsdist handle around 1M qps at 100% cache hit ratio in an ideal setup
in a lab. You should never reach that over a single dnsdist in
production, but still :)
You should first read [1], and tune at least setMaxUDPOutstanding(), use
several threads to handle incoming queries and consider adding several
newServer() directives (adding the same backend twice might help because
you then have more threads handling the responses) and/or use the
'sockets' parameter on the 'newServer()' directive depending on your setup.
Then the next step should be to get some metrics using `carbonServer()`
[2] and metronome [3], for example, so you know what's going on.
Then you might want to upgrade your kernel (recent Linux kernels are a
lot better at handling UDP packets) and tune your OS and your network cards.
I would advise removing any rule that you have in dnsdist's
configuration first and using one of the built-in load-balancing policy
to be sure that Lua isn't your bottleneck. Please also remember that
without the cache, dnsdist won't be faster than your backends, so
perhaps benchmark them as well.
Good luck!
[1]: https://dnsdist.org/advanced/tuning.html
[2]: https://dnsdist.org/guides/carbon.html
[3]: https://github.com/ahupowerdns/metronome
Best regards,
--
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/
IMPORTANT NOTICE
The content of this e-mail is intended for the addressee(s) only and may
contain information that is confidential and/or otherwise protected from
disclosure. If you are not the intended recipient, please note that any
copying, distribution or any other use or dissemination of the information
contained in this e-mail (and its attachments) is strictly prohibited. If you
have received this e-mail in error, kindly notify the sender immediately by
replying to this e-mail and delete the e-mail and any copies thereof.
Tele2 AB (publ) and its subsidiaries ("Tele2 Group") accepts no responsibility
for the consequences of any viruses, corruption or other interference
transmitted by e-mail.
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist capacity
Hi Alexey, On 11/9/18 2:07 PM, Alexey Bokhanovskiy wrote: > Now we are using two DNS loadbalancers with seven servers powerdns and > we found that the capacity loadbalancer has limit approximately 50k qps > > Our version of dnsdist is 1.3.3-1pdns.bionic. > > Is 50kqps the limit of software or hardware? Or need to additional > configuration? This seems ridiculously low for your hardware, but it's hard to know for sure without seeing your configuration. For what it's worth, you should be able to reach easily 450k QPS without the packet cache being enabled and much more with it. I have seen one dnsdist handle around 1M qps at 100% cache hit ratio in an ideal setup in a lab. You should never reach that over a single dnsdist in production, but still :) You should first read [1], and tune at least setMaxUDPOutstanding(), use several threads to handle incoming queries and consider adding several newServer() directives (adding the same backend twice might help because you then have more threads handling the responses) and/or use the 'sockets' parameter on the 'newServer()' directive depending on your setup. Then the next step should be to get some metrics using `carbonServer()` [2] and metronome [3], for example, so you know what's going on. Then you might want to upgrade your kernel (recent Linux kernels are a lot better at handling UDP packets) and tune your OS and your network cards. I would advise removing any rule that you have in dnsdist's configuration first and using one of the built-in load-balancing policy to be sure that Lua isn't your bottleneck. Please also remember that without the cache, dnsdist won't be faster than your backends, so perhaps benchmark them as well. Good luck! [1]: https://dnsdist.org/advanced/tuning.html [2]: https://dnsdist.org/guides/carbon.html [3]: https://github.com/ahupowerdns/metronome Best regards, -- Remi Gacogne PowerDNS.COM BV - https://www.powerdns.com/ signature.asc Description: OpenPGP digital signature ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
