This is a good read on the problem as well: https://rsmitty.github.io/KubeDNS-Tweaks/
Basically greatly reduce the number of calls by tweaking some kube-dns settings. On Thursday, October 5, 2017 at 2:46:55 PM UTC-7, Evan Jones wrote: > My script is always looking up the same domain, and I believe it is cached by > dnsmasq. I think the limit is the kernel NAT connection tracking, because > each DNS query comes from a new ephemeral port, so it ends up using up all > NAT mappings on the node running kube-dns. This is why dnsPolicy: Default > fixes the problem: It uses the host's DNS configuration which avoids the NAT > connection limits. > > > Details including the Python code and configs to reproduce it on a brand new > GKE cluster are at the bottom of > https://github.com/kubernetes/kubernetes/issues/45976 > > > I did a separate test, using a Go DNS query generator, which was able to do > 80000 DNS queries per second, so dnsmasq does not appear to be the limit. > > > Thanks! > > Evan > > > > > On Thu, Oct 5, 2017 at 5:26 PM, Rodrigo Campos <rod...@sdfg.com.ar> wrote: > On Thu, Oct 05, 2017 at 04:29:21PM -0400, Evan Jones wrote: > > > The sustained 1000 qps comes from an application making that many outbound > > > connections. I agree that the application is very inefficient and shouldn't > > > be doing a DNS lookup for every request it sends, but it's a python program > > > that uses urllib2.urlopen so it creates a new connection each time. I > > > suspect this isn't that unusual? This could be a server that hits an > > > external service for every user request, for example. Given the activity on > > > the GitHub issues I linked, it appears I'm not the only person to have run > > > into this. > > > > But is always on different domains? If not, it can probably be cached (as long > > as the TTL allows) by the DNS server and, even if your app makes so many > > requests, it should be answered quite fast. > > > > > > -- > > You received this message because you are subscribed to a topic in the Google > Groups "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/kubernetes-users/7JBq6jhMZHc/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > kubernetes-use...@googlegroups.com. > > To post to this group, send email to kubernet...@googlegroups.com. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
