We encountered this issue too, and tried to counter it by lowering UDP
conntrack timeouts so that these entries expire more quickly. However, at
the time we found that the corresponding sysctls are not propagated into
network namespaces, so we now patch the global defaults in our kernel build
(patch attached). This works for us because we build a somewhat customized
kernel anyway, but it is a bit heavy handed.
/MR
On Fri, Oct 6, 2017 at 2:55 AM Rodrigo Campos <rodrig...@gmail.com> wrote:
> Ohh, sorry. My bad, just ignore my past email :-)
>
> On Thursday, October 5, 2017, Evan Jones <evan.jo...@bluecore.com> wrote:
>
>> My script *is* always looking up the same domain, and I believe it is
>> cached by dnsmasq. I *think* the limit is the kernel NAT connection
>> tracking, because each DNS query comes from a new ephemeral port, so it
>> ends up using up all NAT mappings on the node running kube-dns. This is why
>> dnsPolicy: Default fixes the problem: It uses the host's DNS configuration
>> which avoids the NAT connection limits.
>>
>> Details including the Python code and configs to reproduce it on a brand
>> new GKE cluster are at the bottom of
>> https://github.com/kubernetes/kubernetes/issues/45976
>>
>> I did a separate test, using a Go DNS query generator, which was able to
>> do 80000 DNS queries per second, so dnsmasq does not appear to be the limit.
>>
>> Thanks!
>>
>> Evan
>>
>>
>> On Thu, Oct 5, 2017 at 5:26 PM, Rodrigo Campos <rodr...@sdfg.com.ar>
>> wrote:
>>
>>> On Thu, Oct 05, 2017 at 04:29:21PM -0400, Evan Jones wrote:
>>> > The sustained 1000 qps comes from an application making that many
>>> outbound
>>> > connections. I agree that the application is very inefficient and
>>> shouldn't
>>> > be doing a DNS lookup for every request it sends, but it's a python
>>> program
>>> > that uses urllib2.urlopen so it creates a new connection each time. I
>>> > suspect this isn't that unusual? This could be a server that hits an
>>> > external service for every user request, for example. Given the
>>> activity on
>>> > the GitHub issues I linked, it appears I'm not the only person to have
>>> run
>>> > into this.
>>>
>>> But is always on different domains? If not, it can probably be cached
>>> (as long
>>> as the TTL allows) by the DNS server and, even if your app makes so many
>>> requests, it should be answered quite fast.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Kubernetes user discussion and Q&A" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/kubernetes-users/7JBq6jhMZHc/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to
>>> kubernetes-users+unsubscr...@googlegroups.com.
>>> To post to this group, send email to kubernetes-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kubernetes-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to kubernetes-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
From c073001cb88a9cf4cdb50b4ce259f54a15ce654a Mon Sep 17 00:00:00 2001
From: Rene Treffer <rene.tref...@soundcloud.com>
Date: Mon, 22 Aug 2016 14:24:47 +0200
Subject: [PATCH] Lower conntrack timeouts for udp & tcp
Those settings can be way lower for datacenter usage, thus freeing
natted ports quicker.
---
net/netfilter/nf_conntrack_proto_tcp.c | 2 +-
net/netfilter/nf_conntrack_proto_udp.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/net/netfilter/nf_conntrack_proto_tcp.c
b/net/netfilter/nf_conntrack_proto_tcp.c
index 70c8381..b258c2c 100644
--- a/net/netfilter/nf_conntrack_proto_tcp.c
+++ b/net/netfilter/nf_conntrack_proto_tcp.c
@@ -75,7 +75,7 @@ static unsigned int tcp_timeouts[TCP_CONNTRACK_TIMEOUT_MAX]
__read_mostly = {
[TCP_CONNTRACK_FIN_WAIT] = 2 MINS,
[TCP_CONNTRACK_CLOSE_WAIT] = 60 SECS,
[TCP_CONNTRACK_LAST_ACK] = 30 SECS,
- [TCP_CONNTRACK_TIME_WAIT] = 2 MINS,
+ [TCP_CONNTRACK_TIME_WAIT] = 10 SECS,
[TCP_CONNTRACK_CLOSE] = 10 SECS,
[TCP_CONNTRACK_SYN_SENT2] = 2 MINS,
/* RFC1122 says the R2 limit should be at least 100 seconds.
diff --git a/net/netfilter/nf_conntrack_proto_udp.c
b/net/netfilter/nf_conntrack_proto_udp.c
index 4fd0405..fbf91a6 100644
--- a/net/netfilter/nf_conntrack_proto_udp.c
+++ b/net/netfilter/nf_conntrack_proto_udp.c
@@ -27,8 +27,8 @@
#include <net/netfilter/ipv6/nf_conntrack_ipv6.h>
static unsigned int udp_timeouts[UDP_CT_MAX] = {
- [UDP_CT_UNREPLIED] = 30*HZ,
- [UDP_CT_REPLIED] = 180*HZ,
+ [UDP_CT_UNREPLIED] = 3*HZ,
+ [UDP_CT_REPLIED] = 15*HZ,
};
static inline struct nf_udp_net *udp_pernet(struct net *net)
--
2.7.4
