Muehlenhoff has uploaded a new change for review.
https://gerrit.wikimedia.org/r/240361
Change subject: Lower the conntrack tracking time for TIME_WAIT connections
......................................................................
Lower the conntrack tracking time for TIME_WAIT connections
TCP connections in TIME_WAIT are maintained for sixty seconds
by the Linux kernel.
Note: There's many misleading Google hits indicating that this
is configurable through the sysctl value net.ipv4.tcp_fin_timeout
but after some digging that turned out to be bogus.
It's a constant defined in the Linux lernel in include/net/tcp.h:
However nf_conntrack tracks these for 120 seconds by default
(configurable through the sysctl value
nf_conntrack_tcp_timeout_time_wait)
Reduce this to 65 (the maximum time used by the kernel plus
a five seconds error margin).
I tried this value manually on the initial job runner which has
ferm enabled (mw1010) which had around 100k tracked connections
of which nearly 58k were in TIME_WAIT.
Reducing nf_conntrack_tcp_timeout_time_wait to 65 shrunk that down
to about 70k.
This patch also removes the custom setting for swift (it was
initially reduced to 3 since the Swift performance docs recommend
net.ipv4.tcp_fin_timeout=3 and it was assumed that this also effects
the TIME_WAIT interval) for consistency.
I've also tested this on ms-[fb]e1002 and it works fine.
Bug: T105307
Change-Id: Ia3fb71591a9482daf022b314eb00d25397f5b492
---
M modules/base/manifests/firewall.pp
M modules/swift/manifests/init.pp
2 files changed, 2 insertions(+), 5 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/61/240361/1
diff --git a/modules/base/manifests/firewall.pp
b/modules/base/manifests/firewall.pp
index bf0ae53..937aa71 100644
--- a/modules/base/manifests/firewall.pp
+++ b/modules/base/manifests/firewall.pp
@@ -19,7 +19,8 @@
# Increase the size of conntrack table size (default is 65536)
sysctl::parameters { 'ferm_conntrack':
values => {
- 'net.netfilter.nf_conntrack_max' => 262144,
+ 'net.netfilter.nf_conntrack_max' => 262144,
+ 'net.netfilter.nf_conntrack_tcp_timeout_time_wait' => 65,
},
}
diff --git a/modules/swift/manifests/init.pp b/modules/swift/manifests/init.pp
index cd49cbb..d34ace2 100644
--- a/modules/swift/manifests/init.pp
+++ b/modules/swift/manifests/init.pp
@@ -30,10 +30,6 @@
'net.ipv4.tcp_max_orphans' => 262144,
'net.ipv4.tcp_synack_retries' => 2,
'net.ipv4.tcp_syn_retries' => 2,
-
- # even with NOTRACK enabled, conntrack will still keep track
- # connections in TIME_WAIT, thus lower the respective timeout
- 'net.netfilter.nf_conntrack_tcp_timeout_time_wait' => 3,
},
}
--
To view, visit https://gerrit.wikimedia.org/r/240361
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia3fb71591a9482daf022b314eb00d25397f5b492
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Muehlenhoff <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
