[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: vhtcpd host regex filter
BBlack has submitted this change and it was merged.
Change subject: cache_upload: vhtcpd host regex filter
..
cache_upload: vhtcpd host regex filter
Only about 0.1% (or less) of the total multicast purge flow
actually applies to the upload caches. The rest is the crazy mess
of redundant/template/backref-driven text purges. Reducing the
rate of actual PURGE into varnishd may help a lot with various
problems in cache_upload for now, and vhtcpd is capable of
regex-filtering on the purged URL.
The full flow can still be seen in vhtcpd's inpkts_received and
inpkts_sane, but the reduce number from this fitering will be seen
in inpkts_enqueued and other stats further into the pipeline.
We never got around to this optimization before because the
perfect was the enemy of the good and it was hard to think through
and test the quoting for a proper, perfectly-matching regex with
several layers of interpolation and quoting between puppet and the
actual vhtcpd commandline. However, using the literal simple
string 'upload' as a regex filter does the job fine in this case,
and is all we really need.
Change-Id: I605638d8e60c9c52ecc8368afc4a1705112f57d1
---
M modules/role/manifests/cache/upload.pp
M modules/varnish/manifests/htcppurger.pp
M modules/varnish/templates/vhtcpd-default.erb
3 files changed, 11 insertions(+), 2 deletions(-)
Approvals:
BBlack: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/role/manifests/cache/upload.pp
b/modules/role/manifests/cache/upload.pp
index ee8707a..826a269 100644
--- a/modules/role/manifests/cache/upload.pp
+++ b/modules/role/manifests/cache/upload.pp
@@ -9,7 +9,8 @@
}
class { 'varnish::htcppurger':
-mc_addrs => [ '239.128.0.112', '239.128.0.113' ],
+host_regex => 'upload',
+mc_addrs => [ '239.128.0.112', '239.128.0.113' ],
}
class { 'lvs::realserver':
diff --git a/modules/varnish/manifests/htcppurger.pp
b/modules/varnish/manifests/htcppurger.pp
index 3a5dc0e..a92fa51 100644
--- a/modules/varnish/manifests/htcppurger.pp
+++ b/modules/varnish/manifests/htcppurger.pp
@@ -31,6 +31,7 @@
class varnish::htcppurger(
$mc_addrs,
+$host_regex = '',
$varnishes = [ '127.0.0.1:3127', '127.0.0.1:3128' ],
) {
Class[varnish::packages] -> Class[varnish::htcppurger]
@@ -39,6 +40,13 @@
ensure => latest,
}
+if $host_regex != '' {
+$regex_arg = " -r '${host_regex}'"
+}
+else {
+$regex_arg = ''
+}
+
file { '/etc/default/vhtcpd':
owner => 'root',
group => 'root',
diff --git a/modules/varnish/templates/vhtcpd-default.erb
b/modules/varnish/templates/vhtcpd-default.erb
index 1ada858..6b16ae7 100644
--- a/modules/varnish/templates/vhtcpd-default.erb
+++ b/modules/varnish/templates/vhtcpd-default.erb
@@ -1 +1 @@
-DAEMON_OPTS="-m <%= @mc_addrs.join(' -m ') -%> -c <%= @varnishes.join(' -c ')
-%> -l 1024"
+DAEMON_OPTS="-m <%= @mc_addrs.join(' -m ') -%> -c <%= @varnishes.join(' -c ')
-%><%= @regex_arg -%> -l 1024"
--
To view, visit https://gerrit.wikimedia.org/r/311332
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I605638d8e60c9c52ecc8368afc4a1705112f57d1
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: BBlack
Gerrit-Reviewer: BBlack
Gerrit-Reviewer: Ema
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: vhtcpd host regex filter
BBlack has uploaded a new change for review.
https://gerrit.wikimedia.org/r/311332
Change subject: cache_upload: vhtcpd host regex filter
..
cache_upload: vhtcpd host regex filter
Only about 0.1% (or less) of the total multicast purge flow
actually applies to the upload caches. The rest is the crazy mess
of redundant/template/backref-driven text purges. Reducing the
rate of actual PURGE into varnishd may help a lot with various
problems in cache_upload for now, and vhtcpd is capable of
regex-filtering on the purged URL.
The full flow can still be seen in vhtcpd's inpkts_received and
inpkts_sane, but the reduce number from this fitering will be seen
in inpkts_enqueued and other stats further into the pipeline.
We never got around to this optimization before because the
perfect was the enemy of the good and it was hard to think through
and test the quoting for a proper, perfectly-matching regex with
several layers of interpolation and quoting between puppet and the
actual vhtcpd commandline. However, using the literal simple
string 'upload' as a regex filter does the job fine in this case,
and is all we really need.
Change-Id: I605638d8e60c9c52ecc8368afc4a1705112f57d1
---
M modules/role/manifests/cache/upload.pp
M modules/varnish/manifests/htcppurger.pp
M modules/varnish/templates/vhtcpd-default.erb
3 files changed, 11 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/32/311332/1
diff --git a/modules/role/manifests/cache/upload.pp
b/modules/role/manifests/cache/upload.pp
index ee8707a..826a269 100644
--- a/modules/role/manifests/cache/upload.pp
+++ b/modules/role/manifests/cache/upload.pp
@@ -9,7 +9,8 @@
}
class { 'varnish::htcppurger':
-mc_addrs => [ '239.128.0.112', '239.128.0.113' ],
+host_regex => 'upload',
+mc_addrs => [ '239.128.0.112', '239.128.0.113' ],
}
class { 'lvs::realserver':
diff --git a/modules/varnish/manifests/htcppurger.pp
b/modules/varnish/manifests/htcppurger.pp
index 3a5dc0e..3873fe5 100644
--- a/modules/varnish/manifests/htcppurger.pp
+++ b/modules/varnish/manifests/htcppurger.pp
@@ -31,6 +31,7 @@
class varnish::htcppurger(
$mc_addrs,
+$host_regex = '',
$varnishes = [ '127.0.0.1:3127', '127.0.0.1:3128' ],
) {
Class[varnish::packages] -> Class[varnish::htcppurger]
@@ -39,6 +40,13 @@
ensure => latest,
}
+if $host_regex != '' {
+$regex_arg = " -r '${host_regex}'"
+}
+else {
+$regex_arg = '';
+}
+
file { '/etc/default/vhtcpd':
owner => 'root',
group => 'root',
diff --git a/modules/varnish/templates/vhtcpd-default.erb
b/modules/varnish/templates/vhtcpd-default.erb
index 1ada858..6b16ae7 100644
--- a/modules/varnish/templates/vhtcpd-default.erb
+++ b/modules/varnish/templates/vhtcpd-default.erb
@@ -1 +1 @@
-DAEMON_OPTS="-m <%= @mc_addrs.join(' -m ') -%> -c <%= @varnishes.join(' -c ')
-%> -l 1024"
+DAEMON_OPTS="-m <%= @mc_addrs.join(' -m ') -%> -c <%= @varnishes.join(' -c ')
-%><%= @regex_arg -%> -l 1024"
--
To view, visit https://gerrit.wikimedia.org/r/311332
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I605638d8e60c9c52ecc8368afc4a1705112f57d1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: BBlack
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
