Ema has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/401526 )
Change subject: varnish: add varnishmtail instance for varnish backends
......................................................................
varnish: add varnishmtail instance for varnish backends
We have introduced a service called varnishmtail starting with
4159504325de721e7236c16e8f40073d5a113a2b with the goal of gathering and
exposing metrics related to varnish frontend instances.
Add a new service called varnishmtail-backend, responsible for doing
similar work with varnish backends. We want to keep the two systems
separate for various reasons, including the fact that the work done by
varnish on the backend layer could in the future be done by an entirely
different software. Also, logically speaking serving client requests and
routing HTTP requests through our CDN are different tasks, which should
be cleanly separated at the monitoring level too.
Bug: T177199
Change-Id: Ia4b55413ae2fa2fd0a88bd937366a59474d03871
---
A modules/varnish/files/varnishmtail-backend
M modules/varnish/manifests/logging.pp
A modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
3 files changed, 58 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/26/401526/1
diff --git a/modules/varnish/files/varnishmtail-backend
b/modules/varnish/files/varnishmtail-backend
new file mode 100644
index 0000000..0d9f383
--- /dev/null
+++ b/modules/varnish/files/varnishmtail-backend
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# varnishmtail-backend - pipe varnishncsa output to mtail
+
+PROGS="${1:-/etc/mtail}"
+PORT="${2:-3903}"
+
+fmt_http_status='http_status %s'
+fmt_http_method='http_method %m'
+fmt_backend='backend %{VSL:BackendOpen[2]}x'
+fmt_ttfb='ttfb %{Varnish:time_firstbyte}x'
+
+FMT="${fmt_http_status}\t${fmt_http_method}\t${fmt_backend}\t${fmt_ttfb}\t"
+
+# Exclude client requests resulting in a pipe as they do not generate a backend
+# request. Varnish blindly passes on bytes in both directions in that case, so
+# there is no status and no ttfb.
+VSL_QUERY='BereqMethod ne "PURGE" and VCL_call ne "PIPE"'
+
+/usr/bin/varnishncsa -b -q "${VSL_QUERY}" -F "${FMT}" | mtail -progs
"${PROGS}" -logfds 0 -port "${PORT}"
diff --git a/modules/varnish/manifests/logging.pp
b/modules/varnish/manifests/logging.pp
index a236470..95491fd 100644
--- a/modules/varnish/manifests/logging.pp
+++ b/modules/varnish/manifests/logging.pp
@@ -17,11 +17,21 @@
# [*mtail_progs*]
# Directory with mtail programs. Defaults to /etc/mtail.
#
+# [*varnishmtail_backend_progs*]
+# Directory with varnish backend mtail programs.
+# Defaults to /etc/varnishmtail-backend/.
+#
+# [*varnishmtail_backend_port*]
+# Port on which to bind the varnish backend mtail instance.
+# Defaults to 3904.
+#
class varnish::logging(
$cache_cluster,
$statsd_host,
$forward_syslog='',
$mtail_progs='/etc/mtail',
+ $varnishmtail_backend_progs='/etc/varnishmtail-backend/',
+ $varnishmtail_backend_port=3904,
){
rsyslog::conf { 'varnish':
content => template('varnish/rsyslog.conf.erb'),
@@ -44,6 +54,22 @@
require => File['/usr/local/bin/varnishmtail'],
}
+ file { '/usr/local/bin/varnishmtail-backend':
+ ensure => present,
+ owner => 'root',
+ group => 'root',
+ mode => '0555',
+ source => 'puppet:///modules/varnish/varnishmtail-backend',
+ notify => Systemd::Service['varnishmtail-backend'],
+ }
+
+ systemd::service { 'varnishmtail-backend':
+ ensure => present,
+ content => systemd_template('varnishmtail-backend'),
+ restart => true,
+ require => File['/usr/local/bin/varnishmtail-backend'],
+ }
+
# Client connection stats from the 'X-Connection-Properties'
# header set by the SSL terminators.
::varnish::logging::xcps { 'xcps':
diff --git
a/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
new file mode 100644
index 0000000..632846a
--- /dev/null
+++ b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
@@ -0,0 +1,12 @@
+[Unit]
+Description=Varnish Backend mtail
+After=varnish.service
+Requires=varnish.service
+
+[Service]
+SyslogIdentifier=varnishmtail-backend
+Restart=always
+ExecStart=/usr/local/bin/varnishmtail-backend <%= @varnishmtail_backend_progs
%> <%= @varnishmtail_backend_port %>
+
+[Install]
+WantedBy=multi-user.target
--
To view, visit https://gerrit.wikimedia.org/r/401526
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia4b55413ae2fa2fd0a88bd937366a59474d03871
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
