Ottomata has uploaded a new change for review.
https://gerrit.wikimedia.org/r/86894
Change subject: Adding kafka::udp2log::relay define to consume from Kafka and
send to udp2log.
......................................................................
Adding kafka::udp2log::relay define to consume from Kafka and send to udp2log.
When we finally switch on varnishkafka on mobile varnish frontends, we will
want to disable
the udp2log varnishncsa instances there. This can be used to ensure that
existing consumers of
udp2log mobile data can still do their thing. I intend to run this somewhere
and pipe mobile
log data back into the socat relay running on gadolinium. This should keep the
udp2log
firehose stream looking the same as it did before as we migrate more things to
Kafka.
Change-Id: Ib17ee1cace34a8c899d82d3194dfdd8883ce6a6c
---
M manifests/role/analytics/kafka.pp
A templates/misc/kafka-udp2log-relay.upstart.conf.erb
2 files changed, 71 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/94/86894/1
diff --git a/manifests/role/analytics/kafka.pp
b/manifests/role/analytics/kafka.pp
index 8385ade..23070d4 100644
--- a/manifests/role/analytics/kafka.pp
+++ b/manifests/role/analytics/kafka.pp
@@ -105,3 +105,51 @@
metrics_properties => $metrics_properties,
}
}
+
+
+# == Define role::analytics::kafka::udp2log::relay
+# Consumes varnishkafka JSON data from Kafka, transforms it
+# into the original tabular udp2log format, and sends it to
+# a UDP address via netcat.
+#
+# == Parameters:
+# $destination_ip
+# $destination_port
+# $kafka_topic
+# $kafka_group
+#
+define role::analytics::kafka::udp2log::relay(
+ $destination_ip,
+ $destination_port,
+ $kafka_topic,
+ $kafka_group = 'udp2log',
+)
+{
+ require role::analytics::kafka::client
+ require role::analytics::zookeeper::client
+
+ # Need jq for parsing varnishkafka json into tabular udp2log log format
+ package { 'jq': ensure => 'installed' }
+
+ $daemon_name = "kafka-udp2log-${title}-relay"
+
+ $zookeeper_hosts = $role::analytics::zookeeper::config::hosts_array
+ $zookeeper_chroot = $role::analytics::kafka::config::zookeeper_chroot
+
+ # Create symlink
+ file { "/etc/init.d/${daemon_name}":
+ ensure => 'link',
+ target => '/lib/init/upstart-job';
+ }
+
+ file { "/etc/init/${daemon_name}.conf":
+ content => template('misc/kafka-udp2log-relay.upstart.conf.erb'),
+ }
+
+ service { $daemon_name:
+ ensure => running,
+ require => Package['socat'],
+ subscribe => File["/etc/init/${daemon_name}.conf"],
+ provider => 'upstart',
+ }
+}
\ No newline at end of file
diff --git a/templates/misc/kafka-udp2log-relay.upstart.conf.erb
b/templates/misc/kafka-udp2log-relay.upstart.conf.erb
new file mode 100644
index 0000000..3e28632
--- /dev/null
+++ b/templates/misc/kafka-udp2log-relay.upstart.conf.erb
@@ -0,0 +1,23 @@
+# This file is managed by Puppet!
+#
+# <%= @daemon_name %>
+
+description "Kafka -> udp2log <%= title %> relay"
+
+start on filesystem
+stop on runlevel S
+
+respawn
+respawn limit 10 5
+umask 022
+
+# consume from Kafka, pipe into jq for JSON -> TSV tranform, and then
+# pipe into netcat which sends to a udp2log
+# instance at <%= @destination_ip %>:<%= @destination_port %>
+
+exec /usr/sbin/kafka console-consumer \
+ --topic <%= @kafka_topic %> \
+ --group <%= @kafka_group %> \
+ --zookeeper <%= @zookeeper_hosts.sort.join(',') %><%= @zookeeper_chroot if
@zookeeper_chroot %> \
+ | /root/jq -r '.hostname + "\t" + (.sequence|tostring) + "\t" + .dt + "\t"
+ (.time_firstbyte|tostring) + "\t" + .ip + "\t" +
.cache_status+"/"+.http_status + "\t" + (.response_size|tostring) + "\t" +
.http_method + "\t" + "http://"+.uri_host+.uri_path+.uri_query + "\t" + "-" +
"\t" + .content_type + "\t" + .referer + "\t" + .x_forwarded_for + "\t" +
.user_agent + "\t" + .accept_language + "\t" + .x_analytics' \
+ | /bin/netcat -nu <%= @destination_ip %> <%= @destination_port %>
--
To view, visit https://gerrit.wikimedia.org/r/86894
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib17ee1cace34a8c899d82d3194dfdd8883ce6a6c
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
