Ori.livneh has uploaded a new change for review.
https://gerrit.wikimedia.org/r/92271
Change subject: [WIP] Add Graphite module & role
......................................................................
[WIP] Add Graphite module & role
Work-in-progress.
What is included:
-----------------
- Carbon / Whisper configs.
Still to be done:
-----------------
- Graphite webapp & its dependencies:
- memcached
- apache + mod_wsgi or nginx + gunicorn
- aggregation-rules.conf
- monitoring
- decomissioning
Things that could be controversial:
-----------------------------------
- The Upstart configs are my own invention. Graphite ships with a crummy init.d
script that only works for basic setups. Most people roll their own, but I
thought I could do a better job, esp. since I still had my EventLogging work
to refer to.
- The one-size-fits-all retention schema. I settled on it after concluding that
the uniformity and simplicity provided by having one retention policy was a
better & safer bet than trying to improve resource utilization by auguring
the future.
- File layout. Graphite keeps almost everything in /opt/graphite by default,
but it is possible to configure it to use FHS-compliant locations. I decided
to stick with /opt/graphite because it is routinely taken for granted by
authors of blog posts, mailing list posts, etc., and even the Graphite docs
themselves, and I found that having to constantly map /opt/graphite paths to
their FHS equivalents was mentally taxing.
Change-Id: If563923aa55aef77111637aacbb6a93950d9066e
---
A manifests/role/graphite.pp
M manifests/site.pp
A modules/graphite/files/carbon-upstart/cache.conf
A modules/graphite/files/carbon-upstart/init.conf
A modules/graphite/files/carbon-upstart/relay.conf
A modules/graphite/files/graphite.limits.conf
A modules/graphite/lib/puppet/parser/functions/configparser_format.rb
A modules/graphite/manifests/init.pp
8 files changed, 239 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/71/92271/1
diff --git a/manifests/role/graphite.pp b/manifests/role/graphite.pp
new file mode 100644
index 0000000..7383a31
--- /dev/null
+++ b/manifests/role/graphite.pp
@@ -0,0 +1,72 @@
+class role::graphite {
+ class { 'graphite':
+ retention => {
+ # Retain data at a one-minute resolution for one year and at a
+ # ten-minute resolution for ten years. It's clear & easy to
remember.
+ # Avoid making this more complicated that it needs to be.
+ 'default' => {
+ pattern => '.*',
+ retentions => '1m:1y,10m:10y',
+ },
+ },
+
+ # All metric data goes through a single carbon-relay instance, which
+ # forwards each data point to one of four carbon-cache instances,
using a
+ # consistent hash ring to distribute the load.
+ #
+ # Why is this necessary? Because carbon-cache is CPU-bound, and the
Python
+ # GIL prevents it from utilizing multiple processor cores efficiently.
+ #
+ # cf. "Single node, multiple carbon-caches"
+ # <http://bitprophet.org/blog/2013/03/07/graphite/>
+ #
+ # If we need to scale up, the next step is multi-node.
+ # <http://tinyurl.com/graphite-cluster-setup>.
+ carbon => {
+ 'cache' => {
+ log_updates => false, # Don't log Whisper
updates.
+ user => undef, # Don't suid;
Upstart will do it for us.
+ line_receiver_interface => '127.0.0.1', # Only the relay
binds to 0.0.0.0.
+ pickle_receiver_interface => '127.0.0.1',
+
+ },
+
+ ## Carbon caches ##
+
+ 'cache:a' => {
+ line_receiver_port => 2103,
+ pickle_receiver_port => 2104,
+ cache_query_port => 7102,
+ },
+ 'cache:b' => {
+ line_receiver_port => 2203,
+ pickle_receiver_port => 2204,
+ cache_query_port => 7202,
+ },
+ 'cache:c' => {
+ line_receiver_port => 2303,
+ pickle_receiver_port => 2304,
+ cache_query_port => 7302,
+ },
+ 'cache:d' => {
+ line_receiver_port => 2403,
+ pickle_receiver_port => 2404,
+ cache_query_port => 7402,
+ },
+
+ ## Carbon relay ##
+
+ 'relay' => {
+ line_receiver_interface => '0.0.0.0',
+ pickle_receiver_interface => '0.0.0.0',
+ relay_method => 'consistent-hashing',
+ destinations => [
+ '127.0.0.1:2104:a',
+ '127.0.0.1:2204:b',
+ '127.0.0.1:2304:c',
+ '127.0.0.1:2404:d',
+ ],
+ },
+ },
+ }
+}
diff --git a/manifests/site.pp b/manifests/site.pp
index adae38d..78c4989 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2836,7 +2836,7 @@
# RT #5871
node 'tungsten.eqiad.wmnet' {
# services
- include standard, role::statsd
+ include standard, role::statsd, role::graphite
# access
include groups::wikidev, accounts::olivneh
diff --git a/modules/graphite/files/carbon-upstart/cache.conf
b/modules/graphite/files/carbon-upstart/cache.conf
new file mode 100644
index 0000000..67665c3
--- /dev/null
+++ b/modules/graphite/files/carbon-upstart/cache.conf
@@ -0,0 +1,19 @@
+# carbon/cache
+#
+# carbon-cache.py listens for metrics on a network link, caches them in RAM,
+# and flushes to disk periodically. Part of the Graphite suite.
+#
+description "Graphite metrics listener"
+
+instance $NAME
+
+stop on runlevel [!2345] or carbon.stop
+
+setuid graphite
+setgid graphite
+
+# "--debug" makes carbon-* stay in the foreground and log to stdout, which
+# allows Upstart to take care of daemonizing and logging.
+exec /opt/graphite/bin/carbon-cache.py --instance "$NAME" --debug start
+
+respawn
diff --git a/modules/graphite/files/carbon-upstart/init.conf
b/modules/graphite/files/carbon-upstart/init.conf
new file mode 100644
index 0000000..adcf3ea
--- /dev/null
+++ b/modules/graphite/files/carbon-upstart/init.conf
@@ -0,0 +1,19 @@
+# Upstart job configuration for Carbon, a part of Graphite
+# This file is managed by Puppet
+description "Graphite metrics storage"
+
+start on runlevel [2345] or carbon.start
+stop on runlevel [!2345] or carbon.stop
+
+task
+
+script
+ # Start an instance for each '[cache:<instance name>]' match.
+ instances=$(grep -Po '(?<=^\[cache:).+(?=\]$)'
/opt/graphite/conf/carbon.conf)
+ set -- $instances
+ for name in "$@"; do
+ start carbon/cache NAME="$name" || status carbon/cache NAME="$name"
+ done
+end script
+
+# vim: set ft=upstart:
diff --git a/modules/graphite/files/carbon-upstart/relay.conf
b/modules/graphite/files/carbon-upstart/relay.conf
new file mode 100644
index 0000000..5db1393
--- /dev/null
+++ b/modules/graphite/files/carbon-upstart/relay.conf
@@ -0,0 +1,24 @@
+# graphite/carbon-relay
+#
+# carbon-relay.py listens for metrics and shards or replicates across multiple
+# carbon-cache.py instances. Part of the Graphite suite.
+#
+description "Graphite metrics listener"
+
+# Run after carbon/init has spun up all carbon/cache instances.
+start on stopped carbon/init
+stop on runlevel [!2345] or carbon.stop
+
+setuid graphite
+setgid graphite
+
+pre-start script
+ # Exit normally if no relay daemon is configured for this node.
+ grep -Fqx '[relay]' /opt/graphite/conf/carbon.conf || { stop; exit 0; }
+end script
+
+# "--debug" makes carbon-* stay in the foreground and log to stdout, which
+# allows Upstart to take care of daemonizing and logging.
+exec /opt/graphite/bin/carbon-relay.py --debug start
+
+respawn
diff --git a/modules/graphite/files/graphite.limits.conf
b/modules/graphite/files/graphite.limits.conf
new file mode 100644
index 0000000..381f94a
--- /dev/null
+++ b/modules/graphite/files/graphite.limits.conf
@@ -0,0 +1,6 @@
+# Increase file descriptor limits for Graphite
+# See <http://graphite.readthedocs.org/en/latest/carbon-daemons.html>
+# This file is managed by Puppet
+
+graphite soft nofile 65536
+graphite hard nofile 65536
diff --git
a/modules/graphite/lib/puppet/parser/functions/configparser_format.rb
b/modules/graphite/lib/puppet/parser/functions/configparser_format.rb
new file mode 100644
index 0000000..5fa5a20
--- /dev/null
+++ b/modules/graphite/lib/puppet/parser/functions/configparser_format.rb
@@ -0,0 +1,28 @@
+# == Function: configparser_format
+#
+# Serialize a hash to Python ConfigParser format.
+# See <http://docs.python.org/2/library/configparser.html>
+#
+def configparser_format(config)
+ # Serialize a hash to Python ConfigParser format
+ config.sort.map { |section,items|
+ ["[#{section}]"].concat items.sort.map { |k,v|
+ v = case v
+ when Array then v.join(',')
+ when true, false then v.to_s.capitalize
+ when nil, :undef, :undefined then ''
+ else v
+ end
+ "#{k} = #{v}"
+ }.push []
+ }.join("\n")
+end
+
+module Puppet::Parser::Functions
+ newfunction(:configparser_format, :type => :rvalue) do |args|
+ if args.empty? or not args.first.is_a? Hash
+ fail 'configparser_format() requires a hash argument'
+ end
+ configparser_format(args.first)
+ end
+end
diff --git a/modules/graphite/manifests/init.pp
b/modules/graphite/manifests/init.pp
new file mode 100644
index 0000000..568d52d
--- /dev/null
+++ b/modules/graphite/manifests/init.pp
@@ -0,0 +1,70 @@
+# == Class: graphite
+#
+# Graphite is a monitoring tool that stores numeric time-series data and
+# renders graphs of this data on demand. It consists of three software
+# components:
+#
+# - Carbon, a daemon that listens for time-series data
+# - Whisper, a database library for storing time-series data
+# - Graphite webapp, a webapp which renders graphs on demand
+#
+class graphite(
+ $retention,
+ $carbon,
+) {
+ package { 'python-carbon': }
+ package { 'python-graphite-web': }
+ package { 'python-whisper': }
+
+ file { '/etc/security/limits.d/graphite.conf':
+ source => 'puppet:///modules/graphite/graphite.limits.conf',
+ }
+
+ group { 'graphite':
+ ensure => present,
+ }
+
+ user { 'graphite':
+ ensure => present,
+ gid => 'graphite',
+ shell => '/bin/false',
+ home => '/nonexistent',
+ system => true,
+ }
+
+ file { '/opt/graphite':
+ ensure => directory,
+ require => Package['python-carbon'],
+ }
+
+ file { '/opt/graphite/storage':
+ ensure => directory,
+ owner => 'graphite',
+ group => 'graphite',
+ mode => '0644',
+ require => Package['python-carbon'],
+ }
+
+ file { '/opt/graphite/conf/storage-schemas.conf':
+ content => configparser_format($retention),
+ before => File['/etc/init/carbon'],
+ }
+
+ file { '/opt/graphite/conf/carbon.conf':
+ content => configparser_format($carbon),
+ before => File['/etc/init/carbon'],
+ }
+
+ file { '/etc/init/carbon':
+ source => 'puppet:///modules/graphite/carbon-upstart',
+ recurse => true,
+ }
+
+ service { 'carbon/init':
+ provider => 'upstart',
+ require => [
+ File['/etc/init/carbon'],
+ User['graphite']
+ ],
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/92271
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If563923aa55aef77111637aacbb6a93950d9066e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ori.livneh <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
