jenkins-bot has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/342772 )
Change subject: Update logstash to 5.2.2, kibana to 5.1.2
......................................................................
Update logstash to 5.2.2, kibana to 5.1.2
In preparation for upgrading production to a modern version of the ELK
stack, upgrade mwv.
Kibana needs to be upgraded in lock-step with elasticsearch, so 5.1.2
was chosen instead of 5.2.2 which is the latest release.
Changes:
* This new version of logstash gets stuck in a boot loop OOM'ing with
the previously set 64m ram. Even at 128m ram was able to trigger an
OOM. Increase to 192m. Additionally moved setting from defaults file
to jvm.options, which is the new standard for elastic.co packages.
* update jdk from 7 to 8 to match logstash new requirements
* upstream packaging no longer includes both sets of startup scripts, so
removes the kludge that previously deleted the upstart scripts.
* Adds new logstash::plugin define, as plugins are now handled with a
script rather than all bundled together into a single contrib package.
* Remove /etc/defaults/logstash, this is no longer used in 5.x.
* elasticsearch_http output no longer exists, instead it is
elasticsearchi now which only speaks http. This also no longer has
a replication setting, Elastic consider async writes to be a
Bad Idea™ which can lead to cluster overload.
* Add the haveged package to provide entropy in a vm so that JRuby
doesn't spend minutes booting up waiting on entropy from /dev/random
Bug: T154473
Change-Id: I4f23fd402d13ad33c8b6a72fc96b3b39be6ba364
---
M puppet/hieradata/common.yaml
D puppet/modules/elasticsearch/files/elasticsearch.yml
M puppet/modules/elasticsearch/manifests/init.pp
A puppet/modules/kibana/files/kibana-dump.json
A puppet/modules/kibana/files/kibana-mapping.json
D puppet/modules/kibana/manifests/dashboard.pp
M puppet/modules/kibana/manifests/init.pp
D puppet/modules/kibana/templates/check-dashboard.erb
D puppet/modules/kibana/templates/config.js
D puppet/modules/kibana/templates/save-dashboard.erb
M puppet/modules/logstash/files/elasticsearch-template.json
M puppet/modules/logstash/manifests/init.pp
M puppet/modules/logstash/manifests/output/elasticsearch.pp
A puppet/modules/logstash/manifests/plugin.pp
D puppet/modules/logstash/templates/default.erb
A puppet/modules/logstash/templates/jvm.options.erb
M puppet/modules/logstash/templates/output/elasticsearch.erb
M puppet/modules/role/manifests/elk.pp
M puppet/modules/role/settings/elk.yaml
19 files changed, 541 insertions(+), 256 deletions(-)
Approvals:
BryanDavis: Looks good to me, approved
jenkins-bot: Verified
diff --git a/puppet/hieradata/common.yaml b/puppet/hieradata/common.yaml
index eacaaba..4adc168 100644
--- a/puppet/hieradata/common.yaml
+++ b/puppet/hieradata/common.yaml
@@ -164,10 +164,9 @@
kibana::deploy_dir: "%{hiera('mwv::vendor_dir')}/kibana"
kibana::default_route: /dashboard/elasticsearch/default
-logstash::heap_memory_mb: 64
-logstash::filter_workers: 1
+logstash::heap_memory_mb: 192
+logstash::pipeline_workers: 1
logstash::output::elasticsearch::host: 127.0.0.1
-logstash::output::elasticsearch::replication: async
logstash::output::elasticsearch::manage_indices: true
logstash::output::elasticsearch::priority: 90
diff --git a/puppet/modules/elasticsearch/files/elasticsearch.yml
b/puppet/modules/elasticsearch/files/elasticsearch.yml
deleted file mode 100644
index e69de29..0000000
--- a/puppet/modules/elasticsearch/files/elasticsearch.yml
+++ /dev/null
diff --git a/puppet/modules/elasticsearch/manifests/init.pp
b/puppet/modules/elasticsearch/manifests/init.pp
index 5988ce2..27e0236 100644
--- a/puppet/modules/elasticsearch/manifests/init.pp
+++ b/puppet/modules/elasticsearch/manifests/init.pp
@@ -39,19 +39,15 @@
]
}
+ exec { 'wait-for-elasticsearch':
+ require => Service['elasticsearch'],
+ command => '/usr/bin/wget --tries 20 --retry-connrefused
http://localhost:9200/',
+ }
+
file { '/etc/default/elasticsearch':
source => 'puppet:///modules/elasticsearch/defaults',
require => Package['elasticsearch'],
notify => Service['elasticsearch'],
- }
-
- file { '/etc/elasticsearch/elasticsearch.yml':
- source => 'puppet:///modules/elasticsearch/elasticsearch.yml',
- require => Package['elasticsearch'],
- notify => Service['elasticsearch'],
- owner => 'root',
- group => 'root',
- mode => '0444',
}
file { '/etc/elasticsearch/jvm.options':
diff --git a/puppet/modules/kibana/files/kibana-dump.json
b/puppet/modules/kibana/files/kibana-dump.json
new file mode 100644
index 0000000..ce843be
--- /dev/null
+++ b/puppet/modules/kibana/files/kibana-dump.json
@@ -0,0 +1,14 @@
+{"index":{"_index":".kibana","_type":"config","_id":"5.1.2"}}
+{"buildNum":14588,"defaultIndex":"logstash-*"}
+{"index":{"_index":".kibana","_type":"visualization","_id":"Events-Over-Time"}}
+{"title":"Events Over Time","visState":"{\"title\":\"Events Over
Time\",\"type\":\"line\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"showCircles\":true,\"smoothLines\":false,\"interpolate\":\"linear\",\"scale\":\"linear\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}}],\"listeners\":{}}","uiStateJSON":"{\"vis\":{\"legendOpen\":false}}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"}}
+{"index":{"_index":".kibana","_type":"visualization","_id":"Event-Types"}}
+{"title":"Event Types","visState":"{\"title\":\"Event
Types\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"type\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}","uiStateJSON":"{\"vis\":{\"legendOpen\":false}}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"}}
+{"index":{"_index":".kibana","_type":"visualization","_id":"Event-Level"}}
+{"title":"Event Level","visState":"{\"title\":\"Event
Level\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"level\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}","uiStateJSON":"{}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"}}
+{"index":{"_index":".kibana","_type":"index-pattern","_id":"logstash-*"}}
+{"title":"logstash-*","timeFieldName":"@timestamp","fields":"[{\"name\":\"timeSpentBackend.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"wiki\",\"type\":\"string\",\"count\":1,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"params.format\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"channel\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"reqId.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"params.action.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"ip.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"params.formatversion.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"metaDirty\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"metaDirty.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"http_method\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"host\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"dataDirty\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"params.limit\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"method\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"level\",\"type\":\"string\",\"count\":1,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"params.format.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"ip\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"count\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"runtime\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"mwversion\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"normalized_message\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"params.suggest.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"reqId\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"tags\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"userAgent.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"dataDirty.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"params.action\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":false,\"aggregatable\":false},{\"name\":\"process_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"server\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"private\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"referrer.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"normalized_message.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"params.suggest\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"session\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"session.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"count.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"process_id.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"key.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"method.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"udp_sender\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":false,\"aggregatable\":false},{\"name\":\"private.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"normalized_message.normalized_message\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"params.search.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"uid\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"forcePersist\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"geoip.location\",\"type\":\"geo_point\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"@version\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"sequence_id\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":false,\"aggregatable\":false},{\"name\":\"params.limit.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"http_method.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"forcePersist.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"key\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"hadError\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"params.namespace\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"params.search\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"params.formatversion\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"url.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"wiki.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"server.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"ts.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"callers.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"userAgent\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"message\",\"type\":\"string\",\"count\":1,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"params.namespace.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"url\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"master\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"referrer\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"@timestamp\",\"type\":\"date\",\"count\":1,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"uid.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"mwversion.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"channel.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"callers\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"timeSpentBackend\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"runtime.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"master.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"hadError.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"ts\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"severity_label.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"timestamp.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"pid\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"program\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"facility_label.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"program.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"logsource.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"facility.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"priority.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"timestamp\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"severity\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"logsource\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"priority\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"severity.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"pid.raw\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"facility\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":false,\"doc_values\":true,\"searchable\":true,\"aggregatable\":true},{\"name\":\"severity_label\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"facility_label\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"searchable\":true,\"aggregatable\":false},{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":false,\"aggregatable\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":true,\"aggregatable\":true},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":false,\"aggregatable\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":false,\"aggregatable\":false}]"}
+{"index":{"_index":".kibana","_type":"search","_id":"Default-Event-List"}}
+{"title":"Default Event
List","description":"","hits":0,"columns":["level","wiki","message"],"sort":["@timestamp","desc"],"version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"logstash-*\",\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647},\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}}}"}}
+{"index":{"_index":".kibana","_type":"dashboard","_id":"default"}}
+{"title":"default","hits":0,"description":"","panelsJSON":"[{\"id\":\"Events-Over-Time\",\"type\":\"visualization\",\"panelIndex\":1,\"size_x\":12,\"size_y\":2,\"col\":1,\"row\":1},{\"id\":\"Event-Types\",\"type\":\"visualization\",\"panelIndex\":2,\"size_x\":6,\"size_y\":2,\"col\":1,\"row\":3},{\"id\":\"Event-Level\",\"type\":\"visualization\",\"panelIndex\":3,\"size_x\":6,\"size_y\":2,\"col\":7,\"row\":3},{\"id\":\"Default-Event-List\",\"type\":\"search\",\"panelIndex\":4,\"size_x\":12,\"size_y\":11,\"col\":1,\"row\":5,\"columns\":[\"level\",\"wiki\",\"message\"],\"sort\":[\"@timestamp\",\"desc\"]}]","optionsJSON":"{\"darkTheme\":false}","uiStateJSON":"{\"P-3\":{\"vis\":{\"legendOpen\":false}}}","version":1,"timeRestore":false,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"filter\":[{\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}}}]}"}}
diff --git a/puppet/modules/kibana/files/kibana-mapping.json
b/puppet/modules/kibana/files/kibana-mapping.json
new file mode 100644
index 0000000..c4c6a7f
--- /dev/null
+++ b/puppet/modules/kibana/files/kibana-mapping.json
@@ -0,0 +1,180 @@
+{
+ "aliases": {},
+ "mappings": {
+ "server": {
+ "properties": {
+ "uuid": {
+ "type": "keyword"
+ }
+ }
+ },
+ "config": {
+ "properties": {
+ "buildNum": {
+ "type": "keyword"
+ },
+ "defaultIndex": {
+ "type": "text",
+ "fields": {
+ "keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+ }
+ }
+ }
+ }
+ },
+ "index-pattern": {
+ "properties": {
+ "fieldFormatMap": {
+ "type": "text"
+ },
+ "fields": {
+ "type": "text"
+ },
+ "intervalName": {
+ "type": "text"
+ },
+ "notExpandable": {
+ "type": "boolean"
+ },
+ "sourceFilters": {
+ "type": "text"
+ },
+ "timeFieldName": {
+ "type": "text"
+ },
+ "title": {
+ "type": "text"
+ }
+ }
+ },
+ "dashboard": {
+ "properties": {
+ "description": {
+ "type": "text"
+ },
+ "hits": {
+ "type": "integer"
+ },
+ "kibanaSavedObjectMeta": {
+ "properties": {
+ "searchSourceJSON": {
+ "type": "text"
+ }
+ }
+ },
+ "optionsJSON": {
+ "type": "text"
+ },
+ "panelsJSON": {
+ "type": "text"
+ },
+ "refreshInterval": {
+ "properties": {
+ "display": {
+ "type": "text"
+ },
+ "pause": {
+ "type": "boolean"
+ },
+ "section": {
+ "type": "integer"
+ },
+ "value": {
+ "type": "integer"
+ }
+ }
+ },
+ "timeFrom": {
+ "type": "text"
+ },
+ "timeRestore": {
+ "type": "boolean"
+ },
+ "timeTo": {
+ "type": "text"
+ },
+ "title": {
+ "type": "text"
+ },
+ "uiStateJSON": {
+ "type": "text"
+ },
+ "version": {
+ "type": "integer"
+ }
+ }
+ },
+ "search": {
+ "properties": {
+ "columns": {
+ "type": "text"
+ },
+ "description": {
+ "type": "text"
+ },
+ "hits": {
+ "type": "integer"
+ },
+ "kibanaSavedObjectMeta": {
+ "properties": {
+ "searchSourceJSON": {
+ "type": "text"
+ }
+ }
+ },
+ "sort": {
+ "type": "text"
+ },
+ "title": {
+ "type": "text"
+ },
+ "version": {
+ "type": "integer"
+ }
+ }
+ },
+ "visualization": {
+ "properties": {
+ "description": {
+ "type": "text"
+ },
+ "kibanaSavedObjectMeta": {
+ "properties": {
+ "searchSourceJSON": {
+ "type": "text"
+ }
+ }
+ },
+ "savedSearchId": {
+ "type": "text"
+ },
+ "title": {
+ "type": "text"
+ },
+ "uiStateJSON": {
+ "type": "text"
+ },
+ "version": {
+ "type": "integer"
+ },
+ "visState": {
+ "type": "text"
+ }
+ }
+ }
+ },
+ "settings": {
+ "index": {
+ "creation_date": "1490215360374",
+ "number_of_shards": "1",
+ "number_of_replicas": "1",
+ "uuid": "g5yFFGPuSsGwZHzNSyWKHg",
+ "version": {
+ "created": "5010299"
+ },
+ "provided_name": ".kibana"
+ }
+ }
+}
diff --git a/puppet/modules/kibana/manifests/dashboard.pp
b/puppet/modules/kibana/manifests/dashboard.pp
deleted file mode 100644
index b2d1245..0000000
--- a/puppet/modules/kibana/manifests/dashboard.pp
+++ /dev/null
@@ -1,21 +0,0 @@
-# == Define: kibana::dashboard
-#
-# === Parameters:
-# - $content: Dashboard contents
-#
-define kibana::dashboard(
- $content,
-) {
- $safe_title = regsubst($title, '\W_', '-', 'G')
- $dashboard = {
- 'user' => 'guest',
- 'group' => 'guest',
- 'title' => $safe_title,
- 'dashboard' => $content,
- }
- exec { "save dashboard ${title}":
- command => template('kibana/save-dashboard.erb'),
- unless => template('kibana/check-dashboard.erb'),
- require => Service['elasticsearch'],
- }
-}
diff --git a/puppet/modules/kibana/manifests/init.pp
b/puppet/modules/kibana/manifests/init.pp
index 54fc826..8f455ac 100644
--- a/puppet/modules/kibana/manifests/init.pp
+++ b/puppet/modules/kibana/manifests/init.pp
@@ -18,25 +18,42 @@
$deploy_dir,
$default_route,
) {
- git::clone { 'operations/software/kibana':
- ensure => 'latest',
- directory => $deploy_dir,
- owner => 'root',
- group => 'root',
+ package { 'kibana':
+ ensure => latest,
}
- file { '/etc/kibana':
- ensure => directory,
- owner => 'root',
- group => 'root',
- mode => '0755',
- }
-
- file { '/etc/kibana/config.js':
- ensure => present,
- content => template('kibana/config.js'),
+ file { '/etc/kibana/kibana.yml':
+ ensure => file,
owner => 'root',
group => 'root',
- mode => '0644',
+ mode => '0444',
+ content => ordered_yaml({
+ 'kibana.defaultAppId' => 'dashboard/default',
+ 'logging.quiet' => true,
+ 'server.host' => '0.0.0.0',
+ }),
+ require => Package['kibana']
+ }
+
+ service { 'kibana':
+ ensure => running,
+ enable => true,
+ require => [
+ Package['kibana'],
+ File['/etc/kibana/kibana.yml'],
+ ],
+ }
+
+ exec { 'create-kibana-index':
+ command => 'curl -XPUT localhost:9200/.kibana --data-binary
@/vagrant/puppet/modules/kibana/files/kibana-mapping.json',
+ unless => 'curl -sf -XHEAD localhost:9200/.kibana',
+ require => Exec['wait-for-elasticsearch'],
+ }
+
+ exec { 'preload-kibana-dashboard':
+ command => 'curl -sf -XPOST localhost:9200/.kibana/_bulk
--data-binary @/vagrant/puppet/modules/kibana/files/kibana-dump.json >
/dev/null',
+ refreshonly => true,
+ subscribe => Exec['create-kibana-index'],
+ require => Exec['wait-for-elasticsearch'],
}
}
diff --git a/puppet/modules/kibana/templates/check-dashboard.erb
b/puppet/modules/kibana/templates/check-dashboard.erb
deleted file mode 100644
index fd7c262..0000000
--- a/puppet/modules/kibana/templates/check-dashboard.erb
+++ /dev/null
@@ -1,2 +0,0 @@
-/usr/bin/curl -sf \
- 'http://127.0.0.1:9200/kibana-int/dashboard/<%= @safe_title %>' >/dev/null
diff --git a/puppet/modules/kibana/templates/config.js
b/puppet/modules/kibana/templates/config.js
deleted file mode 100644
index 1f6abd9..0000000
--- a/puppet/modules/kibana/templates/config.js
+++ /dev/null
@@ -1,67 +0,0 @@
-/** @scratch /configuration/config.js/1
- * == Configuration
- * config.js is where you will find the core Kibana configuration. This file
- * contains parameters that must be set before kibana is run for the first
time.
- */
-define(['settings'],
-function (Settings) {
- "use strict";
-
- /** @scratch /configuration/config.js/2
- * === Parameters
- */
- return new Settings({
-
- /** @scratch /configuration/config.js/5
- * ==== elasticsearch
- *
- * Our apache config acts as a reverse proxy to the elasticsearch cluster.
- */
- elasticsearch: '//' + window.location.host,
-
- /** @scratch /configuration/config.js/5
- * ==== default_route
- *
- * This is the default landing page when you don't specify a dashboard to
- * load. You can specify files, scripts or saved dashboards here. For
- * example, if you had saved a dashboard called `WebLogs' to elasticsearch
- * you might use:
- *
- * +default_route: '/dashboard/elasticsearch/WebLogs',+
- */
- default_route : <%= @default_route.to_pson %>,
-
- /** @scratch /configuration/config.js/5
- * ==== kibana-int
- *
- * The default ES index to use for storing Kibana specific object
- * such as stored dashboards
- */
- kibana_index: "kibana-int",
-
- /** @scratch /configuration/config.js/5
- * ==== panel_name
- *
- * An array of panel modules available. Panels will only be loaded when
- * they are defined in the dashboard, but this list is used in the "add
- * panel" interface.
- */
- panel_names: [
- 'histogram',
- 'map',
- 'goal',
- 'table',
- 'filtering',
- 'timepicker',
- 'text',
- 'hits',
- 'column',
- 'trends',
- 'bettermap',
- 'query',
- 'terms',
- 'stats',
- 'sparklines'
- ]
- });
-});
diff --git a/puppet/modules/kibana/templates/save-dashboard.erb
b/puppet/modules/kibana/templates/save-dashboard.erb
deleted file mode 100644
index 56a83e7..0000000
--- a/puppet/modules/kibana/templates/save-dashboard.erb
+++ /dev/null
@@ -1,3 +0,0 @@
-/usr/bin/curl -XPUT -s \
- 'http://127.0.0.1:9200/kibana-int/dashboard/<%= @safe_title %>' \
- --data-binary '<%= @dashboard.to_pson %>'
diff --git a/puppet/modules/logstash/files/elasticsearch-template.json
b/puppet/modules/logstash/files/elasticsearch-template.json
index 0e74c36..7b5dfc0 100644
--- a/puppet/modules/logstash/files/elasticsearch-template.json
+++ b/puppet/modules/logstash/files/elasticsearch-template.json
@@ -1,67 +1,148 @@
{
- "template": "logstash-*",
- "settings": {
- "number_of_shards": 1,
- "number_of_replicas": 0,
- "index.cache.field.type": "soft",
- "index.refresh_interval": "5s",
- "analysis": {
- "analyzer": {
- "default": {
- "type": "standard",
- "stopwords": "_none_"
+ "template" : "logstash-*",
+ "settings" : {
+ "number_of_shards" : 1,
+ "number_of_replicas" : 0,
+ "index.refresh_interval" : "5s",
+ "analysis" : {
+ "analyzer" : {
+ "default" : {
+ "type" : "standard",
+ "stopwords" : "_none_"
}
}
}
},
- "mappings": {
- "_default_": {
- "_all": { "enabled": true },
- "dynamic_templates": [ {
- "string_fields": {
- "match": "*",
- "match_mapping_type": "string",
- "mapping": {
- "type": "multi_field",
- "fields": {
- "{name}": {
- "type": "string",
- "index": "analyzed",
- "omit_norms": true
- },
- "raw": {
- "type": "string",
- "index": "no",
- "ignore_above": 256,
- "fielddata": {
- "format": "doc_values"
+ "mappings" : {
+ "_default_" : {
+ "_all" : {
+ "enabled" : true,
+ "omit_norms": true
+ },
+ "date_detection" : false,
+ "numeric_detection": false,
+ "dynamic_templates" : [
+ {
+ "string_fields" : {
+ "match_mapping_type" : "string",
+ "mapping" : {
+ "type" : "text",
+ "omit_norms" : true,
+ "fields" : {
+ "raw" : {
+ "type" : "keyword",
+ "doc_values": true
+ }
+ }
+ }
+ }
+ },
+ {
+ "boolean_to_string" : {
+ "match_mapping_type" : "boolean",
+ "mapping" : {
+ "type" : "keyword",
+ "omit_norms" : true,
+ "fields" : {
+ "raw" : {
+ "type" : "keyword",
+ "doc_values": true
+ }
+ }
+ }
+ }
+ },
+ {
+ "double_to_string" : {
+ "match_mapping_type" : "double",
+ "mapping" : {
+ "type" : "keyword",
+ "omit_norms" : true,
+ "fields" : {
+ "raw" : {
+ "type" : "keyword",
+ "doc_values": true
+ }
+ }
+ }
+ }
+ },
+ {
+ "long_to_string" : {
+ "match_mapping_type" : "long",
+ "mapping" : {
+ "type" : "keyword",
+ "omit_norms" : true,
+ "fields" : {
+ "raw" : {
+ "type" : "keyword",
+ "doc_values": true
}
}
}
}
}
- } ],
- "properties": {
- "@timestamp": { "type": "date", "index": "not_analyzed" },
- "@version": { "type": "string", "index": "not_analyzed" },
- "geoip" : {
- "type": "object",
- "dynamic": true,
- "properties": { "location": { "type": "geo_point" } }
+ ],
+ "properties" : {
+ "@timestamp" : {
+ "type" : "date",
+ "doc_values": true
},
- "host": { "type": "string", "index": "not_analyzed" },
- "message": {
- "type": "string",
- "index": "analyzed",
- "omit_norms": true
+ "@version" : {
+ "type" : "keyword",
+ "doc_values": true
},
- "sequence_id": { "type": "long" },
- "tags": {
- "type": "string",
- "index": "not_analyzed"
+ "geoip" : {
+ "type" : "object",
+ "dynamic" : true,
+ "properties" : {
+ "location" : {
+ "type" : "geo_point",
+ "doc_values": true
+ }
+ }
},
- "type": { "type": "string", "index": "not_analyzed" },
- "udp_sender": { "type": "string", "index": "not_analyzed" }
+ "host" : {
+ "type" : "keyword",
+ "doc_values": true
+ },
+ "level" : {
+ "type" : "keyword",
+ "doc_values": true
+ },
+ "message" : {
+ "type" : "text",
+ "omit_norms" : true
+ },
+ "normalized_message" : {
+ "type": "text",
+ "fields": {
+ "raw": {
+ "type": "keyword",
+ "doc_values": true
+ },
+ "normalized_message": {
+ "type" : "text",
+ "omit_norms" : true
+ }
+ }
+ },
+ "sequence_id" : {
+ "type": "long",
+ "doc_values": true
+ },
+ "tags" : {
+ "type" : "keyword",
+ "doc_values": true
+ },
+ "type" : {
+ "type" : "keyword",
+ "doc_values": true
+ },
+ "udp_sender" : {
+ "type" : "keyword",
+ "doc_values": true
+ }
}
}
}
diff --git a/puppet/modules/logstash/manifests/init.pp
b/puppet/modules/logstash/manifests/init.pp
index 0e30ec2..e1470ca 100644
--- a/puppet/modules/logstash/manifests/init.pp
+++ b/puppet/modules/logstash/manifests/init.pp
@@ -6,31 +6,35 @@
#
# == Parameters:
# - $heap_memory_mb: amount of memory to allocate to logstash in megabytes.
-# - $filter_workers: number of worker threads to run to process filters
+# - $pipeline_workers: number of worker threads to run to use for
+# filter/output processing
#
# == Sample usage:
#
# class { 'logstash':
# heap_memory_mb => 128,
-# filter_workers => 3,
+# pipeline_workers => 3,
# }
#
class logstash(
$heap_memory_mb,
- $filter_workers,
+ $pipeline_workers,
) {
- require_package('openjdk-7-jre-headless')
+ require_package('openjdk-8-jre-headless')
package { 'logstash':
- ensure => present,
- require => Package['openjdk-7-jre-headless'],
+ ensure => latest,
+ require => Package['openjdk-8-jre-headless'],
}
- file { '/etc/default/logstash':
- content => template('logstash/default.erb'),
- require => Package['logstash'],
- notify => Service['logstash'],
+ # JRuby waits for enough entropy from /dev/random when starting up.
Especially
+ # within VM's very little entropy may be available. haveged adds an
entropy source
+ # to fix this problem.
+ package { 'haveged':
+ ensure => present,
+ before => Service['logstash']
}
+ Package['haveged'] -> Logstash::Plugin <| |>
file { '/etc/logstash/conf.d':
ensure => directory,
@@ -41,31 +45,39 @@
require => Package['logstash'],
}
- service { 'logstash':
- ensure => running,
- provider => 'debian',
- enable => true,
- hasstatus => true,
- hasrestart => true,
- require => Package['logstash'],
+ file { '/etc/logstash/logstash.yml':
+ ensure => file,
+ content => ordered_yaml({
+ 'path.data' => '/var/lib/logstash',
+ 'path.config' => '/etc/logstash/conf.d',
+ 'path.logs' => '/var/log/logstash',
+ 'pipeline.workers' => $pipeline_workers,
+ }),
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
+ require => Package['logstash'],
+ before => Service['logstash'],
+ notify => Service['logstash'],
}
- # Upstream package provides both sysv and upstart startup scripts. Remove
- # the upstart versions so that Puppet and `service logstash ...` don't
- # fight over which to use.
- file { '/etc/init/logstash.conf':
- ensure => absent,
- require => Package['logstash'],
- }
- file { '/etc/init/logstash-web.conf':
- ensure => absent,
- require => Package['logstash'],
- }
-
- exec { 'install logstash contrib plugins':
- command => '/opt/logstash/bin/plugin install contrib',
- unless => '/usr/bin/test -d /opt/logstash/vendor/logstash/',
+ file { '/etc/logstash/jvm.options':
+ ensure => file,
+ content => template('logstash/jvm.options.erb'),
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
require => Package['logstash'],
notify => Service['logstash'],
}
+
+ service { 'logstash':
+ ensure => running,
+ provider => 'systemd',
+ enable => true,
+ require => [
+ Package['logstash'],
+ File['/etc/logstash/jvm.options']
+ ]
+ }
}
diff --git a/puppet/modules/logstash/manifests/output/elasticsearch.pp
b/puppet/modules/logstash/manifests/output/elasticsearch.pp
index 398ac1a..b7bf8ac 100644
--- a/puppet/modules/logstash/manifests/output/elasticsearch.pp
+++ b/puppet/modules/logstash/manifests/output/elasticsearch.pp
@@ -11,8 +11,6 @@
# - $idle_flush_time: Maxmimum seconds to wait between sends. Default 1.
# - $index: Index to write events to. Default 'logstash-%{+YYYY.MM.dd}'.
# - $port: Elasticsearch server port. Default 9200.
-# - $replication: Type of elasticsearch replication to use ('async', 'sync').
-# Default 'sync'.
# - $require_tag: Tag to require on events. Default undef.
# - $manage_indices: Whether cron jobs should be installed to manage
# Elasticsearch indices. Default false.
@@ -23,7 +21,6 @@
#
# class { 'logstash::output::elasticsearch':
# host => '127.0.0.1',
-# replication => 'async',
# require_tag => 'es',
# manage_indices => 'true',
# }
@@ -35,7 +32,6 @@
$idle_flush_time = 1,
$index = 'logstash-%{+YYYY.MM.dd}',
$port = 9200,
- $replication = 'sync',
$require_tag = undef,
$manage_indices = false,
$priority = undef,
diff --git a/puppet/modules/logstash/manifests/plugin.pp
b/puppet/modules/logstash/manifests/plugin.pp
new file mode 100644
index 0000000..ac95aee
--- /dev/null
+++ b/puppet/modules/logstash/manifests/plugin.pp
@@ -0,0 +1,35 @@
+# == Define: logstash::plugin
+#
+# A Logstash plugin.
+#
+# === Parameters
+#
+# [*title*]
+# Name of the plugin. Examples:
+# logstash-filter-prune
+#
+define logstash::plugin(
+ $ensure = present,
+) {
+ case $ensure {
+ present: {
+ exec { "install_logstash_plugin_${title}":
+ command => "/usr/share/logstash/bin/logstash-plugin install
'${title}'",
+ require => Package['logstash'],
+ notify => Service['logstash'],
+ unless => "grep '^gem \"${title}\"$'
/usr/share/logstash/Gemfile"
+ }
+ }
+ absent: {
+ exec { "uninstall_logstash_plugin_${title}":
+ command => "/usr/share/logstash/bin/logstash-plugin remove
'${title}'",
+ require => Package['logstash'],
+ notify => Service['logstash'],
+ onlyif => "grep '^gem \"${title}\"$'
/usr/share/logstash/Gemfile"
+ }
+ }
+ default: {
+ fail("'ensure' may be 'present' or 'absent' (got: '${ensure}').")
+ }
+ }
+}
diff --git a/puppet/modules/logstash/templates/default.erb
b/puppet/modules/logstash/templates/default.erb
deleted file mode 100644
index 0c3aade..0000000
--- a/puppet/modules/logstash/templates/default.erb
+++ /dev/null
@@ -1,30 +0,0 @@
-#####################################################################
-## THIS FILE IS MANAGED BY PUPPET
-## puppet:///modules/logstash/default
-#####################################################################
-
-# Start logstash on boot?
-START=yes
-
-# Location of the logstash jar file
-LS_JAR=/opt/logstash/logstash.jar
-
-# Java JVM options
-LS_JAVA_OPTS="-Xms<%= @heap_memory_mb %>m -Xmx<%= @heap_memory_mb %>m"
-LS_JAVA_OPTS="${LS_JAVA_OPTS} -Djava.io.tmpdir=/var/lib/logstash"
-
-# Location of configuration files.
-# All files in this directory are concatenated in alphabetical order.
-CONF_DIR=/etc/logstash/conf.d
-
-# Logstash will look for ruby plugins in the logstash/{inputs,filters,outputs}
-# subdirectories of plugin_dir
-PLUGIN_DIR=/var/lib
-
-# Logstash options
-# Note: the sysv init script prepends these options with "agent -f ${CONF_DIR}"
-LS_OPTS="--log /var/log/logstash/logstash.log"
-LS_OPTS="${LS_OPTS} --filterworkers <%= @filter_workers %>"
-# XXX: Don't add plugin dir until we need it.
-# Logstash refuses to start if missing
-#LS_OPTS="${LS_OPTS} --pluginpath ${PLUGIN_DIR}"
diff --git a/puppet/modules/logstash/templates/jvm.options.erb
b/puppet/modules/logstash/templates/jvm.options.erb
new file mode 100644
index 0000000..e75cd6a
--- /dev/null
+++ b/puppet/modules/logstash/templates/jvm.options.erb
@@ -0,0 +1,79 @@
+#####################################################################
+## THSI FILE IS MANAGED BY PUPPET
+## puppet://modules/logstash/jvm.options.erb
+#####################################################################
+
+## JVM configuration
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms<%= @heap_memory_mb %>m
+-Xmx<%= @heap_memory_mb %>m
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+-XX:+UseParNewGC
+-XX:+UseConcMarkSweepGC
+-XX:CMSInitiatingOccupancyFraction=75
+-XX:+UseCMSInitiatingOccupancyOnly
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+## locale
+# Set the locale language
+#-Duser.language=en
+
+# Set the locale country
+#-Duser.country=US
+
+# Set the locale variant, if any
+#-Duser.variant=
+
+## basic
+
+# set the I/O temp directory
+#-Djava.io.tmpdir=$HOME
+
+# set to headless, just in case
+-Djava.awt.headless=true
+
+# ensure UTF-8 encoding by default (e.g. filenames)
+-Dfile.encoding=UTF-8
+
+# use our provided JNA always versus the system one
+#-Djna.nosys=true
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+#-XX:HeapDumpPath=${LOGSTASH_HOME}/heapdump.hprof
+
+## GC logging
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${LS_GC_LOG_FILE}
diff --git a/puppet/modules/logstash/templates/output/elasticsearch.erb
b/puppet/modules/logstash/templates/output/elasticsearch.erb
index 0457b72..e60e9ab 100644
--- a/puppet/modules/logstash/templates/output/elasticsearch.erb
+++ b/puppet/modules/logstash/templates/output/elasticsearch.erb
@@ -1,12 +1,10 @@
output {
<%- if @require_tag %>if "<%= @require_tag %>" in [tags] {<% end -%>
- elasticsearch_http {
+ elasticsearch {
flush_size => <%= @flush_size %>
- host => "<%= @host %>"
+ hosts => ["<%= @host %>:<%= @port %>"]
idle_flush_time => <%= @idle_flush_time %>
index => "<%= @index %>"
- port => <%= @port %>
- replication => "<%= @replication %>"
}
<% if @require_tag %>}<% end %>
}
diff --git a/puppet/modules/role/manifests/elk.pp
b/puppet/modules/role/manifests/elk.pp
index 7073a53..c960200 100644
--- a/puppet/modules/role/manifests/elk.pp
+++ b/puppet/modules/role/manifests/elk.pp
@@ -35,14 +35,28 @@
priority => 40,
}
+ logstash::plugin { 'logstash-filter-prune':
+ ensure => present
+ }
logstash::conf { 'filter_gelf':
source => 'puppet:///modules/role/elk/filter-gelf.conf',
priority => 50,
+ require => Logstash::Plugin['logstash-filter-prune'],
}
+ logstash::plugin { 'logstash-filter-multiline':
+ ensure => present
+ }
+ logstash::plugin { 'logstash-filter-anonymize':
+ ensure => present
+ }
logstash::conf { 'filter_syslog':
source => 'puppet:///modules/role/elk/filter-syslog.conf',
priority => 50,
+ require => [
+ Logstash::Plugin['logstash-filter-multiline'],
+ Logstash::Plugin['logstash-filter-anonymize'],
+ ]
}
exec { 'Create logstash index template':
@@ -55,20 +69,6 @@
## Configure Kibana
apache::site { $vhost_name:
content => template('role/elk/apache.conf.erb'),
- }
-
- exec { 'Create kibana index':
- command => template('role/elk/create-kibana-index.erb'),
- unless => template('role/elk/check-kibana-index.erb'),
- require => Service['elasticsearch'],
- }
- kibana::dashboard { 'default':
- content => template('role/elk/dashboard-default.erb'),
- require => Exec['Create kibana index'],
- }
- kibana::dashboard { 'mediawiki':
- content => template('role/elk/dashboard-mediawiki.erb'),
- require => Exec['Create kibana index'],
}
## Configure MediaWiki
diff --git a/puppet/modules/role/settings/elk.yaml
b/puppet/modules/role/settings/elk.yaml
index 961c898..0eceb2d 100644
--- a/puppet/modules/role/settings/elk.yaml
+++ b/puppet/modules/role/settings/elk.yaml
@@ -1,3 +1,4 @@
vagrant_ram: 512
forward_ports:
9200: 9200
+ 5601: 5601
--
To view, visit https://gerrit.wikimedia.org/r/342772
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I4f23fd402d13ad33c8b6a72fc96b3b39be6ba364
Gerrit-PatchSet: 6
Gerrit-Project: mediawiki/vagrant
Gerrit-Branch: master
Gerrit-Owner: EBernhardson <[email protected]>
Gerrit-Reviewer: BryanDavis <[email protected]>
Gerrit-Reviewer: Dduvall <[email protected]>
Gerrit-Reviewer: EBernhardson <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
