EBernhardson has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/341354 )
Change subject: migrate to elasticsearch 5
......................................................................
migrate to elasticsearch 5
Initial work porting to elasticsearch 5. This probably shouldn't be
merged until the es5 branches are merged into cirrussearch and elastica.
It needs to be ready to go by then though.
* Adds wikimedia experimental repo to source elasticsearch 5.x package
for jessie, for trusty adds an openjdk ppa and the elasticsearch
apt repo.
* Adds jvm.options for settings memory usage in es5
* Replaces logging.yml with log4j2.properties
* Installs openjdk-8-jre-headless
* Replaces defaults file with the one from es5. adjusted
to set JAVA_HOME to point at java 8
* updates elasticsearch::plugin to work with the new
elasticsearch-plugins script
todo:
* raita role?
** Doesn't currently install, nodejs is duplicated in npm and raita
roles.
* elk role?
** This role is woefully outdated and isn't particularly useful in it's
current state to support development. It should either be updated
or removed.
* quips role?
** The app uses the Elastica php library. It would need to be upgraded
to elasticsearch 5. Not sure at what point that is desired though, as
IIUC the primary instance of this talks to the tool labs elasticsearch
cluster and I don't know when that upgrade is planned.
* sal role?
** Simliar to the quips role, this uses the Elastica php library and
the primary instance runs against the tool labs elasticsearch
cluster. Not sure what to do with it.
hacks that (might) need resolving:
* Running this in labs w/lxc requires setting lxc.aa_profile = unconfined
because the .deb wants to set /proc/sys/vm/max_map_count and cannot. The
package never finishes installing without this.
Bug: T159002
Change-Id: Ic8cf196d66edeed7964887cd3a02e9da2c7fe366
---
M Vagrantfile
M puppet/modules/elasticsearch/files/defaults
A puppet/modules/elasticsearch/files/elasticsearch-pubkey.asc
A puppet/modules/elasticsearch/files/jvm.options
A puppet/modules/elasticsearch/files/log4j2.properties
M puppet/modules/elasticsearch/files/mwv-elasticsearch-plugin
M puppet/modules/elasticsearch/manifests/init.pp
M puppet/modules/elasticsearch/manifests/plugin.pp
D puppet/modules/elasticsearch/templates/logging.yml.erb
A puppet/modules/elasticsearch/templates/wikimedia-experimental.list.erb
M puppet/modules/kibana/manifests/dashboard.pp
M puppet/modules/role/manifests/cirrussearch.pp
12 files changed, 357 insertions(+), 133 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/vagrant
refs/changes/54/341354/1
diff --git a/Vagrantfile b/Vagrantfile
index c726d6b..ee403cf 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -195,6 +195,9 @@
config.vm.provider :lxc do |lxc|
lxc.customize 'cgroup.memory.limit_in_bytes', "#{settings[:vagrant_ram]}M"
lxc.backingstore = 'none'
+ if mwv.roles_enabled.include?('cirrussearch')
+ lxc.customize 'aa_profile', 'unconfined'
+ end
end
config.vm.provider :libvirt do |libvirt|
diff --git a/puppet/modules/elasticsearch/files/defaults
b/puppet/modules/elasticsearch/files/defaults
index ad47758..acaedfd 100644
--- a/puppet/modules/elasticsearch/files/defaults
+++ b/puppet/modules/elasticsearch/files/defaults
@@ -1,44 +1,67 @@
-# Run ElasticSearch as this user ID and group ID
-#ES_USER=elasticsearch
-#ES_GROUP=elasticsearch
+################################
+# Elasticsearch
+################################
-# Heap Size (defaults to 256m min, 1g max)
-ES_HEAP_SIZE=256m
+# Elasticsearch home directory
+#ES_HOME=/usr/share/elasticsearch
-# Heap new generation
-#ES_HEAP_NEWSIZE=
+# Elasticsearch Java path
+#JAVA_HOME=
-# max direct memory
-#ES_DIRECT_SIZE=
-
-# Maximum number of open files, defaults to 65535.
-#MAX_OPEN_FILES=65535
-
-# Maximum locked memory size. Set to "unlimited" if you use the
-# bootstrap.mlockall option in elasticsearch.yml. You must also set
-# ES_HEAP_SIZE.
-#MAX_LOCKED_MEMORY=unlimited
-
-# Maximum number of VMA (Virtual Memory Areas) a process can own
-#MAX_MAP_COUNT=262144
-
-# ElasticSearch log directory
-#LOG_DIR=/var/log/elasticsearch
-
-# ElasticSearch data directory
-#DATA_DIR=/var/lib/elasticsearch
-
-# ElasticSearch work directory
-#WORK_DIR=/tmp/elasticsearch
-
-# ElasticSearch configuration directory
+# Elasticsearch configuration directory
#CONF_DIR=/etc/elasticsearch
-# ElasticSearch configuration file (elasticsearch.yml)
-#CONF_FILE=/etc/elasticsearch/elasticsearch.yml
+# Elasticsearch data directory
+#DATA_DIR=/var/lib/elasticsearch
+
+# Elasticsearch logs directory
+#LOG_DIR=/var/log/elasticsearch
+
+# Elasticsearch PID directory
+#PID_DIR=/var/run/elasticsearch
# Additional Java OPTS
#ES_JAVA_OPTS=
# Configure restart on package upgrade (true, every other setting will lead to
not restarting)
#RESTART_ON_UPGRADE=true
+
+################################
+# Elasticsearch service
+################################
+
+# SysV init.d
+#
+# When executing the init script, this user will be used to run the
elasticsearch service.
+# The default value is 'elasticsearch' and is declared in the init.d file.
+# Note that this setting is only used by the init script. If changed, make
sure that
+# the configured user can read and write into the data, work, plugins and log
directories.
+# For systemd service, the user is usually configured in file
/usr/lib/systemd/system/elasticsearch.service
+#ES_USER=elasticsearch
+#ES_GROUP=elasticsearch
+
+# The number of seconds to wait before checking if Elasticsearch started
successfully as a daemon process
+ES_STARTUP_SLEEP_TIME=5
+
+################################
+# System properties
+################################
+
+# Specifies the maximum file descriptor number that can be opened by this
process
+# When using Systemd, this setting is ignored and the LimitNOFILE defined in
+# /usr/lib/systemd/system/elasticsearch.service takes precedence
+#MAX_OPEN_FILES=65536
+
+# The maximum number of bytes of memory that may be locked into RAM
+# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
+# in elasticsearch.yml.
+# When using Systemd, the LimitMEMLOCK property must be set
+# in /usr/lib/systemd/system/elasticsearch.service
+#MAX_LOCKED_MEMORY=unlimited
+
+# Maximum number of VMA (Virtual Memory Areas) a process can own
+# When using Systemd, this setting is ignored and the 'vm.max_map_count'
+# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
+#MAX_MAP_COUNT=262144
+
+JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
diff --git a/puppet/modules/elasticsearch/files/elasticsearch-pubkey.asc
b/puppet/modules/elasticsearch/files/elasticsearch-pubkey.asc
new file mode 100644
index 0000000..1b50dcc
--- /dev/null
+++ b/puppet/modules/elasticsearch/files/elasticsearch-pubkey.asc
@@ -0,0 +1,31 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Version: GnuPG v2.0.14 (GNU/Linux)
+
+mQENBFI3HsoBCADXDtbNJnxbPqB1vDNtCsqhe49vFYsZN9IOZsZXgp7aHjh6CJBD
+A+bGFOwyhbd7at35jQjWAw1O3cfYsKAmFy+Ar3LHCMkV3oZspJACTIgCrwnkic/9
+CUliQe324qvObU2QRtP4Fl0zWcfb/S8UYzWXWIFuJqMvE9MaRY1bwUBvzoqavLGZ
+j3SF1SPO+TB5QrHkrQHBsmX+Jda6d4Ylt8/t6CvMwgQNlrlzIO9WT+YN6zS+sqHd
+1YK/aY5qhoLNhp9G/HxhcSVCkLq8SStj1ZZ1S9juBPoXV1ZWNbxFNGwOh/NYGldD
+2kmBf3YgCqeLzHahsAEpvAm8TBa7Q9W21C8vABEBAAG0RUVsYXN0aWNzZWFyY2gg
+KEVsYXN0aWNzZWFyY2ggU2lnbmluZyBLZXkpIDxkZXZfb3BzQGVsYXN0aWNzZWFy
+Y2gub3JnPokBOAQTAQIAIgUCUjceygIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgEC
+F4AACgkQ0n1mbNiOQrRzjAgAlTUQ1mgo3nK6BGXbj4XAJvuZDG0HILiUt+pPnz75
+nsf0NWhqR4yGFlmpuctgCmTD+HzYtV9fp9qW/bwVuJCNtKXk3sdzYABY+Yl0Cez/
+7C2GuGCOlbn0luCNT9BxJnh4mC9h/cKI3y5jvZ7wavwe41teqG14V+EoFSn3NPKm
+TxcDTFrV7SmVPxCBcQze00cJhprKxkuZMPPVqpBS+JfDQtzUQD/LSFfhHj9eD+Xe
+8d7sw+XvxB2aN4gnTlRzjL1nTRp0h2/IOGkqYfIG9rWmSLNlxhB2t+c0RsjdGM4/
+eRlPWylFbVMc5pmDpItrkWSnzBfkmXL3vO2X3WvwmSFiQbkBDQRSNx7KAQgA5JUl
+zcMW5/cuyZR8alSacKqhSbvoSqqbzHKcUQZmlzNMKGTABFG1yRx9r+wa/fvqP6OT
+RzRDvVS/cycws8YX7Ddum7x8uI95b9ye1/Xy5noPEm8cD+hplnpU+PBQZJ5XJ2I+
+1l9Nixx47wPGXeClLqcdn0ayd+v+Rwf3/XUJrvccG2YZUiQ4jWZkoxsA07xx7Bj+
+Lt8/FKG7sHRFvePFU0ZS6JFx9GJqjSBbHRRkam+4emW3uWgVfZxuwcUCn1ayNgRt
+KiFv9jQrg2TIWEvzYx9tywTCxc+FFMWAlbCzi+m4WD+QUWWfDQ009U/WM0ks0Kww
+EwSk/UDuToxGnKU2dQARAQABiQEfBBgBAgAJBQJSNx7KAhsMAAoJENJ9ZmzYjkK0
+c3MIAIE9hAR20mqJWLcsxLtrRs6uNF1VrpB+4n/55QU7oxA1iVBO6IFu4qgsF12J
+TavnJ5MLaETlggXY+zDef9syTPXoQctpzcaNVDmedwo1SiL03uMoblOvWpMR/Y0j
+6rm7IgrMWUDXDPvoPGjMl2q1iTeyHkMZEyUJ8SKsaHh4jV9wp9KmC8C+9CwMukL7
+vM5w8cgvJoAwsp3Fn59AxWthN3XJYcnMfStkIuWgR7U2r+a210W6vnUxU4oN0PmM
+cursYPyeV0NX/KQeUeNMwGTFB6QHS/anRaGQewijkrYYoTNtfllxIu9XYmiBERQ/
+qPDlGRlOgVTd9xUfHFkzB52c70E=
+=92oX
+-----END PGP PUBLIC KEY BLOCK-----
diff --git a/puppet/modules/elasticsearch/files/jvm.options
b/puppet/modules/elasticsearch/files/jvm.options
new file mode 100644
index 0000000..fee163c
--- /dev/null
+++ b/puppet/modules/elasticsearch/files/jvm.options
@@ -0,0 +1,86 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See
https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms256m
+-Xmx256m
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+-XX:+UseConcMarkSweepGC
+-XX:CMSInitiatingOccupancyFraction=75
+-XX:+UseCMSInitiatingOccupancyOnly
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# set to headless, just in case
+-Djava.awt.headless=true
+
+# ensure UTF-8 encoding by default (e.g. filenames)
+-Dfile.encoding=UTF-8
+
+# use our provided JNA always versus the system one
+-Djna.nosys=true
+
+# use old-style file permissions on JDK9
+-Djdk.io.permissionsUseCanonicalPath=true
+
+# flags to configure Netty
+-Dio.netty.noUnsafe=true
+-Dio.netty.noKeySetOptimization=true
+-Dio.netty.recycler.maxCapacityPerThread=0
+
+# log4j 2
+-Dlog4j.shutdownHookEnabled=false
+-Dlog4j2.disable.jmx=true
+-Dlog4j.skipJansi=true
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+#-XX:HeapDumpPath=${heap.dump.path}
diff --git a/puppet/modules/elasticsearch/files/log4j2.properties
b/puppet/modules/elasticsearch/files/log4j2.properties
new file mode 100644
index 0000000..dad0bc9
--- /dev/null
+++ b/puppet/modules/elasticsearch/files/log4j2.properties
@@ -0,0 +1,75 @@
+status = error
+
+# log action execution errors for easier debugging
+logger.action.name = org.elasticsearch.action
+logger.action.level = debug
+
+appender.console.type = Console
+appender.console.name = console
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n
+
+appender.rolling.type = RollingFile
+appender.rolling.name = rolling
+appender.rolling.fileName = ${sys:es.logs}.log
+appender.rolling.layout.type = PatternLayout
+appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}]
%marker%.-10000m%n
+appender.rolling.filePattern = ${sys:es.logs}-%d{yyyy-MM-dd}.log
+appender.rolling.policies.type = Policies
+appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling.policies.time.interval = 1
+appender.rolling.policies.time.modulate = true
+
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = console
+rootLogger.appenderRef.rolling.ref = rolling
+
+appender.deprecation_rolling.type = RollingFile
+appender.deprecation_rolling.name = deprecation_rolling
+appender.deprecation_rolling.fileName = ${sys:es.logs}_deprecation.log
+appender.deprecation_rolling.layout.type = PatternLayout
+appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}]
%marker%.-10000m%n
+appender.deprecation_rolling.filePattern = ${sys:es.logs}_deprecation-%i.log.gz
+appender.deprecation_rolling.policies.type = Policies
+appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling.policies.size.size = 1GB
+appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling.strategy.max = 4
+
+logger.deprecation.name = org.elasticsearch.deprecation
+logger.deprecation.level = warn
+logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
+logger.deprecation.additivity = false
+
+appender.index_search_slowlog_rolling.type = RollingFile
+appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
+appender.index_search_slowlog_rolling.fileName =
${sys:es.logs}_index_search_slowlog.log
+appender.index_search_slowlog_rolling.layout.type = PatternLayout
+appender.index_search_slowlog_rolling.layout.pattern =
[%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n
+appender.index_search_slowlog_rolling.filePattern =
${sys:es.logs}_index_search_slowlog-%d{yyyy-MM-dd}.log
+appender.index_search_slowlog_rolling.policies.type = Policies
+appender.index_search_slowlog_rolling.policies.time.type =
TimeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling.policies.time.interval = 1
+appender.index_search_slowlog_rolling.policies.time.modulate = true
+
+logger.index_search_slowlog_rolling.name = index.search.slowlog
+logger.index_search_slowlog_rolling.level = trace
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref
= index_search_slowlog_rolling
+logger.index_search_slowlog_rolling.additivity = false
+
+appender.index_indexing_slowlog_rolling.type = RollingFile
+appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
+appender.index_indexing_slowlog_rolling.fileName =
${sys:es.logs}_index_indexing_slowlog.log
+appender.index_indexing_slowlog_rolling.layout.type = PatternLayout
+appender.index_indexing_slowlog_rolling.layout.pattern =
[%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n
+appender.index_indexing_slowlog_rolling.filePattern =
${sys:es.logs}_index_indexing_slowlog-%d{yyyy-MM-dd}.log
+appender.index_indexing_slowlog_rolling.policies.type = Policies
+appender.index_indexing_slowlog_rolling.policies.time.type =
TimeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling.policies.time.interval = 1
+appender.index_indexing_slowlog_rolling.policies.time.modulate = true
+
+logger.index_indexing_slowlog.name = index.indexing.slowlog.index
+logger.index_indexing_slowlog.level = trace
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref =
index_indexing_slowlog_rolling
+logger.index_indexing_slowlog.additivity = false
+
diff --git a/puppet/modules/elasticsearch/files/mwv-elasticsearch-plugin
b/puppet/modules/elasticsearch/files/mwv-elasticsearch-plugin
index bc43122..53f627b 100644
--- a/puppet/modules/elasticsearch/files/mwv-elasticsearch-plugin
+++ b/puppet/modules/elasticsearch/files/mwv-elasticsearch-plugin
@@ -13,10 +13,10 @@
if [ -z "$3" ]; then
INSTALL_ARGS="$PLUGIN_NAME"
else
- # In elasticsearch 2 this is group/title, which elasticsearch will use
- # to magic up some urls. in elasticsearch 5 we will have to provide our
- # own urls
- INSTALL_ARGS="$3/$ES_VERSION"
+ # Magic up a url to maven central that will download the appropriate version
of the plugin
+ PLUGIN_GROUP="$3"
+ PLUGIN_ARTIFACT="$4"
+
INSTALL_ARGS="https://repo1.maven.org/maven2/${PLUGIN_GROUP//.//}/${PLUGIN_ARTIFACT}/${ES_VERSION}/${PLUGIN_ARTIFACT}-${ES_VERSION}.zip";
fi
PLUGIN_DIR="/usr/share/elasticsearch/plugins/${PLUGIN_NAME}"
@@ -45,26 +45,28 @@
else
if [ "$PLUGIN_INSTALLED" = "yes" ]; then
echo "Pruning old version ($PLUGIN_VERSION) of $PLUGIN_NAME"
- /usr/share/elasticsearch/bin/plugin remove "$PLUGIN_NAME"
+ /usr/share/elasticsearch/bin/elasticsearch-plugin remove "$PLUGIN_NAME"
fi
# We need to delete all old plugins before trying to install a new
- # one, "bin/plugin install" will simply fail if an unsupported one
+ # one, "bin/elasticsearch-plugin install" will simply fail if an
unsupported one
# is found in the plugins directory.
find /usr/share/elasticsearch/plugins -mindepth 1 -maxdepth 1 -type d \
'!' '(' -exec test -e '{}/plugin-descriptor.properties' ';' -a \
- -exec egrep -q
"(^elasticsearch.version=${ES_VERSION}|^site=true)" \
+ -exec egrep -q "^elasticsearch.version=${ES_VERSION}" \
'{}/plugin-descriptor.properties' ';' \
')' \
- -exec sh -c '/usr/share/elasticsearch/bin/plugin remove `basename {}`'
';'
+ -exec sh -c '/usr/share/elasticsearch/bin/elasticsearch-plugin remove
`basename {}`' ';'
- /usr/share/elasticsearch/bin/plugin install "$INSTALL_ARGS"
+ echo "Installing elasticsearch plugin: $INSTALL_ARGS"
+ /usr/share/elasticsearch/bin/elasticsearch-plugin install "$INSTALL_ARGS"
fi
;;
uninstall)
if [ "$PLUGIN_INSTALLED" = "yes" ]; then
- /usr/share/elasticsearch/bin/plugin remove "$PLUGIN_NAME"
+ echo "Removing version $PLUGIN_VERSION of $PLUGIN_NAME"
+ /usr/share/elasticsearch/bin/elasticsearch-plugin remove "$PLUGIN_NAME"
else
echo "$PLUGIN_NAME not installed, doing nothing"
fi
diff --git a/puppet/modules/elasticsearch/manifests/init.pp
b/puppet/modules/elasticsearch/manifests/init.pp
index 6d95cf2..06db0e9 100644
--- a/puppet/modules/elasticsearch/manifests/init.pp
+++ b/puppet/modules/elasticsearch/manifests/init.pp
@@ -4,11 +4,68 @@
# engine, much like Solr, but with a more user-friendly inteface.
#
class elasticsearch {
- package { 'elasticsearch':
- ensure => latest,
+ if $::lsbdistcodename == 'jessie' {
+ # Elasticsearch 5.x is currently in the experiemental repository
+ file { '/etc/apt/sources.list.d/wikimedia-experimental.list':
+ ensure => present,
+ content =>
template('elasticsearch/wikimedia-experimental.list.erb'),
+ before => Exec['apt-get update'],
+ # Does this work to force apt-get update beyond the scheduled
daily run?
+ notify => Exec['apt-get update'],
+ }
+ } else {
+ # For openjdk-8
+ apt::ppa { 'openjdk-r/ppa': }
+
+ # elasticsearch 5.x packages. One potential downside here is this will
auto-upgrade
+ # to new releases of elasticsearch before we release custom plugins.
Switching to
+ # .deb packaging for plugins should fix that, as it wont upgrade if
the version
+ # constraints aren't met.
+ file { '/usr/local/share/elasticsearch-pubkey.asc':
+ source =>
'puppet:///modules/elasticsearch/elasticsearch-pubkey.asc',
+ owner => 'root',
+ group => 'root',
+ before => File['/etc/apt/sources.list.d/elasticsearch.list'],
+ notify => Exec['add_elasticsearch_apt_key'],
+ }
+ exec { 'add_elasticsearch_apt_key':
+ command => '/usr/bin/apt-key add
/usr/local/share/elasticsearch-pubkey.asc',
+ before => File['/etc/apt/sources.list.d/elasticsearch.list'],
+ refreshonly => true,
+ }
+ file { '/etc/apt/sources.list.d/elasticsearch.list':
+ content => 'deb https://artifacts.elastic.co/packages/5.x/apt
stable main',
+ before => Exec['apt-get update'],
+ notify => Exec['apt-get update'],
+ }
+ # The 5.x package from elasticsearch needs to be pinned higher than
default wikimedia
+ apt::pin { 'elasticsearch':
+ package => 'elasticsearch',
+ pin => 'release o=elastic',
+ priority => 1010,
+ }
+
}
- require_package('openjdk-7-jre-headless')
+ require_package('openjdk-8-jre-headless')
+
+ package { 'elasticsearch':
+ ensure => latest,
+ require => [
+ Apt::Pin['elasticsearch'],
+ Exec['apt-get update'],
+ ],
+ }
+
+ # This is needed when upgrading from 2.x to 5.x, the directory
+ # ends up owned by root and elasticsearch refuses to start
+ file { '/var/run/elasticsearch':
+ ensure => directory,
+ owner => 'elasticsearch',
+ group => 'elasticsearch',
+ mode => '0755',
+ require => Package['elasticsearch'],
+ }
file { '/usr/local/bin/mwv-elasticsearch-plugin':
source => 'puppet:///modules/elasticsearch/mwv-elasticsearch-plugin',
@@ -17,18 +74,12 @@
mode => '0555',
}
- file { '/var/run/elasticsearch/':
- # Temporary and poor work around for
- # https://github.com/elastic/elasticsearch/issues/11594
- ensure => 'directory'
- }
-
service { 'elasticsearch':
ensure => running,
enable => true,
require => [
- Package['elasticsearch', 'openjdk-7-jre-headless'],
- File['/var/run/elasticsearch/'],
+ Package['elasticsearch'],
+ Package['openjdk-8-jre-headless'],
]
}
@@ -42,13 +93,24 @@
source => 'puppet:///modules/elasticsearch/elasticsearch.yml',
require => Package['elasticsearch'],
notify => Service['elasticsearch'],
- }
-
- file { '/etc/elasticsearch/logging.yml':
- ensure => file,
owner => 'root',
group => 'root',
- content => template('elasticsearch/logging.yml.erb'),
+ mode => '0444',
+ }
+
+ file { '/etc/elasticsearch/jvm.options':
+ source => 'puppet:///modules/elasticsearch/jvm.options',
+ require => Package['elasticsearch'],
+ notify => Service['elasticsearch'],
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
+ }
+
+ file { '/etc/elasticsearch/log4j2.properties':
+ source => 'puppet:///modules/elasticsearch/log4j2.properties',
+ owner => 'root',
+ group => 'root',
mode => '0444',
require => Package['elasticsearch'],
}
@@ -62,8 +124,15 @@
# The logrotate above works on size, rather than daily. For this to work
# reasonably well logrotate needs to run multiple times per day
+ file { '/etc/cron.hourly':
+ ensure => 'directory',
+ owner => 'root',
+ group => 'root',
+ mode => '0755',
+ }
file { '/etc/cron.hourly/logrotate':
- ensure => 'link',
- target => '/etc/cron.daily/logrotate',
+ ensure => 'link',
+ target => '/etc/cron.daily/logrotate',
+ require => File['/etc/cron.hourly'],
}
}
diff --git a/puppet/modules/elasticsearch/manifests/plugin.pp
b/puppet/modules/elasticsearch/manifests/plugin.pp
index 888eb54..f14d68a 100644
--- a/puppet/modules/elasticsearch/manifests/plugin.pp
+++ b/puppet/modules/elasticsearch/manifests/plugin.pp
@@ -42,10 +42,10 @@
# Core plugins are part of elastic realease process thus no additional
# information should be provided. External plugins (such as those released
# by wikimedia) provide the group and artifact ids. the
mwv-elasticsearch-plugin
- # script will add the appropriate version.
+ # script will convert this into a url to request from maven central
$plugin_identifier = $core ? {
true => '',
- false => "${group}/${title}"
+ false => "${group} ${title}"
}
case $ensure {
diff --git a/puppet/modules/elasticsearch/templates/logging.yml.erb
b/puppet/modules/elasticsearch/templates/logging.yml.erb
deleted file mode 100644
index 3cd8631..0000000
--- a/puppet/modules/elasticsearch/templates/logging.yml.erb
+++ /dev/null
@@ -1,67 +0,0 @@
-rootLogger: INFO, console, file
-logger:
- # log action execution errors for easier debugging
- action: DEBUG
-
- # This is noisy and already filed upstream:
- # https://github.com/elasticsearch/elasticsearch/issues/4203
- action.admin.cluster.node.stats: INFO
-
- # If you need to know more about shard allocation you to set this to debug.
- # Trace seems to generate enough logs to slow down the process.
- # cluster: DEBUG
-
- # reduce the logging for aws, too much is logged under the default INFO
- com.amazonaws: WARN
- common.jna: DEBUG
-
- # gateway
- #gateway: DEBUG
- #index.gateway: DEBUG
-
- # peer shard recovery
- #indices.recovery: DEBUG
-
- # discovery
- #discovery: TRACE
-
- index.search.slowlog: TRACE, index_search_slow_log_file
- index.indexing.slowlog: INFO, index_indexing_slow_log_file
-
-additivity:
- index.search.slowlog: false
- index.indexing.slowlog: false
-
-appender:
- console:
- type: console
- layout:
- type: consolePattern
- conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
-
- file:
- type: file
- file: ${path.logs}/${cluster.name}.log
- encoding: UTF-8
- append: true
- layout:
- type: pattern
- conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
-
- index_search_slow_log_file:
- type: file
- file: ${path.logs}/${cluster.name}_index_search_slowlog.log
- encoding: UTF-8
- append: true
- layout:
- type: pattern
- conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
-
- index_indexing_slow_log_file:
- type: file
- file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
- encoding: UTF-8
- append: true
- layout:
- type: pattern
- conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
diff --git
a/puppet/modules/elasticsearch/templates/wikimedia-experimental.list.erb
b/puppet/modules/elasticsearch/templates/wikimedia-experimental.list.erb
new file mode 100644
index 0000000..1eafcda
--- /dev/null
+++ b/puppet/modules/elasticsearch/templates/wikimedia-experimental.list.erb
@@ -0,0 +1,2 @@
+deb http://apt.wikimedia.org/wikimedia <%= scope['::lsbdistcodename']
%>-wikimedia experimental
+deb-src http://apt.wikimedia.org/wikimedia <%= scope['::lsbdistcodename']
%>-wikimedia experimental
diff --git a/puppet/modules/kibana/manifests/dashboard.pp
b/puppet/modules/kibana/manifests/dashboard.pp
index 74b321b..b2d1245 100644
--- a/puppet/modules/kibana/manifests/dashboard.pp
+++ b/puppet/modules/kibana/manifests/dashboard.pp
@@ -16,5 +16,6 @@
exec { "save dashboard ${title}":
command => template('kibana/save-dashboard.erb'),
unless => template('kibana/check-dashboard.erb'),
+ require => Service['elasticsearch'],
}
}
diff --git a/puppet/modules/role/manifests/cirrussearch.pp
b/puppet/modules/role/manifests/cirrussearch.pp
index c008e90..2f6a621 100644
--- a/puppet/modules/role/manifests/cirrussearch.pp
+++ b/puppet/modules/role/manifests/cirrussearch.pp
@@ -31,19 +31,16 @@
# Not enabled on production wikis
elasticsearch::plugin { 'analysis-kuromoji':
ensure => absent,
- core => true,
}
# Not enabled on production wikis
elasticsearch::plugin { 'analysis-stempel':
ensure => absent,
- core => true,
}
# Not enabled on production wikis
elasticsearch::plugin { 'analysis-smartcn':
ensure => absent,
- core => true,
}
elasticsearch::plugin { 'elasticsearch-analysis-hebrew':
# Less stable then icu plugin
@@ -63,6 +60,7 @@
mediawiki::extension { 'Elastica':
composer => true,
+ branch => 'es5',
}
mediawiki::extension { 'CirrusSearch':
@@ -72,6 +70,7 @@
Class['eventschemas'],
],
browser_tests => 'tests/browser',
+ branch => 'es5',
}
mediawiki::settings { 'cirrustest:cirrussearch test suite':
--
To view, visit https://gerrit.wikimedia.org/r/341354
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic8cf196d66edeed7964887cd3a02e9da2c7fe366
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/vagrant
Gerrit-Branch: master
Gerrit-Owner: EBernhardson <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
