[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Change should be threadsafe.

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/406019 )

Change subject: Change should be threadsafe.
..

Change should be threadsafe.

Changes are processed by an executor service, they should be threadsafe.

Change-Id: Iaddac7894a15c5bd0cec492607b349886e04ba25
---
M tools/src/main/java/org/wikidata/query/rdf/tool/change/Change.java
1 file changed, 7 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/19/406019/1

diff --git a/tools/src/main/java/org/wikidata/query/rdf/tool/change/Change.java 
b/tools/src/main/java/org/wikidata/query/rdf/tool/change/Change.java
index ce48b99..5e846ac 100644
--- a/tools/src/main/java/org/wikidata/query/rdf/tool/change/Change.java
+++ b/tools/src/main/java/org/wikidata/query/rdf/tool/change/Change.java
@@ -1,6 +1,7 @@
 package org.wikidata.query.rdf.tool.change;
 
 import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Collections.unmodifiableCollection;
 
 import java.util.Collection;
 import java.util.Date;
@@ -245,7 +246,7 @@
  * Set statements collection.
  * @return
  */
-public Collection getStatements() {
+public synchronized Collection getStatements() {
 return statements;
 }
 
@@ -253,15 +254,15 @@
  * Return statements collection.
  * @return
  */
-public void setStatements(Collection statements) {
-this.statements = statements;
+public synchronized void setStatements(Collection statements) {
+this.statements = unmodifiableCollection(statements);
 }
 
 /**
  * Set cleanup list.
  * @return
  */
-public Collection getCleanupList() {
+public synchronized Collection getCleanupList() {
 return cleanupList;
 }
 
@@ -269,7 +270,7 @@
  * Return cleanup list.
  * @param cleanupList
  */
-public void setCleanupList(Collection cleanupList) {
-this.cleanupList = cleanupList;
+public synchronized void setCleanupList(Collection cleanupList) {
+this.cleanupList = unmodifiableCollection(cleanupList);
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/406019
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaddac7894a15c5bd0cec492607b349886e04ba25
Gerrit-PatchSet: 1
Gerrit-Project: wikidata/query/rdf
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Disable jolokia agent.

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405907 )

Change subject: Disable jolokia agent.
..

Disable jolokia agent.

Now that we collect metrics directly with prometheus jmx exporter, having
jolokia is not needed.

Bug: T182773
Depends-On: I8c5a5dfdeff332f90e29a19d21bb32bf67e0de26
Change-Id: I6a2951d2b52a0e245672e3e36169ddd24b22926b
---
M dist/src/script/runUpdate.sh
1 file changed, 1 insertion(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/07/405907/1

diff --git a/dist/src/script/runUpdate.sh b/dist/src/script/runUpdate.sh
index 7b302f2..737e87a 100755
--- a/dist/src/script/runUpdate.sh
+++ b/dist/src/script/runUpdate.sh
@@ -70,8 +70,7 @@
 CP=lib/wikidata-query-tools-*-jar-with-dependencies.jar
 MAIN=org.wikidata.query.rdf.tool.Update
 SPARQL_URL=$HOST/$CONTEXT/namespace/$NAMESPACE/sparql
-AGENT=-javaagent:lib/jolokia-jvm-1.3.7-agent.jar=port=8778,host=localhost,discoveryEnabled=false
 echo "Updating via $SPARQL_URL"
 exec java -cp ${CP} ${MEMORY} ${GC_LOGS} ${LOG_OPTIONS} ${EXTRA_JVM_OPTS} \
- ${TIMEOUT_ARG} ${AGENT} ${UPDATER_OPTS} \
+ ${TIMEOUT_ARG} ${UPDATER_OPTS} \
  ${MAIN} ${ARGS} --sparqlUrl ${SPARQL_URL} "$@"

-- 
To view, visit https://gerrit.wikimedia.org/r/405907
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6a2951d2b52a0e245672e3e36169ddd24b22926b
Gerrit-PatchSet: 1
Gerrit-Project: wikidata/query/rdf
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: replace prometheus-wdqs-updater-exporter with promethe...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405887 )

Change subject: wdqs: replace prometheus-wdqs-updater-exporter with 
prometheus-jmx-exporter
..

wdqs: replace prometheus-wdqs-updater-exporter with prometheus-jmx-exporter

Bug: T182773
Change-Id: I8c5a5dfdeff332f90e29a19d21bb32bf67e0de26
---
A modules/profile/files/wdqs/wdqs-updater-prometheus-jmx.yaml
D modules/profile/manifests/prometheus/wdqs_updater_exporter.pp
M modules/profile/manifests/wdqs.pp
M modules/wdqs/manifests/updater.pp
M modules/wdqs/templates/updater-default.erb
5 files changed, 33 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/87/405887/1

diff --git a/modules/profile/files/wdqs/wdqs-updater-prometheus-jmx.yaml 
b/modules/profile/files/wdqs/wdqs-updater-prometheus-jmx.yaml
new file mode 100644
index 000..19da75d
--- /dev/null
+++ b/modules/profile/files/wdqs/wdqs-updater-prometheus-jmx.yaml
@@ -0,0 +1,6 @@
+---
+lowercaseOutputLabelNames: true
+lowercaseOutputName: false
+startDelaySeconds: 10
+whitelistObjectNames:
+  - 'metrics:name=*'
diff --git a/modules/profile/manifests/prometheus/wdqs_updater_exporter.pp 
b/modules/profile/manifests/prometheus/wdqs_updater_exporter.pp
deleted file mode 100644
index ed2d5ae..000
--- a/modules/profile/manifests/prometheus/wdqs_updater_exporter.pp
+++ /dev/null
@@ -1,18 +0,0 @@
-class profile::prometheus::wdqs_updater_exporter (
-$prometheus_nodes = hiera('prometheus_nodes'),
-) {
-$prometheus_ferm_nodes = join($prometheus_nodes, ' ')
-$ferm_srange = "(@resolve((${prometheus_ferm_nodes})) 
@resolve((${prometheus_ferm_nodes}), ))"
-
-require_package('prometheus-wdqs-updater-exporter')
-
-service { 'prometheus-wdqs-updater-exporter':
-ensure  => running,
-}
-
-ferm::service { 'prometheus-wdqs-updater-exporter':
-proto  => 'tcp',
-port   => '9194',
-srange => $ferm_srange,
-}
-}
diff --git a/modules/profile/manifests/wdqs.pp 
b/modules/profile/manifests/wdqs.pp
index b9af7e4..299d7a6 100644
--- a/modules/profile/manifests/wdqs.pp
+++ b/modules/profile/manifests/wdqs.pp
@@ -9,8 +9,8 @@
 $blazegraph_config_file = hiera('profile::wdqs::blazegraph_config_file'),
 $updater_options = hiera('profile::wdqs::updater_options'),
 $nodes = hiera('profile::wdqs::nodes'),
+$prometheus_nodes = hiera('prometheus_nodes'),
 ) {
-require ::profile::prometheus::wdqs_updater_exporter
 require ::profile::prometheus::blazegraph_exporter
 
 $nagios_contact_group = 'admins,wdqs-admins'
@@ -27,10 +27,30 @@
 logstash_host  => $logstash_host,
 }
 
+$prometheus_agent_path = 
'/usr/share/java/prometheus/jmx_prometheus_javaagent.jar'
+$prometheus_agent_port = '9101'
+$prometheus_agent_config = '/etc/wdqs/wdqs-updater-prometheus-jmx.yaml'
+
+# TODO: this is for cleanup of the previous wdqs-updater prometheus 
exproter
+# it will be removed in a future commit
+package { 'prometheus-wdqs-updater-exporter':
+ensure => absent,
+}
+
 # WDQS Updater service
+profile::prometheus::jmx_exporter { 'wdqs-updater':
+hostname => $::hostname,
+port => $prometheus_agent_port,
+prometheus_nodes => $prometheus_nodes,
+config_file => $prometheus_agent_config,
+content => 
'puppet:///modules/profile/wdqs/wdqs-updater-prometheus-jmx.yaml',
+}
+
 class { 'wdqs::updater':
-options   => $updater_options,
-logstash_host => $logstash_host,
+options=> $updater_options,
+logstash_host  => $logstash_host,
+extra_jvm_opts => 
"-javaagent:${prometheus_agent_path}=${prometheus_agent_port}:${prometheus_agent_config}",
+require => Profile::Prometheus::Jmx_exporter['wdqs-updater'],
 }
 
 # Service Web proxy
diff --git a/modules/wdqs/manifests/updater.pp 
b/modules/wdqs/manifests/updater.pp
index 1382db3..2d38fac 100644
--- a/modules/wdqs/manifests/updater.pp
+++ b/modules/wdqs/manifests/updater.pp
@@ -13,6 +13,7 @@
 $package_dir = $::wdqs::package_dir,
 $username = $::wdqs::username,
 $data_dir = $::wdqs::data_dir,
+$extra_jvm_opts = undef,
 ){
 file { '/etc/default/wdqs-updater':
 ensure  => present,
diff --git a/modules/wdqs/templates/updater-default.erb 
b/modules/wdqs/templates/updater-default.erb
index 76410c8..0f4752c 100644
--- a/modules/wdqs/templates/updater-default.erb
+++ b/modules/wdqs/templates/updater-default.erb
@@ -1 +1,4 @@
 LOG_CONFIG=/etc/wdqs/logback-wdqs-updater.xml
+<%- if @extra_jvm_opts -%>
+EXTRA_JVM_OPTS=<%= @extra_jvm_opts %>
+<%- end -%>

-- 
To view, visit https://gerrit.wikimedia.org/r/405887
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I8c5a5dfdeff332f90e29a19d21bb32bf67e0de26
Gerrit-PatchSet: 1

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: remove cleanup code after migrating to prometheus jmx ...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405888 )

Change subject: wdqs: remove cleanup code after migrating to prometheus jmx 
exporter
..

wdqs: remove cleanup code after migrating to prometheus jmx exporter

Bug: T182773
Change-Id: I4dc83dfd6c34c7921ab792fb71f00b6b62cb010d
---
M modules/profile/manifests/wdqs.pp
1 file changed, 0 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/88/405888/1

diff --git a/modules/profile/manifests/wdqs.pp 
b/modules/profile/manifests/wdqs.pp
index 299d7a6..768fe74 100644
--- a/modules/profile/manifests/wdqs.pp
+++ b/modules/profile/manifests/wdqs.pp
@@ -31,12 +31,6 @@
 $prometheus_agent_port = '9101'
 $prometheus_agent_config = '/etc/wdqs/wdqs-updater-prometheus-jmx.yaml'
 
-# TODO: this is for cleanup of the previous wdqs-updater prometheus 
exproter
-# it will be removed in a future commit
-package { 'prometheus-wdqs-updater-exporter':
-ensure => absent,
-}
-
 # WDQS Updater service
 profile::prometheus::jmx_exporter { 'wdqs-updater':
 hostname => $::hostname,

-- 
To view, visit https://gerrit.wikimedia.org/r/405888
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4dc83dfd6c34c7921ab792fb71f00b6b62cb010d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-blazegraph-exporter[master]: Metrics are exposed by Blazegraph directly

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405878 )

Change subject: Metrics are exposed by Blazegraph directly
..

Metrics are exposed by Blazegraph directly

While the code references Jolokia, the metrics we collect are exposed
directly by Blazegraph. Naming is now corrected and less misleading.

Bug: T182857
Change-Id: Iae70f97875f8c435f6eaf5a3c4c11479df8671bc
---
M prometheus-blazegraph-exporter
1 file changed, 3 insertions(+), 4 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-blazegraph-exporter 
refs/changes/78/405878/1

diff --git a/prometheus-blazegraph-exporter b/prometheus-blazegraph-exporter
index a75ffa2..b7de957 100755
--- a/prometheus-blazegraph-exporter
+++ b/prometheus-blazegraph-exporter
@@ -76,7 +76,7 @@
 
 @scrape_duration.time()
 def collect(self):
-jolokia_metrics = {
+blazegraph_metrics = {
 '/Query Engine/queryStartCount': 
CounterMetricFamily('blazegraph_queries_start', 'Number of queries that have 
started since the start of the application.'),
 '/Query Engine/queryDoneCount': 
CounterMetricFamily('blazegraph_queries_done', 'Number of queries completed 
since the start of the application.'),
 '/Query Engine/queryErrorCount': 
CounterMetricFamily('blazegraph_queries_error', 'Number of queries in error 
since the start of the application.'),
@@ -98,12 +98,11 @@
 '/JVM/Memory/Garbage Collectors/G1 Young Generation/Cumulative 
Collection Time': 
GaugeMetricFamily('blazegraph_jvm_memory_gc_g1_young_cumulative_collection_time',
 'Total time spent in young GC (seconds).'),
 }
 
-for metric_name in jolokia_metrics:
+for metric_name, metric_family in blazegraph_metrics.iteritems():
 if metric_name is None:
 log.warn('Unknown metric %r', metric_name)
 else:
 metric_value = self.get_counter(metric_name)
-metric_family = jolokia_metrics.get(metric_name)
 
 try:
 value = float(metric_value)
@@ -141,7 +140,7 @@
 yield triple_metric
 yield lag_metric
 
-for metric in jolokia_metrics.values():
+for metric in blazegraph_metrics.values():
 yield metric
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/405878
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iae70f97875f8c435f6eaf5a3c4c11479df8671bc
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-blazegraph-exporter
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Align GC parameters with puppet.

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405723 )

Change subject: Align GC parameters with puppet.
..

Align GC parameters with puppet.

After investigation, we have tuned GC via puppet. We should copy those
changes to the default in the startup script so that other may benefit from
them.

Bug: T175919
Change-Id: Ic2fc199e62ae1f4e9f40564f7db8614d56c58d66
---
M dist/src/script/runBlazegraph.sh
1 file changed, 4 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/23/405723/1

diff --git a/dist/src/script/runBlazegraph.sh b/dist/src/script/runBlazegraph.sh
index 625e7b2..c1e1e2a 100755
--- a/dist/src/script/runBlazegraph.sh
+++ b/dist/src/script/runBlazegraph.sh
@@ -12,7 +12,7 @@
 LOG_CONFIG=${LOG_CONFIG:-""}
 LOG_DIR=${LOG_DIR:-"/var/log/wdqs"}
 MEMORY=${MEMORY:-"-Xmx${HEAP_SIZE}"}
-GC_LOGS=${GC_LOGS:-"-Xloggc:${LOG_DIR}/wdqs-blazegraph_jvm_gc.%p.log \
+GC_LOGS=${GC_LOGS:-"-Xloggc:${LOG_DIR}/wdqs-blazegraph_jvm_gc.%p-%t.log \
  -XX:+PrintGCDetails \
  -XX:+PrintGCDateStamps \
  -XX:+PrintGCTimeStamps \
@@ -21,6 +21,9 @@
  -XX:+PrintGCCause \
  -XX:+PrintGCApplicationStoppedTime \
  -XX:+PrintTenuringDistribution \
+ -XX:+UnlockExperimentalVMOptions \
+ -XX:G1NewSizePercent=20 \
+ -XX:+ParallelRefProcEnabled \
  -XX:+UseGCLogFileRotation \
  -XX:NumberOfGCLogFiles=10 \
  -XX:GCLogFileSize=20M"}

-- 
To view, visit https://gerrit.wikimedia.org/r/405723
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic2fc199e62ae1f4e9f40564f7db8614d56c58d66
Gerrit-PatchSet: 1
Gerrit-Project: wikidata/query/rdf
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Updates to enable short URLs for transliteration for...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/405048 )

Change subject: Revert "Updates to enable short URLs for transliteration for 
crhwiki - beta"
..


Revert "Updates to enable short URLs for transliteration for crhwiki - beta"

The deployment of the transliteration mediawiki-config in today's SWAT
did not work, so it's not clear whether this apache config is correct or
not. Best to back it out until we know what's going on, so that if this
approach is not correct, we don't forget to undo it.

This reverts commit 8872f00b2e57fa305209abf5c6093c32b8e94f64.

Bug: T23582
Change-Id: I292257d33196570d4c2407fbea74647cf0ea5bad
---
M modules/mediawiki/files/apache/beta/sites/wikibooks.conf
M modules/mediawiki/files/apache/beta/sites/wikinews.conf
M modules/mediawiki/files/apache/beta/sites/wikipedia.conf
M modules/mediawiki/files/apache/beta/sites/wikiquote.conf
M modules/mediawiki/files/apache/beta/sites/wikisource.conf
M modules/mediawiki/files/apache/beta/sites/wikiversity.conf
M modules/mediawiki/files/apache/beta/sites/wiktionary.conf
7 files changed, 0 insertions(+), 21 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/mediawiki/files/apache/beta/sites/wikibooks.conf 
b/modules/mediawiki/files/apache/beta/sites/wikibooks.conf
index a1dac61..38bc582 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikibooks.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikibooks.conf
@@ -20,9 +20,6 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
-Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
-Alias /crh-latn /srv/mediawiki/docroot/standard-docroot/w/index.php
-Alias /crh-cyrl /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/standard-docroot/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikinews.conf 
b/modules/mediawiki/files/apache/beta/sites/wikinews.conf
index fad3a11..cdf5bd9 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikinews.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikinews.conf
@@ -20,9 +20,6 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
-Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
-Alias /crh-latn /srv/mediawiki/docroot/standard-docroot/w/index.php
-Alias /crh-cyrl /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/standard-docroot/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikipedia.conf 
b/modules/mediawiki/files/apache/beta/sites/wikipedia.conf
index 7752127..c005ccf 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikipedia.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikipedia.conf
@@ -24,9 +24,6 @@
 RewriteRule ^(.*)$ http://commons.wikimedia.beta.wmflabs.org$1 [R=301,L,NE]
 
 # Variant aliases
-Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
-Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
-Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikipedia.org/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikiquote.conf 
b/modules/mediawiki/files/apache/beta/sites/wikiquote.conf
index 33ae5cc..53d87f0 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikiquote.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikiquote.conf
@@ -20,9 +20,6 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
-Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
-Alias /crh-latn /srv/mediawiki/docroot/standard-docroot/w/index.php
-Alias /crh-cyrl /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/standard-docroot/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikisource.conf 
b/modules/mediawiki/files/apache/beta/sites/wikisource.conf
index 271f8e8..862b477 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikisource.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikisource.conf
@@ -20,9 +20,6 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
-Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
-Alias /crh-latn 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mediawiki: remove unused logging configuration of mediawiki

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404980 )

Change subject: mediawiki: remove unused logging configuration of mediawiki
..


mediawiki: remove unused logging configuration of mediawiki

The definition of the logstash host is overwritten in
`profile::mediawiki::common`, so this file can now be removed.

Bug: T182304
Change-Id: Iba48fb5266e0b133ca1fedbecfa0b5805ad4
---
D hieradata/common/mediawiki.yaml
1 file changed, 0 insertions(+), 1 deletion(-)

Approvals:
  Andrew Bogott: Looks good to me, but someone else must approve
  BryanDavis: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/common/mediawiki.yaml b/hieradata/common/mediawiki.yaml
deleted file mode 100644
index 5b303e8..000
--- a/hieradata/common/mediawiki.yaml
+++ /dev/null
@@ -1 +0,0 @@
-mediawiki::forward_syslog: "logstash.svc.eqiad.wmnet:10514"

-- 
To view, visit https://gerrit.wikimedia.org/r/404980
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iba48fb5266e0b133ca1fedbecfa0b5805ad4
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Andrew Bogott 
Gerrit-Reviewer: BryanDavis 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mediawiki: remove unused logging configuration of mediawiki

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404980 )

Change subject: mediawiki: remove unused logging configuration of mediawiki
..

mediawiki: remove unused logging configuration of mediawiki

The definition of the logstash host is overwritten in
`profile::mediawiki::common`, so this file can now be removed.

Bug: T182304
Change-Id: Iba48fb5266e0b133ca1fedbecfa0b5805ad4
---
D hieradata/common/mediawiki.yaml
1 file changed, 0 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/80/404980/1

diff --git a/hieradata/common/mediawiki.yaml b/hieradata/common/mediawiki.yaml
deleted file mode 100644
index 5b303e8..000
--- a/hieradata/common/mediawiki.yaml
+++ /dev/null
@@ -1 +0,0 @@
-mediawiki::forward_syslog: "logstash.svc.eqiad.wmnet:10514"

-- 
To view, visit https://gerrit.wikimedia.org/r/404980
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iba48fb5266e0b133ca1fedbecfa0b5805ad4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...package[master]: Update the service-runner dependency and the NodeJS version ...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398278 )

Change subject: Update the service-runner dependency and the NodeJS version 
used in prod
..


Update the service-runner dependency and the NodeJS version used in prod

Change-Id: I2188d5cd8f7976d12db8e35518dc0015c9d61cc4
---
M package.json
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Gehel: Verified; Looks good to me, approved



diff --git a/package.json b/package.json
index 5721b7b..5e0ad53 100644
--- a/package.json
+++ b/package.json
@@ -52,7 +52,7 @@
   },
   "dependencies": {
 "kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#update-snapshot-dep;,
-"service-runner": "^2.4.6"
+"service-runner": "^2.4.8"
   },
   "optionalDependencies": {
 "bunyan-prettystream": "*"
@@ -66,7 +66,7 @@
 "swagger-router": "^0.4.2"
   },
   "deploy": {
-"node": "6.9.1",
+"node": "6.11.1",
 "target": "debian",
 "install_opts": [
   "--build-from-source=mapnik",

-- 
To view, visit https://gerrit.wikimedia.org/r/398278
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2188d5cd8f7976d12db8e35518dc0015c9d61cc4
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/package
Gerrit-Branch: master
Gerrit-Owner: Mobrovac 
Gerrit-Reviewer: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: RdfRepository uses Apache HC instead of Jetty HTTP client

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402753 )

Change subject: RdfRepository uses Apache HC instead of Jetty HTTP client
..

RdfRepository uses Apache HC instead of Jetty HTTP client

This removes a dependency, and provides a (IMHO) simpler HTTP client
library. This might help our issue with leaked threads in tests.

Change-Id: I2ea367e3f7c77338d487793fe2b9982b8538b017
---
M tools/pom.xml
M tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
M tools/src/main/java/org/wikidata/query/rdf/tool/Update.java
A 
tools/src/main/java/org/wikidata/query/rdf/tool/http/OkOnlyResponseHandler.java
M tools/src/main/java/org/wikidata/query/rdf/tool/rdf/RdfRepository.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
M tools/src/test/java/org/wikidata/query/rdf/tool/IOBlastingIntegrationTest.java
M tools/src/test/java/org/wikidata/query/rdf/tool/MungeIntegrationTest.java
M tools/src/test/java/org/wikidata/query/rdf/tool/RdfRepositoryForTesting.java
A 
tools/src/test/java/org/wikidata/query/rdf/tool/http/OkOnlyResponseHandlerTest.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/rdf/RdfRepositoryIntegrationTest.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryIntegrationTest.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryWireIntegrationTest.java
13 files changed, 323 insertions(+), 274 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/53/402753/8

diff --git a/tools/pom.xml b/tools/pom.xml
index f76c6b7..d18ff86 100644
--- a/tools/pom.xml
+++ b/tools/pom.xml
@@ -39,6 +39,10 @@
 
 
 com.fasterxml.jackson.core
+jackson-core
+
+
+com.fasterxml.jackson.core
 jackson-databind
 
 
@@ -81,18 +85,6 @@
 
 org.apache.httpcomponents
 httpcore
-
-
-org.eclipse.jetty
-jetty-client
-
-
-org.eclipse.jetty
-jetty-http
-
-
-org.eclipse.jetty
-jetty-util
 
 
 org.openrdf.sesame
diff --git 
a/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java 
b/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
index eaec48d..dbb7fe9 100644
--- a/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
+++ b/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
@@ -1,11 +1,18 @@
 package org.wikidata.query.rdf.tool;
 
+import static java.lang.Integer.parseInt;
+
 import java.io.InterruptedIOException;
 import java.net.UnknownHostException;
+import java.security.KeyManagementException;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
 
 import javax.net.ssl.SSLException;
 
 import org.apache.http.HttpEntityEnclosingRequest;
+import org.apache.http.HttpException;
+import org.apache.http.HttpHost;
 import org.apache.http.HttpRequest;
 import org.apache.http.HttpResponse;
 import org.apache.http.HttpStatus;
@@ -16,32 +23,41 @@
 import org.apache.http.client.methods.HttpRequestBase;
 import org.apache.http.client.protocol.HttpClientContext;
 import org.apache.http.conn.ConnectTimeoutException;
+import org.apache.http.conn.routing.HttpRoute;
+import org.apache.http.conn.routing.HttpRoutePlanner;
+import org.apache.http.conn.socket.LayeredConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
 import org.apache.http.impl.client.CloseableHttpClient;
 import org.apache.http.impl.client.DefaultServiceUnavailableRetryStrategy;
 import org.apache.http.impl.client.HttpClients;
+import org.apache.http.impl.conn.DefaultProxyRoutePlanner;
 import org.apache.http.protocol.HttpContext;
+import org.apache.http.ssl.SSLContextBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
  * Utilities for dealing with HttpClient.
  */
+@SuppressWarnings("checkstyle:classfanoutcomplexity") // dependencies are 
mostly about HC configuration
 public final class HttpClientUtils {
 
 private static final Logger log = 
LoggerFactory.getLogger(HttpClientUtils.class);
 
-/**
- * How many retries allowed on error.
- */
+/** Configuration name for proxy host. */
+public static final String HTTP_PROXY = "http.proxyHost";
+
+/** Configuration name for proxy port. */
+public static final String HTTP_PROXY_PORT = "http.proxyPort";
+
+/** How many retries allowed on error. */
 public static final int RETRIES = 3;
-/**
- * Retry interval, in ms.
- */
+
+/** Retry interval, in ms. */
 public static final int RETRY_INTERVAL = 500;
 
-/**
- * Configure request to ignore 

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Extract collaborators of WikibaseRepository

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402395 )

Change subject: Extract collaborators of WikibaseRepository
..

Extract collaborators of WikibaseRepository

This is a preliminary step that will allow sharing HttpClient between
WikibaseRepository and RdfRepository.

Change-Id: I2bb98d5b1c9ff8996a219f398641e9b228fc85aa
---
M tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
M tools/src/main/java/org/wikidata/query/rdf/tool/Update.java
M 
tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryIntegrationTest.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryWireIntegrationTest.java
6 files changed, 180 insertions(+), 181 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/95/402395/5

diff --git 
a/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java 
b/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
index 805a379..eaec48d 100644
--- a/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
+++ b/tools/src/main/java/org/wikidata/query/rdf/tool/HttpClientUtils.java
@@ -1,13 +1,44 @@
 package org.wikidata.query.rdf.tool;
 
+import java.io.InterruptedIOException;
+import java.net.UnknownHostException;
+
+import javax.net.ssl.SSLException;
+
+import org.apache.http.HttpEntityEnclosingRequest;
+import org.apache.http.HttpRequest;
+import org.apache.http.HttpResponse;
+import org.apache.http.HttpStatus;
+import org.apache.http.client.HttpRequestRetryHandler;
+import org.apache.http.client.ServiceUnavailableRetryStrategy;
 import org.apache.http.client.config.CookieSpecs;
 import org.apache.http.client.config.RequestConfig;
 import org.apache.http.client.methods.HttpRequestBase;
+import org.apache.http.client.protocol.HttpClientContext;
+import org.apache.http.conn.ConnectTimeoutException;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.DefaultServiceUnavailableRetryStrategy;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.protocol.HttpContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * Utilities for dealing with HttpClient.
  */
 public final class HttpClientUtils {
+
+private static final Logger log = 
LoggerFactory.getLogger(HttpClientUtils.class);
+
+/**
+ * How many retries allowed on error.
+ */
+public static final int RETRIES = 3;
+/**
+ * Retry interval, in ms.
+ */
+public static final int RETRY_INTERVAL = 500;
+
 /**
  * Configure request to ignore cookies.
  */
@@ -19,4 +50,82 @@
 private HttpClientUtils() {
 // Uncallable utility constructor
 }
+
+public static CloseableHttpClient createHttpClient() {
+return HttpClients.custom()
+.setMaxConnPerRoute(100).setMaxConnTotal(100)
+.setRetryHandler(getRetryHandler(RETRIES))
+.setServiceUnavailableRetryStrategy(getRetryStrategy(RETRIES, 
RETRY_INTERVAL))
+.disableCookieManagement()
+.setUserAgent("Wikidata Query Service Updater")
+.build();
+}
+
+/**
+ * Return retry strategy for "service unavailable".
+ * This one handles 503 and 429 by retrying it after a fixed period.
+ * TODO: 429 may contain header that we may want to use for retrying?
+ * @param max Maximum number of retries.
+ * @param interval Interval between retries, ms.
+ * @see DefaultServiceUnavailableRetryStrategy
+ * @return
+ */
+public static ServiceUnavailableRetryStrategy getRetryStrategy(final int 
max, final int interval) {
+// This is the same as DefaultServiceUnavailableRetryStrategy but also 
handles 429
+return new ServiceUnavailableRetryStrategy() {
+@Override
+public boolean retryRequest(final HttpResponse response, final int 
executionCount, final HttpContext context) {
+return executionCount <= max &&
+(response.getStatusLine().getStatusCode() == 
HttpStatus.SC_SERVICE_UNAVAILABLE ||
+response.getStatusLine().getStatusCode() == 429);
+}
+
+@Override
+public long getRetryInterval() {
+return interval;
+}
+};
+}
+
+/**
+ * Create retry handler.
+ * Note: this is for retrying I/O exceptions.
+ * @param max Maximum retries number.
+ * @return
+ */
+public static HttpRequestRetryHandler getRetryHandler(final int max) {
+return (exception, executionCount, context) -> {
+log.debug("Exception in attempt {}", executionCount, 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: simplify logging of categories reload

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404315 )

Change subject: wdqs: simplify logging of categories reload
..


wdqs: simplify logging of categories reload

The reload of categories was logged to multiple files, including one that
was timestamped. This generates unnecessary complexity.

My guess is that this multiple logging was done to also log manual reload
of categories, which was usefull during a testing phase where categories
were reloaded manually.

Change-Id: I58199e4c0d4397d2da8308187fb6b870fecf5f80
---
M modules/wdqs/files/cron/reloadCategories.sh
M modules/wdqs/manifests/gui.pp
2 files changed, 12 insertions(+), 17 deletions(-)

Approvals:
  Smalyshev: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/files/cron/reloadCategories.sh 
b/modules/wdqs/files/cron/reloadCategories.sh
index 2710a19..2f9dfe9 100755
--- a/modules/wdqs/files/cron/reloadCategories.sh
+++ b/modules/wdqs/files/cron/reloadCategories.sh
@@ -4,12 +4,17 @@
 # sudo systemctl reload nginx
 . /usr/local/bin/cronUtils.sh
 
+echo "$(date --iso-8601=seconds) starting categories reload"
+
 newNamespace="categories${today}"
 # Drop old dumps
 rm -f ${DATA_DIR}/*-categories.ttl.gz
-cd $DEPLOY_DIR
+cd ${DEPLOY_DIR}
 # Create new namespace
-bash createNamespace.sh $newNamespace || exit 1
+bash createNamespace.sh ${newNamespace} || exit 1
 # Load the data
-bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace >> 
"${LOG_DIR}/${newNamespace}.log"
-replaceNamespace categories $newNamespace
+echo "loading categories in ${newNamespace}"
+bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace
+replaceNamespace categories ${newNamespace}
+
+echo "$(date --iso-8601=seconds) categories reload done"
diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index d056705..7ff6dfa 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -83,7 +83,7 @@
 mode   => '0755',
 }
 
-$cron_log = "${log_dir}/reloadCategories.log"
+$reload_categories_log = "${log_dir}/reloadCategories.log"
 # the reload-categories cron needs to reload nginx once the categories are 
up to date
 sudo::user { "${username}-reload-nginx":
   ensure => present,
@@ -96,7 +96,7 @@
 # do not want them to be too far from one another.
 cron { 'reload-categories':
 ensure  => present,
-command => "/usr/local/bin/reloadCategories.sh >> ${cron_log}",
+command => "/usr/local/bin/reloadCategories.sh >> 
${reload_categories_log}",
 user=> $username,
 weekday => 1,
 minute  => fqdn_rand(60),
@@ -114,7 +114,7 @@
 
 logrotate::rule { 'wdqs-reload-categories':
 ensure   => present,
-file_glob=> $cron_log,
+file_glob=> $reload_categories_log,
 frequency=> 'monthly',
 missing_ok   => true,
 not_if_empty => true,
@@ -134,14 +134,4 @@
 su   => "${username} wikidev",
 }
 
-# Remove categories*.log files after 30 days.
-logrotate::rule { 'wdqs-categories-logs':
-ensure   => present,
-file_glob=> "${log_dir}/categories*.log",
-missing_ok   => true,
-not_if_empty => true,
-rotate   => 0,
-max_age  => 30,
-su   => "${username} wikidev",
-}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/404315
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I58199e4c0d4397d2da8308187fb6b870fecf5f80
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Smalyshev 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Updates to enable short URLs for transliteration for crhwiki...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/396283 )

Change subject: Updates to enable short URLs for transliteration for crhwiki - 
beta
..


Updates to enable short URLs for transliteration for crhwiki - beta

Add Apache rules for language variant paths for crh Wikipedia and other wikis.

Bug: T23582
Change-Id: I3b91f08cd665a7ede25e16be6e9fd8fffc75c904
---
M modules/mediawiki/files/apache/beta/sites/wikibooks.conf
M modules/mediawiki/files/apache/beta/sites/wikinews.conf
M modules/mediawiki/files/apache/beta/sites/wikipedia.conf
M modules/mediawiki/files/apache/beta/sites/wikiquote.conf
M modules/mediawiki/files/apache/beta/sites/wikisource.conf
M modules/mediawiki/files/apache/beta/sites/wikiversity.conf
M modules/mediawiki/files/apache/beta/sites/wiktionary.conf
7 files changed, 21 insertions(+), 0 deletions(-)

Approvals:
  Subramanya Sastry: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/mediawiki/files/apache/beta/sites/wikibooks.conf 
b/modules/mediawiki/files/apache/beta/sites/wikibooks.conf
index 38bc582..a1dac61 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikibooks.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikibooks.conf
@@ -20,6 +20,9 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/standard-docroot/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikinews.conf 
b/modules/mediawiki/files/apache/beta/sites/wikinews.conf
index cdf5bd9..fad3a11 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikinews.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikinews.conf
@@ -20,6 +20,9 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/standard-docroot/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikipedia.conf 
b/modules/mediawiki/files/apache/beta/sites/wikipedia.conf
index c005ccf..7752127 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikipedia.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikipedia.conf
@@ -24,6 +24,9 @@
 RewriteRule ^(.*)$ http://commons.wikimedia.beta.wmflabs.org$1 [R=301,L,NE]
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikipedia.org/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikiquote.conf 
b/modules/mediawiki/files/apache/beta/sites/wikiquote.conf
index 53d87f0..33ae5cc 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikiquote.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikiquote.conf
@@ -20,6 +20,9 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/standard-docroot/w/index.php
diff --git a/modules/mediawiki/files/apache/beta/sites/wikisource.conf 
b/modules/mediawiki/files/apache/beta/sites/wikisource.conf
index 862b477..271f8e8 100644
--- a/modules/mediawiki/files/apache/beta/sites/wikisource.conf
+++ b/modules/mediawiki/files/apache/beta/sites/wikisource.conf
@@ -20,6 +20,9 @@
 Include "sites-enabled/public-wiki-rewrites.incl"
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/standard-docroot/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr /srv/mediawiki/docroot/standard-docroot/w/index.php
 Alias /sr-ec 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: simplify logging of categories reload

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404315 )

Change subject: wdqs: simplify logging of categories reload
..

wdqs: simplify logging of categories reload

The reload of categories was logged to multiple files, including one that
was timestamped. This generates unnecessary complexity.

My guess is that this multiple logging was done to also log manual reload
of categories, which was usefull during a testing phase where categories
were reloaded manually.

Change-Id: I58199e4c0d4397d2da8308187fb6b870fecf5f80
---
M modules/wdqs/files/cron/reloadCategories.sh
M modules/wdqs/manifests/gui.pp
2 files changed, 8 insertions(+), 17 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/15/404315/1

diff --git a/modules/wdqs/files/cron/reloadCategories.sh 
b/modules/wdqs/files/cron/reloadCategories.sh
index 2710a19..338fa33 100755
--- a/modules/wdqs/files/cron/reloadCategories.sh
+++ b/modules/wdqs/files/cron/reloadCategories.sh
@@ -7,9 +7,10 @@
 newNamespace="categories${today}"
 # Drop old dumps
 rm -f ${DATA_DIR}/*-categories.ttl.gz
-cd $DEPLOY_DIR
+cd ${DEPLOY_DIR}
 # Create new namespace
-bash createNamespace.sh $newNamespace || exit 1
+bash createNamespace.sh ${newNamespace} || exit 1
 # Load the data
-bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace >> 
"${LOG_DIR}/${newNamespace}.log"
-replaceNamespace categories $newNamespace
+echo "loading categories in ${newNamespace}"
+bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace
+replaceNamespace categories ${newNamespace}
diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index d056705..7ff6dfa 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -83,7 +83,7 @@
 mode   => '0755',
 }
 
-$cron_log = "${log_dir}/reloadCategories.log"
+$reload_categories_log = "${log_dir}/reloadCategories.log"
 # the reload-categories cron needs to reload nginx once the categories are 
up to date
 sudo::user { "${username}-reload-nginx":
   ensure => present,
@@ -96,7 +96,7 @@
 # do not want them to be too far from one another.
 cron { 'reload-categories':
 ensure  => present,
-command => "/usr/local/bin/reloadCategories.sh >> ${cron_log}",
+command => "/usr/local/bin/reloadCategories.sh >> 
${reload_categories_log}",
 user=> $username,
 weekday => 1,
 minute  => fqdn_rand(60),
@@ -114,7 +114,7 @@
 
 logrotate::rule { 'wdqs-reload-categories':
 ensure   => present,
-file_glob=> $cron_log,
+file_glob=> $reload_categories_log,
 frequency=> 'monthly',
 missing_ok   => true,
 not_if_empty => true,
@@ -134,14 +134,4 @@
 su   => "${username} wikidev",
 }
 
-# Remove categories*.log files after 30 days.
-logrotate::rule { 'wdqs-categories-logs':
-ensure   => present,
-file_glob=> "${log_dir}/categories*.log",
-missing_ok   => true,
-not_if_empty => true,
-rotate   => 0,
-max_age  => 30,
-su   => "${username} wikidev",
-}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/404315
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I58199e4c0d4397d2da8308187fb6b870fecf5f80
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-blazegraph-exporter[master]: prometheus blazegraph exporter should not fail when blazegra...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403933 )

Change subject: prometheus blazegraph exporter should not fail when blazegraph 
is down
..


prometheus blazegraph exporter should not fail when blazegraph is down

prometheus client collects all metrics on startup to enumerate the metric
names. If blazegraph isn't available at that time, it crashes. Note that
the same issue occurs if blazegraph goes down while the exporter is running.

Arguably, the prometheus python client itself should be more robust. But we
can already make metric collection more robust by managing at least the most
obvious exceptions.

Bug: T184434
Change-Id: I8bfd360b2ca864e8c21ec8b03bde58b16a2839ae
---
M prometheus-blazegraph-exporter
1 file changed, 30 insertions(+), 21 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/prometheus-blazegraph-exporter b/prometheus-blazegraph-exporter
index 9f9a23e..a75ffa2 100755
--- a/prometheus-blazegraph-exporter
+++ b/prometheus-blazegraph-exporter
@@ -27,6 +27,7 @@
 
 from datetime import timedelta, tzinfo
 from dateutil.parser import parse
+from urllib2 import URLError
 from xml.etree import ElementTree
 
 from prometheus_client import start_http_server, Summary
@@ -52,9 +53,13 @@
 url = self.url + "counters?depth=10&" + \
 urllib.urlencode({'path': cnt_name})
 
-req = urllib2.Request(url)
-req.add_header('Accept', 'application/xml')
-response = urllib2.urlopen(req)
+try:
+req = urllib2.Request(url)
+req.add_header('Accept', 'application/xml')
+response = urllib2.urlopen(req)
+except URLError:
+return None
+
 el = ElementTree.fromstring(response.read())
 last_name = cnt_name.split('/')[-1]
 
@@ -102,33 +107,37 @@
 
 try:
 value = float(metric_value)
-except ValueError:
+except (ValueError, TypeError):
 value = float('nan')
 
 metric_family.add_metric([], value)
 
-sparql_query = """ prefix schema: 
-SELECT * WHERE { {
-  SELECT ( COUNT( * ) AS ?count ) { ?s ?p ?o }
-} UNION {
-  SELECT * WHERE {  
schema:dateModified ?y }
-} }"""
-data = self.execute_sparql(sparql_query)
-
 triple_metric = CounterMetricFamily('blazegraph_triples', '')
 lag_metric = CounterMetricFamily('blazegraph_lastupdated', '')
+try:
+sparql_query = """ prefix schema: 
+SELECT * WHERE { {
+  SELECT ( COUNT( * ) AS ?count ) { ?s ?p ?o }
+} UNION {
+  SELECT * WHERE {  
schema:dateModified ?y }
+} }"""
 
-for binding in data['results']['bindings']:
-if 'count' in binding:
-triple_count = binding['count']['value']
-triple_metric.add_metric([], float(triple_count))
+data = self.execute_sparql(sparql_query)
 
-elif 'y' in binding:
-lastUpdated = parse(binding['y']['value'])
-lag_metric.add_metric([], float(lastUpdated.strftime('%s')))
-else:
-raise ValueError('SPARQL binding returned with unexpected key')
+for binding in data['results']['bindings']:
+if 'count' in binding:
+triple_count = binding['count']['value']
+triple_metric.add_metric([], float(triple_count))
 
+elif 'y' in binding:
+lastUpdated = parse(binding['y']['value'])
+lag_metric.add_metric([], 
float(lastUpdated.strftime('%s')))
+else:
+raise ValueError('SPARQL binding returned with unexpected 
key')
+
+except URLError:
+triple_metric.add_metric([], float('nan'))
+lag_metric.add_metric([], float('nan'))
 yield triple_metric
 yield lag_metric
 

-- 
To view, visit https://gerrit.wikimedia.org/r/403933
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8bfd360b2ca864e8c21ec8b03bde58b16a2839ae
Gerrit-PatchSet: 3
Gerrit-Project: operations/debs/prometheus-blazegraph-exporter
Gerrit-Branch: master
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

[MediaWiki-commits] [Gerrit] wikimedia...discovery-parent-pom[master]: Update dependencies and plugins to latest stable versions

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402362 )

Change subject: Update dependencies and plugins to latest stable versions
..

Update dependencies and plugins to latest stable versions

Change-Id: Id9806a02a54241b9cb813f0299c910797bdb8fd8
---
M pom.xml
1 file changed, 14 insertions(+), 14 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/wikimedia/discovery/discovery-parent-pom 
refs/changes/62/402362/1

diff --git a/pom.xml b/pom.xml
index 8934b17..8caf675 100644
--- a/pom.xml
+++ b/pom.xml
@@ -69,7 +69,7 @@
 UTF-8
 
UTF-8
 1.7.25
-3.1.0-RC7
+3.1.1
 
 
 
@@ -117,7 +117,7 @@
 
 org.assertj
 assertj-core
-3.8.0
+3.9.0
 test
 
 
@@ -298,7 +298,7 @@
 
 org.apache.maven.plugins
 maven-assembly-plugin
-3.0.0
+3.1.0
 
 
 org.apache.maven.plugins
@@ -315,7 +315,7 @@
 
 com.puppycrawl.tools
 checkstyle
-8.3
+8.7
 
 
 org.wikimedia.discovery
@@ -341,7 +341,7 @@
 
 org.apache.maven.plugins
 maven-compiler-plugin
-3.6.1
+3.7.0
 
 -Xlint:all
 
@@ -349,7 +349,7 @@
 
 org.apache.maven.plugins
 maven-dependency-plugin
-3.0.0
+3.0.2
 
 
 org.apache.maven.plugins
@@ -359,7 +359,7 @@
 
 org.apache.maven.plugins
 maven-enforcer-plugin
-1.4.1
+3.0.0-M1
 
 
 
@@ -417,7 +417,7 @@
 
 org.apache.maven.plugins
 maven-javadoc-plugin
-2.10.4
+3.0.0
 
 true
 -Xdoclint:none
@@ -455,7 +455,7 @@
 
 org.apache.maven.plugins
 maven-site-plugin
-3.6
+3.7
 
 
 org.apache.maven.plugins
@@ -465,12 +465,12 @@
 
 org.apache.maven.plugins
 maven-surefire-plugin
-2.20
+2.20.1
 
 
 org.apache.maven.plugins
 maven-war-plugin
-3.0.0
+3.2.0
 
 
 org.basepom.maven
@@ -504,7 +504,7 @@
 
 org.codehaus.mojo
 versions-maven-plugin
-2.4
+2.5
 
 
 org.jacoco
@@ -563,7 +563,7 @@
 
 pl.project13.maven
 git-commit-id-plugin
-2.2.3
+2.2.4
 
 -MM-dd'T'HH:mm:ssZ
 
true
@@ -649,7 +649,7 @@
 
 org.apache.maven.plugins
 maven-project-info-reports-plugin
-2.7
+2.9
 
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/402362
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id9806a02a54241b9cb813f0299c910797bdb8fd8
Gerrit-PatchSet: 1
Gerrit-Project: wikimedia/discovery/discovery-parent-pom
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add loading DCAT-AP data into dcatap namespace on WDQS

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399954 )

Change subject: Add loading DCAT-AP data into dcatap namespace on WDQS
..


Add loading DCAT-AP data into dcatap namespace on WDQS

Bug: T178978
Change-Id: Ie1d7eb7567d1409fb1116a71b23b4d5160aa2c7b
---
A modules/wdqs/files/cron/cronUtils.sh
M modules/wdqs/files/cron/reloadCategories.sh
A modules/wdqs/files/cron/reloadDCAT-AP.sh
M modules/wdqs/manifests/gui.pp
4 files changed, 118 insertions(+), 31 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/files/cron/cronUtils.sh 
b/modules/wdqs/files/cron/cronUtils.sh
new file mode 100755
index 000..4d2487a
--- /dev/null
+++ b/modules/wdqs/files/cron/cronUtils.sh
@@ -0,0 +1,55 @@
+#!/bin/bash
+# Cron helper functions
+if [ -r /etc/wdqs/vars.sh ]; then
+  . /etc/wdqs/vars.sh
+fi
+
+if [ -r /etc/wdqs/gui_vars.sh ]; then
+  . /etc/wdqs/gui_vars.sh
+fi
+
+if [ -z "${DATA_DIR}" -o -z "${LOG_DIR}" -o -z "${DEPLOY_DIR}" ]; then
+   echo "Variables not set up right!"
+   exit 1
+fi
+
+HOST="http://localhost:;
+NAMESPACE_URL="/bigdata/namespace/"
+today=$(date -u +'%Y%m%d')
+
+function loadFileIntoBlazegraph {
+   # source URL
+   local URL=$1
+   # local filename (will be in DATA_DIR)
+   local fileName=$2
+   local sparqlEndpoint=$3
+   curl -s -f -XGET $URL -o ${DATA_DIR}/${fileName}
+   if [ ! -s ${DATA_DIR}/${fileName} ]; then
+   echo "Could not download $URL into ${fileName}"
+   exit 1
+   fi  
+   curl -s -XPOST --data-binary update="LOAD " 
$sparqlEndpoint
+}
+
+# NOTE: This should be run under user that has rights to
+# sudo systemctl reload nginx
+function replaceNamespace {
+   local mainName=$1
+   local currentAlias=$2
+   local oldNamespace=$(cat $ALIAS_FILE | grep $mainName | cut -d' ' -f2 | 
cut -d ';' -f1)
+   if [ "${oldNamespace}" = ${currentAlias} ]; then
+   # nothing to do
+   return
+   fi
+   if [ -n "${oldNamespace}" ]; then
+   sed -i "/${mainName}/c ${mainName} ${currentAlias};" $ALIAS_FILE
+   else
+   echo "${mainName} ${currentAlias};" >> $ALIAS_FILE
+   fi
+   # Bump nginx to reload config
+   sudo systemctl reload nginx
+   if [ -n "${oldNamespace}" ]; then
+   # Drop old namespace
+   curl -s -X DELETE "${HOST}${NAMESPACE_URL}${oldNamespace}"
+   fi
+}
\ No newline at end of file
diff --git a/modules/wdqs/files/cron/reloadCategories.sh 
b/modules/wdqs/files/cron/reloadCategories.sh
index 33b83e9..2710a19 100755
--- a/modules/wdqs/files/cron/reloadCategories.sh
+++ b/modules/wdqs/files/cron/reloadCategories.sh
@@ -2,35 +2,14 @@
 # This script is reloading categories into a new namespace
 # NOTE: This should be run under user that has rights to
 # sudo systemctl reload nginx
-if [ -r /etc/wdqs/vars.sh ]; then
-  . /etc/wdqs/vars.sh
-fi
+. /usr/local/bin/cronUtils.sh
 
-if [ -r /etc/wdqs/gui_vars.sh ]; then
-  . /etc/wdqs/gui_vars.sh
-fi
-
-if [ -z "${DATA_DIR}" -o -z "${LOG_DIR}" -o -z "${DEPLOY_DIR}" ]; then
-   echo "Variables not set up right!"
-   exit 1
-fi
-
-today=$(date -u +'%Y%m%d')
 newNamespace="categories${today}"
 # Drop old dumps
 rm -f ${DATA_DIR}/*-categories.ttl.gz
 cd $DEPLOY_DIR
+# Create new namespace
 bash createNamespace.sh $newNamespace || exit 1
 # Load the data
 bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace >> 
"${LOG_DIR}/${newNamespace}.log"
-# Get old namespace
-oldNamespace=$(cat $ALIAS_FILE | grep categories | cut -d' ' -f2 | cut -d ';' 
-f1)
-# Switch the map
-# NOTE: right now it overrides the map. If we reuse it for other purposes, 
this needs to be made smarter.
-echo "categories ${newNamespace};" > $ALIAS_FILE
-# Bump nginx to reload config
-sudo systemctl reload nginx
-if [ -n "${oldNamespace}" ]; then
-   # Drop old namespace
-   curl -s -X DELETE 
"http://localhost:/bigdata/namespace/${oldNamespace};
-fi
+replaceNamespace categories $newNamespace
diff --git a/modules/wdqs/files/cron/reloadDCAT-AP.sh 
b/modules/wdqs/files/cron/reloadDCAT-AP.sh
new file mode 100755
index 000..618bddc
--- /dev/null
+++ b/modules/wdqs/files/cron/reloadDCAT-AP.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+# This script is reloading DCAT-AP data from Wikidata
+# NOTE: This should be run under user that has rights to
+# sudo systemctl reload nginx
+. /usr/local/bin/cronUtils.sh
+
+DCAT_SOURCE=${DCAT_SOURCE:-"https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf"}
+
+newNamespace="dcatap${today}"
+# Drop old dumps
+rm -f ${DATA_DIR}/dcatap-*.rdf
+cd $DEPLOY_DIR
+# Create new NS
+bash createNamespace.sh $newNamespace || exit 1
+# Load the data
+FILENAME=dcatap-${today}.rdf
+loadFileIntoBlazegraph $DCAT_SOURCE $FILENAME 
"${HOST}${NAMESPACE_URL}${newNamespace}/sparql"

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: auto reload log4j2 configuration

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/388130 )

Change subject: elasticsearch: auto reload log4j2 configuration
..


elasticsearch: auto reload log4j2 configuration

At the moment, we need a full cluster restart to reload simple changes in
logging configuration. Log4j2 has a mechanism to autoreload the
configuration with very minimal performance impact. Let's use it.

Change-Id: I94fa19375ba13ee4a041f3a3fe65628416b8139d
---
M modules/elasticsearch/templates/log4j2.properties.erb
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  DCausse: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/modules/elasticsearch/templates/log4j2.properties.erb 
b/modules/elasticsearch/templates/log4j2.properties.erb
index e079d12..7230945 100644
--- a/modules/elasticsearch/templates/log4j2.properties.erb
+++ b/modules/elasticsearch/templates/log4j2.properties.erb
@@ -1,4 +1,4 @@
-
+monitorInterval=600
 rootLogger.level = info
 rootLogger.appenderRef.file.ref = file
 <% if @send_logs_to_logstash %>

-- 
To view, visit https://gerrit.wikimedia.org/r/388130
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I94fa19375ba13ee4a041f3a3fe65628416b8139d
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: DCausse 
Gerrit-Reviewer: EBernhardson 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch / prometheus: enable prometheus jmx_exporter

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402095 )

Change subject: elasticsearch / prometheus: enable prometheus jmx_exporter
..

elasticsearch / prometheus: enable prometheus jmx_exporter

Bug: T181627
Change-Id: Ib2de51b15676c4607e2a4c6ab21e9f402f9751b6
---
M modules/elasticsearch/manifests/init.pp
M modules/elasticsearch/templates/jvm.options.erb
A 
modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.java.policy
A modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.yaml
M modules/profile/manifests/elasticsearch.pp
A modules/profile/manifests/prometheus/elasticsearch_jmx_exporter.pp
M modules/role/manifests/elasticsearch/beta.pp
M modules/role/manifests/elasticsearch/cirrus.pp
M modules/role/manifests/elasticsearch/relforge.pp
9 files changed, 56 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/95/402095/1

diff --git a/modules/elasticsearch/manifests/init.pp 
b/modules/elasticsearch/manifests/init.pp
index e711aa1..6119e36 100644
--- a/modules/elasticsearch/manifests/init.pp
+++ b/modules/elasticsearch/manifests/init.pp
@@ -120,6 +120,7 @@
 $curator_uses_unicast_hosts = true,
 $reindex_remote_whitelist = undef,
 $script_max_compilations_per_minute = undef,
+$additional_jvm_options = [],
 ) {
 
 # Check arguments
diff --git a/modules/elasticsearch/templates/jvm.options.erb 
b/modules/elasticsearch/templates/jvm.options.erb
index e4e6345..953fc92 100644
--- a/modules/elasticsearch/templates/jvm.options.erb
+++ b/modules/elasticsearch/templates/jvm.options.erb
@@ -126,3 +126,7 @@
 <% @gc_log_flags.each do |gc_log_flag| -%>
 <%= gc_log_flag %>
 <% end -%>
+
+<% @additional_jvm_options.each do |flag| -%>
+  <%= flag %>
+<% end -%>
diff --git 
a/modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.java.policy
 
b/modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.java.policy
new file mode 100644
index 000..7fab013
--- /dev/null
+++ 
b/modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.java.policy
@@ -0,0 +1,9 @@
+grant codeBase "file:/usr/share/java/prometheus/-" {
+
+  permission javax.management.MBeanServerPermission "createMBeanServer";
+  permission javax.management.MBeanPermission "*", "*";
+  permission java.lang.RuntimePermission "accessClassInPackage.sun.management";
+  permission java.io.FilePermission 
"/etc/elasticsearch/prometheus_jmx_exporter.yaml", "read";
+  permission java.io.FilePermission "/proc/self/status", "read";
+
+};
diff --git 
a/modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.yaml 
b/modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.yaml
new file mode 100644
index 000..b8cf8c9
--- /dev/null
+++ 
b/modules/profile/files/prometheus/elasticsearch_prometheus_jmx_exporter.yaml
@@ -0,0 +1,7 @@
+---
+lowercaseOutputLabelNames: true
+lowercaseOutputName: true
+# to ensure we only load a minimum number of MBeans, let's define a dummy 
whitelist
+whitelistObjectNames: [ 'java.lang:type=ClassLoading' ]
+# and since we are only interested in the DefaultExports, let's blacklist 
everything
+blacklistObjectNames: [ '*:*' ]
diff --git a/modules/profile/manifests/elasticsearch.pp 
b/modules/profile/manifests/elasticsearch.pp
index aa873c8..1e966c9 100644
--- a/modules/profile/manifests/elasticsearch.pp
+++ b/modules/profile/manifests/elasticsearch.pp
@@ -36,6 +36,10 @@
 ) {
 $master_eligible = $::fqdn in $unicast_hosts
 
+$jmx_exporter_path = 
'/usr/share/java/prometheus/jmx_prometheus_javaagent.jar'
+$jmx_exporter_port = '9109'
+$jmx_exporter_config = '/etc/elasticsearch/prometheus_jmx_exporter.yaml'
+
 ferm::service { 'elastic-http':
 proto   => 'tcp',
 port=> '9200',
@@ -78,6 +82,8 @@
 before => Class['::elasticsearch'],
 }
 
+
+
 # Install
 class { '::elasticsearch':
 # Production elasticsearch needs these plugins to be loaded in order
@@ -112,6 +118,7 @@
 search_shard_count_limit   => $search_shard_count_limit,
 reindex_remote_whitelist   => $reindex_remote_whitelist,
 script_max_compilations_per_minute => 1,
+additional_jvm_options => [ 
"-javaagent:$jmx_exporter_path=${jmx_exporter_port}:${jmx_exporter_config}" ],
 }
 
 class { '::elasticsearch::https':
diff --git a/modules/profile/manifests/prometheus/elasticsearch_jmx_exporter.pp 
b/modules/profile/manifests/prometheus/elasticsearch_jmx_exporter.pp
new file mode 100644
index 000..2bac681
--- /dev/null
+++ b/modules/profile/manifests/prometheus/elasticsearch_jmx_exporter.pp
@@ -0,0 +1,25 @@
+class profile::prometheus::elasticsearch_jmx_exporter (
+$prometheus_nodes = hiera('prometheus_nodes'),
+) {
+
+$prometheus_jmx_exporter_port = 9109
+
+

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: move prometheus classes to conform with best ...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398444 )

Change subject: elasticsearch: move prometheus classes to conform with best 
practice
..


elasticsearch: move prometheus classes to conform with best practice

Change-Id: Ifd0a97eb3cd5446fcdbd87f7bc725710c1219237
---
M modules/profile/manifests/elasticsearch.pp
R modules/profile/manifests/prometheus/elasticsearch_exporter.pp
M modules/role/manifests/elasticsearch/beta.pp
M modules/role/manifests/elasticsearch/cirrus.pp
M modules/role/manifests/elasticsearch/relforge.pp
M modules/role/manifests/logstash/elasticsearch.pp
M modules/role/manifests/prometheus/ops.pp
7 files changed, 6 insertions(+), 6 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/modules/profile/manifests/elasticsearch.pp 
b/modules/profile/manifests/elasticsearch.pp
index f32c066..aa873c8 100644
--- a/modules/profile/manifests/elasticsearch.pp
+++ b/modules/profile/manifests/elasticsearch.pp
@@ -34,9 +34,6 @@
 $reindex_remote_whitelist = 
hiera('profile::elasticsearch::reindex_remote_whitelist'),
 $storage_device = hiera('profile::elasticsearch::storage_device'),
 ) {
-
-include ::profile::elasticsearch::monitoring
-
 $master_eligible = $::fqdn in $unicast_hosts
 
 ferm::service { 'elastic-http':
diff --git a/modules/profile/manifests/elasticsearch/monitoring.pp 
b/modules/profile/manifests/prometheus/elasticsearch_exporter.pp
similarity index 86%
rename from modules/profile/manifests/elasticsearch/monitoring.pp
rename to modules/profile/manifests/prometheus/elasticsearch_exporter.pp
index e1bf3e0..9d6445f 100644
--- a/modules/profile/manifests/elasticsearch/monitoring.pp
+++ b/modules/profile/manifests/prometheus/elasticsearch_exporter.pp
@@ -1,4 +1,4 @@
-class profile::elasticsearch::monitoring(
+class profile::prometheus::elasticsearch_exporter(
 $prometheus_nodes = hiera('prometheus_nodes'),
 ) {
 $prometheus_nodes_ferm = join($prometheus_nodes, ' ')
diff --git a/modules/role/manifests/elasticsearch/beta.pp 
b/modules/role/manifests/elasticsearch/beta.pp
index cb668a1..7717ccd 100644
--- a/modules/role/manifests/elasticsearch/beta.pp
+++ b/modules/role/manifests/elasticsearch/beta.pp
@@ -5,6 +5,7 @@
 # filtertags: labs-project-deployment-prep labs-project-search 
labs-project-math
 class role::elasticsearch::beta {
 include ::profile::elasticsearch
+include ::profile::prometheus::elasticsearch_exporter
 
 system::role { 'elasticsearch::beta':
 ensure  => 'present',
diff --git a/modules/role/manifests/elasticsearch/cirrus.pp 
b/modules/role/manifests/elasticsearch/cirrus.pp
index 699e854..f91f910 100644
--- a/modules/role/manifests/elasticsearch/cirrus.pp
+++ b/modules/role/manifests/elasticsearch/cirrus.pp
@@ -7,6 +7,7 @@
 include ::base::firewall
 include ::role::lvs::realserver
 include ::profile::elasticsearch
+include ::profile::prometheus::elasticsearch_exporter
 
 system::role { 'elasticsearch::cirrus':
 ensure  => 'present',
diff --git a/modules/role/manifests/elasticsearch/relforge.pp 
b/modules/role/manifests/elasticsearch/relforge.pp
index d88e058..80cbccd 100644
--- a/modules/role/manifests/elasticsearch/relforge.pp
+++ b/modules/role/manifests/elasticsearch/relforge.pp
@@ -6,6 +6,7 @@
 include ::standard
 include ::base::firewall
 include ::profile::elasticsearch
+include ::profile::prometheus::elasticsearch_exporter
 include ::elasticsearch::nagios::check
 include ::profile::mjolnir::kafka_daemon
 
diff --git a/modules/role/manifests/logstash/elasticsearch.pp 
b/modules/role/manifests/logstash/elasticsearch.pp
index 31a3255..390f21d 100644
--- a/modules/role/manifests/logstash/elasticsearch.pp
+++ b/modules/role/manifests/logstash/elasticsearch.pp
@@ -6,7 +6,7 @@
 class role::logstash::elasticsearch {
 include ::standard
 include ::elasticsearch::monitor::diamond
-include ::profile::elasticsearch::monitoring
+include ::profile::prometheus::elasticsearch_exporter
 include ::profile::base::firewall
 
 # the logstash cluster has 3 data nodes, and each shard has 3 replica (each
diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index ab4e3e3..8680d6f 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -761,7 +761,7 @@
 prometheus::class_config { "elasticsearch_${::site}":
 dest   => "${targets_path}/elasticsearch_${::site}.yaml",
 site   => $::site,
-class_name => 'profile::elasticsearch::monitoring',
+class_name => 'profile::prometheus::elasticsearch_exporter',
 port   => '9108',
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/398444
To unsubscribe, 

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Access Archiva over HTTPS

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401709 )

Change subject: Access Archiva over HTTPS
..

Access Archiva over HTTPS

Archiva sends an HTTP 307 (temporary redirect) when accessed over HTTP.
Maven does not handle that well, and since we should only be accessing
it over HTTPS, let's correct our pom.xml

Change-Id: If8a951fc2d5f05581d595abefe25fe79993d478f
---
M pom.xml
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/09/401709/1

diff --git a/pom.xml b/pom.xml
index 69fddaf..5df3940 100644
--- a/pom.xml
+++ b/pom.xml
@@ -619,11 +619,11 @@
 
 
 wmf.mirrored
-http://archiva.wikimedia.org/repository/mirrored
+https://archiva.wikimedia.org/repository/mirrored
 
 
 wmf.releases
-http://archiva.wikimedia.org/repository/releases
+https://archiva.wikimedia.org/repository/releases
 
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/401709
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If8a951fc2d5f05581d595abefe25fe79993d478f
Gerrit-PatchSet: 1
Gerrit-Project: wikidata/query/rdf
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Properly close HC client in WikibaseRepository

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401705 )

Change subject: Properly close HC client in WikibaseRepository
..

Properly close HC client in WikibaseRepository

This could be one of the cause for leaked threads detected in our unit
tests.

Bug: T178721
Change-Id: I882d886382fe73630cfc5d4f372de809e4ddeb29
---
M tools/src/main/java/org/wikidata/query/rdf/tool/Update.java
M 
tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryIntegrationTest.java
4 files changed, 55 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/05/401705/1

diff --git a/tools/src/main/java/org/wikidata/query/rdf/tool/Update.java 
b/tools/src/main/java/org/wikidata/query/rdf/tool/Update.java
index 9ea247e..5d768ac 100644
--- a/tools/src/main/java/org/wikidata/query/rdf/tool/Update.java
+++ b/tools/src/main/java/org/wikidata/query/rdf/tool/Update.java
@@ -50,10 +50,11 @@
 public static void main(String[] args) throws Exception {
 RdfRepository rdfRepository = null;
 Updater updater;
+WikibaseRepository wikibaseRepository;
 
 try {
 UpdateOptions options = handleOptions(UpdateOptions.class, args);
-WikibaseRepository wikibaseRepository = 
buildWikibaseRepository(options);
+wikibaseRepository = buildWikibaseRepository(options);
 URI sparqlUri = sparqlUri(options);
 WikibaseUris uris = new WikibaseUris(options.wikibaseHost());
 rdfRepository = new RdfRepository(sparqlUri, uris);
@@ -67,13 +68,15 @@
 }
 throw e;
 }
-try (RdfRepository r = rdfRepository) {
+try (
+WikibaseRepository w = wikibaseRepository;
+RdfRepository r = rdfRepository;
+Updater u = updater
+) {
 updater.run();
 } catch (Exception e) {
 log.error("Error during updater run.", e);
 throw e;
-} finally {
-updater.close();
 }
 }
 
diff --git 
a/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
 
b/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
index eeccbc5..58c3cfb 100644
--- 
a/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
+++ 
b/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
@@ -1,5 +1,6 @@
 package org.wikidata.query.rdf.tool.wikibase;
 
+import java.io.Closeable;
 import java.io.IOException;
 import java.io.InputStreamReader;
 import java.io.InterruptedIOException;
@@ -72,7 +73,7 @@
  */
 // TODO fan out complexity
 @SuppressWarnings("checkstyle:classfanoutcomplexity")
-public class WikibaseRepository {
+public class WikibaseRepository implements Closeable {
 private static final Logger log = 
LoggerFactory.getLogger(WikibaseRepository.class);
 
 /**
@@ -416,6 +417,11 @@
 return name.matches("^[A-Za-z0-9:]+$");
 }
 
+@Override
+public void close() throws IOException {
+client.close();
+}
+
 /**
  * URIs used for accessing wikibase.
  */
diff --git 
a/tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
 
b/tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
index 22bf8eb..c49cca4 100644
--- 
a/tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
+++ 
b/tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
@@ -1,10 +1,12 @@
 package org.wikidata.query.rdf.tool;
 
+import java.io.IOException;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.ThreadPoolExecutor;
 import java.util.concurrent.TimeUnit;
 
+import org.junit.After;
 import org.junit.Rule;
 import org.junit.runner.RunWith;
 import org.wikidata.query.rdf.common.uri.WikibaseUris;
@@ -37,6 +39,11 @@
 public RdfRepositoryForTesting rdfRepository = new 
RdfRepositoryForTesting("wdq");
 
 
+@After
+public void closeWikibaseRepository() throws IOException {
+wikibaseRepository.close();
+}
+
 /**
  * Update all ids from from to to.
  */
diff --git 
a/tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryIntegrationTest.java
 
b/tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryIntegrationTest.java
index bf4c887..865db7b 100644
--- 
a/tools/src/test/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepositoryIntegrationTest.java
+++ 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: thumbor: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399652 )

Change subject: thumbor: use the canonical definition of logstash host
..


thumbor: use the canonical definition of logstash host

Also extracts a profile::thumbor class to respect our puppet guidelines
at least on new hiera parameters.

Bug: T182304
Change-Id: I87bb508d633b6f39ead3e6075641db608bb9bbf4
---
M hieradata/common.yaml
M hieradata/labs/deployment-prep/common.yaml
M hieradata/role/codfw/thumbor/mediawiki.yaml
M hieradata/role/eqiad/thumbor/mediawiki.yaml
A modules/profile/manifests/thumbor.pp
M modules/role/manifests/thumbor/mediawiki.pp
6 files changed, 44 insertions(+), 34 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 2cfc1b1..71a9a87 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -522,7 +522,10 @@
 logstash_host: "logstash.svc.eqiad.wmnet"
 logstash_syslog_port: 10514
 logstash_gelf_port: 12201
+# TCP json_lines input
 logstash_json_lines_port: 11514
+# UDP logback/json input
+logstash_logback_port: 11514
 udp2log_aggregator: "udplog:8420"
 
 tcpircbot_host: 'icinga.wikimedia.org'
diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index f2c54ce..5bad6f5 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -140,10 +140,15 @@
   default_instance_params:
 rpc_address: '%{::ipaddress}'
 # end cassandra
-"udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
-"logstash_host": deployment-logstash2.deployment-prep.eqiad.wmflabs
-"logstash_syslog_port": 10514
-"logstash_gelf_port": 12201
+udp2log_aggregator: deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
+logstash_host: deployment-logstash2.deployment-prep.eqiad.wmflabs
+logstash_syslog_port: 10514
+logstash_gelf_port: 12201
+# TCP json_lines input
+logstash_json_lines_port: 11514
+# UDP logback/json input
+logstash_logback_port: 11514
+
 mediawiki_memcached_servers:
 - 10.68.23.25:11211:1  # deployment-memc04
 - 10.68.23.49:11211:1  # deployment-memc05
@@ -357,9 +362,6 @@
   recentchange:
 topics: ["%{::site}.mediawiki.recentchange"]
 profile::eventstreams::rdkafka_config: {}
-
-thumbor::logstash_host: 'deployment-logstash2.deployment-prep.eqiad.wmflabs'
-thumbor::logstash_port: 11514
 
 cache::be_transient_gb: 0
 cache::fe_transient_gb: 0
diff --git a/hieradata/role/codfw/thumbor/mediawiki.yaml 
b/hieradata/role/codfw/thumbor/mediawiki.yaml
index 9dd4e02..634df35 100644
--- a/hieradata/role/codfw/thumbor/mediawiki.yaml
+++ b/hieradata/role/codfw/thumbor/mediawiki.yaml
@@ -17,7 +17,4 @@
 
 thumbor::poolcounter_server: 'poolcounter2001.codfw.wmnet'
 
-thumbor::logstash_host: 'logstash.svc.eqiad.wmnet'
-thumbor::logstash_port: 11514
-
 thumbor::stl_support: true
diff --git a/hieradata/role/eqiad/thumbor/mediawiki.yaml 
b/hieradata/role/eqiad/thumbor/mediawiki.yaml
index 12ac392..4163d6c 100644
--- a/hieradata/role/eqiad/thumbor/mediawiki.yaml
+++ b/hieradata/role/eqiad/thumbor/mediawiki.yaml
@@ -17,7 +17,4 @@
 
 thumbor::poolcounter_server: 'poolcounter1001.eqiad.wmnet'
 
-thumbor::logstash_host: 'logstash.svc.eqiad.wmnet'
-thumbor::logstash_port: 11514
-
 thumbor::stl_support: true
diff --git a/modules/profile/manifests/thumbor.pp 
b/modules/profile/manifests/thumbor.pp
new file mode 100644
index 000..c31fa2e
--- /dev/null
+++ b/modules/profile/manifests/thumbor.pp
@@ -0,0 +1,31 @@
+class profile::thumbor(
+$memcached_servers_nutcracker = 
hiera('thumbor_memcached_servers_nutcracker'),
+$logstash_host = hiera('logstash_host'),
+$logstash_port = hiera('logstash_logback_port'),
+$swift_sharded_containers = 
hiera_array('swift::proxy::shard_container_list'),
+) {
+
+class { '::thumbor::nutcracker':
+thumbor_memcached_servers => $memcached_servers_nutcracker,
+}
+
+class { '::thumbor':
+logstash_host => $logstash_host,
+logstash_port => $logstash_port,
+}
+
+include ::swift::params
+$swift_account_keys = $::swift::params::account_keys
+
+class { '::thumbor::swift':
+swift_key=> $swift_account_keys['mw_thumbor'],
+swift_sharded_containers => $swift_sharded_containers,
+}
+
+ferm::service { 'thumbor':
+proto  => 'tcp',
+port   => '8800',
+srange => '$DOMAIN_NETWORKS',
+}
+
+}
\ No newline at end of file
diff --git a/modules/role/manifests/thumbor/mediawiki.pp 
b/modules/role/manifests/thumbor/mediawiki.pp
index 6214b4c..88f5943 100644
--- a/modules/role/manifests/thumbor/mediawiki.pp
+++ b/modules/role/manifests/thumbor/mediawiki.pp
@@ -10,28 +10,8 @@
 include ::mediawiki::packages::fonts
 include 

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: update jackson to version 2.9.3

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/400258 )

Change subject: update jackson to version 2.9.3
..

update jackson to version 2.9.3

The version of jackson that we are using is outdated (> 4 years) and will
cause issues when introducing wiremock (or other lib requiring a reasonably
recent version of jackson). And it is most of the time a good idea to keep
dependencies up to date.

Tests are still passing with this new version and jackson has a good track
record of preserving compatibility.

Change-Id: I4051bd08872a54284796a49383346fb14a00f147
---
M pom.xml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/58/400258/1

diff --git a/pom.xml b/pom.xml
index 0ff4ffb..69fddaf 100644
--- a/pom.xml
+++ b/pom.xml
@@ -56,7 +56,7 @@
 
 
 2.1.5-SNAPSHOT
-2.2.3
+2.9.3
 9.2.3.v20140905
 1.2.3
 2.7.12

-- 
To view, visit https://gerrit.wikimedia.org/r/400258
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4051bd08872a54284796a49383346fb14a00f147
Gerrit-PatchSet: 1
Gerrit-Project: wikidata/query/rdf
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Completely disable cookie management in WikibaseRepository

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399819 )

Change subject: Completely disable cookie management in WikibaseRepository
..

Completely disable cookie management in WikibaseRepository

Instead of disabling cookies on each request, we can just disable cookie
management completely.

Change-Id: I50d20c91a1666c2d75c078c6659de007a01bbe5e
---
M 
tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
1 file changed, 1 insertion(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/19/399819/1

diff --git 
a/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
 
b/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
index 81b15be..eeccbc5 100644
--- 
a/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
+++ 
b/tools/src/main/java/org/wikidata/query/rdf/tool/wikibase/WikibaseRepository.java
@@ -56,7 +56,6 @@
 import org.openrdf.rio.helpers.StatementCollector;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.wikidata.query.rdf.tool.HttpClientUtils;
 import org.wikidata.query.rdf.tool.change.Change;
 import org.wikidata.query.rdf.tool.exception.ContainedException;
 import org.wikidata.query.rdf.tool.exception.FatalException;
@@ -93,6 +92,7 @@
 .setMaxConnPerRoute(100).setMaxConnTotal(100)
 .setRetryHandler(getRetryHandler(RETRIES))
 .setServiceUnavailableRetryStrategy(getRetryStrategy(RETRIES, 
RETRY_INTERVAL))
+.disableCookieManagement()
 .setUserAgent("Wikidata Query Service Updater")
 .build();
 
@@ -239,7 +239,6 @@
 StatementCollector collector = new StatementCollector();
 parser.setRDFHandler(new NormalizingRdfHandler(collector));
 HttpGet request = new HttpGet(uri);
-HttpClientUtils.ignoreCookies(request);
 try {
 try (CloseableHttpResponse response = client.execute(request)) {
 if (response.getStatusLine().getStatusCode() == 404) {
@@ -376,7 +375,6 @@
  * @throws ParseException the json was malformed and couldn't be parsed
  */
 private JSONObject getJson(HttpRequestBase request) throws IOException, 
ParseException {
-HttpClientUtils.ignoreCookies(request);
 try (CloseableHttpResponse response = client.execute(request)) {
 return (JSONObject) new JSONParser().parse(new 
InputStreamReader(response.getEntity().getContent(),
 Charsets.UTF_8));

-- 
To view, visit https://gerrit.wikimedia.org/r/399819
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I50d20c91a1666c2d75c078c6659de007a01bbe5e
Gerrit-PatchSet: 1
Gerrit-Project: wikidata/query/rdf
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399650 )

Change subject: wdqs: use the canonical definition of logstash host
..


wdqs: use the canonical definition of logstash host

Bug: T182304
Change-Id: I07b7dbcd847ef01d5833d80c21b100be4c5c7017
---
M hieradata/role/common/wdqs.yaml
M modules/profile/manifests/wdqs.pp
2 files changed, 1 insertion(+), 2 deletions(-)

Approvals:
  Smalyshev: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/role/common/wdqs.yaml b/hieradata/role/common/wdqs.yaml
index f083ff8..3c91737 100644
--- a/hieradata/role/common/wdqs.yaml
+++ b/hieradata/role/common/wdqs.yaml
@@ -1,4 +1,3 @@
-profile::wdqs::logstash_host: 'logstash.svc.eqiad.wmnet'
 profile::wdqs::use_git_deploy: true
 profile::wdqs::package_dir: '/srv/deployment/wdqs/wdqs'
 profile::wdqs::data_dir: '/srv/wdqs'
diff --git a/modules/profile/manifests/wdqs.pp 
b/modules/profile/manifests/wdqs.pp
index d807d06..b9af7e4 100644
--- a/modules/profile/manifests/wdqs.pp
+++ b/modules/profile/manifests/wdqs.pp
@@ -1,5 +1,5 @@
 class profile::wdqs (
-$logstash_host = hiera('profile::wdqs::logstash_host'),
+$logstash_host = hiera('logstash_host'),
 $use_git_deploy = hiera('profile::wdqs::use_git_deploy'),
 $package_dir = hiera('profile::wdqs::package_dir'),
 $data_dir = hiera('profile::wdqs::data_dir'),

-- 
To view, visit https://gerrit.wikimedia.org/r/399650
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I07b7dbcd847ef01d5833d80c21b100be4c5c7017
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Smalyshev 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: thumbor: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399652 )

Change subject: thumbor: use the canonical definition of logstash host
..

thumbor: use the canonical definition of logstash host

Bug: T182304
Change-Id: I87bb508d633b6f39ead3e6075641db608bb9bbf4
---
M hieradata/common.yaml
M hieradata/labs/deployment-prep/common.yaml
M hieradata/role/codfw/thumbor/mediawiki.yaml
M hieradata/role/eqiad/thumbor/mediawiki.yaml
M modules/role/manifests/thumbor/mediawiki.pp
5 files changed, 16 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/52/399652/1

diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index f55e552..ece30fb 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -520,7 +520,10 @@
 logstash_host: "logstash.svc.eqiad.wmnet"
 logstash_syslog_port: 10514
 logstash_gelf_port: 12201
+# TCP json_lines input
 logstash_json_lines_port: 11514
+# UDP logback/json input
+logstash_logback_port: 11514
 udp2log_aggregator: "udplog:8420"
 
 tcpircbot_host: 'icinga.wikimedia.org'
diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index f2c54ce..5bad6f5 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -140,10 +140,15 @@
   default_instance_params:
 rpc_address: '%{::ipaddress}'
 # end cassandra
-"udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
-"logstash_host": deployment-logstash2.deployment-prep.eqiad.wmflabs
-"logstash_syslog_port": 10514
-"logstash_gelf_port": 12201
+udp2log_aggregator: deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
+logstash_host: deployment-logstash2.deployment-prep.eqiad.wmflabs
+logstash_syslog_port: 10514
+logstash_gelf_port: 12201
+# TCP json_lines input
+logstash_json_lines_port: 11514
+# UDP logback/json input
+logstash_logback_port: 11514
+
 mediawiki_memcached_servers:
 - 10.68.23.25:11211:1  # deployment-memc04
 - 10.68.23.49:11211:1  # deployment-memc05
@@ -357,9 +362,6 @@
   recentchange:
 topics: ["%{::site}.mediawiki.recentchange"]
 profile::eventstreams::rdkafka_config: {}
-
-thumbor::logstash_host: 'deployment-logstash2.deployment-prep.eqiad.wmflabs'
-thumbor::logstash_port: 11514
 
 cache::be_transient_gb: 0
 cache::fe_transient_gb: 0
diff --git a/hieradata/role/codfw/thumbor/mediawiki.yaml 
b/hieradata/role/codfw/thumbor/mediawiki.yaml
index 9dd4e02..634df35 100644
--- a/hieradata/role/codfw/thumbor/mediawiki.yaml
+++ b/hieradata/role/codfw/thumbor/mediawiki.yaml
@@ -17,7 +17,4 @@
 
 thumbor::poolcounter_server: 'poolcounter2001.codfw.wmnet'
 
-thumbor::logstash_host: 'logstash.svc.eqiad.wmnet'
-thumbor::logstash_port: 11514
-
 thumbor::stl_support: true
diff --git a/hieradata/role/eqiad/thumbor/mediawiki.yaml 
b/hieradata/role/eqiad/thumbor/mediawiki.yaml
index 12ac392..4163d6c 100644
--- a/hieradata/role/eqiad/thumbor/mediawiki.yaml
+++ b/hieradata/role/eqiad/thumbor/mediawiki.yaml
@@ -17,7 +17,4 @@
 
 thumbor::poolcounter_server: 'poolcounter1001.eqiad.wmnet'
 
-thumbor::logstash_host: 'logstash.svc.eqiad.wmnet'
-thumbor::logstash_port: 11514
-
 thumbor::stl_support: true
diff --git a/modules/role/manifests/thumbor/mediawiki.pp 
b/modules/role/manifests/thumbor/mediawiki.pp
index 6214b4c..41e189d 100644
--- a/modules/role/manifests/thumbor/mediawiki.pp
+++ b/modules/role/manifests/thumbor/mediawiki.pp
@@ -15,7 +15,10 @@
 thumbor_memcached_servers => 
hiera('thumbor_memcached_servers_nutcracker')
 }
 
-class { '::thumbor': }
+class { '::thumbor':
+logstash_host => hiera('logstash_host'),
+logstash_port => hiera('logstash_logback_port'),
+}
 
 include ::swift::params
 $swift_account_keys = $::swift::params::account_keys

-- 
To view, visit https://gerrit.wikimedia.org/r/399652
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I87bb508d633b6f39ead3e6075641db608bb9bbf4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399650 )

Change subject: wdqs: use the canonical definition of logstash host
..

wdqs: use the canonical definition of logstash host

Bug: T182304
Change-Id: I07b7dbcd847ef01d5833d80c21b100be4c5c7017
---
M hieradata/role/common/wdqs.yaml
M modules/profile/manifests/wdqs.pp
2 files changed, 1 insertion(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/50/399650/1

diff --git a/hieradata/role/common/wdqs.yaml b/hieradata/role/common/wdqs.yaml
index f083ff8..3c91737 100644
--- a/hieradata/role/common/wdqs.yaml
+++ b/hieradata/role/common/wdqs.yaml
@@ -1,4 +1,3 @@
-profile::wdqs::logstash_host: 'logstash.svc.eqiad.wmnet'
 profile::wdqs::use_git_deploy: true
 profile::wdqs::package_dir: '/srv/deployment/wdqs/wdqs'
 profile::wdqs::data_dir: '/srv/wdqs'
diff --git a/modules/profile/manifests/wdqs.pp 
b/modules/profile/manifests/wdqs.pp
index d807d06..b9af7e4 100644
--- a/modules/profile/manifests/wdqs.pp
+++ b/modules/profile/manifests/wdqs.pp
@@ -1,5 +1,5 @@
 class profile::wdqs (
-$logstash_host = hiera('profile::wdqs::logstash_host'),
+$logstash_host = hiera('logstash_host'),
 $use_git_deploy = hiera('profile::wdqs::use_git_deploy'),
 $package_dir = hiera('profile::wdqs::package_dir'),
 $data_dir = hiera('profile::wdqs::data_dir'),

-- 
To view, visit https://gerrit.wikimedia.org/r/399650
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I07b7dbcd847ef01d5833d80c21b100be4c5c7017
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: gerrit: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399644 )

Change subject: gerrit: use the canonical definition of logstash host
..


gerrit: use the canonical definition of logstash host

Bug: T182304
Change-Id: Ifddb902fedb465ff969c80833186b811dd5ef9ee
---
M hieradata/role/common/gerrit.yaml
M modules/gerrit/manifests/init.pp
M modules/profile/manifests/gerrit/server.pp
3 files changed, 9 insertions(+), 7 deletions(-)

Approvals:
  Hashar: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/role/common/gerrit.yaml 
b/hieradata/role/common/gerrit.yaml
index 7f9abdf..2237dd9 100644
--- a/hieradata/role/common/gerrit.yaml
+++ b/hieradata/role/common/gerrit.yaml
@@ -33,8 +33,6 @@
 replicateProjectDeletions: true
 replicateHiddenProjects: true
 
-gerrit::jetty::log_host: 'logstash.svc.eqiad.wmnet'
-
 # Allow use of supplemental key for scap deployment
 profile::base::ssh_server_settings:
 authorized_keys_file: '/etc/ssh/userkeys/%u /etc/ssh/userkeys/%u.d/cumin 
/etc/ssh/userkeys/%u.d/gerrit-scap'
diff --git a/modules/gerrit/manifests/init.pp b/modules/gerrit/manifests/init.pp
index 61dbbf0..c5d0c4f 100644
--- a/modules/gerrit/manifests/init.pp
+++ b/modules/gerrit/manifests/init.pp
@@ -6,14 +6,16 @@
 $ipv6,
 $slave_hosts = [],
 $slave = false,
+$log_host = undef,
 ) {
 
 class { '::gerrit::jetty':
-host   => $host,
-ipv4   => $ipv4,
-ipv6   => $ipv6,
-slave  => $slave,
-config => $config,
+host => $host,
+ipv4 => $ipv4,
+ipv6 => $ipv6,
+slave=> $slave,
+config   => $config,
+log_host => $log_host,
 }
 
 class { '::gerrit::proxy':
diff --git a/modules/profile/manifests/gerrit/server.pp 
b/modules/profile/manifests/gerrit/server.pp
index 042b784..246d794 100644
--- a/modules/profile/manifests/gerrit/server.pp
+++ b/modules/profile/manifests/gerrit/server.pp
@@ -10,6 +10,7 @@
 $bacula = hiera('gerrit::server::bacula'),
 $gerrit_servers = join(hiera('gerrit::servers'), ' '),
 $config = hiera('gerrit::server::config'),
+$log_host = hiera('logstash_host'),
 ) {
 
 interface::alias { 'gerrit server':
@@ -71,6 +72,7 @@
 slave   => $slave,
 slave_hosts => $slave_hosts,
 config  => $config,
+log_host=> $log_host,
 }
 
 class { '::apache::mod::rewrite': }

-- 
To view, visit https://gerrit.wikimedia.org/r/399644
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifddb902fedb465ff969c80833186b811dd5ef9ee
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Chad 
Gerrit-Reviewer: Dzahn 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Hashar 
Gerrit-Reviewer: Paladox 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: gerrit: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399644 )

Change subject: gerrit: use the canonical definition of logstash host
..

gerrit: use the canonical definition of logstash host

Bug: T182304
Change-Id: Ifddb902fedb465ff969c80833186b811dd5ef9ee
---
M hieradata/role/common/gerrit.yaml
M modules/gerrit/manifests/init.pp
M modules/profile/manifests/gerrit/server.pp
3 files changed, 9 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/44/399644/1

diff --git a/hieradata/role/common/gerrit.yaml 
b/hieradata/role/common/gerrit.yaml
index 7f9abdf..2237dd9 100644
--- a/hieradata/role/common/gerrit.yaml
+++ b/hieradata/role/common/gerrit.yaml
@@ -33,8 +33,6 @@
 replicateProjectDeletions: true
 replicateHiddenProjects: true
 
-gerrit::jetty::log_host: 'logstash.svc.eqiad.wmnet'
-
 # Allow use of supplemental key for scap deployment
 profile::base::ssh_server_settings:
 authorized_keys_file: '/etc/ssh/userkeys/%u /etc/ssh/userkeys/%u.d/cumin 
/etc/ssh/userkeys/%u.d/gerrit-scap'
diff --git a/modules/gerrit/manifests/init.pp b/modules/gerrit/manifests/init.pp
index 61dbbf0..c5d0c4f 100644
--- a/modules/gerrit/manifests/init.pp
+++ b/modules/gerrit/manifests/init.pp
@@ -6,14 +6,16 @@
 $ipv6,
 $slave_hosts = [],
 $slave = false,
+$log_host = undef,
 ) {
 
 class { '::gerrit::jetty':
-host   => $host,
-ipv4   => $ipv4,
-ipv6   => $ipv6,
-slave  => $slave,
-config => $config,
+host => $host,
+ipv4 => $ipv4,
+ipv6 => $ipv6,
+slave=> $slave,
+config   => $config,
+log_host => $log_host,
 }
 
 class { '::gerrit::proxy':
diff --git a/modules/profile/manifests/gerrit/server.pp 
b/modules/profile/manifests/gerrit/server.pp
index 042b784..246d794 100644
--- a/modules/profile/manifests/gerrit/server.pp
+++ b/modules/profile/manifests/gerrit/server.pp
@@ -10,6 +10,7 @@
 $bacula = hiera('gerrit::server::bacula'),
 $gerrit_servers = join(hiera('gerrit::servers'), ' '),
 $config = hiera('gerrit::server::config'),
+$log_host = hiera('logstash_host'),
 ) {
 
 interface::alias { 'gerrit server':
@@ -71,6 +72,7 @@
 slave   => $slave,
 slave_hosts => $slave_hosts,
 config  => $config,
+log_host=> $log_host,
 }
 
 class { '::apache::mod::rewrite': }

-- 
To view, visit https://gerrit.wikimedia.org/r/399644
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifddb902fedb465ff969c80833186b811dd5ef9ee
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kafkatee: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399629 )

Change subject: kafkatee: use the canonical definition of logstash host
..


kafkatee: use the canonical definition of logstash host

Bug: T182304
Change-Id: Iaee3be3b2aa3de99bd2448da9f11106683f7cbca
---
M hieradata/common.yaml
D hieradata/role/common/logging/kafkatee/webrequest/ops.yaml
M modules/role/manifests/logging/kafkatee/webrequest/ops.pp
3 files changed, 3 insertions(+), 4 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 4ff228a..f55e552 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -520,6 +520,7 @@
 logstash_host: "logstash.svc.eqiad.wmnet"
 logstash_syslog_port: 10514
 logstash_gelf_port: 12201
+logstash_json_lines_port: 11514
 udp2log_aggregator: "udplog:8420"
 
 tcpircbot_host: 'icinga.wikimedia.org'
diff --git a/hieradata/role/common/logging/kafkatee/webrequest/ops.yaml 
b/hieradata/role/common/logging/kafkatee/webrequest/ops.yaml
deleted file mode 100644
index 7d7a879..000
--- a/hieradata/role/common/logging/kafkatee/webrequest/ops.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-role::logging::kafkatee::webrequest::ops::logstash_host: 
logstash.svc.eqiad.wmnet
-role::logging::kafkatee::webrequest::ops::logstash_port: 11514
diff --git a/modules/role/manifests/logging/kafkatee/webrequest/ops.pp 
b/modules/role/manifests/logging/kafkatee/webrequest/ops.pp
index 67b43e1..14393dd 100644
--- a/modules/role/manifests/logging/kafkatee/webrequest/ops.pp
+++ b/modules/role/manifests/logging/kafkatee/webrequest/ops.pp
@@ -8,8 +8,8 @@
 include ::geoip  # lint:ignore:wmf_styleguide
 
 $webrequest_log_directory = 
$::role::logging::kafkatee::webrequest::base::webrequest_log_directory
-$logstash_host = 
hiera('role::logging::kafkatee::webrequest::ops::logstash_host')
-$logstash_port = 
hiera('role::logging::kafkatee::webrequest::ops::logstash_port')
+$logstash_host = hiera('logstash_host')
+$logstash_port = hiera('logstash_json_lines_port')
 
 kafkatee::output { 'sampled-1000':
 destination => "${webrequest_log_directory}/sampled-1000.json",

-- 
To view, visit https://gerrit.wikimedia.org/r/399629
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iaee3be3b2aa3de99bd2448da9f11106683f7cbca
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: aqs: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399624 )

Change subject: aqs: use the canonical definition of logstash host
..


aqs: use the canonical definition of logstash host

Bug: T182304
Change-Id: I5aec6c6d44c253986576676309da7b3bece67e77
---
M hieradata/role/common/aqs.yaml
M modules/profile/manifests/aqs.pp
2 files changed, 1 insertion(+), 2 deletions(-)

Approvals:
  Elukey: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/role/common/aqs.yaml b/hieradata/role/common/aqs.yaml
index f825bd7..5ad96c5 100644
--- a/hieradata/role/common/aqs.yaml
+++ b/hieradata/role/common/aqs.yaml
@@ -105,7 +105,6 @@
 - aqs1008-b.eqiad.wmnet
 - aqs1009-a.eqiad.wmnet
 - aqs1009-b.eqiad.wmnet
-profile::aqs::logstash_host: logstash.svc.eqiad.wmnet
 profile::aqs::cassandra_default_consistency: localQuorum
 profile::aqs::cassandra_local_dc: "%{::site}"
 profile::aqs::statsd_host: statsd.eqiad.wmnet
diff --git a/modules/profile/manifests/aqs.pp b/modules/profile/manifests/aqs.pp
index 262034e..07092e2 100644
--- a/modules/profile/manifests/aqs.pp
+++ b/modules/profile/manifests/aqs.pp
@@ -9,7 +9,7 @@
 $cassandra_user= hiera('profile::aqs::cassandra_user'),
 $cassandra_password= 
hiera('profile::aqs::cassandra_password'),
 $cassandra_seeds   = hiera('profile::aqs::seeds'),
-$logstash_host = hiera('profile::aqs::logstash_host'),
+$logstash_host = hiera('logstash_host'),
 $cassandra_default_consistency = 
hiera('profile::aqs::cassandra_default_consistency'),
 $cassandra_local_dc= 
hiera('profile::aqs::cassandra_local_dc'),
 $statsd_host   = hiera('profile::aqs::statsd_host'),

-- 
To view, visit https://gerrit.wikimedia.org/r/399624
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5aec6c6d44c253986576676309da7b3bece67e77
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kafkatee: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399629 )

Change subject: kafkatee: use the canonical definition of logstash host
..

kafkatee: use the canonical definition of logstash host

Bug: T182304
Change-Id: Iaee3be3b2aa3de99bd2448da9f11106683f7cbca
---
M hieradata/common.yaml
D hieradata/role/common/logging/kafkatee/webrequest/ops.yaml
M modules/role/manifests/logging/kafkatee/webrequest/ops.pp
3 files changed, 3 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/29/399629/1

diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 4ff228a..f55e552 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -520,6 +520,7 @@
 logstash_host: "logstash.svc.eqiad.wmnet"
 logstash_syslog_port: 10514
 logstash_gelf_port: 12201
+logstash_json_lines_port: 11514
 udp2log_aggregator: "udplog:8420"
 
 tcpircbot_host: 'icinga.wikimedia.org'
diff --git a/hieradata/role/common/logging/kafkatee/webrequest/ops.yaml 
b/hieradata/role/common/logging/kafkatee/webrequest/ops.yaml
deleted file mode 100644
index 7d7a879..000
--- a/hieradata/role/common/logging/kafkatee/webrequest/ops.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-role::logging::kafkatee::webrequest::ops::logstash_host: 
logstash.svc.eqiad.wmnet
-role::logging::kafkatee::webrequest::ops::logstash_port: 11514
diff --git a/modules/role/manifests/logging/kafkatee/webrequest/ops.pp 
b/modules/role/manifests/logging/kafkatee/webrequest/ops.pp
index 67b43e1..14393dd 100644
--- a/modules/role/manifests/logging/kafkatee/webrequest/ops.pp
+++ b/modules/role/manifests/logging/kafkatee/webrequest/ops.pp
@@ -8,8 +8,8 @@
 include ::geoip  # lint:ignore:wmf_styleguide
 
 $webrequest_log_directory = 
$::role::logging::kafkatee::webrequest::base::webrequest_log_directory
-$logstash_host = 
hiera('role::logging::kafkatee::webrequest::ops::logstash_host')
-$logstash_port = 
hiera('role::logging::kafkatee::webrequest::ops::logstash_port')
+$logstash_host = hiera('logstash_host')
+$logstash_port = hiera('logstash_json_lines_port')
 
 kafkatee::output { 'sampled-1000':
 destination => "${webrequest_log_directory}/sampled-1000.json",

-- 
To view, visit https://gerrit.wikimedia.org/r/399629
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaee3be3b2aa3de99bd2448da9f11106683f7cbca
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: aqs: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399624 )

Change subject: aqs: use the canonical definition of logstash host
..

aqs: use the canonical definition of logstash host

Bug: T182304
Change-Id: I5aec6c6d44c253986576676309da7b3bece67e77
---
M hieradata/role/common/aqs.yaml
M modules/profile/manifests/aqs.pp
2 files changed, 1 insertion(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/24/399624/1

diff --git a/hieradata/role/common/aqs.yaml b/hieradata/role/common/aqs.yaml
index f825bd7..5ad96c5 100644
--- a/hieradata/role/common/aqs.yaml
+++ b/hieradata/role/common/aqs.yaml
@@ -105,7 +105,6 @@
 - aqs1008-b.eqiad.wmnet
 - aqs1009-a.eqiad.wmnet
 - aqs1009-b.eqiad.wmnet
-profile::aqs::logstash_host: logstash.svc.eqiad.wmnet
 profile::aqs::cassandra_default_consistency: localQuorum
 profile::aqs::cassandra_local_dc: "%{::site}"
 profile::aqs::statsd_host: statsd.eqiad.wmnet
diff --git a/modules/profile/manifests/aqs.pp b/modules/profile/manifests/aqs.pp
index 262034e..07092e2 100644
--- a/modules/profile/manifests/aqs.pp
+++ b/modules/profile/manifests/aqs.pp
@@ -9,7 +9,7 @@
 $cassandra_user= hiera('profile::aqs::cassandra_user'),
 $cassandra_password= 
hiera('profile::aqs::cassandra_password'),
 $cassandra_seeds   = hiera('profile::aqs::seeds'),
-$logstash_host = hiera('profile::aqs::logstash_host'),
+$logstash_host = hiera('logstash_host'),
 $cassandra_default_consistency = 
hiera('profile::aqs::cassandra_default_consistency'),
 $cassandra_local_dc= 
hiera('profile::aqs::cassandra_local_dc'),
 $statsd_host   = hiera('profile::aqs::statsd_host'),

-- 
To view, visit https://gerrit.wikimedia.org/r/399624
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5aec6c6d44c253986576676309da7b3bece67e77
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Updates to enable short URLs for transliteration for crhwiki...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398832 )

Change subject: Updates to enable short URLs for transliteration for crhwiki 
production
..

Updates to enable short URLs for transliteration for crhwiki production

Add Apache rules for language variant paths for crh Wikipedia and other wikis.

Bug: T23582
Change-Id: Id2bbac3038952c495b9274aedf1c88e0f9b1d356
---
M modules/mediawiki/files/apache/sites/main.conf
1 file changed, 21 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/32/398832/1

diff --git a/modules/mediawiki/files/apache/sites/main.conf 
b/modules/mediawiki/files/apache/sites/main.conf
index 0809264..c3c0dc7 100644
--- a/modules/mediawiki/files/apache/sites/main.conf
+++ b/modules/mediawiki/files/apache/sites/main.conf
@@ -139,6 +139,9 @@
 Include "sites-enabled/wikimedia-legacy.incl"
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wiktionary.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wiktionary.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wiktionary.org/w/index.php
@@ -194,6 +197,9 @@
 RewriteRule ^/math/(.*) %{ENV:RW_PROTO}://upload.wikimedia.org/math/$1 
[R=301]
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikiquote.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikiquote.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikiquote.org/w/index.php
@@ -293,6 +299,9 @@
 RewriteRule ^/wikistats(/(.*$)|$) %{ENV:RW_PROTO}://stats.wikimedia.org/$2 
[R=302,L]
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikipedia.org/w/index.php
@@ -351,6 +360,9 @@
 RewriteRule ^/math/(.*) %{ENV:RW_PROTO}://upload.wikimedia.org/math/$1 
[R=301]
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikibooks.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikibooks.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikibooks.org/w/index.php
@@ -407,6 +419,9 @@
 RewriteRule ^/math/(.*) %{ENV:RW_PROTO}://upload.wikimedia.org/math/$1 
[R=301]
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikisource.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikisource.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikisource.org/w/index.php
@@ -463,6 +478,9 @@
 Include "sites-enabled/wikimedia-legacy.incl"
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikinews.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikinews.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikinews.org/w/index.php
@@ -520,6 +538,9 @@
 RewriteRule ^/math/(.*) %{ENV:RW_PROTO}://upload.wikimedia.org/math/$1 
[R=301]
 
 # Variant aliases
+Alias /crh /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-latn /srv/mediawiki/docroot/wikipedia.org/w/index.php
+Alias /crh-cyrl /srv/mediawiki/docroot/wikipedia.org/w/index.php
 Alias /sr /srv/mediawiki/docroot/wikiversity.org/w/index.php
 Alias /sr-ec /srv/mediawiki/docroot/wikiversity.org/w/index.php
 Alias /sr-el /srv/mediawiki/docroot/wikiversity.org/w/index.php

-- 
To view, visit https://gerrit.wikimedia.org/r/398832
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id2bbac3038952c495b9274aedf1c88e0f9b1d356
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elastic: provide elastic55 component also for Stretch

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398801 )

Change subject: elastic: provide elastic55 component also for Stretch
..


elastic: provide elastic55 component also for Stretch

While we don't yet have elasticsearch production servers on Stretch, we
do have users interested in testing it in mediawiki-vagrant.

Change-Id: I5fe06f3e8cd45d9afa7ba0ac67bee23df4018b42
---
M modules/aptrepo/files/distributions-wikimedia
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/aptrepo/files/distributions-wikimedia 
b/modules/aptrepo/files/distributions-wikimedia
index d0b3706..ef3f019 100644
--- a/modules/aptrepo/files/distributions-wikimedia
+++ b/modules/aptrepo/files/distributions-wikimedia
@@ -36,9 +36,9 @@
 AlsoAcceptFor: stretch stretch-backports
 Version: 9
 Architectures: source amd64 i386
-Components: main thirdparty/cloudera thirdparty/ci thirdparty/confluent 
thirdparty/hwraid thirdparty/k8s
+Components: main thirdparty/cloudera thirdparty/ci thirdparty/confluent 
thirdparty/hwraid thirdparty/k8s component/elastic55 thirdparty/elastic55
 UDebComponents: main
-Update: hwraid-stretch hp-mcp-stretch jenkins-stretch confluent-stretch
+Update: hwraid-stretch hp-mcp-stretch jenkins-stretch confluent-stretch 
elastic55
 # NOTE:  Cloudera does not (yet) provide Stretch packages.  We use the Jessie 
ones,
 # along with a fake equivs to satisfy hadoop-mapreduce's (unnecessary) 
dependency on libssl1.0.0
 # which is not available in Stretch.  Once Cloudera releases Stretch packages, 
we can

-- 
To view, visit https://gerrit.wikimedia.org/r/398801
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5fe06f3e8cd45d9afa7ba0ac67bee23df4018b42
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: Paladox 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elastic: provide elastic55 component also for Stretch

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398801 )

Change subject: elastic: provide elastic55 component also for Stretch
..

elastic: provide elastic55 component also for Stretch

While we don't yet have elasticsearch production servers on Stretch, we
do have users interested in testing it in mediawiki-vagrant.

Change-Id: I5fe06f3e8cd45d9afa7ba0ac67bee23df4018b42
---
M modules/aptrepo/files/distributions-wikimedia
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/01/398801/1

diff --git a/modules/aptrepo/files/distributions-wikimedia 
b/modules/aptrepo/files/distributions-wikimedia
index d0b3706..ef3f019 100644
--- a/modules/aptrepo/files/distributions-wikimedia
+++ b/modules/aptrepo/files/distributions-wikimedia
@@ -36,9 +36,9 @@
 AlsoAcceptFor: stretch stretch-backports
 Version: 9
 Architectures: source amd64 i386
-Components: main thirdparty/cloudera thirdparty/ci thirdparty/confluent 
thirdparty/hwraid thirdparty/k8s
+Components: main thirdparty/cloudera thirdparty/ci thirdparty/confluent 
thirdparty/hwraid thirdparty/k8s component/elastic55 thirdparty/elastic55
 UDebComponents: main
-Update: hwraid-stretch hp-mcp-stretch jenkins-stretch confluent-stretch
+Update: hwraid-stretch hp-mcp-stretch jenkins-stretch confluent-stretch 
elastic55
 # NOTE:  Cloudera does not (yet) provide Stretch packages.  We use the Jessie 
ones,
 # along with a fake equivs to satisfy hadoop-mapreduce's (unnecessary) 
dependency on libssl1.0.0
 # which is not available in Stretch.  Once Cloudera releases Stretch packages, 
we can

-- 
To view, visit https://gerrit.wikimedia.org/r/398801
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5fe06f3e8cd45d9afa7ba0ac67bee23df4018b42
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: move prometheus classes to conform with best ...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398444 )

Change subject: elasticsearch: move prometheus classes to conform with best 
practice
..

elasticsearch: move prometheus classes to conform with best practice

Change-Id: Ifd0a97eb3cd5446fcdbd87f7bc725710c1219237
---
M modules/profile/manifests/elasticsearch.pp
R modules/profile/manifests/prometheus/elasticsearch_exporter.pp
M modules/role/manifests/elasticsearch/beta.pp
M modules/role/manifests/elasticsearch/cirrus.pp
M modules/role/manifests/elasticsearch/relforge.pp
M modules/role/manifests/logstash/elasticsearch.pp
M modules/role/manifests/prometheus/ops.pp
7 files changed, 6 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/44/398444/1

diff --git a/modules/profile/manifests/elasticsearch.pp 
b/modules/profile/manifests/elasticsearch.pp
index f32c066..aa873c8 100644
--- a/modules/profile/manifests/elasticsearch.pp
+++ b/modules/profile/manifests/elasticsearch.pp
@@ -34,9 +34,6 @@
 $reindex_remote_whitelist = 
hiera('profile::elasticsearch::reindex_remote_whitelist'),
 $storage_device = hiera('profile::elasticsearch::storage_device'),
 ) {
-
-include ::profile::elasticsearch::monitoring
-
 $master_eligible = $::fqdn in $unicast_hosts
 
 ferm::service { 'elastic-http':
diff --git a/modules/profile/manifests/elasticsearch/monitoring.pp 
b/modules/profile/manifests/prometheus/elasticsearch_exporter.pp
similarity index 86%
rename from modules/profile/manifests/elasticsearch/monitoring.pp
rename to modules/profile/manifests/prometheus/elasticsearch_exporter.pp
index e1bf3e0..9d6445f 100644
--- a/modules/profile/manifests/elasticsearch/monitoring.pp
+++ b/modules/profile/manifests/prometheus/elasticsearch_exporter.pp
@@ -1,4 +1,4 @@
-class profile::elasticsearch::monitoring(
+class profile::prometheus::elasticsearch_exporter(
 $prometheus_nodes = hiera('prometheus_nodes'),
 ) {
 $prometheus_nodes_ferm = join($prometheus_nodes, ' ')
diff --git a/modules/role/manifests/elasticsearch/beta.pp 
b/modules/role/manifests/elasticsearch/beta.pp
index cb668a1..7717ccd 100644
--- a/modules/role/manifests/elasticsearch/beta.pp
+++ b/modules/role/manifests/elasticsearch/beta.pp
@@ -5,6 +5,7 @@
 # filtertags: labs-project-deployment-prep labs-project-search 
labs-project-math
 class role::elasticsearch::beta {
 include ::profile::elasticsearch
+include ::profile::prometheus::elasticsearch_exporter
 
 system::role { 'elasticsearch::beta':
 ensure  => 'present',
diff --git a/modules/role/manifests/elasticsearch/cirrus.pp 
b/modules/role/manifests/elasticsearch/cirrus.pp
index 699e854..f91f910 100644
--- a/modules/role/manifests/elasticsearch/cirrus.pp
+++ b/modules/role/manifests/elasticsearch/cirrus.pp
@@ -7,6 +7,7 @@
 include ::base::firewall
 include ::role::lvs::realserver
 include ::profile::elasticsearch
+include ::profile::prometheus::elasticsearch_exporter
 
 system::role { 'elasticsearch::cirrus':
 ensure  => 'present',
diff --git a/modules/role/manifests/elasticsearch/relforge.pp 
b/modules/role/manifests/elasticsearch/relforge.pp
index d88e058..80cbccd 100644
--- a/modules/role/manifests/elasticsearch/relforge.pp
+++ b/modules/role/manifests/elasticsearch/relforge.pp
@@ -6,6 +6,7 @@
 include ::standard
 include ::base::firewall
 include ::profile::elasticsearch
+include ::profile::prometheus::elasticsearch_exporter
 include ::elasticsearch::nagios::check
 include ::profile::mjolnir::kafka_daemon
 
diff --git a/modules/role/manifests/logstash/elasticsearch.pp 
b/modules/role/manifests/logstash/elasticsearch.pp
index 31a3255..390f21d 100644
--- a/modules/role/manifests/logstash/elasticsearch.pp
+++ b/modules/role/manifests/logstash/elasticsearch.pp
@@ -6,7 +6,7 @@
 class role::logstash::elasticsearch {
 include ::standard
 include ::elasticsearch::monitor::diamond
-include ::profile::elasticsearch::monitoring
+include ::profile::prometheus::elasticsearch_exporter
 include ::profile::base::firewall
 
 # the logstash cluster has 3 data nodes, and each shard has 3 replica (each
diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index bc2b7b3..30f2655 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -721,7 +721,7 @@
 prometheus::class_config { "elasticsearch_${::site}":
 dest   => "${targets_path}/elasticsearch_${::site}.yaml",
 site   => $::site,
-class_name => 'profile::elasticsearch::monitoring',
+class_name => 'profile::prometheus::elasticsearch_exporter',
 port   => '9108',
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/398444
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: configure prometheus to collect metrics from ...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398059 )

Change subject: elasticsearch: configure prometheus to collect metrics from 
elasticsearch
..


elasticsearch: configure prometheus to collect metrics from elasticsearch

This enables metric collection based on the
profile::elasticsearch::monitoring class, which is common to both logstash
and elasticsearch

Bug: T181627
Change-Id: If0b2e47f8a7584bfa62e3a48601f8c32600f728d
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 862df8a..bc2b7b3 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -721,7 +721,7 @@
 prometheus::class_config { "elasticsearch_${::site}":
 dest   => "${targets_path}/elasticsearch_${::site}.yaml",
 site   => $::site,
-class_name => 'role::logstash::elasticsearch',
+class_name => 'profile::elasticsearch::monitoring',
 port   => '9108',
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/398059
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If0b2e47f8a7584bfa62e3a48601f8c32600f728d
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: configure prometheus to collect metrics from ...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398051 )

Change subject: elasticsearch: configure prometheus to collect metrics from 
logstash
..


elasticsearch: configure prometheus to collect metrics from logstash

Bug: T181627
Change-Id: I7e3ab94a0c015823a88a79d48f7086dc79733403
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 17 insertions(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 57ddbf5..862df8a 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -709,6 +709,22 @@
 port   => '9199',
 }
 
+$elasticsearch_jobs = [
+{
+'job_name'=> 'elasticsearch',
+'scheme'  => 'http',
+'file_sd_configs' => [
+{ 'files' => [ "${targets_path}/elasticsearch_*.yaml" ]}
+],
+},
+]
+prometheus::class_config { "elasticsearch_${::site}":
+dest   => "${targets_path}/elasticsearch_${::site}.yaml",
+site   => $::site,
+class_name => 'role::logstash::elasticsearch',
+port   => '9108',
+}
+
 prometheus::server { 'ops':
 storage_encoding  => '2',
 listen_address=> '127.0.0.1:9900',
@@ -720,7 +736,7 @@
 $apache_jobs, $etcd_jobs, $etcdmirror_jobs, $pdu_jobs,
 $nginx_jobs, $pybal_jobs, $blackbox_jobs, $jmx_exporter_jobs,
 $redis_jobs, $mtail_jobs, $ldap_jobs, $ircd_jobs, $pdns_rec_jobs,
-$etherpad_jobs,
+$etherpad_jobs, $elasticsearch_jobs,
 ),
 global_config_extra   => $config_extra,
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/398051
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7e3ab94a0c015823a88a79d48f7086dc79733403
Gerrit-PatchSet: 6
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: configure prometheus to collect metrics from ...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398059 )

Change subject: elasticsearch: configure prometheus to collect metrics from 
both logstash and elasticsearch
..

elasticsearch: configure prometheus to collect metrics from both
logstash and elasticsearch

Bug: T181627
Change-Id: If0b2e47f8a7584bfa62e3a48601f8c32600f728d
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/59/398059/1

diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 862df8a..bc2b7b3 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -721,7 +721,7 @@
 prometheus::class_config { "elasticsearch_${::site}":
 dest   => "${targets_path}/elasticsearch_${::site}.yaml",
 site   => $::site,
-class_name => 'role::logstash::elasticsearch',
+class_name => 'profile::elasticsearch::monitoring',
 port   => '9108',
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/398059
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If0b2e47f8a7584bfa62e3a48601f8c32600f728d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: activate prometheus elasticsearch exporter

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398027 )

Change subject: elasticsearch: activate prometheus elasticsearch exporter
..


elasticsearch: activate prometheus elasticsearch exporter

Bug: T181627
Change-Id: If2f5e752a8248b0d1fb0d88d2bdd3c221dc09935
---
M modules/profile/manifests/elasticsearch.pp
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/profile/manifests/elasticsearch.pp 
b/modules/profile/manifests/elasticsearch.pp
index aa873c8..f32c066 100644
--- a/modules/profile/manifests/elasticsearch.pp
+++ b/modules/profile/manifests/elasticsearch.pp
@@ -34,6 +34,9 @@
 $reindex_remote_whitelist = 
hiera('profile::elasticsearch::reindex_remote_whitelist'),
 $storage_device = hiera('profile::elasticsearch::storage_device'),
 ) {
+
+include ::profile::elasticsearch::monitoring
+
 $master_eligible = $::fqdn in $unicast_hosts
 
 ferm::service { 'elastic-http':

-- 
To view, visit https://gerrit.wikimedia.org/r/398027
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If2f5e752a8248b0d1fb0d88d2bdd3c221dc09935
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: activate prometheus elasticsearch exporter

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398026 )

Change subject: logstash: activate prometheus elasticsearch exporter
..


logstash: activate prometheus elasticsearch exporter

Bug: T181627
Change-Id: I8421cb77b1a9cb54bfa7120e5a3d309598e3b8be
---
M modules/role/manifests/logstash/elasticsearch.pp
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/modules/role/manifests/logstash/elasticsearch.pp 
b/modules/role/manifests/logstash/elasticsearch.pp
index 50a9fd3..31a3255 100644
--- a/modules/role/manifests/logstash/elasticsearch.pp
+++ b/modules/role/manifests/logstash/elasticsearch.pp
@@ -6,6 +6,7 @@
 class role::logstash::elasticsearch {
 include ::standard
 include ::elasticsearch::monitor::diamond
+include ::profile::elasticsearch::monitoring
 include ::profile::base::firewall
 
 # the logstash cluster has 3 data nodes, and each shard has 3 replica (each

-- 
To view, visit https://gerrit.wikimedia.org/r/398026
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8421cb77b1a9cb54bfa7120e5a3d309598e3b8be
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: deploy prometheus-elasticsearch-exporter

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398025 )

Change subject: elasticsearch: deploy prometheus-elasticsearch-exporter
..


elasticsearch: deploy prometheus-elasticsearch-exporter

This just creates the class to deploy the exporter, subsequent CR will
enable it for cirrus / logstash.

Bug: T181627
Change-Id: I70f934e129df31fffbefdb2ed152bac8fedbcffa
---
A modules/profile/manifests/elasticsearch/monitoring.pp
A modules/prometheus/manifests/elasticsearch_exporter.pp
2 files changed, 22 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/modules/profile/manifests/elasticsearch/monitoring.pp 
b/modules/profile/manifests/elasticsearch/monitoring.pp
new file mode 100644
index 000..e1bf3e0
--- /dev/null
+++ b/modules/profile/manifests/elasticsearch/monitoring.pp
@@ -0,0 +1,14 @@
+class profile::elasticsearch::monitoring(
+$prometheus_nodes = hiera('prometheus_nodes'),
+) {
+$prometheus_nodes_ferm = join($prometheus_nodes, ' ')
+
+class { '::prometheus::elasticsearch_exporter': }
+
+ferm::service { 'prometheus_elasticsearch_exporter':
+proto  => 'tcp',
+port   => '9108',
+srange => "@resolve((${prometheus_nodes_ferm}))",
+}
+
+}
diff --git a/modules/prometheus/manifests/elasticsearch_exporter.pp 
b/modules/prometheus/manifests/elasticsearch_exporter.pp
new file mode 100644
index 000..fff7625
--- /dev/null
+++ b/modules/prometheus/manifests/elasticsearch_exporter.pp
@@ -0,0 +1,8 @@
+class prometheus::elasticsearch_exporter {
+  ensure_packages('prometheus-elasticsearch-exporter')
+
+  service { 'prometheus-elasticsearch-exporter':
+ensure  => 'running',
+require => Package['prometheus-elasticsearch-exporter'],
+  }
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/398025
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I70f934e129df31fffbefdb2ed152bac8fedbcffa
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: configure prometheus to collect metrics from ...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398051 )

Change subject: elasticsearch: configure prometheus to collect metrics from 
elasticsearch
..

elasticsearch: configure prometheus to collect metrics from elasticsearch

Bug: T181627
Change-Id: I7e3ab94a0c015823a88a79d48f7086dc79733403
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 17 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/51/398051/1

diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 7fcd48b..cf7c3df 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -652,6 +652,22 @@
 port   => '9142',
 }
 
+$elasticsearch_jobs = [
+{
+'job_name'=> 'elasticsearch',
+'scheme'  => 'http',
+'file_sd_configs' => [
+{ 'files' => [ "${targets_path}/elasticsearch_*.yaml" ]}
+],
+},
+]
+prometheus::class_config { "elasticsearch_${::site}":
+dest   => "${targets_path}/elasticsearch_${::site}.yaml",
+site   => $::site,
+class_name => 'profile::elasticsearch::monitoring',
+port   => '9108',
+}
+
 prometheus::server { 'ops':
 storage_encoding  => '2',
 listen_address=> '127.0.0.1:9900',
@@ -662,7 +678,7 @@
 $mysql_jobs, $varnish_jobs, $memcached_jobs, $hhvm_jobs,
 $apache_jobs, $etcd_jobs, $etcdmirror_jobs, $pdu_jobs,
 $nginx_jobs, $pybal_jobs, $blackbox_jobs, $jmx_exporter_jobs,
-$redis_jobs, $mtail_jobs, $ldap_jobs,
+$redis_jobs, $mtail_jobs, $ldap_jobs, $elasticsearch_jobs,
 ),
 global_config_extra   => $config_extra,
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/398051
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7e3ab94a0c015823a88a79d48f7086dc79733403
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: activate prometheus elasticsearch exporter

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398027 )

Change subject: elasticsearch: activate prometheus elasticsearch exporter
..

elasticsearch: activate prometheus elasticsearch exporter

Bug: T181627
Change-Id: If2f5e752a8248b0d1fb0d88d2bdd3c221dc09935
---
M modules/profile/manifests/elasticsearch.pp
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/27/398027/1

diff --git a/modules/profile/manifests/elasticsearch.pp 
b/modules/profile/manifests/elasticsearch.pp
index aa873c8..f32c066 100644
--- a/modules/profile/manifests/elasticsearch.pp
+++ b/modules/profile/manifests/elasticsearch.pp
@@ -34,6 +34,9 @@
 $reindex_remote_whitelist = 
hiera('profile::elasticsearch::reindex_remote_whitelist'),
 $storage_device = hiera('profile::elasticsearch::storage_device'),
 ) {
+
+include ::profile::elasticsearch::monitoring
+
 $master_eligible = $::fqdn in $unicast_hosts
 
 ferm::service { 'elastic-http':

-- 
To view, visit https://gerrit.wikimedia.org/r/398027
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If2f5e752a8248b0d1fb0d88d2bdd3c221dc09935
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: activate prometheus elasticsearch exporter

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398026 )

Change subject: logstash: activate prometheus elasticsearch exporter
..

logstash: activate prometheus elasticsearch exporter

Bug: T181627
Change-Id: I8421cb77b1a9cb54bfa7120e5a3d309598e3b8be
---
M modules/role/manifests/logstash/elasticsearch.pp
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/26/398026/1

diff --git a/modules/role/manifests/logstash/elasticsearch.pp 
b/modules/role/manifests/logstash/elasticsearch.pp
index 50a9fd3..31a3255 100644
--- a/modules/role/manifests/logstash/elasticsearch.pp
+++ b/modules/role/manifests/logstash/elasticsearch.pp
@@ -6,6 +6,7 @@
 class role::logstash::elasticsearch {
 include ::standard
 include ::elasticsearch::monitor::diamond
+include ::profile::elasticsearch::monitoring
 include ::profile::base::firewall
 
 # the logstash cluster has 3 data nodes, and each shard has 3 replica (each

-- 
To view, visit https://gerrit.wikimedia.org/r/398026
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I8421cb77b1a9cb54bfa7120e5a3d309598e3b8be
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: deploy prometheus-elasticsearch-exporter

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398025 )

Change subject: elasticsearch: deploy prometheus-elasticsearch-exporter
..

elasticsearch: deploy prometheus-elasticsearch-exporter

This just creates the class to deploy the exporter, subsequent CR will
enable it for cirrus / logstash.

Bug: T181627
Change-Id: I70f934e129df31fffbefdb2ed152bac8fedbcffa
---
A modules/elasticsearch/manifests/monitor/prometheus.pp
A modules/profile/manifests/elasticsearch/monitoring.pp
2 files changed, 21 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/25/398025/1

diff --git a/modules/elasticsearch/manifests/monitor/prometheus.pp 
b/modules/elasticsearch/manifests/monitor/prometheus.pp
new file mode 100644
index 000..3709ffb
--- /dev/null
+++ b/modules/elasticsearch/manifests/monitor/prometheus.pp
@@ -0,0 +1,7 @@
+class elasticsearch::monitor::prometheus {
+ensure_packages('prometheus-elasticsearch-exporter')
+
+service { 'prometheus-elasticsearch-exporter':
+ensure => 'running',
+}
+}
diff --git a/modules/profile/manifests/elasticsearch/monitoring.pp 
b/modules/profile/manifests/elasticsearch/monitoring.pp
new file mode 100644
index 000..1f2beca
--- /dev/null
+++ b/modules/profile/manifests/elasticsearch/monitoring.pp
@@ -0,0 +1,14 @@
+class profile::elasticsearch::monitoring(
+$prometheus_nodes = hiera('prometheus_nodes'),
+) {
+$prometheus_nodes_ferm = join($prometheus_nodes, ' ')
+
+class { 'profile::elasticsearch::monitoring::prometheus': }
+
+ferm::service { 'prometheus_elasticsearch_exporter':
+proto  => 'tcp',
+port   => '9108',
+srange => "@resolve((${prometheus_nodes_ferm}))",
+}
+
+}
\ No newline at end of file

-- 
To view, visit https://gerrit.wikimedia.org/r/398025
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I70f934e129df31fffbefdb2ed152bac8fedbcffa
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: maps: update style to match latest kartotherian version

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397846 )

Change subject: maps: update style to match latest kartotherian version
..


maps: update style to match latest kartotherian version

Bug: T162241
Change-Id: I4731943f4da1057e545bf4bd203b2f093e7ea3c0
---
M hieradata/role/common/maps/test/master.yaml
M hieradata/role/common/maps/test/slave.yaml
M hieradata/role/common/maps/test/vectortiles_master.yaml
M modules/profile/manifests/maps/apps.pp
4 files changed, 6 insertions(+), 6 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/role/common/maps/test/master.yaml 
b/hieradata/role/common/maps/test/master.yaml
index b15c7d7..05aaf58 100644
--- a/hieradata/role/common/maps/test/master.yaml
+++ b/hieradata/role/common/maps/test/master.yaml
@@ -29,5 +29,5 @@
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
 
-profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::kartotherian_style: '''@kartotherian/osm-bright-style'''
 profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/slave.yaml 
b/hieradata/role/common/maps/test/slave.yaml
index cbfa643..780bf31 100644
--- a/hieradata/role/common/maps/test/slave.yaml
+++ b/hieradata/role/common/maps/test/slave.yaml
@@ -25,5 +25,5 @@
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
 
-profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::kartotherian_style: '''@kartotherian/osm-bright-style'''
 profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/vectortiles_master.yaml 
b/hieradata/role/common/maps/test/vectortiles_master.yaml
index 885dab3..f81ac01 100644
--- a/hieradata/role/common/maps/test/vectortiles_master.yaml
+++ b/hieradata/role/common/maps/test/vectortiles_master.yaml
@@ -31,5 +31,5 @@
 
 standard::has_ganglia: false
 
-profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::kartotherian_style: '''@kartotherian/osm-bright-style'''
 profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/modules/profile/manifests/maps/apps.pp 
b/modules/profile/manifests/maps/apps.pp
index 26b4c7d..2788551 100644
--- a/modules/profile/manifests/maps/apps.pp
+++ b/modules/profile/manifests/maps/apps.pp
@@ -23,17 +23,17 @@
 profile::maps::sources_config { 'kartotherian':
 mode   => 'kartotherian',
 storage_id => $storage_id,
-style  => 'osm-bright-style',
+style  => $kartotherian_style,
 }
 profile::maps::sources_config { 'tilerator':
 mode   => 'tilerator',
 storage_id => $storage_id,
-style  => '\'@kartotherian/osm-bright-style\'',
+style  => $tilerator_style,
 }
 profile::maps::sources_config { 'tileratorui':
 mode   => 'tilerator',
 storage_id => $storage_id,
-style  => '\'@kartotherian/osm-bright-style\'',
+style  => $tilerator_style,
 }
 
 class { '::tilerator':

-- 
To view, visit https://gerrit.wikimedia.org/r/397846
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4731943f4da1057e545bf4bd203b2f093e7ea3c0
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: maps: update style to match latest kartotherian version

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/397846 )

Change subject: maps: update style to match latest kartotherian version
..

maps: update style to match latest kartotherian version

Bug: T162241
Change-Id: I4731943f4da1057e545bf4bd203b2f093e7ea3c0
---
M hieradata/role/common/maps/test/master.yaml
M hieradata/role/common/maps/test/slave.yaml
M hieradata/role/common/maps/test/vectortiles_master.yaml
3 files changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/46/397846/1

diff --git a/hieradata/role/common/maps/test/master.yaml 
b/hieradata/role/common/maps/test/master.yaml
index b15c7d7..05aaf58 100644
--- a/hieradata/role/common/maps/test/master.yaml
+++ b/hieradata/role/common/maps/test/master.yaml
@@ -29,5 +29,5 @@
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
 
-profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::kartotherian_style: '''@kartotherian/osm-bright-style'''
 profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/slave.yaml 
b/hieradata/role/common/maps/test/slave.yaml
index cbfa643..780bf31 100644
--- a/hieradata/role/common/maps/test/slave.yaml
+++ b/hieradata/role/common/maps/test/slave.yaml
@@ -25,5 +25,5 @@
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
 
-profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::kartotherian_style: '''@kartotherian/osm-bright-style'''
 profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/vectortiles_master.yaml 
b/hieradata/role/common/maps/test/vectortiles_master.yaml
index 885dab3..f81ac01 100644
--- a/hieradata/role/common/maps/test/vectortiles_master.yaml
+++ b/hieradata/role/common/maps/test/vectortiles_master.yaml
@@ -31,5 +31,5 @@
 
 standard::has_ganglia: false
 
-profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::kartotherian_style: '''@kartotherian/osm-bright-style'''
 profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''

-- 
To view, visit https://gerrit.wikimedia.org/r/397846
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4731943f4da1057e545bf4bd203b2f093e7ea3c0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: maps: extract style used by kartotherian / tilerator as para...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397836 )

Change subject: maps: extract style used by kartotherian / tilerator as 
parameter
..


maps: extract style used by kartotherian / tilerator as parameter

Bug: T162241
Change-Id: I135c644fc312f1586156d0048f00074d6ec45a55
---
M hieradata/role/common/maps/master.yaml
M hieradata/role/common/maps/slave.yaml
M hieradata/role/common/maps/test/master.yaml
M hieradata/role/common/maps/test/slave.yaml
M hieradata/role/common/maps/test/vectortiles_master.yaml
M modules/profile/manifests/maps/apps.pp
M modules/profile/manifests/maps/sources_config.pp
M modules/profile/templates/maps/sources.yaml.erb
8 files changed, 29 insertions(+), 9 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/role/common/maps/master.yaml 
b/hieradata/role/common/maps/master.yaml
index b8e28e8..6dd07a3 100644
--- a/hieradata/role/common/maps/master.yaml
+++ b/hieradata/role/common/maps/master.yaml
@@ -28,3 +28,6 @@
 profile::redis::master::instances: ['6379']
 profile::redis::master::settings:
   bind: "0.0.0.0"
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/slave.yaml 
b/hieradata/role/common/maps/slave.yaml
index 010758d..155b8fa 100644
--- a/hieradata/role/common/maps/slave.yaml
+++ b/hieradata/role/common/maps/slave.yaml
@@ -24,3 +24,6 @@
 
 cassandra::metrics::blacklist:
   - .*\.metrics\.Table\..*$
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/master.yaml 
b/hieradata/role/common/maps/test/master.yaml
index 6cf990b..b15c7d7 100644
--- a/hieradata/role/common/maps/test/master.yaml
+++ b/hieradata/role/common/maps/test/master.yaml
@@ -28,3 +28,6 @@
 
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/slave.yaml 
b/hieradata/role/common/maps/test/slave.yaml
index 4cd555b..cbfa643 100644
--- a/hieradata/role/common/maps/test/slave.yaml
+++ b/hieradata/role/common/maps/test/slave.yaml
@@ -24,3 +24,6 @@
 
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/vectortiles_master.yaml 
b/hieradata/role/common/maps/test/vectortiles_master.yaml
index 317a441..885dab3 100644
--- a/hieradata/role/common/maps/test/vectortiles_master.yaml
+++ b/hieradata/role/common/maps/test/vectortiles_master.yaml
@@ -30,3 +30,6 @@
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
 
 standard::has_ganglia: false
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/modules/profile/manifests/maps/apps.pp 
b/modules/profile/manifests/maps/apps.pp
index 327af52..26b4c7d 100644
--- a/modules/profile/manifests/maps/apps.pp
+++ b/modules/profile/manifests/maps/apps.pp
@@ -8,6 +8,10 @@
 $pgsql_tileratorui_pass = 
hiera('profile::maps::osm_master::tileratorui_pass'),
 $redis_server = hiera('profile::maps::apps::redis_server'),
 $redis_pass = hiera('profile::maps::apps::redis_pass'),
+# for "historical reasons", tilerator and kartotherian load styles 
differently.
+# once this is fixed in application code, the 2 parameters below should be 
merged.
+$kartotherian_style = hiera('profile::maps::apps::kartotherian_style'),
+$tilerator_style = hiera('profile::maps::apps::tilerator_style'),
 ) {
 
 $conf_sources = 'sources.prod2.yaml'
@@ -19,14 +23,17 @@
 profile::maps::sources_config { 'kartotherian':
 mode   => 'kartotherian',
 storage_id => $storage_id,
+style  => 'osm-bright-style',
 }
 profile::maps::sources_config { 'tilerator':
 mode   => 'tilerator',
 storage_id => $storage_id,
+style  => '\'@kartotherian/osm-bright-style\'',
 }
 profile::maps::sources_config { 'tileratorui':
 mode   => 'tilerator',
 storage_id => $storage_id,
+style  => '\'@kartotherian/osm-bright-style\'',
 }
 
 class { '::tilerator':
diff --git a/modules/profile/manifests/maps/sources_config.pp 
b/modules/profile/manifests/maps/sources_config.pp
index 8780db3..a960c56 100644
--- a/modules/profile/manifests/maps/sources_config.pp
+++ 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: maps: extract style used by kartotherian / tilerator as para...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/397836 )

Change subject: maps: extract style used by kartotherian / tilerator as 
parameter
..

maps: extract style used by kartotherian / tilerator as parameter

Bug: T162241
Change-Id: I135c644fc312f1586156d0048f00074d6ec45a55
---
M hieradata/role/common/maps/master.yaml
M hieradata/role/common/maps/slave.yaml
M hieradata/role/common/maps/test/master.yaml
M hieradata/role/common/maps/test/slave.yaml
M hieradata/role/common/maps/test/vectortiles_master.yaml
M modules/profile/manifests/maps/apps.pp
M modules/profile/manifests/maps/sources_config.pp
M modules/profile/templates/maps/sources.yaml.erb
8 files changed, 29 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/36/397836/1

diff --git a/hieradata/role/common/maps/master.yaml 
b/hieradata/role/common/maps/master.yaml
index b8e28e8..6dd07a3 100644
--- a/hieradata/role/common/maps/master.yaml
+++ b/hieradata/role/common/maps/master.yaml
@@ -28,3 +28,6 @@
 profile::redis::master::instances: ['6379']
 profile::redis::master::settings:
   bind: "0.0.0.0"
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/slave.yaml 
b/hieradata/role/common/maps/slave.yaml
index 010758d..155b8fa 100644
--- a/hieradata/role/common/maps/slave.yaml
+++ b/hieradata/role/common/maps/slave.yaml
@@ -24,3 +24,6 @@
 
 cassandra::metrics::blacklist:
   - .*\.metrics\.Table\..*$
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/master.yaml 
b/hieradata/role/common/maps/test/master.yaml
index 6cf990b..b15c7d7 100644
--- a/hieradata/role/common/maps/test/master.yaml
+++ b/hieradata/role/common/maps/test/master.yaml
@@ -28,3 +28,6 @@
 
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/slave.yaml 
b/hieradata/role/common/maps/test/slave.yaml
index 4cd555b..cbfa643 100644
--- a/hieradata/role/common/maps/test/slave.yaml
+++ b/hieradata/role/common/maps/test/slave.yaml
@@ -24,3 +24,6 @@
 
 profile::maps::postgresql_common::shared_buffers: '4GB'
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/hieradata/role/common/maps/test/vectortiles_master.yaml 
b/hieradata/role/common/maps/test/vectortiles_master.yaml
index 317a441..885dab3 100644
--- a/hieradata/role/common/maps/test/vectortiles_master.yaml
+++ b/hieradata/role/common/maps/test/vectortiles_master.yaml
@@ -30,3 +30,6 @@
 profile::maps::postgresql_common::maintenance_work_mem: '3GB'
 
 standard::has_ganglia: false
+
+profile::maps::apps::kartotherian_style: 'osm-bright-style'
+profile::maps::apps::tilerator_style: '''@kartotherian/osm-bright-style'''
diff --git a/modules/profile/manifests/maps/apps.pp 
b/modules/profile/manifests/maps/apps.pp
index 327af52..26b4c7d 100644
--- a/modules/profile/manifests/maps/apps.pp
+++ b/modules/profile/manifests/maps/apps.pp
@@ -8,6 +8,10 @@
 $pgsql_tileratorui_pass = 
hiera('profile::maps::osm_master::tileratorui_pass'),
 $redis_server = hiera('profile::maps::apps::redis_server'),
 $redis_pass = hiera('profile::maps::apps::redis_pass'),
+# for "historical reasons", tilerator and kartotherian load styles 
differently.
+# once this is fixed in application code, the 2 parameters below should be 
merged.
+$kartotherian_style = hiera('profile::maps::apps::kartotherian_style'),
+$tilerator_style = hiera('profile::maps::apps::tilerator_style'),
 ) {
 
 $conf_sources = 'sources.prod2.yaml'
@@ -19,14 +23,17 @@
 profile::maps::sources_config { 'kartotherian':
 mode   => 'kartotherian',
 storage_id => $storage_id,
+style  => 'osm-bright-style',
 }
 profile::maps::sources_config { 'tilerator':
 mode   => 'tilerator',
 storage_id => $storage_id,
+style  => '\'@kartotherian/osm-bright-style\'',
 }
 profile::maps::sources_config { 'tileratorui':
 mode   => 'tilerator',
 storage_id => $storage_id,
+style  => '\'@kartotherian/osm-bright-style\'',
 }
 
 class { '::tilerator':
diff --git a/modules/profile/manifests/maps/sources_config.pp 
b/modules/profile/manifests/maps/sources_config.pp
index 8780db3..a960c56 100644
--- a/modules/profile/manifests/maps/sources_config.pp
+++ 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: admin: allow Paul Norman (pnorman) to deploy kartotherian / ...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395481 )

Change subject: admin: allow Paul Norman (pnorman) to deploy kartotherian / 
tilerator
..


admin: allow Paul Norman (pnorman) to deploy kartotherian / tilerator

Bug: T182066
Change-Id: I0025d1391302217e8a18c1457f2ec7877c500266
---
M modules/admin/data/data.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index ca88681..419e22a 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -64,7 +64,7 @@
   mobrovac, nuria, thcipriani, joal, eevans, mforns, dpatrick, 
dcausse,
   bsitzmann, mholloway-shell, dduvall, dereckson,
   ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko, 
pmiazga,
-  musikanimal, jdrewniak]
+  musikanimal, jdrewniak, pnorman]
 privileges: ['ALL = (www-data,apache,mwdeploy,l10nupdate) NOPASSWD: ALL',
  'ALL = NOPASSWD: /sbin/restart hhvm',
  'ALL = NOPASSWD: /sbin/start hhvm',

-- 
To view, visit https://gerrit.wikimedia.org/r/395481
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0025d1391302217e8a18c1457f2ec7877c500266
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Cmjohnson 
Gerrit-Reviewer: Dzahn 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Greg Grossmeier 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: adding "su" directive to log rotation

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/396419 )

Change subject: wdqs: adding "su" directive to log rotation
..


wdqs: adding "su" directive to log rotation

categories update logs should be rotated as blazegraph

Change-Id: I6944cd67c44a3c1ec11e44b96e7778dc339a29db
---
M modules/wdqs/manifests/gui.pp
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index d842eb5..458e38e 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -96,6 +96,7 @@
 not_if_empty => true,
 rotate   => 3,
 compress => true,
+su   => "${username} wikidev",
 }
 
 # Remove categories*.log files after 30 days.
@@ -106,5 +107,6 @@
 not_if_empty => true,
 rotate   => 0,
 max_age  => 30,
+su   => "${username} wikidev",
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/396419
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6944cd67c44a3c1ec11e44b96e7778dc339a29db
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Smalyshev 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: adding "su" directive to log rotation

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/396419 )

Change subject: wdqs: adding "su" directive to log rotation
..

wdqs: adding "su" directive to log rotation

categories update logs should be rotated as blazegraph

Change-Id: I6944cd67c44a3c1ec11e44b96e7778dc339a29db
---
M modules/wdqs/manifests/gui.pp
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/19/396419/1

diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index d842eb5..458e38e 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -96,6 +96,7 @@
 not_if_empty => true,
 rotate   => 3,
 compress => true,
+su   => "${username} wikidev",
 }
 
 # Remove categories*.log files after 30 days.
@@ -106,5 +107,6 @@
 not_if_empty => true,
 rotate   => 0,
 max_age  => 30,
+su   => "${username} wikidev",
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/396419
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6944cd67c44a3c1ec11e44b96e7778dc339a29db
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...deploy[master]: Update maps/tilerator/package to fc4838c

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/396394 )

Change subject: Update maps/tilerator/package to fc4838c
..


Update maps/tilerator/package to fc4838c

List of changes:
fc4838c delegate all logic to main tilerator package
xxx Update node module dependencies

Change-Id: I7084b2059bca8ee96d9b249a82d08360773b9916
---
A node_modules/aws-sdk/.changes/2.164.0.json
A node_modules/aws-sdk/.changes/2.165.0.json
A node_modules/aws-sdk/.changes/2.166.0.json
M node_modules/aws-sdk/apis/clouddirectory-2016-05-10.min.json
M node_modules/aws-sdk/apis/email-2010-12-01.min.json
M node_modules/aws-sdk/apis/email-2010-12-01.paginators.json
M node_modules/aws-sdk/apis/es-2015-01-01.min.json
M node_modules/aws-sdk/apis/iot-2015-05-28.min.json
M node_modules/aws-sdk/apis/metadata.json
M node_modules/aws-sdk/apis/sagemaker-2017-07-24.waiters2.json
M node_modules/aws-sdk/apis/servicecatalog-2015-12-10.min.json
A node_modules/aws-sdk/apis/servicediscovery-2017-03-14.examples.json
A node_modules/aws-sdk/apis/servicediscovery-2017-03-14.min.json
A node_modules/aws-sdk/apis/servicediscovery-2017-03-14.paginators.json
M node_modules/aws-sdk/clients/all.d.ts
M node_modules/aws-sdk/clients/all.js
M node_modules/aws-sdk/clients/clouddirectory.d.ts
M node_modules/aws-sdk/clients/elasticbeanstalk.d.ts
M node_modules/aws-sdk/clients/es.d.ts
M node_modules/aws-sdk/clients/iot.d.ts
M node_modules/aws-sdk/clients/sagemaker.d.ts
M node_modules/aws-sdk/clients/servicecatalog.d.ts
A node_modules/aws-sdk/clients/servicediscovery.d.ts
A node_modules/aws-sdk/clients/servicediscovery.js
M node_modules/aws-sdk/clients/ses.d.ts
M node_modules/aws-sdk/dist/aws-sdk-core-react-native.js
M node_modules/aws-sdk/dist/aws-sdk-react-native.js
M node_modules/aws-sdk/dist/aws-sdk.js
M node_modules/aws-sdk/dist/aws-sdk.min.js
M node_modules/aws-sdk/lib/config_service_placeholders.d.ts
M node_modules/aws-sdk/lib/core.js
M node_modules/aws-sdk/package.json
M node_modules/gc-stats/.travis.yml
M node_modules/gc-stats/appveyor.yml
D 
node_modules/gc-stats/build/gcstats/v1.0.2/Release/node-v48-linux-x64/gcstats.node
A 
node_modules/gc-stats/build/gcstats/v1.1.0/Release/node-v48-linux-x64/gcstats.node
D node_modules/gc-stats/node_modules/.bin/mkdirp
D node_modules/gc-stats/node_modules/.bin/uuid
D node_modules/gc-stats/node_modules/debug/.coveralls.yml
D node_modules/gc-stats/node_modules/debug/.eslintrc
D node_modules/gc-stats/node_modules/debug/.npmignore
D node_modules/gc-stats/node_modules/debug/.travis.yml
D node_modules/gc-stats/node_modules/debug/LICENSE
D node_modules/gc-stats/node_modules/debug/Makefile
D node_modules/gc-stats/node_modules/debug/component.json
D node_modules/gc-stats/node_modules/debug/karma.conf.js
D node_modules/gc-stats/node_modules/debug/node.js
D node_modules/gc-stats/node_modules/debug/package.json
D node_modules/gc-stats/node_modules/debug/src/browser.js
D node_modules/gc-stats/node_modules/debug/src/debug.js
D node_modules/gc-stats/node_modules/debug/src/index.js
D node_modules/gc-stats/node_modules/debug/src/node.js
D node_modules/gc-stats/node_modules/extend/.eslintrc
D node_modules/gc-stats/node_modules/extend/.jscs.json
D node_modules/gc-stats/node_modules/extend/.npmignore
D node_modules/gc-stats/node_modules/extend/.travis.yml
D node_modules/gc-stats/node_modules/extend/LICENSE
D node_modules/gc-stats/node_modules/extend/component.json
D node_modules/gc-stats/node_modules/extend/index.js
D node_modules/gc-stats/node_modules/extend/package.json
D node_modules/gc-stats/node_modules/extsprintf/LICENSE
D node_modules/gc-stats/node_modules/extsprintf/Makefile
D node_modules/gc-stats/node_modules/extsprintf/Makefile.deps
D node_modules/gc-stats/node_modules/extsprintf/Makefile.targ
D node_modules/gc-stats/node_modules/extsprintf/examples/simple.js
D node_modules/gc-stats/node_modules/extsprintf/jsl.node.conf
D node_modules/gc-stats/node_modules/extsprintf/lib/extsprintf.js
D node_modules/gc-stats/node_modules/extsprintf/package.json
M node_modules/gc-stats/node_modules/glob/package.json
D node_modules/gc-stats/node_modules/hawk/.npmignore
D node_modules/gc-stats/node_modules/hawk/.travis.yml
D node_modules/gc-stats/node_modules/hawk/LICENSE
D node_modules/gc-stats/node_modules/hawk/bower.json
D node_modules/gc-stats/node_modules/hawk/component.json
D node_modules/gc-stats/node_modules/hawk/dist/client.js
D node_modules/gc-stats/node_modules/hawk/example/usage.js
D node_modules/gc-stats/node_modules/hawk/images/hawk.png
D node_modules/gc-stats/node_modules/hawk/images/logo.png
D node_modules/gc-stats/node_modules/hawk/lib/browser.js
D node_modules/gc-stats/node_modules/hawk/lib/client.js
D node_modules/gc-stats/node_modules/hawk/lib/crypto.js
D node_modules/gc-stats/node_modules/hawk/lib/index.js
D node_modules/gc-stats/node_modules/hawk/lib/server.js
D node_modules/gc-stats/node_modules/hawk/lib/utils.js
D 

[MediaWiki-commits] [Gerrit] maps...deploy[master]: Update maps/tilerator/package to fc4838c

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/396394 )

Change subject: Update maps/tilerator/package to fc4838c
..

Update maps/tilerator/package to fc4838c

List of changes:
fc4838c delegate all logic to main tilerator package
xxx Update node module dependencies

Change-Id: I7084b2059bca8ee96d9b249a82d08360773b9916
---
A node_modules/aws-sdk/.changes/2.164.0.json
A node_modules/aws-sdk/.changes/2.165.0.json
A node_modules/aws-sdk/.changes/2.166.0.json
M node_modules/aws-sdk/apis/clouddirectory-2016-05-10.min.json
M node_modules/aws-sdk/apis/email-2010-12-01.min.json
M node_modules/aws-sdk/apis/email-2010-12-01.paginators.json
M node_modules/aws-sdk/apis/es-2015-01-01.min.json
M node_modules/aws-sdk/apis/iot-2015-05-28.min.json
M node_modules/aws-sdk/apis/metadata.json
M node_modules/aws-sdk/apis/sagemaker-2017-07-24.waiters2.json
M node_modules/aws-sdk/apis/servicecatalog-2015-12-10.min.json
A node_modules/aws-sdk/apis/servicediscovery-2017-03-14.examples.json
A node_modules/aws-sdk/apis/servicediscovery-2017-03-14.min.json
A node_modules/aws-sdk/apis/servicediscovery-2017-03-14.paginators.json
M node_modules/aws-sdk/clients/all.d.ts
M node_modules/aws-sdk/clients/all.js
M node_modules/aws-sdk/clients/clouddirectory.d.ts
M node_modules/aws-sdk/clients/elasticbeanstalk.d.ts
M node_modules/aws-sdk/clients/es.d.ts
M node_modules/aws-sdk/clients/iot.d.ts
M node_modules/aws-sdk/clients/sagemaker.d.ts
M node_modules/aws-sdk/clients/servicecatalog.d.ts
A node_modules/aws-sdk/clients/servicediscovery.d.ts
A node_modules/aws-sdk/clients/servicediscovery.js
M node_modules/aws-sdk/clients/ses.d.ts
M node_modules/aws-sdk/dist/aws-sdk-core-react-native.js
M node_modules/aws-sdk/dist/aws-sdk-react-native.js
M node_modules/aws-sdk/dist/aws-sdk.js
M node_modules/aws-sdk/dist/aws-sdk.min.js
M node_modules/aws-sdk/lib/config_service_placeholders.d.ts
M node_modules/aws-sdk/lib/core.js
M node_modules/aws-sdk/package.json
M node_modules/gc-stats/.travis.yml
M node_modules/gc-stats/appveyor.yml
D 
node_modules/gc-stats/build/gcstats/v1.0.2/Release/node-v48-linux-x64/gcstats.node
A 
node_modules/gc-stats/build/gcstats/v1.1.0/Release/node-v48-linux-x64/gcstats.node
D node_modules/gc-stats/node_modules/.bin/mkdirp
D node_modules/gc-stats/node_modules/.bin/uuid
D node_modules/gc-stats/node_modules/debug/.coveralls.yml
D node_modules/gc-stats/node_modules/debug/.eslintrc
D node_modules/gc-stats/node_modules/debug/.npmignore
D node_modules/gc-stats/node_modules/debug/.travis.yml
D node_modules/gc-stats/node_modules/debug/LICENSE
D node_modules/gc-stats/node_modules/debug/Makefile
D node_modules/gc-stats/node_modules/debug/component.json
D node_modules/gc-stats/node_modules/debug/karma.conf.js
D node_modules/gc-stats/node_modules/debug/node.js
D node_modules/gc-stats/node_modules/debug/package.json
D node_modules/gc-stats/node_modules/debug/src/browser.js
D node_modules/gc-stats/node_modules/debug/src/debug.js
D node_modules/gc-stats/node_modules/debug/src/index.js
D node_modules/gc-stats/node_modules/debug/src/node.js
D node_modules/gc-stats/node_modules/extend/.eslintrc
D node_modules/gc-stats/node_modules/extend/.jscs.json
D node_modules/gc-stats/node_modules/extend/.npmignore
D node_modules/gc-stats/node_modules/extend/.travis.yml
D node_modules/gc-stats/node_modules/extend/LICENSE
D node_modules/gc-stats/node_modules/extend/component.json
D node_modules/gc-stats/node_modules/extend/index.js
D node_modules/gc-stats/node_modules/extend/package.json
D node_modules/gc-stats/node_modules/extsprintf/LICENSE
D node_modules/gc-stats/node_modules/extsprintf/Makefile
D node_modules/gc-stats/node_modules/extsprintf/Makefile.deps
D node_modules/gc-stats/node_modules/extsprintf/Makefile.targ
D node_modules/gc-stats/node_modules/extsprintf/examples/simple.js
D node_modules/gc-stats/node_modules/extsprintf/jsl.node.conf
D node_modules/gc-stats/node_modules/extsprintf/lib/extsprintf.js
D node_modules/gc-stats/node_modules/extsprintf/package.json
M node_modules/gc-stats/node_modules/glob/package.json
D node_modules/gc-stats/node_modules/hawk/.npmignore
D node_modules/gc-stats/node_modules/hawk/.travis.yml
D node_modules/gc-stats/node_modules/hawk/LICENSE
D node_modules/gc-stats/node_modules/hawk/bower.json
D node_modules/gc-stats/node_modules/hawk/component.json
D node_modules/gc-stats/node_modules/hawk/dist/client.js
D node_modules/gc-stats/node_modules/hawk/example/usage.js
D node_modules/gc-stats/node_modules/hawk/images/hawk.png
D node_modules/gc-stats/node_modules/hawk/images/logo.png
D node_modules/gc-stats/node_modules/hawk/lib/browser.js
D node_modules/gc-stats/node_modules/hawk/lib/client.js
D node_modules/gc-stats/node_modules/hawk/lib/crypto.js
D node_modules/gc-stats/node_modules/hawk/lib/index.js
D node_modules/gc-stats/node_modules/hawk/lib/server.js
D node_modules/gc-stats/node_modules/hawk/lib/utils.js
D 

[MediaWiki-commits] [Gerrit] wikidata...rdf[master]: Extract RdfRepositoryForTesting to a JUnit @Rule

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/396380 )

Change subject: Extract RdfRepositoryForTesting to a JUnit @Rule
..

Extract RdfRepositoryForTesting to a JUnit @Rule

Rules allow better composition of test infrastructure than abstract classes.
In this case, it opens the way to better test the RdfRepositoryForTesting
itself as well.

Bug: T178721
Change-Id: I77139a6d64b3610c2621676c9b686dfbfd22b260
---
D 
tools/src/test/java/org/wikidata/query/rdf/tool/AbstractRdfRepositoryIntegrationTestBase.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/AbstractUpdaterIntegrationTestBase.java
M tools/src/test/java/org/wikidata/query/rdf/tool/IOBlastingIntegrationTest.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/MultipleResultsQueryIntegrationTest.java
M tools/src/test/java/org/wikidata/query/rdf/tool/MungeIntegrationTest.java
A tools/src/test/java/org/wikidata/query/rdf/tool/RdfRepositoryForTesting.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/WikibaseDateExtensionIntegrationTest.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/WikibaseGeoExtensionIntegrationTest.java
M 
tools/src/test/java/org/wikidata/query/rdf/tool/rdf/RdfRepositoryIntegrationTest.java
9 files changed, 364 insertions(+), 373 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikidata/query/rdf 
refs/changes/80/396380/1

diff --git 
a/tools/src/test/java/org/wikidata/query/rdf/tool/AbstractRdfRepositoryIntegrationTestBase.java
 
b/tools/src/test/java/org/wikidata/query/rdf/tool/AbstractRdfRepositoryIntegrationTestBase.java
deleted file mode 100644
index 5f439a3..000
--- 
a/tools/src/test/java/org/wikidata/query/rdf/tool/AbstractRdfRepositoryIntegrationTestBase.java
+++ /dev/null
@@ -1,160 +0,0 @@
-package org.wikidata.query.rdf.tool;
-
-import java.net.URI;
-
-import org.junit.After;
-import org.junit.Before;
-import org.junit.runner.RunWith;
-import org.wikidata.query.rdf.common.uri.WikibaseUris;
-import org.wikidata.query.rdf.tool.rdf.RdfRepository;
-
-import com.carrotsearch.randomizedtesting.RandomizedRunner;
-import com.carrotsearch.randomizedtesting.RandomizedTest;
-
-/**
- * Superclass of integration tests that an RDF repository and clear it between
- * test methods.
- */
-@RunWith(RandomizedRunner.class)
-public abstract class AbstractRdfRepositoryIntegrationTestBase extends 
RandomizedTest {
-
-/**
- * Take a relative path and create a URL with the full path to Blazegraph 
on
- * localhost.
- */
-private static URI url(String path) {
-return URI.create("http://localhost:/bigdata; + path);
-}
-
-/**
- * Wikibase uris to test with.
- */
-private final WikibaseUris uris;
-/**
- * Repository to test with.
- */
-private RdfRepositoryForTesting rdfRepository;
-
-/**
- * Build the test against prod wikidata.
- */
-public AbstractRdfRepositoryIntegrationTestBase() {
-this(WikibaseUris.getURISystem());
-}
-
-public AbstractRdfRepositoryIntegrationTestBase(WikibaseUris uris) {
-this.uris = uris;
-}
-
-/**
- * Initializes the {@link RdfRepository} before each test.
- *
- * Since randomized testing ThreadLeakControl checks for leaked thread, not
- * closing properly the RdfRepository after each test causes random false
- * negative in the test results. Initializing the RdfRepository for each
- * test might be slightly less performant, but at least it ensures
- * reproducible tests.
- */
-@Before
-public void initRdfRepository() {
-rdfRepository = new RdfRepositoryForTesting("wdq");
-rdfRepository.clear();
-}
-
-/**
- * Closes the {@link RdfRepository} after each test.
- *
- * @throws Exception on error
- */
-@After
-public void clearAndShutdownRdfRepository() throws Exception {
-if (rdfRepository != null) {
-rdfRepository.clear();
-rdfRepository.close();
-}
-}
-
-/**
- * Uris to test with.
- */
-public WikibaseUris uris() {
-return uris;
-}
-
-/**
- * Repository to test against.
- */
-public RdfRepositoryForTesting rdfRepository() {
-return rdfRepository;
-}
-
-/**
- * RdfRepository extension used for testing. We don't want to anyone to
- * accidentally use clear() so we don't put it in the repository.
- */
-public static class RdfRepositoryForTesting extends RdfRepository {
-
-/**
- * The namespace of the local RDF repository, e.g. "kb" or "wdq".
- */
-private final String namespace;
-
-public RdfRepositoryForTesting(String namespace) {
-super(url("/namespace/" + namespace + "/sparql"), 
WikibaseUris.WIKIDATA);
-this.namespace = namespace;
-}
-
-/**
- * Clear's the whole repository.
- */

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395970 )

Change subject: elasticsearch: use the canonical definition of logstash host
..


elasticsearch: use the canonical definition of logstash host

Bug: T182304
Change-Id: I7f0689a6f808f6a17849303b81900abf1520b834
---
M hieradata/common.yaml
M hieradata/labs/deployment-prep/common.yaml
M hieradata/role/codfw/elasticsearch/cirrus.yaml
M hieradata/role/eqiad/elasticsearch/cirrus.yaml
M hieradata/role/eqiad/elasticsearch/relforge.yaml
M modules/elasticsearch/manifests/init.pp
M modules/elasticsearch/spec/classes/ship_logs_to_logstash_rspec.rb
M modules/elasticsearch/templates/log4j2.properties.erb
M modules/elasticsearch/templates/logging.yml.erb
M modules/profile/manifests/elasticsearch.pp
10 files changed, 18 insertions(+), 30 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 4b8594c..099761c 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -517,6 +517,7 @@
 # Logging: logstash, udp2log
 logstash_host: "logstash.svc.eqiad.wmnet"
 logstash_syslog_port: 10514
+logstash_gelf_port: 12201
 udp2log_aggregator: "udplog:8420"
 
 tcpircbot_host: 'icinga.wikimedia.org'
diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index f0f3209..3c3fbe2 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -144,6 +144,7 @@
 "udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
 "logstash_host": deployment-logstash2.deployment-prep.eqiad.wmflabs
 "logstash_syslog_port": 10514
+"logstash_gelf_port": 12201
 mediawiki_memcached_servers:
 - 10.68.23.25:11211:1  # deployment-memc04
 - 10.68.23.49:11211:1  # deployment-memc05
@@ -230,8 +231,6 @@
 - deployment-elastic06
 - deployment-elastic07
 "profile::elasticsearch::cluster_name": beta-search
-"profile::elasticsearch::graylog_hosts":
-  - deployment-logstash2.deployment-prep.eqiad.wmflabs
 "profile::elasticsearch::reindex_remote_whitelist": ''
 profile::elasticsearch::search_shard_count_limit: 1000
 # By default elasticsearch sets this to ((# of available_processors * 3) / 2) 
+ 1,
diff --git a/hieradata/role/codfw/elasticsearch/cirrus.yaml 
b/hieradata/role/codfw/elasticsearch/cirrus.yaml
index ed089c7..7bbd947 100644
--- a/hieradata/role/codfw/elasticsearch/cirrus.yaml
+++ b/hieradata/role/codfw/elasticsearch/cirrus.yaml
@@ -53,9 +53,6 @@
 # We should be able to run "OK" with 11 servers.
 profile::elasticsearch::recover_after_nodes: 11
 
-profile::elasticsearch::graylog_hosts:
-  - logstash.svc.eqiad.wmnet
-
 profile::elasticsearch::certificate_name: 'search.svc.codfw.wmnet'
 profile::base::dns_alt_names: 'search.svc.codfw.wmnet'
 
diff --git a/hieradata/role/eqiad/elasticsearch/cirrus.yaml 
b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
index 7592320..eee5704 100644
--- a/hieradata/role/eqiad/elasticsearch/cirrus.yaml
+++ b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
@@ -54,9 +54,6 @@
 # We should be able to run "OK" with 24 servers.
 profile::elasticsearch::recover_after_nodes: 24
 
-profile::elasticsearch::graylog_hosts:
-  - logstash.svc.eqiad.wmnet
-
 profile::elasticsearch::certificate_name: 'search.svc.eqiad.wmnet'
 profile::base::dns_alt_names: 'search.svc.eqiad.wmnet'
 
diff --git a/hieradata/role/eqiad/elasticsearch/relforge.yaml 
b/hieradata/role/eqiad/elasticsearch/relforge.yaml
index 39e498d..4ef87a4 100644
--- a/hieradata/role/eqiad/elasticsearch/relforge.yaml
+++ b/hieradata/role/eqiad/elasticsearch/relforge.yaml
@@ -13,9 +13,6 @@
 profile::elasticsearch::heap_memory: '30G'
 profile::elasticsearch::minimum_master_nodes: 1
 
-profile::elasticsearch::graylog_hosts:
-  - logstash.svc.eqiad.wmnet
-
 profile::elasticsearch::certificate_name: 'relforge.svc.eqiad.wmnet'
 profile::base::dns_alt_names: 'relforge.svc.eqiad.wmnet'
 
diff --git a/modules/elasticsearch/manifests/init.pp 
b/modules/elasticsearch/manifests/init.pp
index 077c2ad..e711aa1 100644
--- a/modules/elasticsearch/manifests/init.pp
+++ b/modules/elasticsearch/manifests/init.pp
@@ -111,8 +111,8 @@
 $bulk_thread_pool_capacity = undef,
 $search_thread_pool_executors = undef,
 $load_fixed_bitset_filters_eagerly = true,
-$graylog_hosts = undef,
-$graylog_port = 12201,
+$logstash_host = undef,
+$logstash_gelf_port = 12201,
 $gc_log = true,
 $java_package = 'openjdk-8-jdk',
 $version = 5,
@@ -138,15 +138,10 @@
 fail('script_max_compilations_per_minute should be > 0')
 }
 
-# if no graylog_host is given, do not send logs
-$send_logs_to_logstash = is_array($graylog_hosts)
+$send_logs_to_logstash = $logstash_host != undef
 
-if $send_logs_to_logstash {
-validate_array($graylog_hosts)
- 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: service: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/396072 )

Change subject: service: use the canonical definition of logstash host
..

service: use the canonical definition of logstash host

This introduces an ugly call to hiera in service::configuration. Avoiding
that call would require more refactoring of the service module than I am
willing to invest at this point.

Bug: T182304
Change-Id: I072bc7570503024a2c18981aaef91a73504cdb00
---
M hieradata/common.yaml
M hieradata/common/service/configuration.yaml
M hieradata/labs/deployment-prep/common.yaml
M modules/service/manifests/configuration.pp
4 files changed, 5 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/72/396072/1

diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 099761c..d409ba5 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -517,6 +517,7 @@
 # Logging: logstash, udp2log
 logstash_host: "logstash.svc.eqiad.wmnet"
 logstash_syslog_port: 10514
+logstash_logback_port: 11514
 logstash_gelf_port: 12201
 udp2log_aggregator: "udplog:8420"
 
diff --git a/hieradata/common/service/configuration.yaml 
b/hieradata/common/service/configuration.yaml
index 22fc404..0955a05 100644
--- a/hieradata/common/service/configuration.yaml
+++ b/hieradata/common/service/configuration.yaml
@@ -1,3 +1,2 @@
 service::configuration::http_proxy: 
"http://url-downloader.%{::site}.wikimedia.org:8080;
 service::configuration::statsd_host: statsd.eqiad.wmnet # statsd is (still) 
eqiad-only
-service::configuration::logstash_host: logstash.svc.eqiad.wmnet # logstash is 
(still) eqiad-only
diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 31cbe6f..b0b9afb 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -9,7 +9,6 @@
 service::configuration::restbase_uri: 
http://deployment-restbase02.deployment-prep.eqiad.wmflabs:7231
 #service::configuration::http_proxy:
 service::configuration::statsd_host: labmon1001.eqiad.wmnet
-service::configuration::logstash_host: 
deployment-logstash2.deployment-prep.eqiad.wmflabs
 profile::citoid::zotero_host: deployment-zotero01.deployment-prep.eqiad.wmflabs
 profile::citoid::zotero_port: 1969
 profile::changeprop::purge_host: 
deployment-cache-text04.deployment-prep.eqiad.wmflabs
@@ -144,6 +143,7 @@
 "udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
 "logstash_host": deployment-logstash2.deployment-prep.eqiad.wmflabs
 "logstash_syslog_port": 10514
+"logstash_logback_port": 11514
 "logstash_gelf_port": 12201
 mediawiki_memcached_servers:
 - 10.68.23.25:11211:1  # deployment-memc04
diff --git a/modules/service/manifests/configuration.pp 
b/modules/service/manifests/configuration.pp
index 24a501b..f684da9 100644
--- a/modules/service/manifests/configuration.pp
+++ b/modules/service/manifests/configuration.pp
@@ -49,9 +49,9 @@
 $http_proxy= undef,
 $statsd_host   = 'localhost',
 $statsd_port   = 8125,
-$logstash_host = 'localhost',
-$logstash_port = 12201,
-$logstash_port_logback = 11514,
+$logstash_host = hiera('logstash_host', 'localhost'),
+$logstash_port = hiera('logstash_gelf_port', 12201),
+$logstash_port_logback = hiera('logstash_logback_port', 11514),
 $log_dir   = '/srv/log',
 $use_dev_pkgs  = false,
 ){

-- 
To view, visit https://gerrit.wikimedia.org/r/396072
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I072bc7570503024a2c18981aaef91a73504cdb00
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: elasticsearch: use the canonical definition of logstash host

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395970 )

Change subject: elasticsearch: use the canonical definition of logstash host
..

elasticsearch: use the canonical definition of logstash host

Bug: T182304
Change-Id: I7f0689a6f808f6a17849303b81900abf1520b834
---
M hieradata/common.yaml
M hieradata/labs/deployment-prep/common.yaml
M hieradata/role/codfw/elasticsearch/cirrus.yaml
M hieradata/role/eqiad/elasticsearch/cirrus.yaml
M hieradata/role/eqiad/elasticsearch/relforge.yaml
M modules/profile/manifests/elasticsearch.pp
6 files changed, 5 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/70/395970/1

diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 4b8594c..099761c 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -517,6 +517,7 @@
 # Logging: logstash, udp2log
 logstash_host: "logstash.svc.eqiad.wmnet"
 logstash_syslog_port: 10514
+logstash_gelf_port: 12201
 udp2log_aggregator: "udplog:8420"
 
 tcpircbot_host: 'icinga.wikimedia.org'
diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 9930669..31cbe6f 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -144,6 +144,7 @@
 "udp2log_aggregator": deployment-fluorine02.deployment-prep.eqiad.wmflabs:8420
 "logstash_host": deployment-logstash2.deployment-prep.eqiad.wmflabs
 "logstash_syslog_port": 10514
+"logstash_gelf_port": 12201
 mediawiki_memcached_servers:
 - 10.68.23.25:11211:1  # deployment-memc04
 - 10.68.23.49:11211:1  # deployment-memc05
@@ -230,8 +231,6 @@
 - deployment-elastic06
 - deployment-elastic07
 "profile::elasticsearch::cluster_name": beta-search
-"profile::elasticsearch::graylog_hosts":
-  - deployment-logstash2.deployment-prep.eqiad.wmflabs
 "profile::elasticsearch::reindex_remote_whitelist": ''
 profile::elasticsearch::search_shard_count_limit: 1000
 # By default elasticsearch sets this to ((# of available_processors * 3) / 2) 
+ 1,
diff --git a/hieradata/role/codfw/elasticsearch/cirrus.yaml 
b/hieradata/role/codfw/elasticsearch/cirrus.yaml
index ed089c7..7bbd947 100644
--- a/hieradata/role/codfw/elasticsearch/cirrus.yaml
+++ b/hieradata/role/codfw/elasticsearch/cirrus.yaml
@@ -53,9 +53,6 @@
 # We should be able to run "OK" with 11 servers.
 profile::elasticsearch::recover_after_nodes: 11
 
-profile::elasticsearch::graylog_hosts:
-  - logstash.svc.eqiad.wmnet
-
 profile::elasticsearch::certificate_name: 'search.svc.codfw.wmnet'
 profile::base::dns_alt_names: 'search.svc.codfw.wmnet'
 
diff --git a/hieradata/role/eqiad/elasticsearch/cirrus.yaml 
b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
index 7592320..eee5704 100644
--- a/hieradata/role/eqiad/elasticsearch/cirrus.yaml
+++ b/hieradata/role/eqiad/elasticsearch/cirrus.yaml
@@ -54,9 +54,6 @@
 # We should be able to run "OK" with 24 servers.
 profile::elasticsearch::recover_after_nodes: 24
 
-profile::elasticsearch::graylog_hosts:
-  - logstash.svc.eqiad.wmnet
-
 profile::elasticsearch::certificate_name: 'search.svc.eqiad.wmnet'
 profile::base::dns_alt_names: 'search.svc.eqiad.wmnet'
 
diff --git a/hieradata/role/eqiad/elasticsearch/relforge.yaml 
b/hieradata/role/eqiad/elasticsearch/relforge.yaml
index 39e498d..4ef87a4 100644
--- a/hieradata/role/eqiad/elasticsearch/relforge.yaml
+++ b/hieradata/role/eqiad/elasticsearch/relforge.yaml
@@ -13,9 +13,6 @@
 profile::elasticsearch::heap_memory: '30G'
 profile::elasticsearch::minimum_master_nodes: 1
 
-profile::elasticsearch::graylog_hosts:
-  - logstash.svc.eqiad.wmnet
-
 profile::elasticsearch::certificate_name: 'relforge.svc.eqiad.wmnet'
 profile::base::dns_alt_names: 'relforge.svc.eqiad.wmnet'
 
diff --git a/modules/profile/manifests/elasticsearch.pp 
b/modules/profile/manifests/elasticsearch.pp
index 1b816c0..ca1 100644
--- a/modules/profile/manifests/elasticsearch.pp
+++ b/modules/profile/manifests/elasticsearch.pp
@@ -20,7 +20,8 @@
 $minimum_master_nodes = 
hiera('profile::elasticsearch::minimum_master_nodes'),
 $heap_memory = hiera('profile::elasticsearch::heap_memory'),
 $expected_nodes = hiera('profile::elasticsearch::expected_nodes'),
-$graylog_hosts = hiera('profile::elasticsearch::graylog_hosts'),
+$graylog_hosts = hiera('logstash_host'),
+$graylog_port = hiera('logstash_gelf_port'),
 $rack = hiera('profile::elasticsearch::rack'),
 $row = hiera('profile::elasticsearch::row'),
 $awareness_attributes = 
hiera('profile::elasticsearch::awareness_attributes'),
@@ -106,6 +107,7 @@
 expected_nodes => $expected_nodes,
 master_eligible=> $master_eligible,
 graylog_hosts  => $graylog_hosts,
+graylog_port   => $graylog_port,
 version 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kibana: increase max payload size

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395958 )

Change subject: kibana: increase max payload size
..


kibana: increase max payload size

We are running into issues related to kibana upgrade and the high number
of fields we use. Increasing to 2MB was not sufficient, 4MB should work.

Bug: T178412
Change-Id: Ie77358b1d6641a5647f20d3885faac77b8edeb94
---
M modules/kibana/manifests/init.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  DCausse: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/modules/kibana/manifests/init.pp b/modules/kibana/manifests/init.pp
index 5922cf1..4e77a1d 100644
--- a/modules/kibana/manifests/init.pp
+++ b/modules/kibana/manifests/init.pp
@@ -25,7 +25,7 @@
 content => ordered_yaml({
 'kibana.defaultAppId'=> $default_app_id,
 'logging.quiet'  => true,
-'server.maxPayloadBytes' => 2097152,
+'server.maxPayloadBytes' => 4194304, # 4MB (yes, this is a crazy 
limit, we need to reduce the number of fields)
 }),
 mode=> '0444',
 require => Package['kibana'],

-- 
To view, visit https://gerrit.wikimedia.org/r/395958
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie77358b1d6641a5647f20d3885faac77b8edeb94
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: DCausse 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kibana: increase max payload size

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395958 )

Change subject: kibana: increase max payload size
..

kibana: increase max payload size

We are running into issues related to kibana upgrade and the high number
of fields we use. Increasing to 2MB was not sufficient, 4MB should work.

Bug: T178412
Change-Id: Ie77358b1d6641a5647f20d3885faac77b8edeb94
---
M modules/kibana/manifests/init.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/58/395958/1

diff --git a/modules/kibana/manifests/init.pp b/modules/kibana/manifests/init.pp
index 5922cf1..4e77a1d 100644
--- a/modules/kibana/manifests/init.pp
+++ b/modules/kibana/manifests/init.pp
@@ -25,7 +25,7 @@
 content => ordered_yaml({
 'kibana.defaultAppId'=> $default_app_id,
 'logging.quiet'  => true,
-'server.maxPayloadBytes' => 2097152,
+'server.maxPayloadBytes' => 4194304, # 4MB (yes, this is a crazy 
limit, we need to reduce the number of fields)
 }),
 mode=> '0444',
 require => Package['kibana'],

-- 
To view, visit https://gerrit.wikimedia.org/r/395958
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie77358b1d6641a5647f20d3885faac77b8edeb94
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kibana: increase max payload size

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395957 )

Change subject: kibana: increase max payload size
..


kibana: increase max payload size

We are running into issues related to kibana upgrade and the high number
of fields we use.

Bug: T178412
Change-Id: I5fa73eb899aee5f97e6f0b4c8a0c422f8fff95f5
---
M modules/kibana/manifests/init.pp
1 file changed, 3 insertions(+), 2 deletions(-)

Approvals:
  jenkins-bot: Verified
  DCausse: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/modules/kibana/manifests/init.pp b/modules/kibana/manifests/init.pp
index 59597a0..5922cf1 100644
--- a/modules/kibana/manifests/init.pp
+++ b/modules/kibana/manifests/init.pp
@@ -23,8 +23,9 @@
 owner   => 'root',
 group   => 'root',
 content => ordered_yaml({
-'kibana.defaultAppId' => $default_app_id,
-'logging.quiet'   => true,
+'kibana.defaultAppId'=> $default_app_id,
+'logging.quiet'  => true,
+'server.maxPayloadBytes' => 2097152,
 }),
 mode=> '0444',
 require => Package['kibana'],

-- 
To view, visit https://gerrit.wikimedia.org/r/395957
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5fa73eb899aee5f97e6f0b4c8a0c422f8fff95f5
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: DCausse 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kibana: increase max payload size

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395957 )

Change subject: kibana: increase max payload size
..

kibana: increase max payload size

We are running into issues related to kibana upgrade and the high number
of fields we use.

Bug: T178412
Change-Id: I5fa73eb899aee5f97e6f0b4c8a0c422f8fff95f5
---
M modules/kibana/manifests/init.pp
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/57/395957/1

diff --git a/modules/kibana/manifests/init.pp b/modules/kibana/manifests/init.pp
index 59597a0..cb152ba 100644
--- a/modules/kibana/manifests/init.pp
+++ b/modules/kibana/manifests/init.pp
@@ -25,6 +25,7 @@
 content => ordered_yaml({
 'kibana.defaultAppId' => $default_app_id,
 'logging.quiet'   => true,
+'server.maxPayloadBytes' => 2097152,
 }),
 mode=> '0444',
 require => Package['kibana'],

-- 
To view, visit https://gerrit.wikimedia.org/r/395957
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5fa73eb899aee5f97e6f0b4c8a0c422f8fff95f5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: dedicated components in our APT repository

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/390433 )

Change subject: logstash: dedicated components in our APT repository
..


logstash: dedicated components in our APT repository

Since we now have components dedicated to elastic in our APT repo, it
is now time to use them.

Note: the same implementation is done after moving logstash to role /
profiles (https://gerrit.wikimedia.org/r/#/c/390402/), but it is blocked
by https://gerrit.wikimedia.org/r/#/c/390039/ (T180257).

Bug: T179964
Change-Id: I2fa32b858f9c18ccb48f6aa911fdccef5335419a
---
M modules/role/manifests/logstash/elasticsearch.pp
1 file changed, 7 insertions(+), 0 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/role/manifests/logstash/elasticsearch.pp 
b/modules/role/manifests/logstash/elasticsearch.pp
index c81695c..fd2c6d4 100644
--- a/modules/role/manifests/logstash/elasticsearch.pp
+++ b/modules/role/manifests/logstash/elasticsearch.pp
@@ -24,6 +24,13 @@
 mode   => '0755',
 }
 
+apt::repository { 'wikimedia-elastic':
+uri=> 'http://apt.wikimedia.org/wikimedia',
+dist   => "${::lsbdistcodename}-wikimedia",
+components => 'component/elastic55 thirdparty/elastic55',
+before => Class['::elasticsearch'],
+}
+
 class { '::elasticsearch':
 require=> File['/usr/share/elasticsearch/plugins'],
 curator_uses_unicast_hosts => false, # elasticsearch API is only 
exposed to localhost

-- 
To view, visit https://gerrit.wikimedia.org/r/390433
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2fa32b858f9c18ccb48f6aa911fdccef5335419a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: DCausse 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Revert "Deploy MjoLniR with new deploy repository""

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394002 )

Change subject: Revert "Revert "Deploy MjoLniR with new deploy repository""
..


Revert "Revert "Deploy MjoLniR with new deploy repository""

The problems with mjolnir vs MjoLniR have been resolved, it
required adjustments to scap.cfg in the repo.

This is a standard revert with one addition, we now create an
empty directory /etc/mjolnir owned by deploy-service. This gives
the deploy repo a sane place to install a configuration script
to that can be auto-magicaly found by mjolnir.

This reverts commit 6a7753a14ac3cb66593eabbad30e8ac72e184751.

Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754
---
M hieradata/role/common/deployment_server.yaml
A modules/mjolnir/manifests/init.pp
M modules/profile/manifests/mjolnir/kafka_daemon.pp
M modules/profile/templates/mjolnir/kafka-daemon.service.erb
M modules/role/manifests/elasticsearch/analytics.pp
5 files changed, 28 insertions(+), 14 deletions(-)

Approvals:
  EBernhardson: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/hieradata/role/common/deployment_server.yaml 
b/hieradata/role/common/deployment_server.yaml
index 4f899a6..5c5e154 100644
--- a/hieradata/role/common/deployment_server.yaml
+++ b/hieradata/role/common/deployment_server.yaml
@@ -173,8 +173,8 @@
   # Netbox software
   netbox/deploy:
 repository: operations/software/netbox-deploy
-  relforge/mjolnir:
-repository: search/MjoLniR
+  search/mjolnir/deploy:
+repository: search/MjoLniR/deploy
   statsv/statsv:
 repository: analytics/statsv
   "docker-pkg/deploy":
diff --git a/modules/mjolnir/manifests/init.pp 
b/modules/mjolnir/manifests/init.pp
new file mode 100644
index 000..07e559f
--- /dev/null
+++ b/modules/mjolnir/manifests/init.pp
@@ -0,0 +1,21 @@
+# = Class: mjolnir
+#
+# This class installs the MjoLniR (Machine Learned Ranking) data
+# processing package.
+#
+class mjolnir {
+require_package('virtualenv', 'zip')
+
+file { '/etc/mjolnir':
+ensure => 'directory',
+owner  => 'deploy-service',
+group  => 'deploy-service',
+mode   => '0755',
+}
+
+scap::target { 'search/mjolnir/deploy':
+deploy_user => 'deploy-service',
+}
+}
+
+
diff --git a/modules/profile/manifests/mjolnir/kafka_daemon.pp 
b/modules/profile/manifests/mjolnir/kafka_daemon.pp
index c1dcaf9..ba8cc1c 100644
--- a/modules/profile/manifests/mjolnir/kafka_daemon.pp
+++ b/modules/profile/manifests/mjolnir/kafka_daemon.pp
@@ -9,18 +9,11 @@
 # it is named just 'eqiad'.
 $kafka_config = kafka_config('eqiad'),
 ) {
-scap::target { 'relforge/mjolnir':
-  deploy_user => 'deploy-service',
-}
-
-# This is a limited subset of what the full mjolnir package requires 
because
-# the daemon is a small part of the overall application. The daemon only 
needs
-# to read/write kafka topics and send requests to localhost.
-require_package('python-kafka', 'python-requests')
+class { 'mjolnir': }
 
 systemd::service { 'mjolnir-kafka-daemon':
 content => template('profile/mjolnir/kafka-daemon.service.erb'),
-require => Scap::Target['relforge/mjolnir'],
+require => Class['mjolnir'],
 }
 
 }
diff --git a/modules/profile/templates/mjolnir/kafka-daemon.service.erb 
b/modules/profile/templates/mjolnir/kafka-daemon.service.erb
index b6947ba..7e40bd8 100644
--- a/modules/profile/templates/mjolnir/kafka-daemon.service.erb
+++ b/modules/profile/templates/mjolnir/kafka-daemon.service.erb
@@ -5,9 +5,7 @@
 [Service]
 User=nobody
 Group=nogroup
-WorkingDirectory=/srv/deployment/relforge/mjolnir
-Environment=PYTHONPATH=/srv/deployment/relforge/mjolnir
-ExecStart=/usr/bin/python2 
/srv/deployment/relforge/mjolnir/mjolnir/cli/kafka_daemon.py --brokers <%= 
@kafka_config['brokers']['string'] %>
+ExecStart=/srv/deployment/search/mjolnir/venv/bin/mjolnir-utilities.py 
kafka_daemon --brokers <%= @kafka_config['brokers']['string'] %>
 StandardInput=null
 StandardOutput=journal
 StandardError=journal
diff --git a/modules/role/manifests/elasticsearch/analytics.pp 
b/modules/role/manifests/elasticsearch/analytics.pp
index a46391e..66d7789 100644
--- a/modules/role/manifests/elasticsearch/analytics.pp
+++ b/modules/role/manifests/elasticsearch/analytics.pp
@@ -1,5 +1,7 @@
 # Supports CirrusSearch usage on the analytics cluster
 class role::elasticsearch::analytics {
+class { 'mjolnir': }
+
 # wikimedia/discovery/analytics will be deployed to this node
 scap::target { 'wikimedia/discovery/analytics':
 deploy_user => 'deploy-service',

-- 
To view, visit https://gerrit.wikimedia.org/r/394002
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I599341bd16ecba0a2b8d8132fde6fe3d1443d754
Gerrit-PatchSet: 5
Gerrit-Project: 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: decommission logstash100[1-3]

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395565 )

Change subject: logstash: decommission logstash100[1-3]
..


logstash: decommission logstash100[1-3]

Bug: T175830
Change-Id: I28615bd637dd1811af02b99af5c4d71003736286
---
M conftool-data/node/eqiad.yaml
M hieradata/role/common/logstash/collector.yaml
M hieradata/role/common/logstash/elasticsearch.yaml
M manifests/site.pp
4 files changed, 2 insertions(+), 12 deletions(-)

Approvals:
  RobH: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index 593299a..83e79e6 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -299,9 +299,6 @@
 wdqs1004.eqiad.wmnet: [wdqs]
 wdqs1005.eqiad.wmnet: [wdqs]
   logstash:
-logstash1001.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
-logstash1002.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
-logstash1003.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
 logstash1007.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
 logstash1008.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
 logstash1009.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
diff --git a/hieradata/role/common/logstash/collector.yaml 
b/hieradata/role/common/logstash/collector.yaml
index c533845..c0717f0 100644
--- a/hieradata/role/common/logstash/collector.yaml
+++ b/hieradata/role/common/logstash/collector.yaml
@@ -33,9 +33,6 @@
 
 
 logstash::cluster_hosts:
-  - logstash1001.eqiad.wmnet
-  - logstash1002.eqiad.wmnet
-  - logstash1003.eqiad.wmnet
   - logstash1004.eqiad.wmnet
   - logstash1005.eqiad.wmnet
   - logstash1006.eqiad.wmnet
diff --git a/hieradata/role/common/logstash/elasticsearch.yaml 
b/hieradata/role/common/logstash/elasticsearch.yaml
index 0f56951..79672b0 100644
--- a/hieradata/role/common/logstash/elasticsearch.yaml
+++ b/hieradata/role/common/logstash/elasticsearch.yaml
@@ -7,7 +7,7 @@
 
 # ES-specific
 # NOTE: cluster_name must be kept in sync with the backend
-# node config in hieradata/role/common/logstash.yaml
+# node config in hieradata/role/common/collector.yaml
 elasticsearch::auto_create_index: true
 elasticsearch::cluster_name: production-logstash-eqiad
 elasticsearch::expected_nodes: 3
@@ -20,9 +20,6 @@
   - logstash1005.eqiad.wmnet
   - logstash1006.eqiad.wmnet
 logstash::cluster_hosts:
-  - logstash1001.eqiad.wmnet
-  - logstash1002.eqiad.wmnet
-  - logstash1003.eqiad.wmnet
   - logstash1004.eqiad.wmnet
   - logstash1005.eqiad.wmnet
   - logstash1006.eqiad.wmnet
diff --git a/manifests/site.pp b/manifests/site.pp
index 29233b2..c1e433d 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1376,8 +1376,7 @@
 }
 
 node /^logstash100[1-3]\.eqiad\.wmnet$/ {
-role(logstash::collector, kibana, logstash::apifeatureusage)
-include ::lvs::realserver
+role(spare::system)
 }
 
 node /^logstash100[4-6]\.eqiad\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/395565
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I28615bd637dd1811af02b99af5c4d71003736286
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: DCausse 
Gerrit-Reviewer: EBernhardson 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: RobH 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: schedule cronjob to reload categories

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394021 )

Change subject: wdqs: schedule cronjob to reload categories
..


wdqs: schedule cronjob to reload categories

Bug: T173772
Change-Id: Id71919150d57aad5feb15c610b2fa4f72602b93d
---
M modules/wdqs/manifests/gui.pp
1 file changed, 39 insertions(+), 0 deletions(-)

Approvals:
  Smalyshev: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index 617695f..d842eb5 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -68,4 +68,43 @@
 require => File['/etc/wdqs/gui_vars.sh'],
 }
 
+$cron_log = "${log_dir}/reloadCategories.log"
+# the reload-categories cron needs to reload nginx once the categories are 
up to date
+sudo::user { "${username}-reload-nginx":
+  ensure => present,
+  user   => $username,
+  privileges => [ 'ALL = NOPASSWD: /bin/systemctl reload nginx' ],
+}
+
+# Category dumps start on Sat 20:00. By Mon, they should be done.
+# We want random time so that hosts don't reboot at the same time, but we
+# do not want them to be too far from one another.
+cron { 'reload-categories':
+ensure  => present,
+command => "/usr/local/bin/reloadCategories.sh >> ${cron_log}",
+user=> $username,
+weekday => 1,
+minute  => fqdn_rand(60),
+hour=> fqdn_rand(2),
+}
+
+logrotate::rule { 'wdqs-reload-categories':
+ensure   => present,
+file_glob=> $cron_log,
+frequency=> 'monthly',
+missing_ok   => true,
+not_if_empty => true,
+rotate   => 3,
+compress => true,
+}
+
+# Remove categories*.log files after 30 days.
+logrotate::rule { 'wdqs-categories-logs':
+ensure   => present,
+file_glob=> "${log_dir}/categories*.log",
+missing_ok   => true,
+not_if_empty => true,
+rotate   => 0,
+max_age  => 30,
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/394021
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id71919150d57aad5feb15c610b2fa4f72602b93d
Gerrit-PatchSet: 9
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Smalyshev 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: move eventlogging collection to logstash1007

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395564 )

Change subject: logstash: move eventlogging collection to logstash1007
..


logstash: move eventlogging collection to logstash1007

since logstash100[1-3] are being decommissionned, eventlogging collection
needs to move to new VMs.

Bug: T175830
Change-Id: Ieea876c62516a161d154588f700427c1c63120db
---
M manifests/site.pp
1 file changed, 6 insertions(+), 8 deletions(-)

Approvals:
  EBernhardson: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/manifests/site.pp b/manifests/site.pp
index 8ede359..29233b2 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1375,23 +1375,21 @@
 role(syslog::centralserver)
 }
 
-node /^logstash100[1-2]\.eqiad\.wmnet$/ {
+node /^logstash100[1-3]\.eqiad\.wmnet$/ {
 role(logstash::collector, kibana, logstash::apifeatureusage)
 include ::lvs::realserver
 }
 
-node /^logstash1003\.eqiad\.wmnet$/ {
-# Before decommissionning logstash1003, role::logstash::eventlogging needs
-# to be moved to another node
-role(logstash::collector, kibana, logstash::apifeatureusage, 
logstash::eventlogging)
-include ::lvs::realserver
-}
 node /^logstash100[4-6]\.eqiad\.wmnet$/ {
 role(logstash::elasticsearch)
 }
 
 # logstash collectors (Ganeti VM)
-node /^logstash100[7-9]\.eqiad\.wmnet$/ {
+node 'logstash1007.eqiad.wmnet' {
+role(logstash::collector, kibana, logstash::apifeatureusage, 
logstash::eventlogging)
+include ::lvs::realserver
+}
+node /^logstash100[8-9]\.eqiad\.wmnet$/ {
 role(logstash::collector, kibana, logstash::apifeatureusage)
 include ::lvs::realserver
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/395564
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ieea876c62516a161d154588f700427c1c63120db
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: DCausse 
Gerrit-Reviewer: EBernhardson 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: use the /readiness-probe in WDQS icinga checks

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394987 )

Change subject: wdqs: use the /readiness-probe in WDQS icinga checks
..


wdqs: use the /readiness-probe in WDQS icinga checks

Bug: T181989
Change-Id: Ic42863eb87fe936183189644ef9b1635c29f568d
---
M modules/wdqs/manifests/monitor/services.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Smalyshev: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/manifests/monitor/services.pp 
b/modules/wdqs/manifests/monitor/services.pp
index 29cf7a3..ec67508 100644
--- a/modules/wdqs/manifests/monitor/services.pp
+++ b/modules/wdqs/manifests/monitor/services.pp
@@ -8,7 +8,7 @@
 
 nrpe::monitor_service { 'WDQS_Internal_HTTP_endpoint':
 description  => 'WDQS HTTP Port',
-nrpe_command => '/usr/lib/nagios/plugins/check_tcp -H 127.0.0.1 -p 80',
+nrpe_command => '/usr/lib/nagios/plugins/check_http -H 127.0.0.1 -p 80 
-w 10 -u /readiness-probe',
 }
 
 nrpe::monitor_service { 'WDQS_Local_Blazegraph_endpoint':

-- 
To view, visit https://gerrit.wikimedia.org/r/394987
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic42863eb87fe936183189644ef9b1635c29f568d
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Smalyshev 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: move eventlogging collection to logstash1007

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395564 )

Change subject: logstash: move eventlogging collection to logstash1007
..

logstash: move eventlogging collection to logstash1007

since logstash100[1-3] are being decommissionned, eventlogging collection
needs to move to new VMs.

Bug: T175830
Change-Id: Ieea876c62516a161d154588f700427c1c63120db
---
M manifests/site.pp
1 file changed, 6 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/64/395564/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 8ede359..29233b2 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1375,23 +1375,21 @@
 role(syslog::centralserver)
 }
 
-node /^logstash100[1-2]\.eqiad\.wmnet$/ {
+node /^logstash100[1-3]\.eqiad\.wmnet$/ {
 role(logstash::collector, kibana, logstash::apifeatureusage)
 include ::lvs::realserver
 }
 
-node /^logstash1003\.eqiad\.wmnet$/ {
-# Before decommissionning logstash1003, role::logstash::eventlogging needs
-# to be moved to another node
-role(logstash::collector, kibana, logstash::apifeatureusage, 
logstash::eventlogging)
-include ::lvs::realserver
-}
 node /^logstash100[4-6]\.eqiad\.wmnet$/ {
 role(logstash::elasticsearch)
 }
 
 # logstash collectors (Ganeti VM)
-node /^logstash100[7-9]\.eqiad\.wmnet$/ {
+node 'logstash1007.eqiad.wmnet' {
+role(logstash::collector, kibana, logstash::apifeatureusage, 
logstash::eventlogging)
+include ::lvs::realserver
+}
+node /^logstash100[8-9]\.eqiad\.wmnet$/ {
 role(logstash::collector, kibana, logstash::apifeatureusage)
 include ::lvs::realserver
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/395564
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ieea876c62516a161d154588f700427c1c63120db
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: decommission logstash100[1-3]

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395565 )

Change subject: logstash: decommission logstash100[1-3]
..

logstash: decommission logstash100[1-3]

Bug: T175830
Change-Id: I28615bd637dd1811af02b99af5c4d71003736286
---
M conftool-data/node/eqiad.yaml
M hieradata/role/common/logstash/collector.yaml
M hieradata/role/common/logstash/elasticsearch.yaml
M manifests/site.pp
4 files changed, 2 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/65/395565/1

diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index 593299a..83e79e6 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -299,9 +299,6 @@
 wdqs1004.eqiad.wmnet: [wdqs]
 wdqs1005.eqiad.wmnet: [wdqs]
   logstash:
-logstash1001.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
-logstash1002.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
-logstash1003.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
 logstash1007.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
 logstash1008.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
 logstash1009.eqiad.wmnet: [kibana, logstash, logstash-log4j, 
logstash-gelf, logstash-json, logstash-syslog, logstash-udp2log]
diff --git a/hieradata/role/common/logstash/collector.yaml 
b/hieradata/role/common/logstash/collector.yaml
index c533845..c0717f0 100644
--- a/hieradata/role/common/logstash/collector.yaml
+++ b/hieradata/role/common/logstash/collector.yaml
@@ -33,9 +33,6 @@
 
 
 logstash::cluster_hosts:
-  - logstash1001.eqiad.wmnet
-  - logstash1002.eqiad.wmnet
-  - logstash1003.eqiad.wmnet
   - logstash1004.eqiad.wmnet
   - logstash1005.eqiad.wmnet
   - logstash1006.eqiad.wmnet
diff --git a/hieradata/role/common/logstash/elasticsearch.yaml 
b/hieradata/role/common/logstash/elasticsearch.yaml
index 0f56951..79672b0 100644
--- a/hieradata/role/common/logstash/elasticsearch.yaml
+++ b/hieradata/role/common/logstash/elasticsearch.yaml
@@ -7,7 +7,7 @@
 
 # ES-specific
 # NOTE: cluster_name must be kept in sync with the backend
-# node config in hieradata/role/common/logstash.yaml
+# node config in hieradata/role/common/collector.yaml
 elasticsearch::auto_create_index: true
 elasticsearch::cluster_name: production-logstash-eqiad
 elasticsearch::expected_nodes: 3
@@ -20,9 +20,6 @@
   - logstash1005.eqiad.wmnet
   - logstash1006.eqiad.wmnet
 logstash::cluster_hosts:
-  - logstash1001.eqiad.wmnet
-  - logstash1002.eqiad.wmnet
-  - logstash1003.eqiad.wmnet
   - logstash1004.eqiad.wmnet
   - logstash1005.eqiad.wmnet
   - logstash1006.eqiad.wmnet
diff --git a/manifests/site.pp b/manifests/site.pp
index 29233b2..c1e433d 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1376,8 +1376,7 @@
 }
 
 node /^logstash100[1-3]\.eqiad\.wmnet$/ {
-role(logstash::collector, kibana, logstash::apifeatureusage)
-include ::lvs::realserver
+role(spare::system)
 }
 
 node /^logstash100[4-6]\.eqiad\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/395565
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I28615bd637dd1811af02b99af5c4d71003736286
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...deploy[master]: New packaging repository for tilerator

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395543 )

Change subject: New packaging repository for tilerator
..


New packaging repository for tilerator

Bug: T162241
Change-Id: I4322cd08965aa24fd790e74036754bd6b924eef6
---
M .gitmodules
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Verified; Looks good to me, approved



diff --git a/.gitmodules b/.gitmodules
index 57707e1..4e12625 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,3 @@
 [submodule "src"]
path = src
-   url = https://gerrit.wikimedia.org/r/maps/tilerator
+   url = https://gerrit.wikimedia.org/r/maps/tilerator/package

-- 
To view, visit https://gerrit.wikimedia.org/r/395543
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4322cd08965aa24fd790e74036754bd6b924eef6
Gerrit-PatchSet: 1
Gerrit-Project: maps/tilerator/deploy
Gerrit-Branch: master
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...deploy[master]: New packaging repository for tilerator

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395543 )

Change subject: New packaging repository for tilerator
..

New packaging repository for tilerator

Bug: T162241
Change-Id: I4322cd08965aa24fd790e74036754bd6b924eef6
---
M .gitmodules
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/tilerator/deploy 
refs/changes/43/395543/1

diff --git a/.gitmodules b/.gitmodules
index 57707e1..4e12625 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,3 @@
 [submodule "src"]
path = src
-   url = https://gerrit.wikimedia.org/r/maps/tilerator
+   url = https://gerrit.wikimedia.org/r/maps/tilerator/package

-- 
To view, visit https://gerrit.wikimedia.org/r/395543
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4322cd08965aa24fd790e74036754bd6b924eef6
Gerrit-PatchSet: 1
Gerrit-Project: maps/tilerator/deploy
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...package[master]: initial commit

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395478 )

Change subject: initial commit
..


initial commit

Change-Id: Iffebc2da34e0a1c64a02d12a5cf0a0a0f1216c61
---
A .dockerignore
A .gitignore
A .gitreview
A app.js
A config.dev.yaml
A package.json
A server.js
A targets.yaml
8 files changed, 412 insertions(+), 0 deletions(-)

Approvals:
  Gehel: Verified; Looks good to me, approved



diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 000..563df11
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,3 @@
+.git
+coverage
+node_modules
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..a0a1c47
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,8 @@
+Dockerfile
+.idea/
+coverage
+config.yaml
+node_modules
+npm-debug.log
+vectors
+variables.yaml
diff --git a/.gitreview b/.gitreview
new file mode 100644
index 000..fd59be6
--- /dev/null
+++ b/.gitreview
@@ -0,0 +1,6 @@
+[gerrit]
+host=gerrit.wikimedia.org
+port=29418
+project=maps/tilerator/package.git
+defaultbranch=master
+defaultrebase=0
diff --git a/app.js b/app.js
new file mode 100644
index 000..f866804
--- /dev/null
+++ b/app.js
@@ -0,0 +1,203 @@
+'use strict';
+
+
+var http = require('http');
+var BBPromise = require('bluebird');
+var express = require('express');
+var compression = require('compression');
+var bodyParser = require('body-parser');
+var fs = BBPromise.promisifyAll(require('fs'));
+var sUtil = require('./lib/util');
+var packageInfo = require('./package.json');
+var yaml = require('js-yaml');
+
+
+/**
+ * Creates an express app and initialises it
+ * @param {Object} options the options to initialise the app with
+ * @return {bluebird} the promise resolving to the app object
+ */
+function initApp(options) {
+
+// the main application object
+var app = express();
+
+// get the options and make them available in the app
+app.logger = options.logger;// the logging device
+app.metrics = options.metrics;  // the metrics
+app.conf = options.config;  // this app's config options
+app.info = packageInfo; // this app's package info
+
+// ensure some sane defaults
+if(!app.conf.port) { app.conf.port = ; }
+if(!app.conf.interface) { app.conf.interface = '0.0.0.0'; }
+if(app.conf.compression_level === undefined) { app.conf.compression_level 
= 3; }
+if(app.conf.cors === undefined) { app.conf.cors = '*'; }
+if(app.conf.csp === undefined) {
+app.conf.csp =
+"default-src 'self'; object-src 'none'; media-src *; img-src *; 
style-src *; frame-ancestors 'self'";
+}
+
+// set outgoing proxy
+if(app.conf.proxy) {
+process.env.HTTP_PROXY = app.conf.proxy;
+// if there is a list of domains which should
+// not be proxied, set it
+if(app.conf.no_proxy_list) {
+if(Array.isArray(app.conf.no_proxy_list)) {
+process.env.NO_PROXY = app.conf.no_proxy_list.join(',');
+} else {
+process.env.NO_PROXY = app.conf.no_proxy_list;
+}
+}
+}
+
+// set up header whitelisting for logging
+if(!app.conf.log_header_whitelist) {
+app.conf.log_header_whitelist = [
+'cache-control', 'content-type', 'content-length', 'if-match',
+'user-agent', 'x-request-id'
+];
+}
+app.conf.log_header_whitelist = new RegExp('^(?:' + 
app.conf.log_header_whitelist.map(function(item) {
+return item.trim();
+}).join('|') + ')$', 'i');
+
+// set up the spec
+if(!app.conf.spec) {
+app.conf.spec = __dirname + '/spec.yaml';
+}
+if(app.conf.spec.constructor !== Object) {
+try {
+app.conf.spec = yaml.safeLoad(fs.readFileSync(app.conf.spec));
+} catch(e) {
+app.logger.log('warn/spec', 'Could not load the spec: ' + e);
+app.conf.spec = {};
+}
+}
+if(!app.conf.spec.swagger) {
+app.conf.spec.swagger = '2.0';
+}
+if(!app.conf.spec.info) {
+app.conf.spec.info = {
+version: app.info.version,
+title: app.info.name,
+description: app.info.description
+};
+}
+app.conf.spec.info.version = app.info.version;
+if(!app.conf.spec.paths) {
+app.conf.spec.paths = {};
+}
+
+// set the CORS and CSP headers
+app.all('*', function(req, res, next) {
+//
+// Tilerator is an admin app, there is no point to set app.conf.cors 
and app.conf.csp
+//
+sUtil.initAndLogRequest(req, app);
+next();
+});
+
+// disable the X-Powered-By header
+app.set('x-powered-by', false);
+// disable the ETag header - users should provide them!
+app.set('etag', false);
+// enable compression
+app.use(compression({level: app.conf.compression_level}));
+// use the JSON body 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix deleting old categories

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395082 )

Change subject: Fix deleting old categories
..


Fix deleting old categories

Change-Id: I201acddc402a15ac9cf0b50b3d87f8f8ac1d698a
---
M modules/wdqs/files/cron/reloadCategories.sh
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/files/cron/reloadCategories.sh 
b/modules/wdqs/files/cron/reloadCategories.sh
index b9b4859..33b83e9 100755
--- a/modules/wdqs/files/cron/reloadCategories.sh
+++ b/modules/wdqs/files/cron/reloadCategories.sh
@@ -18,7 +18,7 @@
 today=$(date -u +'%Y%m%d')
 newNamespace="categories${today}"
 # Drop old dumps
-rm -f "${DATA_DIR}/*-categories.ttl.gz"
+rm -f ${DATA_DIR}/*-categories.ttl.gz
 cd $DEPLOY_DIR
 bash createNamespace.sh $newNamespace || exit 1
 # Load the data

-- 
To view, visit https://gerrit.wikimedia.org/r/395082
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I201acddc402a15ac9cf0b50b3d87f8f8ac1d698a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Smalyshev 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: admin: allow maps-admins access to deployment servers

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395481 )

Change subject: admin: allow maps-admins access to deployment servers
..

admin: allow maps-admins access to deployment servers

It looks like we forgot to add maps-admins to the list of admins on
deployment servers. This was not an issue until now as maps-admins were
already in other access groups. This needs to be fixed to allow Paul
to deploy tilerator / kartotherian.

Bug: T182066
Change-Id: I0025d1391302217e8a18c1457f2ec7877c500266
---
M hieradata/role/common/deployment_server.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/81/395481/1

diff --git a/hieradata/role/common/deployment_server.yaml 
b/hieradata/role/common/deployment_server.yaml
index 7c55e05..e668cde 100644
--- a/hieradata/role/common/deployment_server.yaml
+++ b/hieradata/role/common/deployment_server.yaml
@@ -15,6 +15,7 @@
   - deploy-phabricator
   - analytics-admins
   - gerrit-root
+  - maps-admins
 
 # screen sessions are common on deployment servers (T165348)
 monitor_screens: false

-- 
To view, visit https://gerrit.wikimedia.org/r/395481
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0025d1391302217e8a18c1457f2ec7877c500266
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...package[master]: initial commit

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395478 )

Change subject: initial commit
..

initial commit

Change-Id: Iffebc2da34e0a1c64a02d12a5cf0a0a0f1216c61
---
A .dockerignore
A .gitignore
A app.js
A config.dev.yaml
A package.json
A server.js
A targets.yaml
7 files changed, 406 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/tilerator/package 
refs/changes/78/395478/1

diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 000..563df11
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,3 @@
+.git
+coverage
+node_modules
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..a0a1c47
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,8 @@
+Dockerfile
+.idea/
+coverage
+config.yaml
+node_modules
+npm-debug.log
+vectors
+variables.yaml
diff --git a/app.js b/app.js
new file mode 100644
index 000..f866804
--- /dev/null
+++ b/app.js
@@ -0,0 +1,203 @@
+'use strict';
+
+
+var http = require('http');
+var BBPromise = require('bluebird');
+var express = require('express');
+var compression = require('compression');
+var bodyParser = require('body-parser');
+var fs = BBPromise.promisifyAll(require('fs'));
+var sUtil = require('./lib/util');
+var packageInfo = require('./package.json');
+var yaml = require('js-yaml');
+
+
+/**
+ * Creates an express app and initialises it
+ * @param {Object} options the options to initialise the app with
+ * @return {bluebird} the promise resolving to the app object
+ */
+function initApp(options) {
+
+// the main application object
+var app = express();
+
+// get the options and make them available in the app
+app.logger = options.logger;// the logging device
+app.metrics = options.metrics;  // the metrics
+app.conf = options.config;  // this app's config options
+app.info = packageInfo; // this app's package info
+
+// ensure some sane defaults
+if(!app.conf.port) { app.conf.port = ; }
+if(!app.conf.interface) { app.conf.interface = '0.0.0.0'; }
+if(app.conf.compression_level === undefined) { app.conf.compression_level 
= 3; }
+if(app.conf.cors === undefined) { app.conf.cors = '*'; }
+if(app.conf.csp === undefined) {
+app.conf.csp =
+"default-src 'self'; object-src 'none'; media-src *; img-src *; 
style-src *; frame-ancestors 'self'";
+}
+
+// set outgoing proxy
+if(app.conf.proxy) {
+process.env.HTTP_PROXY = app.conf.proxy;
+// if there is a list of domains which should
+// not be proxied, set it
+if(app.conf.no_proxy_list) {
+if(Array.isArray(app.conf.no_proxy_list)) {
+process.env.NO_PROXY = app.conf.no_proxy_list.join(',');
+} else {
+process.env.NO_PROXY = app.conf.no_proxy_list;
+}
+}
+}
+
+// set up header whitelisting for logging
+if(!app.conf.log_header_whitelist) {
+app.conf.log_header_whitelist = [
+'cache-control', 'content-type', 'content-length', 'if-match',
+'user-agent', 'x-request-id'
+];
+}
+app.conf.log_header_whitelist = new RegExp('^(?:' + 
app.conf.log_header_whitelist.map(function(item) {
+return item.trim();
+}).join('|') + ')$', 'i');
+
+// set up the spec
+if(!app.conf.spec) {
+app.conf.spec = __dirname + '/spec.yaml';
+}
+if(app.conf.spec.constructor !== Object) {
+try {
+app.conf.spec = yaml.safeLoad(fs.readFileSync(app.conf.spec));
+} catch(e) {
+app.logger.log('warn/spec', 'Could not load the spec: ' + e);
+app.conf.spec = {};
+}
+}
+if(!app.conf.spec.swagger) {
+app.conf.spec.swagger = '2.0';
+}
+if(!app.conf.spec.info) {
+app.conf.spec.info = {
+version: app.info.version,
+title: app.info.name,
+description: app.info.description
+};
+}
+app.conf.spec.info.version = app.info.version;
+if(!app.conf.spec.paths) {
+app.conf.spec.paths = {};
+}
+
+// set the CORS and CSP headers
+app.all('*', function(req, res, next) {
+//
+// Tilerator is an admin app, there is no point to set app.conf.cors 
and app.conf.csp
+//
+sUtil.initAndLogRequest(req, app);
+next();
+});
+
+// disable the X-Powered-By header
+app.set('x-powered-by', false);
+// disable the ETag header - users should provide them!
+app.set('etag', false);
+// enable compression
+app.use(compression({level: app.conf.compression_level}));
+// use the JSON body parser
+app.use(bodyParser.json());
+// use the application/x-www-form-urlencoded parser
+app.use(bodyParser.urlencoded({extended: true}));
+
+return BBPromise.resolve(app);
+
+}
+
+
+/**
+ * Loads all routes declared in 

[MediaWiki-commits] [Gerrit] maps...deploy[master]: Update maps/kartotherian/package to 29a680f

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395088 )

Change subject: Update maps/kartotherian/package to 29a680f
..


Update maps/kartotherian/package to 29a680f

List of changes:
29a680f move to kartotherian update-snapshot-dep to iterate on fixing 
depencencies
xxx Update node module dependencies

Change-Id: Id4911703fd88f79354c8dcd368a92e512a77707e
---
M node_modules/@kartotherian/snapshot/lib/snapshot.js
M node_modules/@kartotherian/snapshot/package.json
A node_modules/aws-sdk/.changes/2.163.0.json
M node_modules/aws-sdk/apis/budgets-2016-10-20.min.json
M node_modules/aws-sdk/clients/budgets.d.ts
M node_modules/aws-sdk/dist/aws-sdk-core-react-native.js
M node_modules/aws-sdk/dist/aws-sdk-react-native.js
M node_modules/aws-sdk/dist/aws-sdk.js
M node_modules/aws-sdk/dist/aws-sdk.min.js
M node_modules/aws-sdk/lib/core.js
M node_modules/aws-sdk/package.json
M node_modules/kad/package.json
M node_modules/kartotherian/package.json
M node_modules/osm-bright-fonts/package.json
M node_modules/postgis-vt-util/package.json
M node_modules/wikimedia-mapdata/package.json
M node_modules/xmldoc/package.json
M npm-shrinkwrap.json
M src
19 files changed, 170 insertions(+), 103 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved




-- 
To view, visit https://gerrit.wikimedia.org/r/395088
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id4911703fd88f79354c8dcd368a92e512a77707e
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/deploy
Gerrit-Branch: master
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...deploy[master]: Update maps/kartotherian/package to 29a680f

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395088 )

Change subject: Update maps/kartotherian/package to 29a680f
..

Update maps/kartotherian/package to 29a680f

List of changes:
29a680f move to kartotherian update-snapshot-dep to iterate on fixing 
depencencies
xxx Update node module dependencies

Change-Id: Id4911703fd88f79354c8dcd368a92e512a77707e
---
M node_modules/@kartotherian/snapshot/lib/snapshot.js
M node_modules/@kartotherian/snapshot/package.json
A node_modules/aws-sdk/.changes/2.163.0.json
M node_modules/aws-sdk/apis/budgets-2016-10-20.min.json
M node_modules/aws-sdk/clients/budgets.d.ts
M node_modules/aws-sdk/dist/aws-sdk-core-react-native.js
M node_modules/aws-sdk/dist/aws-sdk-react-native.js
M node_modules/aws-sdk/dist/aws-sdk.js
M node_modules/aws-sdk/dist/aws-sdk.min.js
M node_modules/aws-sdk/lib/core.js
M node_modules/aws-sdk/package.json
M node_modules/kad/package.json
M node_modules/kartotherian/package.json
M node_modules/osm-bright-fonts/package.json
M node_modules/postgis-vt-util/package.json
M node_modules/wikimedia-mapdata/package.json
M node_modules/xmldoc/package.json
M npm-shrinkwrap.json
M src
19 files changed, 170 insertions(+), 103 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/kartotherian/deploy 
refs/changes/88/395088/1


-- 
To view, visit https://gerrit.wikimedia.org/r/395088
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id4911703fd88f79354c8dcd368a92e512a77707e
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/deploy
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...package[master]: move to kartotherian update-snapshot-dep to iterate on fixin...

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395078 )

Change subject: move to kartotherian update-snapshot-dep to iterate on fixing 
depencencies
..

move to kartotherian update-snapshot-dep to iterate on fixing depencencies

Change-Id: I2b0fce73ba1a0188a8da985db21ee26d3a43cc46
---
M package.json
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/kartotherian/package 
refs/changes/78/395078/1

diff --git a/package.json b/package.json
index 8bd60d3..5721b7b 100644
--- a/package.json
+++ b/package.json
@@ -51,7 +51,7 @@
 ]
   },
   "dependencies": {
-"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#master;,
+"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#update-snapshot-dep;,
 "service-runner": "^2.4.6"
   },
   "optionalDependencies": {

-- 
To view, visit https://gerrit.wikimedia.org/r/395078
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2b0fce73ba1a0188a8da985db21ee26d3a43cc46
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/package
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...package[master]: move to kartotherian update-snapshot-dep to iterate on fixin...

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395078 )

Change subject: move to kartotherian update-snapshot-dep to iterate on fixing 
depencencies
..


move to kartotherian update-snapshot-dep to iterate on fixing depencencies

Change-Id: I2b0fce73ba1a0188a8da985db21ee26d3a43cc46
---
M package.json
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Gehel: Verified; Looks good to me, approved



diff --git a/package.json b/package.json
index 8bd60d3..5721b7b 100644
--- a/package.json
+++ b/package.json
@@ -51,7 +51,7 @@
 ]
   },
   "dependencies": {
-"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#master;,
+"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#update-snapshot-dep;,
 "service-runner": "^2.4.6"
   },
   "optionalDependencies": {

-- 
To view, visit https://gerrit.wikimedia.org/r/395078
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2b0fce73ba1a0188a8da985db21ee26d3a43cc46
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/package
Gerrit-Branch: master
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...deploy[master]: Update maps/kartotherian/package to 9a5020a

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395070 )

Change subject: Update maps/kartotherian/package to 9a5020a
..

Update maps/kartotherian/package to 9a5020a

List of changes:
9a5020a use master branch for kartotherian
xxx Update node module dependencies

Change-Id: I83d8b9e974926c686af83babb05ebcc1e015b16e
---
M node_modules/kad/package.json
M node_modules/kartotherian/package.json
M node_modules/osm-bright-fonts/package.json
M node_modules/postgis-vt-util/package.json
M node_modules/wikimedia-mapdata/package.json
M node_modules/xmldoc/package.json
M npm-shrinkwrap.json
M src
8 files changed, 28 insertions(+), 28 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/kartotherian/deploy 
refs/changes/70/395070/1

diff --git a/node_modules/kad/package.json b/node_modules/kad/package.json
index c726e80..f7e46f2 100644
--- a/node_modules/kad/package.json
+++ b/node_modules/kad/package.json
@@ -49,7 +49,7 @@
 "/limitation"
   ],
   "_resolved": 
"git+https://github.com/gwicke/kad.git#936c91652d757ea6f9dd30e44698afb0daaa1d17;,
-  "_shasum": "1d8da62b59d3ddfc92327f0474f7281000606009",
+  "_shasum": "9ee999708fe8bd6095c1525ce82e4d45693bd6aa",
   "_shrinkwrap": null,
   "_spec": "kad@git+https://github.com/gwicke/kad.git#master;,
   "_where": "/opt/service/node_modules/limitation",
diff --git a/node_modules/kartotherian/package.json 
b/node_modules/kartotherian/package.json
index 1a83a16..14ccdf2 100644
--- a/node_modules/kartotherian/package.json
+++ b/node_modules/kartotherian/package.json
@@ -2,56 +2,56 @@
   "_args": [
 [
   {
-"raw": 
"kartotherian@git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
+"raw": 
"kartotherian@git+https://github.com/kartotherian/kartotherian.git#master;,
 "scope": null,
 "escapedName": "kartotherian",
 "name": "kartotherian",
-"rawSpec": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
-"spec": "git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
+"rawSpec": 
"git+https://github.com/kartotherian/kartotherian.git#master;,
+"spec": "git+https://github.com/kartotherian/kartotherian.git#master;,
 "type": "hosted",
 "hosted": {
   "type": "github",
-  "ssh": "g...@github.com:kartotherian/kartotherian.git#v0.0.32",
-  "sshUrl": 
"git+ssh://g...@github.com/kartotherian/kartotherian.git#v0.0.32",
-  "httpsUrl": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
-  "gitUrl": "git://github.com/kartotherian/kartotherian.git#v0.0.32",
-  "shortcut": "github:kartotherian/kartotherian#v0.0.32",
-  "directUrl": 
"https://raw.githubusercontent.com/kartotherian/kartotherian/v0.0.32/package.json;
+  "ssh": "g...@github.com:kartotherian/kartotherian.git#master",
+  "sshUrl": 
"git+ssh://g...@github.com/kartotherian/kartotherian.git#master",
+  "httpsUrl": 
"git+https://github.com/kartotherian/kartotherian.git#master;,
+  "gitUrl": "git://github.com/kartotherian/kartotherian.git#master",
+  "shortcut": "github:kartotherian/kartotherian#master",
+  "directUrl": 
"https://raw.githubusercontent.com/kartotherian/kartotherian/master/package.json;
 }
   },
   "/opt/service"
 ]
   ],
-  "_from": "git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
+  "_from": "git+https://github.com/kartotherian/kartotherian.git#master;,
   "_id": "kartotherian@0.0.32",
   "_inCache": true,
   "_location": "/kartotherian",
   "_phantomChildren": {},
   "_requested": {
-"raw": 
"kartotherian@git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
+"raw": 
"kartotherian@git+https://github.com/kartotherian/kartotherian.git#master;,
 "scope": null,
 "escapedName": "kartotherian",
 "name": "kartotherian",
-"rawSpec": "git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
-"spec": "git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
+"rawSpec": "git+https://github.com/kartotherian/kartotherian.git#master;,
+"spec": "git+https://github.com/kartotherian/kartotherian.git#master;,
 "type": "hosted",
 "hosted": {
   "type": "github",
-  "ssh": "g...@github.com:kartotherian/kartotherian.git#v0.0.32",
-  "sshUrl": 
"git+ssh://g...@github.com/kartotherian/kartotherian.git#v0.0.32",
-  "httpsUrl": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
-  "gitUrl": "git://github.com/kartotherian/kartotherian.git#v0.0.32",
-  "shortcut": "github:kartotherian/kartotherian#v0.0.32",
-  "directUrl": 
"https://raw.githubusercontent.com/kartotherian/kartotherian/v0.0.32/package.json;
+  "ssh": "g...@github.com:kartotherian/kartotherian.git#master",
+  "sshUrl": 
"git+ssh://g...@github.com/kartotherian/kartotherian.git#master",
+   

[MediaWiki-commits] [Gerrit] maps...package[master]: use master branch for kartotherian

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395068 )

Change subject: use master branch for kartotherian
..

use master branch for kartotherian

As long as we are iterating on fixing the kartotherian deployment, let's
point directly to master. Once we are good, we can tag things properly.

Change-Id: I7a9a1597d965f59cfa645a24e45611936fcc4613
---
M package.json
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/kartotherian/package 
refs/changes/68/395068/1

diff --git a/package.json b/package.json
index 1d3d434..8bd60d3 100644
--- a/package.json
+++ b/package.json
@@ -51,7 +51,7 @@
 ]
   },
   "dependencies": {
-"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
+"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#master;,
 "service-runner": "^2.4.6"
   },
   "optionalDependencies": {

-- 
To view, visit https://gerrit.wikimedia.org/r/395068
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7a9a1597d965f59cfa645a24e45611936fcc4613
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/package
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Enable wdqs-admins to restart nginx

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/393814 )

Change subject: Enable wdqs-admins to restart nginx
..


Enable wdqs-admins to restart nginx

Bug: T181540
Change-Id: I9bbf59ed996534fefa299baced30e4bfc3056813
---
M modules/admin/data/data.yaml
1 file changed, 5 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved
  Dzahn: Looks good to me, but someone else must approve



diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index f0eb5af..b4711cf 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -369,6 +369,11 @@
 members: [smalyshev, hoo]
 privileges: ['ALL = NOPASSWD: /usr/sbin/service wdqs-blazegraph *',
  'ALL = NOPASSWD: /usr/sbin/service wdqs-updater *',
+ 'ALL = NOPASSWD: /usr/sbin/service nginx *',
+ 'ALL = NOPASSWD: /bin/systemctl reload nginx',
+ 'ALL = NOPASSWD: /bin/systemctl restart nginx',
+ 'ALL = NOPASSWD: /bin/systemctl start nginx',
+ 'ALL = NOPASSWD: /bin/systemctl stop nginx',
  'ALL = NOPASSWD: /bin/systemctl enable wdqs-updater',
  'ALL = NOPASSWD: /bin/systemctl disable wdqs-updater',
  'ALL = NOPASSWD: /bin/systemctl mask wdqs-updater',

-- 
To view, visit https://gerrit.wikimedia.org/r/393814
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9bbf59ed996534fefa299baced30e4bfc3056813
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Smalyshev 
Gerrit-Reviewer: Alex Monk 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Dzahn 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: udp2log: use LVS endpoint for logstash

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/388426 )

Change subject: udp2log: use LVS endpoint for logstash
..


udp2log: use LVS endpoint for logstash

* externalized variables used for configuration
* moved to hiera parameters given directly in site.pp

Bug: T175242
Change-Id: I9001dfdb82d336d9325c569d5562921a16553484
---
M hieradata/hosts/mwlog1001.yaml
M hieradata/hosts/mwlog2001.yaml
M hieradata/labs.yaml
A hieradata/role/common/logging/mediawiki/udp2log.yaml
M manifests/site.pp
M modules/role/manifests/logging/mediawiki/udp2log.pp
6 files changed, 8 insertions(+), 17 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Gehel: Looks good to me, approved



diff --git a/hieradata/hosts/mwlog1001.yaml b/hieradata/hosts/mwlog1001.yaml
index a8474a6..2c5d68a 100644
--- a/hieradata/hosts/mwlog1001.yaml
+++ b/hieradata/hosts/mwlog1001.yaml
@@ -4,3 +4,4 @@
   - mw-log-readers
 role::logging::mediawiki::udp2log::rsync_slow_parse: true
 role::logging::mediawiki::udp2log::forward_messages: true
+role::logging::mediawiki::udp2log::monitor: false
diff --git a/hieradata/hosts/mwlog2001.yaml b/hieradata/hosts/mwlog2001.yaml
index 19e695d..c899747 100644
--- a/hieradata/hosts/mwlog2001.yaml
+++ b/hieradata/hosts/mwlog2001.yaml
@@ -4,3 +4,4 @@
   - mw-log-readers
 role::logging::mediawiki::udp2log::rsync_slow_parse: false
 role::logging::mediawiki::udp2log::forward_messages: false
+role::logging::mediawiki::udp2log::monitor: false
diff --git a/hieradata/labs.yaml b/hieradata/labs.yaml
index 9cbccbf..ce53a17 100644
--- a/hieradata/labs.yaml
+++ b/hieradata/labs.yaml
@@ -133,3 +133,5 @@
 
 # User for jenkins master-slave connections
 jenkins_agent_username: 'jenkins-deploy'
+
+role::logging::mediawiki::udp2log::logstash_host: 
'deployment-logstash2.deployment-prep.eqiad.wmflabs'
diff --git a/hieradata/role/common/logging/mediawiki/udp2log.yaml 
b/hieradata/role/common/logging/mediawiki/udp2log.yaml
new file mode 100644
index 000..fd7a401
--- /dev/null
+++ b/hieradata/role/common/logging/mediawiki/udp2log.yaml
@@ -0,0 +1 @@
+role::logging::mediawiki::udp2log::logstash_host: 'logstash.svc.eqiad.wmnet'
diff --git a/manifests/site.pp b/manifests/site.pp
index 120e30a..2558865 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1817,26 +1817,18 @@
 
 # mw logging host codfw
 node 'mwlog2001.codfw.wmnet' {
-role(xenon)
+role(xenon, logging::mediawiki::udp2log)
 
 include ::base::firewall
 include ::standard
-
-class { 'role::logging::mediawiki::udp2log':
-monitor => false,
-}
 }
 
 # mw logging host eqiad
 node 'mwlog1001.eqiad.wmnet' {
-role(xenon)
+role(xenon, logging::mediawiki::udp2log)
 
 include ::base::firewall
 include ::standard
-
-class { 'role::logging::mediawiki::udp2log':
-monitor => false,
-}
 }
 
 node 'mx1001.wikimedia.org' {
diff --git a/modules/role/manifests/logging/mediawiki/udp2log.pp 
b/modules/role/manifests/logging/mediawiki/udp2log.pp
index 3176b42..69ad73d 100644
--- a/modules/role/manifests/logging/mediawiki/udp2log.pp
+++ b/modules/role/manifests/logging/mediawiki/udp2log.pp
@@ -2,6 +2,7 @@
 #
 # filtertags: labs-project-deployment-prep
 class role::logging::mediawiki::udp2log(
+$logstash_host,
 $monitor = true,
 $log_directory = '/srv/mw-log',
 $rotate = 1000,
@@ -55,13 +56,6 @@
 owner  => 'root',
 group  => 'root',
 source => 'puppet:///modules/udp2log/udpmirror.py',
-}
-
-$logstash_host = $::realm ? {
-# TODO: Find a way to use multicast that doesn't cause duplicate
-# messages to be stored in logstash. This is a SPOF.
-production => 'logstash1001.eqiad.wmnet',
-labs   => 'deployment-logstash2.deployment-prep.eqiad.wmflabs',
 }
 
 $logstash_port = 8324

-- 
To view, visit https://gerrit.wikimedia.org/r/388426
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9001dfdb82d336d9325c569d5562921a16553484
Gerrit-PatchSet: 6
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wdqs: use the /readiness-probe in WDQS icinga checks

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394987 )

Change subject: wdqs: use the /readiness-probe in WDQS icinga checks
..

wdqs: use the /readiness-probe in WDQS icinga checks

Bug: T181989
Change-Id: Ic42863eb87fe936183189644ef9b1635c29f568d
---
M modules/wdqs/manifests/monitor/services.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/87/394987/1

diff --git a/modules/wdqs/manifests/monitor/services.pp 
b/modules/wdqs/manifests/monitor/services.pp
index 29cf7a3..ec67508 100644
--- a/modules/wdqs/manifests/monitor/services.pp
+++ b/modules/wdqs/manifests/monitor/services.pp
@@ -8,7 +8,7 @@
 
 nrpe::monitor_service { 'WDQS_Internal_HTTP_endpoint':
 description  => 'WDQS HTTP Port',
-nrpe_command => '/usr/lib/nagios/plugins/check_tcp -H 127.0.0.1 -p 80',
+nrpe_command => '/usr/lib/nagios/plugins/check_http -H 127.0.0.1 -p 80 
-w 10 -u /readiness-probe',
 }
 
 nrpe::monitor_service { 'WDQS_Local_Blazegraph_endpoint':

-- 
To view, visit https://gerrit.wikimedia.org/r/394987
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic42863eb87fe936183189644ef9b1635c29f568d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: maps: Bump maximum zoom to 19

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394948 )

Change subject: maps: Bump maximum zoom to 19
..

maps: Bump maximum zoom to 19

Per Community Wishlist, community wants this direly.

Bug: T180907
Change-Id: I356602e3e77ab373372d958e96976ebf992cf34a
---
M modules/profile/templates/maps/sources.yaml.erb
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/48/394948/1

diff --git a/modules/profile/templates/maps/sources.yaml.erb 
b/modules/profile/templates/maps/sources.yaml.erb
index 04cb88c..a964d99 100644
--- a/modules/profile/templates/maps/sources.yaml.erb
+++ b/modules/profile/templates/maps/sources.yaml.erb
@@ -82,7 +82,7 @@
   uri: overzoom://
   params:
 source: {ref: <%= @storage_id %>}
-maxzoom: 18
+maxzoom: 19
   overrideInfo:
 attribution: 'Map data © http://openstreetmap.org/copyright;>OpenStreetMap contributors'
 tiles: ['https://maps.wikimedia.org/osm-pbf/{z}/{x}/{y}.pbf']
@@ -92,7 +92,7 @@
   public: true
   formats: [png,json,headers,svg,jpeg]
   scales: [1.3, 1.5, 2, 2.6, 3]
-  maxzoom: 18
+  maxzoom: 19
   static: true
   maxheight: 2048
   maxwidth: 2048
@@ -110,7 +110,7 @@
   public: true
   formats: [png,json,headers,svg,jpeg]
   scales: [1.3, 1.5, 2, 2.6, 3]
-  maxzoom: 18
+  maxzoom: 19
   static: true
   maxheight: 2048
   maxwidth: 2048

-- 
To view, visit https://gerrit.wikimedia.org/r/394948
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I356602e3e77ab373372d958e96976ebf992cf34a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...package[master]: Bump Kartotherian version

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394576 )

Change subject: Bump Kartotherian version
..


Bump Kartotherian version

This fixes the transitive dependency issue that was bringing mapnik 3.6.2

Bug: T181718
Change-Id: I8d1333f0ad82f0c12cbb79982b28d3b23fa630b2
---
M package.json
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Gehel: Verified; Looks good to me, approved



diff --git a/package.json b/package.json
index 207282c..1d3d434 100644
--- a/package.json
+++ b/package.json
@@ -51,7 +51,7 @@
 ]
   },
   "dependencies": {
-"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.31;,
+"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
 "service-runner": "^2.4.6"
   },
   "optionalDependencies": {

-- 
To view, visit https://gerrit.wikimedia.org/r/394576
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8d1333f0ad82f0c12cbb79982b28d3b23fa630b2
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/package
Gerrit-Branch: master
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...deploy[master]: New packaging repository for kartotherian

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394575 )

Change subject: New packaging repository for kartotherian
..


New packaging repository for kartotherian

Bug: T162241
Change-Id: I154c9e5dd67cb9b70b7f629edcbcd249a9fc6453
---
M .gitmodules
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Verified; Looks good to me, approved



diff --git a/.gitmodules b/.gitmodules
index 593d9a5..69c5a2e 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,3 @@
 [submodule "src"]
path = src
-   url = https://gerrit.wikimedia.org/r/maps/kartotherian
+   url = https://gerrit.wikimedia.org/r/maps/kartotherian/package

-- 
To view, visit https://gerrit.wikimedia.org/r/394575
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I154c9e5dd67cb9b70b7f629edcbcd249a9fc6453
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/deploy
Gerrit-Branch: master
Gerrit-Owner: Gehel 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...package[master]: Bump Kartotherian version

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394576 )

Change subject: Bump Kartotherian version
..

Bump Kartotherian version

This fixes the transitive dependency issue that was bringing mapnik 3.6.2

Bug: T181718
Change-Id: I8d1333f0ad82f0c12cbb79982b28d3b23fa630b2
---
M package.json
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/kartotherian/package 
refs/changes/76/394576/1

diff --git a/package.json b/package.json
index 207282c..1d3d434 100644
--- a/package.json
+++ b/package.json
@@ -51,7 +51,7 @@
 ]
   },
   "dependencies": {
-"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.31;,
+"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.32;,
 "service-runner": "^2.4.6"
   },
   "optionalDependencies": {

-- 
To view, visit https://gerrit.wikimedia.org/r/394576
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I8d1333f0ad82f0c12cbb79982b28d3b23fa630b2
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/package
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] maps...deploy[master]: New packaging repository for kartotherian

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394575 )

Change subject: New packaging repository for kartotherian
..

New packaging repository for kartotherian

Bug: T162241
Change-Id: I154c9e5dd67cb9b70b7f629edcbcd249a9fc6453
---
M .gitmodules
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/kartotherian/deploy 
refs/changes/75/394575/1

diff --git a/.gitmodules b/.gitmodules
index 593d9a5..69c5a2e 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,3 @@
 [submodule "src"]
path = src
-   url = https://gerrit.wikimedia.org/r/maps/kartotherian
+   url = https://gerrit.wikimedia.org/r/maps/kartotherian/package

-- 
To view, visit https://gerrit.wikimedia.org/r/394575
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I154c9e5dd67cb9b70b7f629edcbcd249a9fc6453
Gerrit-PatchSet: 1
Gerrit-Project: maps/kartotherian/deploy
Gerrit-Branch: master
Gerrit-Owner: Gehel 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix another place where gui-vars location was wrong

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394141 )

Change subject: Fix another place where gui-vars location was wrong
..


Fix another place where gui-vars location was wrong

Change-Id: Ief68b82ae47b0bb31fe3c477b2f93fa0b7cf5b27
---
M modules/wdqs/manifests/gui.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index 7778c16..617695f 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -65,7 +65,7 @@
 owner   => 'root',
 group   => 'root',
 mode=> '0755',
-require => File['/etc/default/gui_vars.sh'],
+require => File['/etc/wdqs/gui_vars.sh'],
 }
 
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/394141
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ief68b82ae47b0bb31fe3c477b2f93fa0b7cf5b27
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Smalyshev 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix gui-vars location

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394139 )

Change subject: Fix gui-vars location
..


Fix gui-vars location

Change-Id: Ib10ae49f256f2d0c8ddb8b57b4e7807ac1529fc4
---
M modules/wdqs/manifests/gui.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index f677ce0..7778c16 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -53,7 +53,7 @@
 
 file { '/etc/wdqs/gui_vars.sh':
 ensure  => present,
-content => template('wdqs/cron/gui_vars.erb'),
+content => template('wdqs/gui_vars.erb'),
 owner   => 'root',
 group   => 'root',
 mode=> '0644',

-- 
To view, visit https://gerrit.wikimedia.org/r/394139
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib10ae49f256f2d0c8ddb8b57b4e7807ac1529fc4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Smalyshev 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix reloadCategories.sh - nginx needs semicolon in maps

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394134 )

Change subject: Fix reloadCategories.sh - nginx needs semicolon in maps
..


Fix reloadCategories.sh - nginx needs semicolon in maps

Change-Id: I54762de5c99eb845af82048383f9b9575a7d2ac7
---
M modules/wdqs/files/cron/reloadCategories.sh
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/files/cron/reloadCategories.sh 
b/modules/wdqs/files/cron/reloadCategories.sh
index 2d21a2d..b9b4859 100755
--- a/modules/wdqs/files/cron/reloadCategories.sh
+++ b/modules/wdqs/files/cron/reloadCategories.sh
@@ -24,10 +24,10 @@
 # Load the data
 bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace >> 
"${LOG_DIR}/${newNamespace}.log"
 # Get old namespace
-oldNamespace=$(cat $ALIAS_FILE | grep categories | cut -d' ' -f2)
+oldNamespace=$(cat $ALIAS_FILE | grep categories | cut -d' ' -f2 | cut -d ';' 
-f1)
 # Switch the map
 # NOTE: right now it overrides the map. If we reuse it for other purposes, 
this needs to be made smarter.
-echo "categories ${newNamespace}" > $ALIAS_FILE
+echo "categories ${newNamespace};" > $ALIAS_FILE
 # Bump nginx to reload config
 sudo systemctl reload nginx
 if [ -n "${oldNamespace}" ]; then

-- 
To view, visit https://gerrit.wikimedia.org/r/394134
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I54762de5c99eb845af82048383f9b9575a7d2ac7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Smalyshev 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Create script for automatic reload of categories

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/392736 )

Change subject: Create script for automatic reload of categories
..


Create script for automatic reload of categories

Bug: T173772
Change-Id: I96e0863aff73a3cece8c93c8f03ee4901759b7b1
---
A modules/wdqs/files/cron/reloadCategories.sh
M modules/wdqs/manifests/gui.pp
M modules/wdqs/manifests/init.pp
A modules/wdqs/templates/gui_vars.erb
M modules/wdqs/templates/vars.yaml.erb
5 files changed, 72 insertions(+), 4 deletions(-)

Approvals:
  jenkins-bot: Verified
  Gehel: Looks good to me, approved



diff --git a/modules/wdqs/files/cron/reloadCategories.sh 
b/modules/wdqs/files/cron/reloadCategories.sh
new file mode 100755
index 000..2d21a2d
--- /dev/null
+++ b/modules/wdqs/files/cron/reloadCategories.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+# This script is reloading categories into a new namespace
+# NOTE: This should be run under user that has rights to
+# sudo systemctl reload nginx
+if [ -r /etc/wdqs/vars.sh ]; then
+  . /etc/wdqs/vars.sh
+fi
+
+if [ -r /etc/wdqs/gui_vars.sh ]; then
+  . /etc/wdqs/gui_vars.sh
+fi
+
+if [ -z "${DATA_DIR}" -o -z "${LOG_DIR}" -o -z "${DEPLOY_DIR}" ]; then
+   echo "Variables not set up right!"
+   exit 1
+fi
+
+today=$(date -u +'%Y%m%d')
+newNamespace="categories${today}"
+# Drop old dumps
+rm -f "${DATA_DIR}/*-categories.ttl.gz"
+cd $DEPLOY_DIR
+bash createNamespace.sh $newNamespace || exit 1
+# Load the data
+bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace >> 
"${LOG_DIR}/${newNamespace}.log"
+# Get old namespace
+oldNamespace=$(cat $ALIAS_FILE | grep categories | cut -d' ' -f2)
+# Switch the map
+# NOTE: right now it overrides the map. If we reuse it for other purposes, 
this needs to be made smarter.
+echo "categories ${newNamespace}" > $ALIAS_FILE
+# Bump nginx to reload config
+sudo systemctl reload nginx
+if [ -n "${oldNamespace}" ]; then
+   # Drop old namespace
+   curl -s -X DELETE 
"http://localhost:/bigdata/namespace/${oldNamespace};
+fi
diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index 532c5c2..f677ce0 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -3,19 +3,25 @@
 # Provisions WDQS GUI
 #
 # == Parameters:
+# - $logstash_host: Where to send the logs for the service in syslog format
+# - $logstash_syslog_port: port on which to send logs in syslog format
 # - $package_dir:  Directory where the service is installed.
-# GUI files are expected to be under its gui/ directory.
+#   GUI files are expected to be under its gui/ directory.
 # - $data_dir: Where the data is installed.
-# - $logstash_host: Where to send the logs for the service in syslog format.
-#
+# - $log_dir: Directory where the logs go
+# - $username: Username owning the service
+# - $port: main GUI service port
+# - $additional_port: secondary port for internal requests
 class wdqs::gui(
 $logstash_host = undef,
 $logstash_syslog_port = 10514,
 $package_dir = $::wdqs::package_dir,
 $data_dir = $::wdqs::data_dir,
+$log_dir = $::wdqs::log_dir,
 $username = $::wdqs::username,
 $port = 80,
 $additional_port = ,
+$use_git_deploy = $::wdqs::use_git_deploy,
 ) {
 
 $alias_map = "${data_dir}/aliases.map"
@@ -25,7 +31,7 @@
 require => File[$alias_map],
 }
 
-# List of namespace aliases in format: 
+# List of namespace aliases in format:
 # ALIAS REAL_NAME
 # This map is generated manually or by category update script
 file { $alias_map:
@@ -44,4 +50,22 @@
 # Because nginx site creates /var/lib/nginx
 require => Nginx::Site['wdqs'],
 }
+
+file { '/etc/wdqs/gui_vars.sh':
+ensure  => present,
+content => template('wdqs/cron/gui_vars.erb'),
+owner   => 'root',
+group   => 'root',
+mode=> '0644',
+}
+
+file { '/usr/local/bin/reloadCategories.sh':
+ensure  => present,
+source  => 'puppet:///modules/wdqs/cron/reloadCategories.sh',
+owner   => 'root',
+group   => 'root',
+mode=> '0755',
+require => File['/etc/default/gui_vars.sh'],
+}
+
 }
diff --git a/modules/wdqs/manifests/init.pp b/modules/wdqs/manifests/init.pp
index 567660f..8849c6e 100644
--- a/modules/wdqs/manifests/init.pp
+++ b/modules/wdqs/manifests/init.pp
@@ -6,6 +6,7 @@
 # == Parameters:
 # - $logstash_host: hostname where to send logs
 # - $logstash_json_port: port on which to send logs in json format
+# - $use_git_deploy: whether scap deployment is being used
 # - $username: Username owning the service
 # - $package_dir:  Directory where the service should be installed.
 # - $data_dir: Directory where the database should be stored
diff --git a/modules/wdqs/templates/gui_vars.erb 
b/modules/wdqs/templates/gui_vars.erb
new file mode 100644
index 000..6b18fff
--- /dev/null
+++ 

[MediaWiki-commits] [Gerrit] maps...package[master]: align package.json with currently deployed version

[Gehel (Code Review)]

Gehel has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394027 )

Change subject: align package.json with currently deployed version
..


align package.json with currently deployed version

Bug: T162241
Change-Id: I2528c345802084163d384d7d57b4f083f2f82077
---
A .gitreview
M app.js
M package.json
3 files changed, 58 insertions(+), 63 deletions(-)

Approvals:
  Gehel: Verified; Looks good to me, approved



diff --git a/.gitreview b/.gitreview
new file mode 100644
index 000..61c6841
--- /dev/null
+++ b/.gitreview
@@ -0,0 +1,6 @@
+[gerrit]
+host=gerrit.wikimedia.org
+port=29418
+project=maps/kartotherian/package.git
+defaultbranch=master
+defaultrebase=0
diff --git a/app.js b/app.js
index 951251a..4c44358 100644
--- a/app.js
+++ b/app.js
@@ -1,47 +1,4 @@
 'use strict';
 
-var kartotherian = require('kartotherian')
-
-/**
- * Creates an express app and initialises it
- * @param {Object} options the options to initialise the app with
- * @return {bluebird} the promise resolving to the app object
- */
-function initApp(options) {
-return kartotherian.app.initApp(options);
-}
-
-
-/**
- * Loads all routes declared in routes/ into the app
- * @param {Application} app the application object to load routes into
- * @returns {bluebird} a promise resolving to the app object
- */
-function loadRoutes (app) {
-return kartotherian.app.loadRoutes(app);
-}
-
-
-/**
- * Creates and start the service's web server
- * @param {Application} app the app object to use in the service
- * @returns {bluebird} a promise creating the web server
- */
-function createServer(app) {
-return kartotherian.app.createServer(app);
-}
-
-
-/**
- * The service's entry point. It takes over the configuration
- * options and the logger- and metrics-reporting objects from
- * service-runner and starts an HTTP server, attaching the application
- * object to it.
- */
-module.exports = function(options) {
-
-return initApp(options)
-.then(loadRoutes)
-.then(createServer);
-
-};
+// just delegate everything to the main kartotherian package
+module.exports = require('kartotherian')
diff --git a/package.json b/package.json
index eacb587..207282c 100644
--- a/package.json
+++ b/package.json
@@ -1,5 +1,5 @@
 {
-  "name": "kartotherian-packaging",
+  "name": "kartotherian-package",
   "version": "0.0.1",
   "description": "Packaging of Kartotherian for use at WMF",
   "main": "./app.js",
@@ -12,7 +12,7 @@
   },
   "repository": {
 "type": "git",
-"url": "git://github.com/kartotherian/kartotherian-packaging.git"
+"url": "https://gerrit.wikimedia.org/r/maps/kartotherian/package;
   },
   "keywords": [
 "REST",
@@ -36,33 +36,42 @@
 "registerSourceLibs": [
   "tilelive-bridge",
   "tilelive-vector",
-  "@kartotherian/autogen",
-  "@kartotherian/babel",
-  "@kartotherian/cassandra",
-  "@kartotherian/demultiplexer",
-  "@kartotherian/layermixer",
-  "@kartotherian/overzoom",
-  "@kartotherian/postgres",
-  "@kartotherian/substantial",
-  "@kartotherian/tilelive-http",
-  "tilejson"
+  "kartotherian-autogen",
+  "kartotherian-cassandra",
+  "kartotherian-postgres",
+  "kartotherian-demultiplexer",
+  "kartotherian-layermixer",
+  "kartotherian-overzoom",
+  "kartotherian-substantial"
 ],
 "requestHandlers": [
-  "@kartotherian/geoshapes",
-  "@kartotherian/maki",
-  "@kartotherian/snapshot"
+  "kartotherian-geoshapes",
+  "kartotherian-maki",
+  "kartotherian-snapshot"
 ]
   },
   "dependencies": {
-"kartotherian": "git+https://github.com/kartotherian/kartotherian.git;,
+"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.31;,
 "service-runner": "^2.4.6"
   },
   "optionalDependencies": {
 "bunyan-prettystream": "*"
   },
+  "devDependencies": {
+"extend": "^3.0.0",
+"istanbul": "^0.4.3",
+"mocha": "^2.4.5",
+"mocha-jshint": "^2.3.1",
+"mocha-lcov-reporter": "^1.2.0",
+"swagger-router": "^0.4.2"
+  },
   "deploy": {
-"node": "6.11",
+"node": "6.9.1",
 "target": "debian",
+"install_opts": [
+  "--build-from-source=mapnik",
+  "--fallback-to-build=false"
+],
 "dependencies": {
   "_all": [
 "libcairo2-dev",
@@ -73,7 +82,30 @@
 "libjpeg62-dev"
   ],
   "debian": [
-"libjpeg62-turbo-dev"
+"libjpeg62-turbo-dev",
+"fonts-dejavu",
+"libboost-filesystem-dev",
+"libboost-program-options-dev",
+"libboost-regex-dev",
+"libboost-system-dev",
+"libboost-thread-dev",
+"libgdal-dev",
+"libicu-dev",
+"libpq-dev",
+"libcurl4-gnutls-dev",
+"libproj-dev",
+"libtiff-dev",
+"libwebp5",
+{
+  "repo_url": "https://apt.wikimedia.org/wikimedia;,
+  "release": 

[MediaWiki-commits] [Gerrit] maps...package[master]: align package.json with currently deployed version

[Gehel (Code Review)]

Gehel has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394027 )

Change subject: align package.json with currently deployed version
..

align package.json with currently deployed version

Bug: T162241
Change-Id: I2528c345802084163d384d7d57b4f083f2f82077
---
A .gitreview
M app.js
M package.json
3 files changed, 55 insertions(+), 71 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/maps/kartotherian/package 
refs/changes/27/394027/1

diff --git a/.gitreview b/.gitreview
new file mode 100644
index 000..61c6841
--- /dev/null
+++ b/.gitreview
@@ -0,0 +1,6 @@
+[gerrit]
+host=gerrit.wikimedia.org
+port=29418
+project=maps/kartotherian/package.git
+defaultbranch=master
+defaultrebase=0
diff --git a/app.js b/app.js
index 951251a..4c44358 100644
--- a/app.js
+++ b/app.js
@@ -1,47 +1,4 @@
 'use strict';
 
-var kartotherian = require('kartotherian')
-
-/**
- * Creates an express app and initialises it
- * @param {Object} options the options to initialise the app with
- * @return {bluebird} the promise resolving to the app object
- */
-function initApp(options) {
-return kartotherian.app.initApp(options);
-}
-
-
-/**
- * Loads all routes declared in routes/ into the app
- * @param {Application} app the application object to load routes into
- * @returns {bluebird} a promise resolving to the app object
- */
-function loadRoutes (app) {
-return kartotherian.app.loadRoutes(app);
-}
-
-
-/**
- * Creates and start the service's web server
- * @param {Application} app the app object to use in the service
- * @returns {bluebird} a promise creating the web server
- */
-function createServer(app) {
-return kartotherian.app.createServer(app);
-}
-
-
-/**
- * The service's entry point. It takes over the configuration
- * options and the logger- and metrics-reporting objects from
- * service-runner and starts an HTTP server, attaching the application
- * object to it.
- */
-module.exports = function(options) {
-
-return initApp(options)
-.then(loadRoutes)
-.then(createServer);
-
-};
+// just delegate everything to the main kartotherian package
+module.exports = require('kartotherian')
diff --git a/package.json b/package.json
index eacb587..5047427 100644
--- a/package.json
+++ b/package.json
@@ -1,5 +1,5 @@
 {
-  "name": "kartotherian-packaging",
+  "name": "kartotherian-package",
   "version": "0.0.1",
   "description": "Packaging of Kartotherian for use at WMF",
   "main": "./app.js",
@@ -12,7 +12,7 @@
   },
   "repository": {
 "type": "git",
-"url": "git://github.com/kartotherian/kartotherian-packaging.git"
+"url": "https://gerrit.wikimedia.org/r/maps/kartotherian/package;
   },
   "keywords": [
 "REST",
@@ -32,48 +32,69 @@
 "url": "https://phabricator.wikimedia.org/tag/maps/;
   },
   "homepage": "https://github.com/kartotherian/kartotherian;,
+
   "kartotherian": {
 "registerSourceLibs": [
   "tilelive-bridge",
   "tilelive-vector",
-  "@kartotherian/autogen",
-  "@kartotherian/babel",
-  "@kartotherian/cassandra",
-  "@kartotherian/demultiplexer",
-  "@kartotherian/layermixer",
-  "@kartotherian/overzoom",
-  "@kartotherian/postgres",
-  "@kartotherian/substantial",
-  "@kartotherian/tilelive-http",
-  "tilejson"
+  "kartotherian-autogen",
+  "kartotherian-cassandra",
+  "kartotherian-postgres",
+  "kartotherian-demultiplexer",
+  "kartotherian-layermixer",
+  "kartotherian-overzoom",
+  "kartotherian-substantial"
 ],
 "requestHandlers": [
-  "@kartotherian/geoshapes",
-  "@kartotherian/maki",
-  "@kartotherian/snapshot"
+  "kartotherian-geoshapes",
+  "kartotherian-maki",
+  "kartotherian-snapshot"
 ]
   },
+
   "dependencies": {
-"kartotherian": "git+https://github.com/kartotherian/kartotherian.git;,
+"kartotherian": 
"git+https://github.com/kartotherian/kartotherian.git#v0.0.31;,
 "service-runner": "^2.4.6"
   },
   "optionalDependencies": {
 "bunyan-prettystream": "*"
   },
+  "devDependencies": {
+"extend": "^3.0.0",
+"istanbul": "^0.4.3",
+"mocha": "^2.4.5",
+"mocha-jshint": "^2.3.1",
+"mocha-lcov-reporter": "^1.2.0",
+"swagger-router": "^0.4.2"
+  },
   "deploy": {
-"node": "6.11",
+"node": "6.9.1",
 "target": "debian",
+"install_opts": ["--build-from-source=mapnik", 
"--fallback-to-build=false"] ,
 "dependencies": {
-  "_all": [
-"libcairo2-dev",
-"libgif-dev",
-"libpango1.0-dev"
-  ],
-  "ubuntu": [
-"libjpeg62-dev"
-  ],
+  "_all": ["libcairo2-dev", "libgif-dev", "libpango1.0-dev"],
+  "ubuntu": ["libjpeg62-dev"],
   "debian": [
-"libjpeg62-turbo-dev"
+"libjpeg62-turbo-dev",
+"fonts-dejavu",
+"libboost-filesystem-dev",
+"libboost-program-options-dev",
+"libboost-regex-dev",
+