[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase200[789]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/405312 )

Change subject: restbase: reprovision restbase200[789]
..


restbase: reprovision restbase200[789]

These hosts have been already reimaged and have all cassandra instances masked 
already. Therefore we
can assign the restbase role for all, then enable instances one by one as 
bootstrap complete:

  systemctl unmask cassandra-
  puppet agent --test

Bug: T184100
Change-Id: I5a4c99b2a49da13c8d2f5ad13d786aa7d18b028c
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
3 files changed, 139 insertions(+), 26 deletions(-)

Approvals:
  Eevans: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index d5a4fa0..c7ed3a1 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -190,27 +190,27 @@
   #listen_address: 10.192.48.50
   #  c:
   #listen_address: 10.192.48.51
-  "restbase2007.codfw.wmnet":
-a:
-  listen_address: 10.192.16.176
-b:
-  listen_address: 10.192.16.177
-c:
-  listen_address: 10.192.16.178
-  "restbase2008.codfw.wmnet":
-a:
-  listen_address: 10.192.32.143
-b:
-  listen_address: 10.192.32.144
-c:
-  listen_address: 10.192.32.145
-  "restbase2009.codfw.wmnet":
-a:
-  listen_address: 10.192.48.54
-b:
-  listen_address: 10.192.48.55
-c:
-  listen_address: 10.192.48.56
+  #"restbase2007.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.16.176
+  #  b:
+  #listen_address: 10.192.16.177
+  #  c:
+  #listen_address: 10.192.16.178
+  #"restbase2008.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.32.143
+  #  b:
+  #listen_address: 10.192.32.144
+  #  c:
+  #listen_address: 10.192.32.145
+  #"restbase2009.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.48.54
+  #  b:
+  #listen_address: 10.192.48.55
+  #  c:
+  #listen_address: 10.192.48.56
   #"restbase2010.codfw.wmnet":
   #  a:
   #listen_address: 10.192.16.186
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index fc8c48b..8ae06f3 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -789,6 +789,123 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase2007.codfw.wmnet":
+a:
+  listen_address: 10.192.16.176
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.192.16.177
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+- /srv/sde4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.192.16.178
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+- /srv/sde4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
+  "restbase2008.codfw.wmnet":
+a:
+  listen_address: 10.192.32.143
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase200[789]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405312 )

Change subject: restbase: reprovision restbase200[789]
..

restbase: reprovision restbase200[789]

These hosts have been already reimaged and have all cassandra instances masked 
already. Therefore we
can assign the restbase role for all, then enable instances one by one as 
bootstrap complete:

  systemctl unmask cassandra-
  puppet agent --test

Bug: T184100
Change-Id: I5a4c99b2a49da13c8d2f5ad13d786aa7d18b028c
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
3 files changed, 139 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/12/405312/1

diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index d5a4fa0..c7ed3a1 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -190,27 +190,27 @@
   #listen_address: 10.192.48.50
   #  c:
   #listen_address: 10.192.48.51
-  "restbase2007.codfw.wmnet":
-a:
-  listen_address: 10.192.16.176
-b:
-  listen_address: 10.192.16.177
-c:
-  listen_address: 10.192.16.178
-  "restbase2008.codfw.wmnet":
-a:
-  listen_address: 10.192.32.143
-b:
-  listen_address: 10.192.32.144
-c:
-  listen_address: 10.192.32.145
-  "restbase2009.codfw.wmnet":
-a:
-  listen_address: 10.192.48.54
-b:
-  listen_address: 10.192.48.55
-c:
-  listen_address: 10.192.48.56
+  #"restbase2007.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.16.176
+  #  b:
+  #listen_address: 10.192.16.177
+  #  c:
+  #listen_address: 10.192.16.178
+  #"restbase2008.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.32.143
+  #  b:
+  #listen_address: 10.192.32.144
+  #  c:
+  #listen_address: 10.192.32.145
+  #"restbase2009.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.48.54
+  #  b:
+  #listen_address: 10.192.48.55
+  #  c:
+  #listen_address: 10.192.48.56
   #"restbase2010.codfw.wmnet":
   #  a:
   #listen_address: 10.192.16.186
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index fc8c48b..8ae06f3 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -789,6 +789,123 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase2007.codfw.wmnet":
+a:
+  listen_address: 10.192.16.176
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.192.16.177
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+- /srv/sde4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.192.16.178
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+- /srv/sde4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
+  "restbase2008.codfw.wmnet":
+a:
+  listen_address: 10.192.32.143
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: install_server: swap restbase2009 disk layout

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/405019 )

Change subject: install_server: swap restbase2009 disk layout
..


install_server: swap restbase2009 disk layout

=> pd all show

Smart Array P440ar in Slot 0 (Embedded) (HBA Mode)

   HBA Drives

  physicaldrive 1I:1:1 (port 1I:box 1:bay 1, Solid State SATA, 1600.3 GB, 
OK)
  physicaldrive 1I:1:2 (port 1I:box 1:bay 2, Solid State SATA, 1600.3 GB, 
OK)
  physicaldrive 1I:1:3 (port 1I:box 1:bay 3, Solid State SATA, 1600.3 GB, 
OK)
  physicaldrive 1I:1:4 (port 1I:box 1:bay 4, Solid State SATA, 1600.3 GB, 
OK)

Bug: T184100
Change-Id: Ib3d5a3fd35294035381cef7c6ca128fc17c3d0fc
---
M modules/install_server/files/autoinstall/netboot.cfg
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index b815c69..00ee98e 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -157,8 +157,8 @@
 restbase101[678]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 restbase-dev100[456]) echo partman/cassandrahosts-4ssd.cfg ;; \
 restbase-test2*)  echo partman/cassandrahosts-2ssd.cfg ;; \
-restbase200[1-9]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
-restbase201[012]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
+restbase200[1-8]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
+restbase201[012]|restbase2009) echo 
partman/cassandrahosts-4ssd-jbod.cfg ;; \
 rhenium) echo partman/raid1-gpt.cfg ;; \
 sarin) echo partman/raid1-lvm-ext4-srv.cfg ;; \
 snapshot1001) echo partman/snapshot.cfg ;; \

-- 
To view, visit https://gerrit.wikimedia.org/r/405019
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib3d5a3fd35294035381cef7c6ca128fc17c3d0fc
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: install_server: swap restbase2009 disk layout

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405019 )

Change subject: install_server: swap restbase2009 disk layout
..

install_server: swap restbase2009 disk layout

=> pd all show

Smart Array P440ar in Slot 0 (Embedded) (HBA Mode)

   HBA Drives

  physicaldrive 1I:1:1 (port 1I:box 1:bay 1, Solid State SATA, 1600.3 GB, 
OK)
  physicaldrive 1I:1:2 (port 1I:box 1:bay 2, Solid State SATA, 1600.3 GB, 
OK)
  physicaldrive 1I:1:3 (port 1I:box 1:bay 3, Solid State SATA, 1600.3 GB, 
OK)
  physicaldrive 1I:1:4 (port 1I:box 1:bay 4, Solid State SATA, 1600.3 GB, 
OK)

Bug: T184100
Change-Id: Ib3d5a3fd35294035381cef7c6ca128fc17c3d0fc
---
M modules/install_server/files/autoinstall/netboot.cfg
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/19/405019/1

diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index b815c69..00ee98e 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -157,8 +157,8 @@
 restbase101[678]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 restbase-dev100[456]) echo partman/cassandrahosts-4ssd.cfg ;; \
 restbase-test2*)  echo partman/cassandrahosts-2ssd.cfg ;; \
-restbase200[1-9]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
-restbase201[012]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
+restbase200[1-8]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
+restbase201[012]|restbase2009) echo 
partman/cassandrahosts-4ssd-jbod.cfg ;; \
 rhenium) echo partman/raid1-gpt.cfg ;; \
 sarin) echo partman/raid1-lvm-ext4-srv.cfg ;; \
 snapshot1001) echo partman/snapshot.cfg ;; \

-- 
To view, visit https://gerrit.wikimedia.org/r/405019
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib3d5a3fd35294035381cef7c6ca128fc17c3d0fc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: install_server: reprovision restbase200[789]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/405008 )

Change subject: install_server: reprovision restbase200[789]
..


install_server: reprovision restbase200[789]

These are HP with 5ssd

Bug: T184100
Change-Id: I08eaf71e0ae45690349ef8d9d584a1ab762afea7
---
M modules/install_server/files/autoinstall/netboot.cfg
1 file changed, 1 insertion(+), 2 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index 995a601..b815c69 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -157,8 +157,7 @@
 restbase101[678]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 restbase-dev100[456]) echo partman/cassandrahosts-4ssd.cfg ;; \
 restbase-test2*)  echo partman/cassandrahosts-2ssd.cfg ;; \
-restbase200[123456]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
-restbase200[7-9]) echo partman/cassandrahosts-4ssd.cfg ;; \
+restbase200[1-9]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase201[012]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 rhenium) echo partman/raid1-gpt.cfg ;; \
 sarin) echo partman/raid1-lvm-ext4-srv.cfg ;; \

-- 
To view, visit https://gerrit.wikimedia.org/r/405008
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I08eaf71e0ae45690349ef8d9d584a1ab762afea7
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: install_server: reprovision restbase200[789]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405008 )

Change subject: install_server: reprovision restbase200[789]
..

install_server: reprovision restbase200[789]

These are HP with 5ssd

Bug: T184100
Change-Id: I08eaf71e0ae45690349ef8d9d584a1ab762afea7
---
M modules/install_server/files/autoinstall/netboot.cfg
1 file changed, 1 insertion(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/08/405008/1

diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index 995a601..b815c69 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -157,8 +157,7 @@
 restbase101[678]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 restbase-dev100[456]) echo partman/cassandrahosts-4ssd.cfg ;; \
 restbase-test2*)  echo partman/cassandrahosts-2ssd.cfg ;; \
-restbase200[123456]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
-restbase200[7-9]) echo partman/cassandrahosts-4ssd.cfg ;; \
+restbase200[1-9]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase201[012]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 rhenium) echo partman/raid1-gpt.cfg ;; \
 sarin) echo partman/raid1-lvm-ext4-srv.cfg ;; \

-- 
To view, visit https://gerrit.wikimedia.org/r/405008
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I08eaf71e0ae45690349ef8d9d584a1ab762afea7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: scap: require sudo rules to be in place before deploy

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404945 )

Change subject: scap: require sudo rules to be in place before deploy
..


scap: require sudo rules to be in place before deploy

For services requiring a config reload, the sudo rules might not be there yet
on the first puppet run.

Note that since puppet 4 ordering in the manifest matters, thus reorder the
resources.

Bug: T185189
Change-Id: I6c0276c7aa0ae1ca2596c114c419b65c645276ac
---
M modules/scap/manifests/target.pp
1 file changed, 31 insertions(+), 29 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/scap/manifests/target.pp b/modules/scap/manifests/target.pp
index 1b0769b..256eec5 100644
--- a/modules/scap/manifests/target.pp
+++ b/modules/scap/manifests/target.pp
@@ -110,35 +110,6 @@
 }
 }
 
-package { $package_name:
-install_options => [{
-  owner => $deploy_user}],
-provider=> 'scap3',
-require => [Package['scap'], User[$deploy_user]],
-}
-
-# XXX: Temporary work-around for switching services from Trebuchet to Scap3
-# The Scap3 provider doesn't touch the target dir if it's already a git 
repo
-# which means that even after switching the provider we end up with the
-# wrong user (root) owning it. Therefore, as a temporary measure we chown
-# the target dir's parent so that the subsequent invocation of deploy-local
-# is able to create the needed dirs and symlinks
-$chown_user = "${deploy_user}:${deploy_user}"
-$name_array = split($package_name, '/')
-$pkg_root = inline_template(
-'<%= @name_array[0,@name_array.size - 1].join("/") %>'
-)
-$chown_target = "/srv/deployment/${pkg_root}"
-$exec_name = "chown ${chown_target} for ${deploy_user}"
-if !defined(Exec[$exec_name]) {
-exec { $exec_name:
-command => "/bin/chown -R ${chown_user} ${chown_target}",
-# perform the chown only if root is the effective owner
-onlyif  => "/usr/bin/test -O /srv/deployment/${package_name}",
-require => [User[$deploy_user], Group[$deploy_user]]
-}
-}
-
 # Allow deploy user user to sudo -u $user, and to sudo /usr/sbin/service
 # if $service_name is defined.
 #
@@ -173,4 +144,35 @@
 }
 }
 
+# Have scap actually deploy the source, restart the service if needed, etc
+# Assume $deploy_user already has sudo permissions because of the block 
above.
+package { $package_name:
+install_options => [{
+  owner => $deploy_user}],
+provider=> 'scap3',
+require => [Package['scap'], User[$deploy_user]],
+}
+
+# XXX: Temporary work-around for switching services from Trebuchet to Scap3
+# The Scap3 provider doesn't touch the target dir if it's already a git 
repo
+# which means that even after switching the provider we end up with the
+# wrong user (root) owning it. Therefore, as a temporary measure we chown
+# the target dir's parent so that the subsequent invocation of deploy-local
+# is able to create the needed dirs and symlinks
+$chown_user = "${deploy_user}:${deploy_user}"
+$name_array = split($package_name, '/')
+$pkg_root = inline_template(
+'<%= @name_array[0,@name_array.size - 1].join("/") %>'
+)
+$chown_target = "/srv/deployment/${pkg_root}"
+$exec_name = "chown ${chown_target} for ${deploy_user}"
+if !defined(Exec[$exec_name]) {
+exec { $exec_name:
+command => "/bin/chown -R ${chown_user} ${chown_target}",
+# perform the chown only if root is the effective owner
+onlyif  => "/usr/bin/test -O /srv/deployment/${package_name}",
+require => [User[$deploy_user], Group[$deploy_user]]
+}
+}
+
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/404945
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6c0276c7aa0ae1ca2596c114c419b65c645276ac
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: 20after4 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Thcipriani 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: scap: require sudo rules to be in place before deploy

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404945 )

Change subject: scap: require sudo rules to be in place before deploy
..

scap: require sudo rules to be in place before deploy

For services requiring a config reload, the sudo rules might not be there yet
on the first puppet run.

Bug: T185189
Change-Id: I6c0276c7aa0ae1ca2596c114c419b65c645276ac
---
M modules/scap/manifests/target.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/45/404945/1

diff --git a/modules/scap/manifests/target.pp b/modules/scap/manifests/target.pp
index 1b0769b..19ce8eb 100644
--- a/modules/scap/manifests/target.pp
+++ b/modules/scap/manifests/target.pp
@@ -114,7 +114,7 @@
 install_options => [{
   owner => $deploy_user}],
 provider=> 'scap3',
-require => [Package['scap'], User[$deploy_user]],
+require => [Package['scap'], User[$deploy_user], 
Sudo::User["scap_${deploy_user}"]],
 }
 
 # XXX: Temporary work-around for switching services from Trebuchet to Scap3

-- 
To view, visit https://gerrit.wikimedia.org/r/404945
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6c0276c7aa0ae1ca2596c114c419b65c645276ac
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase201[012]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404652 )

Change subject: restbase: reprovision restbase201[012]
..


restbase: reprovision restbase201[012]

These hosts have been already reimaged and have all cassandra instances masked 
already. Therefore we
can assign the restbase role for all, then enable instances one by one as 
bootstrap complete:

  systemctl unmask cassandra-
  puppet agent --test

Bug: T184100
Change-Id: I9604283730214439cadb418bccb3a861fc13e561
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
3 files changed, 134 insertions(+), 23 deletions(-)

Approvals:
  Eevans: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index f120bb9..d5a4fa0 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -211,27 +211,27 @@
   listen_address: 10.192.48.55
 c:
   listen_address: 10.192.48.56
-  "restbase2010.codfw.wmnet":
-a:
-  listen_address: 10.192.16.186
-b:
-  listen_address: 10.192.16.187
-c:
-  listen_address: 10.192.16.188
-  "restbase2011.codfw.wmnet":
-a:
-  listen_address: 10.192.32.152
-b:
-  listen_address: 10.192.32.153
-c:
-  listen_address: 10.192.32.154
-  "restbase2012.codfw.wmnet":
-a:
-  listen_address: 10.192.48.68
-b:
-  listen_address: 10.192.48.69
-c:
-  listen_address: 10.192.48.70
+  #"restbase2010.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.16.186
+  #  b:
+  #listen_address: 10.192.16.187
+  #  c:
+  #listen_address: 10.192.16.188
+  #"restbase2011.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.32.152
+  #  b:
+  #listen_address: 10.192.32.153
+  #  c:
+  #listen_address: 10.192.32.154
+  #"restbase2012.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.48.68
+  #  b:
+  #listen_address: 10.192.48.69
+  #  c:
+  #listen_address: 10.192.48.70
 
 
 profile::cassandra::settings:
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 4290581..fc8c48b 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -789,6 +789,117 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase2010.codfw.wmnet":
+a:
+  listen_address: 10.192.16.186
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.192.16.187
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.192.16.188
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
+  "restbase2011.codfw.wmnet":
+a:
+  listen_address: 10.192.32.152
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase101[35]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404675 )

Change subject: restbase: reprovision restbase101[35]
..


restbase: reprovision restbase101[35]

These hosts have been already reimaged and have all cassandra instances masked 
already. Therefore we
can assign the restbase role for all, then enable instances one by one as 
bootstrap complete:

  systemctl unmask cassandra-
  puppet agent --test

Bug: T184100
Change-Id: I8893de5a90ce55613279734357318b2e4e52de8e
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
4 files changed, 96 insertions(+), 21 deletions(-)

Approvals:
  Eevans: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index 538b08f..f120bb9 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -106,13 +106,13 @@
   #listen_address: 10.64.32.203
   #  c:
   #listen_address: 10.64.32.204
-  "restbase1013.eqiad.wmnet":
-a:
-  listen_address: 10.64.32.205
-b:
-  listen_address: 10.64.32.206
-c:
-  listen_address: 10.64.32.207
+  #"restbase1013.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.32.205
+  #  b:
+  #listen_address: 10.64.32.206
+  #  c:
+  #listen_address: 10.64.32.207
   #"restbase1014.eqiad.wmnet":
   #  a:
   #listen_address: 10.64.48.135
@@ -120,13 +120,13 @@
   #listen_address: 10.64.48.136
   #  c:
   #listen_address: 10.64.48.137
-  "restbase1015.eqiad.wmnet":
-a:
-  listen_address: 10.64.48.138
-b:
-  listen_address: 10.64.48.139
-c:
-  listen_address: 10.64.48.140
+  #"restbase1015.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.48.138
+  #  b:
+  #listen_address: 10.64.48.139
+  #  c:
+  #listen_address: 10.64.48.140
   #"restbase1016.eqiad.wmnet":
   #  a:
   #listen_address: 10.64.0.32
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index e986220..4290581 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -318,6 +318,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1013.eqiad.wmnet":
+a:
+  listen_address: 10.64.32.205
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.32.206
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+- /srv/sde4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.32.207
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+- /srv/sde4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase1014.eqiad.wmnet":
 a:
   listen_address: 10.64.48.135
@@ -358,6 +398,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1015.eqiad.wmnet":
+a:
+  listen_address: 10.64.48.138
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase101[35]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404675 )

Change subject: restbase: reprovision restbase101[35]
..

restbase: reprovision restbase101[35]

These hosts have been already reimaged and have all cassandra instances masked 
already. Therefore we
can assign the restbase role for all, then enable instances one by one as 
bootstrap complete:

  systemctl unmask cassandra-
  puppet agent --test

Bug: T184100
Change-Id: I8893de5a90ce55613279734357318b2e4e52de8e
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
4 files changed, 96 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/75/404675/1

diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index 538b08f..f120bb9 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -106,13 +106,13 @@
   #listen_address: 10.64.32.203
   #  c:
   #listen_address: 10.64.32.204
-  "restbase1013.eqiad.wmnet":
-a:
-  listen_address: 10.64.32.205
-b:
-  listen_address: 10.64.32.206
-c:
-  listen_address: 10.64.32.207
+  #"restbase1013.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.32.205
+  #  b:
+  #listen_address: 10.64.32.206
+  #  c:
+  #listen_address: 10.64.32.207
   #"restbase1014.eqiad.wmnet":
   #  a:
   #listen_address: 10.64.48.135
@@ -120,13 +120,13 @@
   #listen_address: 10.64.48.136
   #  c:
   #listen_address: 10.64.48.137
-  "restbase1015.eqiad.wmnet":
-a:
-  listen_address: 10.64.48.138
-b:
-  listen_address: 10.64.48.139
-c:
-  listen_address: 10.64.48.140
+  #"restbase1015.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.48.138
+  #  b:
+  #listen_address: 10.64.48.139
+  #  c:
+  #listen_address: 10.64.48.140
   #"restbase1016.eqiad.wmnet":
   #  a:
   #listen_address: 10.64.0.32
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index e986220..4290581 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -318,6 +318,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1013.eqiad.wmnet":
+a:
+  listen_address: 10.64.32.205
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.32.206
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+- /srv/sde4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.32.207
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+- /srv/sde4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase1014.eqiad.wmnet":
 a:
   listen_address: 10.64.48.135
@@ -358,6 +398,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1015.eqiad.wmnet":
+a:
+  listen_address: 10.64.48.138
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase201[012]

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404652 )

Change subject: restbase: reprovision restbase201[012]
..

restbase: reprovision restbase201[012]

These hosts have been already reimaged and have all cassandra instances masked 
already. Therefore we
can assign the restbase role for all, then enable instances one by one as 
bootstrap complete:

  systemctl unmask cassandra-
  puppet agent --test

Bug: T184100
Change-Id: I9604283730214439cadb418bccb3a861fc13e561
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
3 files changed, 134 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/52/404652/1

diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index 538b08f..c659a0f 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -211,27 +211,27 @@
   listen_address: 10.192.48.55
 c:
   listen_address: 10.192.48.56
-  "restbase2010.codfw.wmnet":
-a:
-  listen_address: 10.192.16.186
-b:
-  listen_address: 10.192.16.187
-c:
-  listen_address: 10.192.16.188
-  "restbase2011.codfw.wmnet":
-a:
-  listen_address: 10.192.32.152
-b:
-  listen_address: 10.192.32.153
-c:
-  listen_address: 10.192.32.154
-  "restbase2012.codfw.wmnet":
-a:
-  listen_address: 10.192.48.68
-b:
-  listen_address: 10.192.48.69
-c:
-  listen_address: 10.192.48.70
+  #"restbase2010.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.16.186
+  #  b:
+  #listen_address: 10.192.16.187
+  #  c:
+  #listen_address: 10.192.16.188
+  #"restbase2011.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.32.152
+  #  b:
+  #listen_address: 10.192.32.153
+  #  c:
+  #listen_address: 10.192.32.154
+  #"restbase2012.codfw.wmnet":
+  #  a:
+  #listen_address: 10.192.48.68
+  #  b:
+  #listen_address: 10.192.48.69
+  #  c:
+  #listen_address: 10.192.48.70
 
 
 profile::cassandra::settings:
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index e986220..e013b55 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -709,6 +709,117 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase2010.codfw.wmnet":
+a:
+  listen_address: 10.192.16.186
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.192.16.187
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.192.16.188
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
+  "restbase2011.codfw.wmnet":
+a:
+  listen_address: 10.192.32.152
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase1016

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404638 )

Change subject: restbase: reprovision restbase1016
..


restbase: reprovision restbase1016

Bug: T184100
Change-Id: I1347aa226442dbf705981ede1e1d134f6494fd3a
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
3 files changed, 46 insertions(+), 9 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index d1f6b2a..538b08f 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -127,13 +127,13 @@
   listen_address: 10.64.48.139
 c:
   listen_address: 10.64.48.140
-  "restbase1016.eqiad.wmnet":
-a:
-  listen_address: 10.64.0.32
-b:
-  listen_address: 10.64.0.33
-c:
-  listen_address: 10.64.0.34
+  #"restbase1016.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.0.32
+  #  b:
+  #listen_address: 10.64.0.33
+  #  c:
+  #listen_address: 10.64.0.34
   # "restbase1017.eqiad.wmnet":
   #   a:
   # listen_address: 10.64.32.130
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index c3d55a9..e986220 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -358,6 +358,43 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1016.eqiad.wmnet":
+a:
+  listen_address: 10.64.0.32
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.0.33
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.0.34
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase1017.eqiad.wmnet":
 a:
   listen_address: 10.64.32.130
diff --git a/manifests/site.pp b/manifests/site.pp
index 3a27719..5bcacc3 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1881,11 +1881,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[356]\.eqiad\.wmnet$/ {
+node /^restbase101[35]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[012478])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[0124678])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/404638
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1347aa226442dbf705981ede1e1d134f6494fd3a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Eevans 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Mobrovac 
Gerrit-Reviewer: Ppchelko 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase1016

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404638 )

Change subject: restbase: reprovision restbase1016
..

restbase: reprovision restbase1016

Bug: T184100
Change-Id: I1347aa226442dbf705981ede1e1d134f6494fd3a
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
3 files changed, 46 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/38/404638/1

diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index d1f6b2a..538b08f 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -127,13 +127,13 @@
   listen_address: 10.64.48.139
 c:
   listen_address: 10.64.48.140
-  "restbase1016.eqiad.wmnet":
-a:
-  listen_address: 10.64.0.32
-b:
-  listen_address: 10.64.0.33
-c:
-  listen_address: 10.64.0.34
+  #"restbase1016.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.0.32
+  #  b:
+  #listen_address: 10.64.0.33
+  #  c:
+  #listen_address: 10.64.0.34
   # "restbase1017.eqiad.wmnet":
   #   a:
   # listen_address: 10.64.32.130
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index c3d55a9..e986220 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -358,6 +358,43 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1016.eqiad.wmnet":
+a:
+  listen_address: 10.64.0.32
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.0.33
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.0.34
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase1017.eqiad.wmnet":
 a:
   listen_address: 10.64.32.130
diff --git a/manifests/site.pp b/manifests/site.pp
index 3a27719..5bcacc3 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1881,11 +1881,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[356]\.eqiad\.wmnet$/ {
+node /^restbase101[35]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[012478])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[0124678])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/404638
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1347aa226442dbf705981ede1e1d134f6494fd3a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: install_server: switch remaining cassandra hosts to jbod

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404455 )

Change subject: install_server: switch remaining cassandra hosts to jbod
..


install_server: switch remaining cassandra hosts to jbod

Change just install_server for Dell machines to unblock reimaging, changing the
role will happen when the time comes.

Bug: T184100
Change-Id: I1b6d1537db22b6f206bb97091fb17d6e33e6e0a4
---
M modules/install_server/files/autoinstall/netboot.cfg
1 file changed, 3 insertions(+), 3 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index aef94c4..03ebab5 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -155,12 +155,12 @@
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
 restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase101[35]) echo partman/cassandrahosts-5ssd.cfg ;; \
-restbase101[78]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
-restbase1016|restbase-dev100[456]) echo 
partman/cassandrahosts-4ssd.cfg ;; \
+restbase101[678]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
+restbase-dev100[456]) echo partman/cassandrahosts-4ssd.cfg ;; \
 restbase-test2*)  echo partman/cassandrahosts-2ssd.cfg ;; \
 restbase200[123456]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase200[7-9]) echo partman/cassandrahosts-4ssd.cfg ;; \
-restbase201[0-2]) echo partman/cassandrahosts-4ssd.cfg ;; \
+restbase201[012]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 rhenium) echo partman/raid1-gpt.cfg ;; \
 sarin) echo partman/raid1-lvm-ext4-srv.cfg ;; \
 snapshot1001) echo partman/snapshot.cfg ;; \

-- 
To view, visit https://gerrit.wikimedia.org/r/404455
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1b6d1537db22b6f206bb97091fb17d6e33e6e0a4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: install_server: switch remaining cassandra hosts to jbod

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404455 )

Change subject: install_server: switch remaining cassandra hosts to jbod
..

install_server: switch remaining cassandra hosts to jbod

Change just install_server for Dell machines to unblock reimaging, changing the
role will happen when the time comes.

Bug: T184100
Change-Id: I1b6d1537db22b6f206bb97091fb17d6e33e6e0a4
---
M modules/install_server/files/autoinstall/netboot.cfg
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/55/404455/1

diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index aef94c4..03ebab5 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -155,12 +155,12 @@
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
 restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase101[35]) echo partman/cassandrahosts-5ssd.cfg ;; \
-restbase101[78]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
-restbase1016|restbase-dev100[456]) echo 
partman/cassandrahosts-4ssd.cfg ;; \
+restbase101[678]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
+restbase-dev100[456]) echo partman/cassandrahosts-4ssd.cfg ;; \
 restbase-test2*)  echo partman/cassandrahosts-2ssd.cfg ;; \
 restbase200[123456]) echo partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase200[7-9]) echo partman/cassandrahosts-4ssd.cfg ;; \
-restbase201[0-2]) echo partman/cassandrahosts-4ssd.cfg ;; \
+restbase201[012]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
 rhenium) echo partman/raid1-gpt.cfg ;; \
 sarin) echo partman/raid1-lvm-ext4-srv.cfg ;; \
 snapshot1001) echo partman/snapshot.cfg ;; \

-- 
To view, visit https://gerrit.wikimedia.org/r/404455
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1b6d1537db22b6f206bb97091fb17d6e33e6e0a4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: allow override of fs-related node-exporter options

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404324 )

Change subject: prometheus: allow override of fs-related node-exporter options
..


prometheus: allow override of fs-related node-exporter options

Bug: T184469
Change-Id: Ic756cb208a10982dba1197d80af1ed50554613bc
---
M modules/prometheus/manifests/node_exporter.pp
M modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
2 files changed, 9 insertions(+), 1 deletion(-)

Approvals:
  Rush: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/prometheus/manifests/node_exporter.pp 
b/modules/prometheus/manifests/node_exporter.pp
index 541e61f..44ce9a9 100644
--- a/modules/prometheus/manifests/node_exporter.pp
+++ b/modules/prometheus/manifests/node_exporter.pp
@@ -11,6 +11,12 @@
 # [*$ignored_devices*]
 #  Regular expression to exclude block devices from being reported
 #
+# [*$ignored_fs_types*]
+#  Regular expression to exclude filesystem types from being reported
+#
+# [*$ignored_mount_points*]
+#  Regular expression to exclude mount points from being reported
+#
 # [*$collectors_extra*]
 #  List of extra collectors to be enabled.
 #
@@ -25,6 +31,8 @@
 
 class prometheus::node_exporter (
 $ignored_devices  = "^(ram|loop|fd)d+\$",
+$ignored_fs_types  = '^(sys|proc|auto)fs$',
+$ignored_mount_points  = '^/(sys|proc|dev)($|/)',
 $collectors_extra = [],
 $web_listen_address = ':9100',
 ) {
diff --git 
a/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb 
b/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
index 5353a08..2358c1a 100644
--- a/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
+++ b/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
@@ -1 +1 @@
-ARGS='-collector.diskstats.ignored-devices=<%= @ignored_devices %> 
-collector.textfile.directory=<%= @textfile_directory %> 
-collectors.enabled=<%= @collectors_enabled %> -web.listen-address=<%= 
@web_listen_address %>'
+ARGS='-collector.diskstats.ignored-devices=<%= @ignored_devices %> 
-collector.filesystem.ignored-fs-types <%= @ignored_fs_types %> 
-collector.filesystem.ignored-mount-points <%= @ignored_mount_points %> 
-collector.textfile.directory=<%= @textfile_directory %> 
-collectors.enabled=<%= @collectors_enabled %> -web.listen-address=<%= 
@web_listen_address %>'

-- 
To view, visit https://gerrit.wikimedia.org/r/404324
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic756cb208a10982dba1197d80af1ed50554613bc
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Andrew Bogott 
Gerrit-Reviewer: Arturo Borrero Gonzalez 
Gerrit-Reviewer: Chasemp 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Madhuvishy 
Gerrit-Reviewer: Rush 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: bump global retention to 15 months

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404434 )

Change subject: prometheus: bump global retention to 15 months
..

prometheus: bump global retention to 15 months

Change-Id: Idfaac4b32be4c3a69a5686625213fad25f8cc8a8
---
M modules/role/manifests/prometheus/global.pp
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/34/404434/1

diff --git a/modules/role/manifests/prometheus/global.pp 
b/modules/role/manifests/prometheus/global.pp
index 1c0533e..2def060 100644
--- a/modules/role/manifests/prometheus/global.pp
+++ b/modules/role/manifests/prometheus/global.pp
@@ -53,8 +53,8 @@
 ]
 
 prometheus::server { 'global':
-# one year
-storage_retention=> '8760h0m0s',
+# fifteen months
+storage_retention=> '10950h0m0s',
 listen_address   => '127.0.0.1:9904',
 scrape_configs_extra => $federation_jobs,
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/404434
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Idfaac4b32be4c3a69a5686625213fad25f8cc8a8
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Whitelist X-MediaWiki-Patrol-Status header in Swift

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402471 )

Change subject: Whitelist X-MediaWiki-Patrol-Status header in Swift
..


Whitelist X-MediaWiki-Patrol-Status header in Swift

See If488170949a8102b0eb5c044eb57bb789728b816 for background.

Bug: T167400
Change-Id: Ic7a940a0b74f83f2de507ecbaccd09d73180b140
---
M modules/swift/templates/object-server.conf.erb
1 file changed, 2 insertions(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/swift/templates/object-server.conf.erb 
b/modules/swift/templates/object-server.conf.erb
index 38e9f2a..4ef4d93 100644
--- a/modules/swift/templates/object-server.conf.erb
+++ b/modules/swift/templates/object-server.conf.erb
@@ -27,7 +27,8 @@
x-delete-at,
x-object-manifest,
x-content-duration,
-   x-content-dimensions
+   x-content-dimensions,
+   x-mediawiki-patrol-status
 
 [object-replicator]
 concurrency = 3

-- 
To view, visit https://gerrit.wikimedia.org/r/402471
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic7a940a0b74f83f2de507ecbaccd09d73180b140
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gergő Tisza 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: tweak node_exporter ignored_devices

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404430 )

Change subject: prometheus: tweak node_exporter ignored_devices
..

prometheus: tweak node_exporter ignored_devices

Also quote arguments to node_exporter, in case one of those contains a 
whitespace

Change-Id: Ib71f482fa4fb7396dca7897921d812c5dbc330be
---
M modules/prometheus/manifests/node_exporter.pp
M modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
2 files changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/30/404430/1

diff --git a/modules/prometheus/manifests/node_exporter.pp 
b/modules/prometheus/manifests/node_exporter.pp
index 44ce9a9..dec1dce 100644
--- a/modules/prometheus/manifests/node_exporter.pp
+++ b/modules/prometheus/manifests/node_exporter.pp
@@ -30,7 +30,7 @@
 
 
 class prometheus::node_exporter (
-$ignored_devices  = "^(ram|loop|fd)d+\$",
+$ignored_devices  = 
'^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvmed+nd+p)d+$',
 $ignored_fs_types  = '^(sys|proc|auto)fs$',
 $ignored_mount_points  = '^/(sys|proc|dev)($|/)',
 $collectors_extra = [],
diff --git 
a/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb 
b/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
index 2358c1a..98f2082 100644
--- a/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
+++ b/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
@@ -1 +1 @@
-ARGS='-collector.diskstats.ignored-devices=<%= @ignored_devices %> 
-collector.filesystem.ignored-fs-types <%= @ignored_fs_types %> 
-collector.filesystem.ignored-mount-points <%= @ignored_mount_points %> 
-collector.textfile.directory=<%= @textfile_directory %> 
-collectors.enabled=<%= @collectors_enabled %> -web.listen-address=<%= 
@web_listen_address %>'
+ARGS='-collector.diskstats.ignored-devices="<%= @ignored_devices %>" 
-collector.filesystem.ignored-fs-types "<%= @ignored_fs_types %>" 
-collector.filesystem.ignored-mount-points "<%= @ignored_mount_points %>" 
-collector.textfile.directory="<%= @textfile_directory %>" 
-collectors.enabled="<%= @collectors_enabled %>" -web.listen-address="<%= 
@web_listen_address %>"'

-- 
To view, visit https://gerrit.wikimedia.org/r/404430
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib71f482fa4fb7396dca7897921d812c5dbc330be
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase1018

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404325 )

Change subject: restbase: reprovision restbase1018
..


restbase: reprovision restbase1018

Bug: T184100
Change-Id: I3a2ad3c40f24903b9faae012351a310365b01a88
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
4 files changed, 48 insertions(+), 11 deletions(-)

Approvals:
  Mobrovac: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index 6326ff5..d1f6b2a 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -141,13 +141,13 @@
   # listen_address: 10.64.32.131
   #   c:
   # listen_address: 10.64.32.132
-  "restbase1018.eqiad.wmnet":
-a:
-  listen_address: 10.64.48.98
-b:
-  listen_address: 10.64.48.99
-c:
-  listen_address: 10.64.48.100
+  #"restbase1018.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.48.98
+  #  b:
+  #listen_address: 10.64.48.99
+  #  c:
+  #listen_address: 10.64.48.100
   # "restbase2001.codfw.wmnet":
   #   a:
   # listen_address: 10.192.16.162
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 4118263..c3d55a9 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -395,6 +395,43 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1018.eqiad.wmnet":
+a:
+  listen_address: 10.64.48.98
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.48.99
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.48.100
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase2001.codfw.wmnet":
 a:
   listen_address: 10.192.16.162
diff --git a/manifests/site.pp b/manifests/site.pp
index 456e8a7..1f83f6a 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1843,11 +1843,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[3568]\.eqiad\.wmnet$/ {
+node /^restbase101[356]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[01247])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[012478])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 
diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index ee406c4..aef94c4 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -155,8 +155,8 @@
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
 restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase101[35]) echo partman/cassandrahosts-5ssd.cfg ;; \
-restbase1017) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
-restbase101[68]|restbase-dev100[456]) echo 
partman/cassandrahosts-4ssd.cfg ;; \
+restbase101[78]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
+restbase1016|restbase-dev100[456]) echo 
partman/cassandrahosts-4ssd.cfg ;; \
 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase1018

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404325 )

Change subject: restbase: reprovision restbase1018
..

restbase: reprovision restbase1018

Bug: T184100
Change-Id: I3a2ad3c40f24903b9faae012351a310365b01a88
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
4 files changed, 48 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/25/404325/1

diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index 6326ff5..d1f6b2a 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -141,13 +141,13 @@
   # listen_address: 10.64.32.131
   #   c:
   # listen_address: 10.64.32.132
-  "restbase1018.eqiad.wmnet":
-a:
-  listen_address: 10.64.48.98
-b:
-  listen_address: 10.64.48.99
-c:
-  listen_address: 10.64.48.100
+  #"restbase1018.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.48.98
+  #  b:
+  #listen_address: 10.64.48.99
+  #  c:
+  #listen_address: 10.64.48.100
   # "restbase2001.codfw.wmnet":
   #   a:
   # listen_address: 10.192.16.162
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 4118263..c3d55a9 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -395,6 +395,43 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1018.eqiad.wmnet":
+a:
+  listen_address: 10.64.48.98
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.48.99
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.48.100
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase2001.codfw.wmnet":
 a:
   listen_address: 10.192.16.162
diff --git a/manifests/site.pp b/manifests/site.pp
index 967ed6a..03417e7 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1834,11 +1834,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[3568]\.eqiad\.wmnet$/ {
+node /^restbase101[356]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[01247])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[012478])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 
diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index ee406c4..aef94c4 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -155,8 +155,8 @@
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
 restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase101[35]) echo partman/cassandrahosts-5ssd.cfg ;; \
-restbase1017) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
-restbase101[68]|restbase-dev100[456]) echo 
partman/cassandrahosts-4ssd.cfg ;; \
+restbase101[78]) echo partman/cassandrahosts-4ssd-jbod.cfg ;; \
+restbase1016|restbase-dev100[456]) echo 
partman/cassandrahosts-4ssd.cfg ;; \
 restbase-test2*)  echo partman/cassandrahosts-2ssd.cfg ;; \
 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: allow override of fs-related node-exporter options

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404324 )

Change subject: prometheus: allow override of fs-related node-exporter options
..

prometheus: allow override of fs-related node-exporter options

Bug: T184469
Change-Id: Ic756cb208a10982dba1197d80af1ed50554613bc
---
M modules/prometheus/manifests/node_exporter.pp
M modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
2 files changed, 3 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/24/404324/1

diff --git a/modules/prometheus/manifests/node_exporter.pp 
b/modules/prometheus/manifests/node_exporter.pp
index 541e61f..adc9c75 100644
--- a/modules/prometheus/manifests/node_exporter.pp
+++ b/modules/prometheus/manifests/node_exporter.pp
@@ -25,6 +25,8 @@
 
 class prometheus::node_exporter (
 $ignored_devices  = "^(ram|loop|fd)d+\$",
+$ignored_fs_types  = "^(sys|proc|auto)fs$",
+$ignored_mount_points  = "^/(sys|proc|dev)($|/)",
 $collectors_extra = [],
 $web_listen_address = ':9100',
 ) {
diff --git 
a/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb 
b/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
index 5353a08..2358c1a 100644
--- a/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
+++ b/modules/prometheus/templates/etc/default/prometheus-node-exporter.erb
@@ -1 +1 @@
-ARGS='-collector.diskstats.ignored-devices=<%= @ignored_devices %> 
-collector.textfile.directory=<%= @textfile_directory %> 
-collectors.enabled=<%= @collectors_enabled %> -web.listen-address=<%= 
@web_listen_address %>'
+ARGS='-collector.diskstats.ignored-devices=<%= @ignored_devices %> 
-collector.filesystem.ignored-fs-types <%= @ignored_fs_types %> 
-collector.filesystem.ignored-mount-points <%= @ignored_mount_points %> 
-collector.textfile.directory=<%= @textfile_directory %> 
-collectors.enabled=<%= @collectors_enabled %> -web.listen-address=<%= 
@web_listen_address %>'

-- 
To view, visit https://gerrit.wikimedia.org/r/404324
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic756cb208a10982dba1197d80af1ed50554613bc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-blazegraph-exporter[master]: Don't depend on blazegraph

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404316 )

Change subject: Don't depend on blazegraph
..


Don't depend on blazegraph

Partially revert 5c3d9802f6, we'll ship an override in puppet instead

Bug: T184434
Change-Id: I89864487fca1aa87ec5428a7af0192843a17594a
---
M debian/changelog
M debian/service
2 files changed, 6 insertions(+), 2 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved
  Gehel: Looks good to me, but someone else must approve



diff --git a/debian/changelog b/debian/changelog
index 20c2079..9738003 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+prometheus-blazegraph-exporter (0.4) jessie-wikimedia; urgency=medium
+
+  * Don't start after blazegraph
+
+ -- Filippo Giunchedi   Mon, 15 Jan 2018 17:25:45 +0100
+
 prometheus-blazegraph-exporter (0.3) jessie-wikimedia; urgency=medium
 
   * start after blazegraph
diff --git a/debian/service b/debian/service
index 7761739..764352f 100644
--- a/debian/service
+++ b/debian/service
@@ -1,7 +1,5 @@
 [Unit]
 Description=Prometheus Blazegraph Exporter
-Requires=blazegraph.service
-After=blazegraph.service
 
 [Service]
 Restart=always

-- 
To view, visit https://gerrit.wikimedia.org/r/404316
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I89864487fca1aa87ec5428a7af0192843a17594a
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-blazegraph-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-blazegraph-exporter[master]: Don't depend on blazegraph

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404316 )

Change subject: Don't depend on blazegraph
..

Don't depend on blazegraph

Partially revert 5c3d9802f6, we'll ship an override in puppet instead

Bug: T184434
Change-Id: I89864487fca1aa87ec5428a7af0192843a17594a
---
M debian/changelog
M debian/service
2 files changed, 6 insertions(+), 2 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-blazegraph-exporter 
refs/changes/16/404316/1

diff --git a/debian/changelog b/debian/changelog
index 20c2079..9738003 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+prometheus-blazegraph-exporter (0.4) jessie-wikimedia; urgency=medium
+
+  * Don't start after blazegraph
+
+ -- Filippo Giunchedi   Mon, 15 Jan 2018 17:25:45 +0100
+
 prometheus-blazegraph-exporter (0.3) jessie-wikimedia; urgency=medium
 
   * start after blazegraph
diff --git a/debian/service b/debian/service
index 7761739..764352f 100644
--- a/debian/service
+++ b/debian/service
@@ -1,7 +1,5 @@
 [Unit]
 Description=Prometheus Blazegraph Exporter
-Requires=blazegraph.service
-After=blazegraph.service
 
 [Service]
 Restart=always

-- 
To view, visit https://gerrit.wikimedia.org/r/404316
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I89864487fca1aa87ec5428a7af0192843a17594a
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-blazegraph-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: enable remaining restbase1017 instances

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404300 )

Change subject: hieradata: enable remaining restbase1017 instances
..


hieradata: enable remaining restbase1017 instances

It has been noted in the past that enabling all instances together causes 
issues because of
concurrent bootstraps. As an experiment I've masked cassandra-c already to test 
if starting out with
all instances masked and enabling one by one would work.

Bug: T184100
Change-Id: I17af71c72f05bb0492742b68db5ce2013f4e58d4
---
M hieradata/role/common/restbase/production_ng.yaml
1 file changed, 24 insertions(+), 24 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 170d601..4118263 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -371,30 +371,30 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
   jmx_exporter_enabled: true
-#b:
-#  listen_address: 10.64.32.131
-#  data_file_directories:
-#- /srv/sda4/cassandra-b/data
-#- /srv/sdb4/cassandra-b/data
-#- /srv/sdc4/cassandra-b/data
-#- /srv/sdd4/cassandra-b/data
-#  heapdump_directory: /srv/sdb4/cassandra-b
-#  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
-#  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
-#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
-#  jmx_exporter_enabled: true
-#c:
-#  listen_address: 10.64.32.132
-#  data_file_directories:
-#- /srv/sda4/cassandra-c/data
-#- /srv/sdb4/cassandra-c/data
-#- /srv/sdc4/cassandra-c/data
-#- /srv/sdd4/cassandra-c/data
-#  heapdump_directory: /srv/sdc4/cassandra-c
-#  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
-#  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
-#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
-#  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.32.131
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.32.132
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase2001.codfw.wmnet":
 a:
   listen_address: 10.192.16.162

-- 
To view, visit https://gerrit.wikimedia.org/r/404300
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I17af71c72f05bb0492742b68db5ce2013f4e58d4
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Eevans 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Mobrovac 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: enable remaining restbase1017 instances

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404300 )

Change subject: hieradata: enable remaining restbase1017 instances
..

hieradata: enable remaining restbase1017 instances

It has been noted in the past that enabling all instances together causes 
issues because of
concurrent bootstraps. As an experiment I've masked cassandra-c already to test 
if starting out with
all instances masked and enabling one by one would work.

Bug: T184100
Change-Id: I17af71c72f05bb0492742b68db5ce2013f4e58d4
---
M hieradata/role/common/restbase/production_ng.yaml
1 file changed, 24 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/00/404300/1

diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 170d601..4118263 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -371,30 +371,30 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
   jmx_exporter_enabled: true
-#b:
-#  listen_address: 10.64.32.131
-#  data_file_directories:
-#- /srv/sda4/cassandra-b/data
-#- /srv/sdb4/cassandra-b/data
-#- /srv/sdc4/cassandra-b/data
-#- /srv/sdd4/cassandra-b/data
-#  heapdump_directory: /srv/sdb4/cassandra-b
-#  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
-#  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
-#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
-#  jmx_exporter_enabled: true
-#c:
-#  listen_address: 10.64.32.132
-#  data_file_directories:
-#- /srv/sda4/cassandra-c/data
-#- /srv/sdb4/cassandra-c/data
-#- /srv/sdc4/cassandra-c/data
-#- /srv/sdd4/cassandra-c/data
-#  heapdump_directory: /srv/sdc4/cassandra-c
-#  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
-#  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
-#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
-#  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.32.131
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.32.132
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase2001.codfw.wmnet":
 a:
   listen_address: 10.192.16.162

-- 
To view, visit https://gerrit.wikimedia.org/r/404300
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I17af71c72f05bb0492742b68db5ce2013f4e58d4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Scap: bump version to 3.7.6-1

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404219 )

Change subject: Scap: bump version to 3.7.6-1
..


Scap: bump version to 3.7.6-1

Bug: T127762
Change-Id: I4e98b01b1371d846243ebe2eed96e16082d34f60
---
M modules/scap/manifests/init.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/scap/manifests/init.pp b/modules/scap/manifests/init.pp
index 292993d..1be0a97 100644
--- a/modules/scap/manifests/init.pp
+++ b/modules/scap/manifests/init.pp
@@ -12,7 +12,7 @@
 class scap (
 $deployment_server = 'deployment',
 $wmflabs_master = 'deployment-tin.deployment-prep.eqiad.wmflabs',
-$version = '3.7.5-1',
+$version = '3.7.6-1',
 ) {
 package { 'scap':
 ensure => $version,

-- 
To view, visit https://gerrit.wikimedia.org/r/404219
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4e98b01b1371d846243ebe2eed96e16082d34f60
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Thcipriani 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: don't consider sde for restbase1017

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404277 )

Change subject: restbase: don't consider sde for restbase1017
..


restbase: don't consider sde for restbase1017

Bug: T184100
Change-Id: I6b456e7ca0868c993b083c69b747640f77e704d3
---
M hieradata/role/common/restbase/production_ng.yaml
1 file changed, 0 insertions(+), 3 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index c29b1d7..170d601 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -366,7 +366,6 @@
 - /srv/sdb4/cassandra-a/data
 - /srv/sdc4/cassandra-a/data
 - /srv/sdd4/cassandra-a/data
-- /srv/sde4/cassandra-a/data
   heapdump_directory: /srv/sda4/cassandra-a
   commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
   hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
@@ -379,7 +378,6 @@
 #- /srv/sdb4/cassandra-b/data
 #- /srv/sdc4/cassandra-b/data
 #- /srv/sdd4/cassandra-b/data
-#- /srv/sde4/cassandra-b/data
 #  heapdump_directory: /srv/sdb4/cassandra-b
 #  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
 #  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
@@ -392,7 +390,6 @@
 #- /srv/sdb4/cassandra-c/data
 #- /srv/sdc4/cassandra-c/data
 #- /srv/sdd4/cassandra-c/data
-#- /srv/sde4/cassandra-c/data
 #  heapdump_directory: /srv/sdc4/cassandra-c
 #  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
 #  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints

-- 
To view, visit https://gerrit.wikimedia.org/r/404277
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6b456e7ca0868c993b083c69b747640f77e704d3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: don't consider sde for restbase1017

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404277 )

Change subject: restbase: don't consider sde for restbase1017
..

restbase: don't consider sde for restbase1017

Bug: T184100
Change-Id: I6b456e7ca0868c993b083c69b747640f77e704d3
---
M hieradata/role/common/restbase/production_ng.yaml
1 file changed, 0 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/77/404277/1

diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index c29b1d7..170d601 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -366,7 +366,6 @@
 - /srv/sdb4/cassandra-a/data
 - /srv/sdc4/cassandra-a/data
 - /srv/sdd4/cassandra-a/data
-- /srv/sde4/cassandra-a/data
   heapdump_directory: /srv/sda4/cassandra-a
   commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
   hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
@@ -379,7 +378,6 @@
 #- /srv/sdb4/cassandra-b/data
 #- /srv/sdc4/cassandra-b/data
 #- /srv/sdd4/cassandra-b/data
-#- /srv/sde4/cassandra-b/data
 #  heapdump_directory: /srv/sdb4/cassandra-b
 #  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
 #  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
@@ -392,7 +390,6 @@
 #- /srv/sdb4/cassandra-c/data
 #- /srv/sdc4/cassandra-c/data
 #- /srv/sdd4/cassandra-c/data
-#- /srv/sde4/cassandra-c/data
 #  heapdump_directory: /srv/sdc4/cassandra-c
 #  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
 #  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints

-- 
To view, visit https://gerrit.wikimedia.org/r/404277
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6b456e7ca0868c993b083c69b747640f77e704d3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase1017

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404262 )

Change subject: restbase: reprovision restbase1017
..


restbase: reprovision restbase1017

This host has 4 ssds, thus also introduce the corresponding partman recipe for 
JBOD.

Bug: T184100
Change-Id: I1f7e0e0f3bf32d7255c24f8c1cb7ff4f1cf21d9d
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
A modules/install_server/files/autoinstall/partman/cassandrahosts-4ssd-jbod.cfg
5 files changed, 120 insertions(+), 10 deletions(-)

Approvals:
  Mobrovac: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index a7dc889..6326ff5 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -134,13 +134,13 @@
   listen_address: 10.64.0.33
 c:
   listen_address: 10.64.0.34
-  "restbase1017.eqiad.wmnet":
-a:
-  listen_address: 10.64.32.130
-b:
-  listen_address: 10.64.32.131
-c:
-  listen_address: 10.64.32.132
+  # "restbase1017.eqiad.wmnet":
+  #   a:
+  # listen_address: 10.64.32.130
+  #   b:
+  # listen_address: 10.64.32.131
+  #   c:
+  # listen_address: 10.64.32.132
   "restbase1018.eqiad.wmnet":
 a:
   listen_address: 10.64.48.98
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 1bc6155..c29b1d7 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -358,6 +358,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1017.eqiad.wmnet":
+a:
+  listen_address: 10.64.32.130
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+#b:
+#  listen_address: 10.64.32.131
+#  data_file_directories:
+#- /srv/sda4/cassandra-b/data
+#- /srv/sdb4/cassandra-b/data
+#- /srv/sdc4/cassandra-b/data
+#- /srv/sdd4/cassandra-b/data
+#- /srv/sde4/cassandra-b/data
+#  heapdump_directory: /srv/sdb4/cassandra-b
+#  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+#  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+#  jmx_exporter_enabled: true
+#c:
+#  listen_address: 10.64.32.132
+#  data_file_directories:
+#- /srv/sda4/cassandra-c/data
+#- /srv/sdb4/cassandra-c/data
+#- /srv/sdc4/cassandra-c/data
+#- /srv/sdd4/cassandra-c/data
+#- /srv/sde4/cassandra-c/data
+#  heapdump_directory: /srv/sdc4/cassandra-c
+#  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+#  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+#  jmx_exporter_enabled: true
   "restbase2001.codfw.wmnet":
 a:
   listen_address: 10.192.16.162
diff --git a/manifests/site.pp b/manifests/site.pp
index 8803b67..cc900c4 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1834,11 +1834,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[35678]\.eqiad\.wmnet$/ {
+node /^restbase101[3568]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[0124])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[01247])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 
diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index 8232895..ee406c4 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -155,7 +155,8 @@
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
 restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase101[35]) echo partman/cassandrahosts-5ssd.cfg ;; \
-

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reprovision restbase1017

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404262 )

Change subject: restbase: reprovision restbase1017
..

restbase: reprovision restbase1017

This host has 4 ssds, thus also introduce the corresponding partman recipe for 
JBOD.

Bug: T184100
Change-Id: I1f7e0e0f3bf32d7255c24f8c1cb7ff4f1cf21d9d
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
A modules/install_server/files/autoinstall/partman/cassandrahosts-4ssd-jbod.cfg
5 files changed, 120 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/62/404262/1

diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index a7dc889..6326ff5 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -134,13 +134,13 @@
   listen_address: 10.64.0.33
 c:
   listen_address: 10.64.0.34
-  "restbase1017.eqiad.wmnet":
-a:
-  listen_address: 10.64.32.130
-b:
-  listen_address: 10.64.32.131
-c:
-  listen_address: 10.64.32.132
+  # "restbase1017.eqiad.wmnet":
+  #   a:
+  # listen_address: 10.64.32.130
+  #   b:
+  # listen_address: 10.64.32.131
+  #   c:
+  # listen_address: 10.64.32.132
   "restbase1018.eqiad.wmnet":
 a:
   listen_address: 10.64.48.98
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 1bc6155..c29b1d7 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -358,6 +358,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1017.eqiad.wmnet":
+a:
+  listen_address: 10.64.32.130
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+#b:
+#  listen_address: 10.64.32.131
+#  data_file_directories:
+#- /srv/sda4/cassandra-b/data
+#- /srv/sdb4/cassandra-b/data
+#- /srv/sdc4/cassandra-b/data
+#- /srv/sdd4/cassandra-b/data
+#- /srv/sde4/cassandra-b/data
+#  heapdump_directory: /srv/sdb4/cassandra-b
+#  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+#  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+#  jmx_exporter_enabled: true
+#c:
+#  listen_address: 10.64.32.132
+#  data_file_directories:
+#- /srv/sda4/cassandra-c/data
+#- /srv/sdb4/cassandra-c/data
+#- /srv/sdc4/cassandra-c/data
+#- /srv/sdd4/cassandra-c/data
+#- /srv/sde4/cassandra-c/data
+#  heapdump_directory: /srv/sdc4/cassandra-c
+#  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+#  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+#  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+#  jmx_exporter_enabled: true
   "restbase2001.codfw.wmnet":
 a:
   listen_address: 10.192.16.162
diff --git a/manifests/site.pp b/manifests/site.pp
index 8803b67..cc900c4 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1834,11 +1834,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[35678]\.eqiad\.wmnet$/ {
+node /^restbase101[3568]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[0124])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[01247])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 
diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index 8232895..ee406c4 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -155,7 +155,8 @@
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
 restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
 restbase101[35]) echo partman/cassandrahosts-5ssd.cfg ;; \
-restbase101[6-8]|restbase-dev100[456]) echo 
partman/cassandrahosts-4ssd.cfg ;; \
+

[MediaWiki-commits] [Gerrit] operations...prometheus-blazegraph-exporter[master]: debian: start after blazegraph

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403915 )

Change subject: debian: start after blazegraph
..


debian: start after blazegraph

Bug: T184434
Change-Id: I31d5adef7bd42761b90660845c5250057e3df3ef
---
M debian/changelog
M debian/service
2 files changed, 8 insertions(+), 0 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved
  Gehel: Looks good to me, but someone else must approve



diff --git a/debian/changelog b/debian/changelog
index 12b38c6..20c2079 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+prometheus-blazegraph-exporter (0.3) jessie-wikimedia; urgency=medium
+
+  * start after blazegraph
+
+ -- Filippo Giunchedi   Fri, 12 Jan 2018 13:54:00 +0100
+
 prometheus-blazegraph-exporter (0.2) jessie-wikimedia; urgency=medium
 
   * Depend on python-dateutil
diff --git a/debian/service b/debian/service
index 764352f..7761739 100644
--- a/debian/service
+++ b/debian/service
@@ -1,5 +1,7 @@
 [Unit]
 Description=Prometheus Blazegraph Exporter
+Requires=blazegraph.service
+After=blazegraph.service
 
 [Service]
 Restart=always

-- 
To view, visit https://gerrit.wikimedia.org/r/403915
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I31d5adef7bd42761b90660845c5250057e3df3ef
Gerrit-PatchSet: 2
Gerrit-Project: operations/debs/prometheus-blazegraph-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-blazegraph-exporter[master]: debian: start after blazegraph

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403915 )

Change subject: debian: start after blazegraph
..

debian: start after blazegraph

Bug: T184434
Change-Id: I31d5adef7bd42761b90660845c5250057e3df3ef
---
M debian/service
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-blazegraph-exporter 
refs/changes/15/403915/1

diff --git a/debian/service b/debian/service
index 764352f..7761739 100644
--- a/debian/service
+++ b/debian/service
@@ -1,5 +1,7 @@
 [Unit]
 Description=Prometheus Blazegraph Exporter
+Requires=blazegraph.service
+After=blazegraph.service
 
 [Service]
 Restart=always

-- 
To view, visit https://gerrit.wikimedia.org/r/403915
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I31d5adef7bd42761b90660845c5250057e3df3ef
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-blazegraph-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Upgrade scap package to 3.7.5-1

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403775 )

Change subject: Upgrade scap package to 3.7.5-1
..


Upgrade scap package to 3.7.5-1

This change depends on the new package being uploaded to apt.

Bug: T184774
Change-Id: I7510e1f656bfeb01a98ec7f05e2dda1fa934cbe7
---
M modules/scap/manifests/init.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  20after4: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/scap/manifests/init.pp b/modules/scap/manifests/init.pp
index 32af20bc..292993d 100644
--- a/modules/scap/manifests/init.pp
+++ b/modules/scap/manifests/init.pp
@@ -12,7 +12,7 @@
 class scap (
 $deployment_server = 'deployment',
 $wmflabs_master = 'deployment-tin.deployment-prep.eqiad.wmflabs',
-$version = '3.7.4-3',
+$version = '3.7.5-1',
 ) {
 package { 'scap':
 ensure => $version,

-- 
To view, visit https://gerrit.wikimedia.org/r/403775
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7510e1f656bfeb01a98ec7f05e2dda1fa934cbe7
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: 20after4 
Gerrit-Reviewer: 20after4 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: statistics: Install php5-dom for wmde scripts

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403366 )

Change subject: statistics: Install php5-dom for wmde scripts
..


statistics: Install php5-dom for wmde scripts

See https://stackoverflow.com/a/14395414
And https://phabricator.wikimedia.org/T165463#3880126

Bug: T165463
Change-Id: Ibfd0844b65069b8e974619dc6fef9f79612595b6
---
M modules/statistics/manifests/wmde/graphite.pp
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/statistics/manifests/wmde/graphite.pp 
b/modules/statistics/manifests/wmde/graphite.pp
index d04185f..1093f90 100644
--- a/modules/statistics/manifests/wmde/graphite.pp
+++ b/modules/statistics/manifests/wmde/graphite.pp
@@ -32,6 +32,7 @@
 require_package(
 'php',
 'php-cli',
+'php-dom'
 )
 
 include ::passwords::mysql::research

-- 
To view, visit https://gerrit.wikimedia.org/r/403366
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibfd0844b65069b8e974619dc6fef9f79612595b6
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ladsgroup 
Gerrit-Reviewer: Addshore 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Ladsgroup 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: extend SMART eqiad deployment

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403621 )

Change subject: hieradata: extend SMART eqiad deployment
..

hieradata: extend SMART eqiad deployment

Bug: T86552
Change-Id: If947d58ca39e153311e69627d9bbd9a8caaa361f
---
M hieradata/regex.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/21/403621/1

diff --git a/hieradata/regex.yaml b/hieradata/regex.yaml
index b0f9d22..6bb518f 100644
--- a/hieradata/regex.yaml
+++ b/hieradata/regex.yaml
@@ -254,7 +254,7 @@
   profile::base::check_smart: true
 
 smart_health_eqiad:
-  __regex: !ruby/regexp 
/^(analytics10[234][0-9]|aqs1004|conf1001|cp1045|db1001|db1087|db1011|druid1001|elastic1017|es1015|etcd1001|ganeti1001|graphite1.*|kafka1001|kafka-jumbo1001|kube.*|labmon1001|labcontrol1001|labvirt1001|labvirt1010|labsdb1009|labstore1006|lvs1007logstash1008|mw11.*|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|phab1001|poolcounter1001|mwlog1001|netmon1.*|puppetmaster1001|pc1004|sca1003|scb1001|thumbor1.*|tin|tungsten|wdqs1003|wtp1025|xenon).eqiad\.wmnet$/
+  __regex: !ruby/regexp 
/^(analytics1.*|aqs1.*|conf1.*|cp1045|db1039|db1001|db1087|db1011|druid1.*|elastic1.*|es1015|etcd1001|ganeti1001|graphite1.*|kafka1.*|kube1.*|labmon1.*|labcontrol1.*|labvirt1.*|labsdb1009|labstore1.*|lvs1007|logstash1.*|mw1.*|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|phab1.*|poolcounter1.*|mwlog1.*|netmon1.*|puppetmaster1.*|pc1004|sca1003|scb1001|thumbor1.*|tin|tungsten|wdqs1.*|wtp1.*|xenon).eqiad\.wmnet$/
   profile::base::check_smart: true
 
 labtestpuppetmaster:

-- 
To view, visit https://gerrit.wikimedia.org/r/403621
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If947d58ca39e153311e69627d9bbd9a8caaa361f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: graphite: cleanup stale ORES metrics

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401917 )

Change subject: graphite: cleanup stale ORES metrics
..


graphite: cleanup stale ORES metrics

Bug: T169969
Change-Id: I01b37944e614a3e2e5a310094282381b67ade6dc
---
M modules/role/manifests/graphite/production.pp
1 file changed, 6 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved
  Halfak: Looks good to me, but someone else must approve



diff --git a/modules/role/manifests/graphite/production.pp 
b/modules/role/manifests/graphite/production.pp
index e46cac1..bf8df2f 100644
--- a/modules/role/manifests/graphite/production.pp
+++ b/modules/role/manifests/graphite/production.pp
@@ -65,6 +65,12 @@
 keep_days => 182,
 }
 
+# ORES metrics - T169969
+graphite::whisper_cleanup { 'graphite-ores':
+directory => "${storage_dir}/whisper/ores",
+keep_days => 30,
+}
+
 $graphite_hosts = [
 'graphite1001.eqiad.wmnet',
 'graphite1003.eqiad.wmnet',

-- 
To view, visit https://gerrit.wikimedia.org/r/401917
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I01b37944e614a3e2e5a310094282381b67ade6dc
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Awight 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Halfak 
Gerrit-Reviewer: Ladsgroup 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reimage restbase1011 as cassandra 3

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403389 )

Change subject: restbase: reimage restbase1011 as cassandra 3
..


restbase: reimage restbase1011 as cassandra 3

Bug: T184100
Change-Id: I3d65e4be4670a4fc65ed54620e299419e3b288dc
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
4 files changed, 51 insertions(+), 11 deletions(-)

Approvals:
  Mobrovac: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index 7c41926..a7dc889 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -92,13 +92,13 @@
   #listen_address: 10.64.0.115
   #  c:
   #listen_address: 10.64.0.116
-  "restbase1011.eqiad.wmnet":
-a:
-  listen_address: 10.64.0.117
-b:
-  listen_address: 10.64.0.118
-c:
-  listen_address: 10.64.0.119
+  #"restbase1011.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.0.117
+  #  b:
+  #listen_address: 10.64.0.118
+  #  c:
+  #listen_address: 10.64.0.119
   #"restbase1012.eqiad.wmnet":
   #  a:
   #listen_address: 10.64.32.202
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 8be132f..1bc6155 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -238,6 +238,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1011.eqiad.wmnet":
+a:
+  listen_address: 10.64.0.117
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.0.118
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+- /srv/sde4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.0.119
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+- /srv/sde4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase1012.eqiad.wmnet":
 a:
   listen_address: 10.64.32.202
diff --git a/manifests/site.pp b/manifests/site.pp
index 0d3ee07..020e122 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1876,11 +1876,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[135678]\.eqiad\.wmnet$/ {
+node /^restbase101[35678]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[024])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[0124])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 
diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index 78bf327..8232895 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -153,8 +153,8 @@
 rdb100[7-8]) echo partman/raid1.cfg ;; \
 rdb200[1-6]) echo partman/raid1-lvm-ext4-srv.cfg ;; \
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
-restbase100[789]|restbase101[024]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
-restbase101[135]) echo partman/cassandrahosts-5ssd.cfg ;; \
+restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
+

[MediaWiki-commits] [Gerrit] operations/puppet[production]: restbase: reimage restbase1011 as cassandra 3

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403389 )

Change subject: restbase: reimage restbase1011 as cassandra 3
..

restbase: reimage restbase1011 as cassandra 3

Bug: T184100
Change-Id: I3d65e4be4670a4fc65ed54620e299419e3b288dc
---
M hieradata/role/common/restbase/production.yaml
M hieradata/role/common/restbase/production_ng.yaml
M manifests/site.pp
M modules/install_server/files/autoinstall/netboot.cfg
4 files changed, 51 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/89/403389/1

diff --git a/hieradata/role/common/restbase/production.yaml 
b/hieradata/role/common/restbase/production.yaml
index 7c41926..a7dc889 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -92,13 +92,13 @@
   #listen_address: 10.64.0.115
   #  c:
   #listen_address: 10.64.0.116
-  "restbase1011.eqiad.wmnet":
-a:
-  listen_address: 10.64.0.117
-b:
-  listen_address: 10.64.0.118
-c:
-  listen_address: 10.64.0.119
+  #"restbase1011.eqiad.wmnet":
+  #  a:
+  #listen_address: 10.64.0.117
+  #  b:
+  #listen_address: 10.64.0.118
+  #  c:
+  #listen_address: 10.64.0.119
   #"restbase1012.eqiad.wmnet":
   #  a:
   #listen_address: 10.64.32.202
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index 8be132f..1bc6155 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -238,6 +238,46 @@
   hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
   saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
   jmx_exporter_enabled: true
+  "restbase1011.eqiad.wmnet":
+a:
+  listen_address: 10.64.0.117
+  data_file_directories:
+- /srv/sda4/cassandra-a/data
+- /srv/sdb4/cassandra-a/data
+- /srv/sdc4/cassandra-a/data
+- /srv/sdd4/cassandra-a/data
+- /srv/sde4/cassandra-a/data
+  heapdump_directory: /srv/sda4/cassandra-a
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-a/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-a/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-a/saved_caches
+  jmx_exporter_enabled: true
+b:
+  listen_address: 10.64.0.118
+  data_file_directories:
+- /srv/sda4/cassandra-b/data
+- /srv/sdb4/cassandra-b/data
+- /srv/sdc4/cassandra-b/data
+- /srv/sdd4/cassandra-b/data
+- /srv/sde4/cassandra-b/data
+  heapdump_directory: /srv/sdb4/cassandra-b
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-b/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-b/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-b/saved_caches
+  jmx_exporter_enabled: true
+c:
+  listen_address: 10.64.0.119
+  data_file_directories:
+- /srv/sda4/cassandra-c/data
+- /srv/sdb4/cassandra-c/data
+- /srv/sdc4/cassandra-c/data
+- /srv/sdd4/cassandra-c/data
+- /srv/sde4/cassandra-c/data
+  heapdump_directory: /srv/sdc4/cassandra-c
+  commitlog_directory: /srv/cassandra/instance-data/cassandra-c/commitlog
+  hints_directory: /srv/cassandra/instance-data/cassandra-c/hints
+  saved_caches_directory: 
/srv/cassandra/instance-data/cassandra-c/saved_caches
+  jmx_exporter_enabled: true
   "restbase1012.eqiad.wmnet":
 a:
   listen_address: 10.64.32.202
diff --git a/manifests/site.pp b/manifests/site.pp
index 0d3ee07..020e122 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1876,11 +1876,11 @@
 }
 
 # restbase eqiad cluster
-node /^restbase101[135678]\.eqiad\.wmnet$/ {
+node /^restbase101[35678]\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-node /^restbase10(0[789]|1[024])\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[0124])\.eqiad\.wmnet$/ {
 role(restbase::production_ng)
 }
 
diff --git a/modules/install_server/files/autoinstall/netboot.cfg 
b/modules/install_server/files/autoinstall/netboot.cfg
index 78bf327..8232895 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -153,8 +153,8 @@
 rdb100[7-8]) echo partman/raid1.cfg ;; \
 rdb200[1-6]) echo partman/raid1-lvm-ext4-srv.cfg ;; \
 restbase100[0-6]) echo partman/cassandrahosts-5ssd.cfg ;; \
-restbase100[789]|restbase101[024]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
-restbase101[135]) echo partman/cassandrahosts-5ssd.cfg ;; \
+restbase100[789]|restbase101[0124]) echo 
partman/cassandrahosts-5ssd-jbod.cfg ;; \
+restbase101[35]) echo partman/cassandrahosts-5ssd.cfg ;; \
 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site: spare::system vs system::spare

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403376 )

Change subject: site: spare::system vs system::spare
..


site: spare::system vs system::spare

Change-Id: I5e4f5a9f4e14276717cea71bc067c8ed29f95e22
---
M manifests/site.pp
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/manifests/site.pp b/manifests/site.pp
index 276cd4a..0d3ee07 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1877,7 +1877,7 @@
 
 # restbase eqiad cluster
 node /^restbase101[135678]\.eqiad\.wmnet$/ {
-role(system::spare)
+role(spare::system)
 }
 
 node /^restbase10(0[789]|1[024])\.eqiad\.wmnet$/ {
@@ -1886,7 +1886,7 @@
 
 # restbase codfw cluster
 node /^restbase20(0[789]|1[012])\.codfw\.wmnet$/ {
-role(system::spare)
+role(spare::system)
 }
 
 node /^restbase200[123456]\.codfw\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/403376
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5e4f5a9f4e14276717cea71bc067c8ed29f95e22
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site: spare::system vs system::spare

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403376 )

Change subject: site: spare::system vs system::spare
..

site: spare::system vs system::spare

Change-Id: I5e4f5a9f4e14276717cea71bc067c8ed29f95e22
---
M manifests/site.pp
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/76/403376/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 276cd4a..0d3ee07 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1877,7 +1877,7 @@
 
 # restbase eqiad cluster
 node /^restbase101[135678]\.eqiad\.wmnet$/ {
-role(system::spare)
+role(spare::system)
 }
 
 node /^restbase10(0[789]|1[024])\.eqiad\.wmnet$/ {
@@ -1886,7 +1886,7 @@
 
 # restbase codfw cluster
 node /^restbase20(0[789]|1[012])\.codfw\.wmnet$/ {
-role(system::spare)
+role(spare::system)
 }
 
 node /^restbase200[123456]\.codfw\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/403376
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5e4f5a9f4e14276717cea71bc067c8ed29f95e22
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: decom legacy restbase/cassandra 2 cluster

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403208 )

Change subject: decom legacy restbase/cassandra 2 cluster
..


decom legacy restbase/cassandra 2 cluster

While reimages are in progress mark the systems as spare to get basic updates 
and otherwise unmanage
the rest of the services.

Bug: T184100
Change-Id: Ib2cea1f71764963a9371577b3168a17b42c9dd7c
---
M manifests/site.pp
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Eevans: Looks good to me, but someone else must approve
  Mobrovac: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/manifests/site.pp b/manifests/site.pp
index 3d3ff63..276cd4a 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1877,7 +1877,7 @@
 
 # restbase eqiad cluster
 node /^restbase101[135678]\.eqiad\.wmnet$/ {
-role(restbase::production)
+role(system::spare)
 }
 
 node /^restbase10(0[789]|1[024])\.eqiad\.wmnet$/ {
@@ -1886,7 +1886,7 @@
 
 # restbase codfw cluster
 node /^restbase20(0[789]|1[012])\.codfw\.wmnet$/ {
-role(restbase::production)
+role(system::spare)
 }
 
 node /^restbase200[123456]\.codfw\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/403208
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib2cea1f71764963a9371577b3168a17b42c9dd7c
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Eevans 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Mobrovac 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: decom legacy restbase/cassandra 2 cluster

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403208 )

Change subject: decom legacy restbase/cassandra 2 cluster
..

decom legacy restbase/cassandra 2 cluster

While reimages are in progress mark the systems as spare to get basic updates 
and otherwise unmanage
the rest of the services.

Bug: T184100
Change-Id: Ib2cea1f71764963a9371577b3168a17b42c9dd7c
---
M manifests/site.pp
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/08/403208/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 9ddd340..b63ec3b 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1929,7 +1929,7 @@
 
 # restbase eqiad cluster
 node /^restbase101[135678]\.eqiad\.wmnet$/ {
-role(restbase::production)
+role(system::spare)
 }
 
 node /^restbase10(0[789]|1[024])\.eqiad\.wmnet$/ {
@@ -1938,7 +1938,7 @@
 
 # restbase codfw cluster
 node /^restbase20(0[789]|1[012])\.codfw\.wmnet$/ {
-role(restbase::production)
+role(system::spare)
 }
 
 node /^restbase200[123456]\.codfw\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/403208
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib2cea1f71764963a9371577b3168a17b42c9dd7c
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...python-thumbor-wikimedia[master]: Upgrade to 1.10

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402906 )

Change subject: Upgrade to 1.10
..


Upgrade to 1.10

Bug: T183907
Change-Id: I63e62815461d968dd35d704d1e62fe6fc8af1030
---
M debian/changelog
M setup.py
M tests/integration/test_images_handler.py
M wikimedia_thumbor/handler/images/images.py
4 files changed, 29 insertions(+), 7 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/debian/changelog b/debian/changelog
index 31a1f0f..d1fe280 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+python-thumbor-wikimedia (1.10-1) jessie-wikimedia; urgency=low
+
+  * New upstream release
+
+ -- Gilles Dubuc   Mon, 8 Jan 2018 19:35:00 +
+
 python-thumbor-wikimedia (1.9-1) jessie-wikimedia; urgency=low
 
   * New upstream release
diff --git a/setup.py b/setup.py
index 5d9cac0..382d109 100644
--- a/setup.py
+++ b/setup.py
@@ -12,7 +12,7 @@
 
 setup(
 name='wikimedia_thumbor',
-version='1.9',
+version='1.10',
 url='https://phabricator.wikimedia.org/diffusion/THMBREXT/',
 license='MIT',
 author='Gilles Dubuc, Wikimedia Foundation',
diff --git a/tests/integration/test_images_handler.py 
b/tests/integration/test_images_handler.py
index 135ffed..0a77310 100644
--- a/tests/integration/test_images_handler.py
+++ b/tests/integration/test_images_handler.py
@@ -37,7 +37,9 @@
 cfg.SWIFT_SHARDED_CONTAINERS = [
 'wikipedia-en-local-public',
 'wikipedia-en-local-temp',
-'wikipedia-en-local-thumb'
+'wikipedia-en-local-thumb',
+'wikipedia-commons-local-public',
+'wikipedia-commons-local-thumb'
 ]
 cfg.SWIFT_PATH_PREFIX = 'thumbor/'
 cfg.SWIFT_CONNECTION_TIMEOUT = 1
@@ -289,7 +291,7 @@
 '400',
 
'http://swifthost/swift/v1/api/path/wikipedia-en-local-public.d3/archive/d/d3/20160729183014!1Mcolors.jpg',
 'conditional_sharpen(0.0,0.8,1.0,0.0,0.85):format(jpg)',
-'inline;filename*=UTF-8\'\'20160729183014!1Mcolors.jpg'
+'inline;filename*=UTF-8\'\'20160729183014%211Mcolors.jpg'
 )
 
 def test_lossless_tiff(self):
@@ -318,7 +320,7 @@
 'http://swifthost/swift/v1/api/path/'
 + 'wikipedia-en-local-public.d3/d/d3/1M\ncolors.tif',
 'format(png):page(1)',
-None
+'inline;filename*=UTF-8\'\'1M%0Acolors.tif.png'
 )
 
 def test_question_mark_original(self):
@@ -332,7 +334,21 @@
 '400',
 
'http://swifthost/swift/v1/api/path/wikipedia-en-local-public.d3/d/d3/1M?colors.tif',
 'format(png):page(1)',
-'inline;filename*=UTF-8\'\'1M?colors.tif.png'
+'inline;filename*=UTF-8\'\'1M%3Fcolors.tif.png'
+)
+
+def test_percentage_original(self):
+self.run_and_check_headers(
+
'/wikipedia/commons/thumb/c/ce/Dramatische_daling_insectenpopulatie%2C_75%25_is_verdwenen.webm/800px--Dramatische_daling_insectenpopulatie%2C_75%25_is_verdwenen.webm.jpg',
+'File:Dramatische_daling_insectenpopulatie,_75%_is_verdwenen.webm',
+'wikipedia-commons-local-public.ce',
+'c/ce/Dramatische_daling_insectenpopulatie,_75%_is_verdwenen.webm',
+'wikipedia-commons-local-thumb.ce',
+
'thumbor/c/ce/Dramatische_daling_insectenpopulatie,_75%_is_verdwenen.webm/800px--Dramatische_daling_insectenpopulatie,_75%_is_verdwenen.webm.jpg',
+'800',
+
'http://swifthost/swift/v1/api/path/wikipedia-commons-local-public.ce/c/ce/Dramatische_daling_insectenpopulatie,_75%_is_verdwenen.webm',
+'format(jpg)',
+
'inline;filename*=UTF-8\'\'Dramatische_daling_insectenpopulatie%2C_75%25_is_verdwenen.webm.jpg'
 )
 
 def test_temp(self):
@@ -346,7 +362,7 @@
 '71',
 
'http://swifthost/swift/v1/api/path/wikipedia-en-local-temp.57/5/57/20161115090130!fYJSjm.pdf',
 'format(jpg):page(1)',
-'inline;filename*=UTF-8\'\'20161115090130!fYJSjm.pdf.jpg'
+'inline;filename*=UTF-8\'\'20161115090130%21fYJSjm.pdf.jpg'
 )
 
 def run_and_check_error(
diff --git a/wikimedia_thumbor/handler/images/images.py 
b/wikimedia_thumbor/handler/images/images.py
index 9094809..6c6e933 100644
--- a/wikimedia_thumbor/handler/images/images.py
+++ b/wikimedia_thumbor/handler/images/images.py
@@ -335,7 +335,7 @@
 
 self.safe_set_header(
 'Content-Disposition',
-u'inline;filename*=UTF-8\'\'%s' % content_disposition
+u'inline;filename*=UTF-8\'\'%s' % 
quote(content_disposition.encode('utf-8'))
 )
 
 self.safe_set_header(

-- 
To view, visit https://gerrit.wikimedia.org/r/402906
To unsubscribe, visit 

[MediaWiki-commits] [Gerrit] operations...python-thumbor-wikimedia[master]: Upgrade to 1.9

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402582 )

Change subject: Upgrade to 1.9
..


Upgrade to 1.9

Bug: T169144
Change-Id: I1cb3d133ca3522dd64e52c6b9cb2764863ffc0ad
---
M debian/changelog
M setup.py
M wikimedia_thumbor/handler/images/images.py
3 files changed, 24 insertions(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/debian/changelog b/debian/changelog
index 59b5c76..31a1f0f 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+python-thumbor-wikimedia (1.9-1) jessie-wikimedia; urgency=low
+
+  * New upstream release
+
+ -- Gilles Dubuc   Sat, 6 Jan 2018 13:02:00 +
+
 python-thumbor-wikimedia (1.8-1) jessie-wikimedia; urgency=low
 
   * New upstream release
diff --git a/setup.py b/setup.py
index 36d..5d9cac0 100644
--- a/setup.py
+++ b/setup.py
@@ -12,7 +12,7 @@
 
 setup(
 name='wikimedia_thumbor',
-version='1.8',
+version='1.9',
 url='https://phabricator.wikimedia.org/diffusion/THMBREXT/',
 license='MIT',
 author='Gilles Dubuc, Wikimedia Foundation',
diff --git a/wikimedia_thumbor/handler/images/images.py 
b/wikimedia_thumbor/handler/images/images.py
index 3625506..9094809 100644
--- a/wikimedia_thumbor/handler/images/images.py
+++ b/wikimedia_thumbor/handler/images/images.py
@@ -197,6 +197,11 @@
 if hasattr(self.context.config, 'SWIFT_SHARDED_CONTAINERS'):
 sharded_containers = self.context.config.SWIFT_SHARDED_CONTAINERS
 
+private_containers = []
+
+if hasattr(self.context.config, 'SWIFT_PRIVATE_CONTAINERS'):
+private_containers = self.context.config.SWIFT_PRIVATE_CONTAINERS
+
 projlang = '-'.join((kw['project'], kw['language']))
 original_container = projlang + '-local-public'
 thumbnail_container = projlang + '-local-thumb'
@@ -221,6 +226,8 @@
 hashed = md5.new(hashed_name).hexdigest()
 original_shard1 = hashed[:1]
 original_shard2 = hashed[:2]
+
+self.context.private = original_container in private_containers
 
 if original_container in sharded_containers:
 original_container += '.' + original_shard2
@@ -374,6 +381,16 @@
 )
 return
 
+if self.context.private:
+received_secret = self.request.headers.get('X-Swift-Secret', False)
+secret = self.context.config.get('SWIFT_PRIVATE_SECRET', False)
+if not secret or not received_secret or received_secret != secret:
+self._error(
+401,
+'Unauthorized access to private Swift container'
+)
+return
+
 xkey = self.set_headers(translated_kw)
 mc = self.failure_memcache()
 

-- 
To view, visit https://gerrit.wikimedia.org/r/402582
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1cb3d133ca3522dd64e52c6b9cb2764863ffc0ad
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/python-thumbor-wikimedia
Gerrit-Branch: master
Gerrit-Owner: Gilles 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: fix counting for frontend varnish http_method

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403158 )

Change subject: mtail: fix counting for frontend varnish http_method
..


mtail: fix counting for frontend varnish http_method

IOW match method name as a whole, not prefix

Bug: T183926
Change-Id: I302f6b0033e85ad30dd4685abce645f2118864bb
---
M modules/mtail/files/programs/varnishreqstats.mtail
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/mtail/files/programs/varnishreqstats.mtail 
b/modules/mtail/files/programs/varnishreqstats.mtail
index a324f12..7c85f6f 100644
--- a/modules/mtail/files/programs/varnishreqstats.mtail
+++ b/modules/mtail/files/programs/varnishreqstats.mtail
@@ -1,7 +1,7 @@
 counter varnish_requests by status, method
 
 /\thttp_status (?P[0-9][0-9][0-9])\thttp_method (?P[A-Z]+)\t/ {
-/http_method (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT|PURGE|PATCH)/ 
{
+/http_method 
(OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT|PURGE|PATCH)\t/ {
   varnish_requests[$status][$method]++
 } else {
   varnish_requests[$status]["invalid"]++

-- 
To view, visit https://gerrit.wikimedia.org/r/403158
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I302f6b0033e85ad30dd4685abce645f2118864bb
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: fix counting for frontend varnish http_method

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403158 )

Change subject: mtail: fix counting for frontend varnish http_method
..

mtail: fix counting for frontend varnish http_method

IOW match method name as a whole, not prefix

Bug: T183926
Change-Id: I302f6b0033e85ad30dd4685abce645f2118864bb
---
M modules/mtail/files/programs/varnishreqstats.mtail
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/58/403158/1

diff --git a/modules/mtail/files/programs/varnishreqstats.mtail 
b/modules/mtail/files/programs/varnishreqstats.mtail
index a324f12..7c85f6f 100644
--- a/modules/mtail/files/programs/varnishreqstats.mtail
+++ b/modules/mtail/files/programs/varnishreqstats.mtail
@@ -1,7 +1,7 @@
 counter varnish_requests by status, method
 
 /\thttp_status (?P[0-9][0-9][0-9])\thttp_method (?P[A-Z]+)\t/ {
-/http_method (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT|PURGE|PATCH)/ 
{
+/http_method 
(OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT|PURGE|PATCH)\t/ {
   varnish_requests[$status][$method]++
 } else {
   varnish_requests[$status]["invalid"]++

-- 
To view, visit https://gerrit.wikimedia.org/r/403158
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I302f6b0033e85ad30dd4685abce645f2118864bb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: RESTBase: Add LVS pools to the production_ng role

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403133 )

Change subject: RESTBase: Add LVS pools to the production_ng role
..


RESTBase: Add LVS pools to the production_ng role

Bug: T184110
Change-Id: Iab131a92465734f5933d4ee78e24eccfe7549209
---
M hieradata/role/common/restbase/production_ng.yaml
1 file changed, 4 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index c73e81b..8be132f 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -586,6 +586,10 @@
 #
 # General
 #
+role::lvs::realserver::pools:
+  restbase: {}
+service::configuration::log_dir: "/var/log"
+
 admin::groups:
   - restbase-roots
   - restbase-admins

-- 
To view, visit https://gerrit.wikimedia.org/r/403133
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iab131a92465734f5933d4ee78e24eccfe7549209
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mobrovac 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Mobrovac 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: RESTBase: Set up RESTBase for the production_ng role as well

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401784 )

Change subject: RESTBase: Set up RESTBase for the production_ng role as well
..


RESTBase: Set up RESTBase for the production_ng role as well

During the migration from Cassandra 2 to Cassandra 3 we chose not to set
up RESTBase on the new nodes, but now the time has come to transition
RESTBase to them as well.

Bug: T184110
Change-Id: Ie2484c2175e56ed40e5108b6ec41adbccfa19bfa
---
M hieradata/role/codfw/restbase/production.yaml
A hieradata/role/codfw/restbase/production_ng.yaml
M hieradata/role/common/restbase/production_ng.yaml
M hieradata/role/eqiad/restbase/production.yaml
A hieradata/role/eqiad/restbase/production_ng.yaml
M modules/profile/manifests/restbase.pp
M modules/role/manifests/restbase/production_ng.pp
7 files changed, 100 insertions(+), 16 deletions(-)

Approvals:
  Eevans: Looks good to me, but someone else must approve
  Mobrovac: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/role/codfw/restbase/production.yaml 
b/hieradata/role/codfw/restbase/production.yaml
index ffebfa4..c820561 100644
--- a/hieradata/role/codfw/restbase/production.yaml
+++ b/hieradata/role/codfw/restbase/production.yaml
@@ -1,7 +1,3 @@
-profile::restbase::seeds:
-  - restbase2007-a.codfw.wmnet
-  - restbase2007-b.codfw.wmnet
-  - restbase2007-c.codfw.wmnet
 profile::restbase::seeds_ng:
   - restbase2001-a.codfw.wmnet
   - restbase2001-b.codfw.wmnet
@@ -17,9 +13,18 @@
   - restbase2005-c.codfw.wmnet
 # used for rate limiting
 profile::restbase::hosts:
+  - restbase2001.codfw.wmnet
+  - restbase2002.codfw.wmnet
+  - restbase2003.codfw.wmnet
+  - restbase2004.codfw.wmnet
+  - restbase2005.codfw.wmnet
+  - restbase2006.codfw.wmnet
   - restbase2007.codfw.wmnet
   - restbase2008.codfw.wmnet
   - restbase2009.codfw.wmnet
+  - restbase2010.codfw.wmnet
+  - restbase2011.codfw.wmnet
+  - restbase2012.codfw.wmnet
 
 lvs::realserver::realserver_ips:
   - '10.2.1.17' # restbase.svc.codfw.wmnet
diff --git a/hieradata/role/codfw/restbase/production_ng.yaml 
b/hieradata/role/codfw/restbase/production_ng.yaml
new file mode 100644
index 000..c820561
--- /dev/null
+++ b/hieradata/role/codfw/restbase/production_ng.yaml
@@ -0,0 +1,30 @@
+profile::restbase::seeds_ng:
+  - restbase2001-a.codfw.wmnet
+  - restbase2001-b.codfw.wmnet
+  - restbase2001-c.codfw.wmnet
+  - restbase2002-a.codfw.wmnet
+  - restbase2002-b.codfw.wmnet
+  - restbase2002-c.codfw.wmnet
+  - restbase2003-a.codfw.wmnet
+  - restbase2003-b.codfw.wmnet
+  - restbase2003-c.codfw.wmnet
+  - restbase2005-a.codfw.wmnet
+  - restbase2005-b.codfw.wmnet
+  - restbase2005-c.codfw.wmnet
+# used for rate limiting
+profile::restbase::hosts:
+  - restbase2001.codfw.wmnet
+  - restbase2002.codfw.wmnet
+  - restbase2003.codfw.wmnet
+  - restbase2004.codfw.wmnet
+  - restbase2005.codfw.wmnet
+  - restbase2006.codfw.wmnet
+  - restbase2007.codfw.wmnet
+  - restbase2008.codfw.wmnet
+  - restbase2009.codfw.wmnet
+  - restbase2010.codfw.wmnet
+  - restbase2011.codfw.wmnet
+  - restbase2012.codfw.wmnet
+
+lvs::realserver::realserver_ips:
+  - '10.2.1.17' # restbase.svc.codfw.wmnet
diff --git a/hieradata/role/common/restbase/production_ng.yaml 
b/hieradata/role/common/restbase/production_ng.yaml
index bbeaea3..c73e81b 100644
--- a/hieradata/role/common/restbase/production_ng.yaml
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -1,4 +1,29 @@
 cluster: restbase
+
+#
+# RESTBase
+#
+profile::restbase::cassandra_local_dc: "%{::site}"
+profile::restbase::logging_label: restbase
+profile::restbase::cassandra_datacenters:
+- "eqiad"
+- "codfw"
+profile::restbase::parsoid_uri: "http://parsoid.discovery.wmnet:8000;
+profile::restbase::graphoid_uri: "http://graphoid.discovery.wmnet:19000;
+profile::restbase::mobileapps_uri: "http://mobileapps.discovery.wmnet:;
+profile::restbase::mathoid_uri: "http://mathoid.discovery.wmnet:10042;
+profile::restbase::aqs_uri: 
"http://aqs.svc.%{::aqs_site}.wmnet:7232/analytics.wikimedia.org/v1;
+profile::restbase::pdfrender_uri: "http://pdfrender.discovery.wmnet:5252;
+profile::restbase::citoid_uri: "http://citoid.discovery.wmnet:1970;
+profile::restbase::eventlogging_service_uri: 
"http://eventbus.discovery.wmnet:8085/v1/events;
+profile::restbase::cxserver_uri: "http://cxserver.discovery.wmnet:8080;
+profile::restbase::recommendation_uri: 
"http://recommendation-api.discovery.wmnet:9632;
+profile::restbase::cassandra_user: restb
+profile::restbase::cassandra_password: "%{passwords::cassandra::restbase}"
+profile::restbase::monitor_domain: en.wikipedia.org
+profile::restbase::cassandra_tls:
+  ca: /dev/null
+
 #
 # Cassandra
 #
diff --git a/hieradata/role/eqiad/restbase/production.yaml 
b/hieradata/role/eqiad/restbase/production.yaml
index ff559bd..8d770bd 100644
--- 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: aggregate varnish_requests rate

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402328 )

Change subject: prometheus: aggregate varnish_requests rate
..


prometheus: aggregate varnish_requests rate

Make it possible to account for counter resets even when aggregating globally.

Bug: T177199
Change-Id: I3ea09933f16a93aea3a10082c9cce3c6c9e113c1
---
M modules/role/files/prometheus/rules_ops.conf
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index b315385..31ef36c 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -138,6 +138,7 @@
 
 # reqstats aggregation
 job_method_status:varnish_requests:sum = sum(varnish_requests) by (job, 
method, status)
+job_method_status:varnish_requests:rate5m = sum(rate(varnish_requests[5m])) by 
(job, method, status)
 
 # MySQL aggregated stats
 job_role_shard:mysql_global_status_queries:rate5m = sum by (job, role, shard) 
(rate(mysql_global_status_queries[5m]))

-- 
To view, visit https://gerrit.wikimedia.org/r/402328
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3ea09933f16a93aea3a10082c9cce3c6c9e113c1
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: aggregate varnish_requests rate

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402328 )

Change subject: prometheus: aggregate varnish_requests rate
..

prometheus: aggregate varnish_requests rate

Make it possible to account for counter resets even when aggregating globally.

Bug: T177199
Change-Id: I3ea09933f16a93aea3a10082c9cce3c6c9e113c1
---
M modules/role/files/prometheus/rules_ops.conf
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/28/402328/1

diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index b315385..31ef36c 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -138,6 +138,7 @@
 
 # reqstats aggregation
 job_method_status:varnish_requests:sum = sum(varnish_requests) by (job, 
method, status)
+job_method_status:varnish_requests:rate5m = sum(rate(varnish_requests[5m])) by 
(job, method, status)
 
 # MySQL aggregated stats
 job_role_shard:mysql_global_status_queries:rate5m = sum by (job, role, shard) 
(rate(mysql_global_status_queries[5m]))

-- 
To view, visit https://gerrit.wikimedia.org/r/402328
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3ea09933f16a93aea3a10082c9cce3c6c9e113c1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: group invalid methods under a single metric

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402318 )

Change subject: mtail: group invalid methods under a single metric
..


mtail: group invalid methods under a single metric

This field is essentially user input, to avoid cardinality explosions and
general metric clutter group all HTTP methods we consider invalid under a single
metric.

Bug: T183926
Change-Id: I27b490f7c2e5acc323053dd1c72f2d75a9c9382b
---
M modules/mtail/files/programs/varnishreqstats.mtail
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
3 files changed, 8 insertions(+), 2 deletions(-)

Approvals:
  Ema: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/mtail/files/programs/varnishreqstats.mtail 
b/modules/mtail/files/programs/varnishreqstats.mtail
index 764f0e2..a324f12 100644
--- a/modules/mtail/files/programs/varnishreqstats.mtail
+++ b/modules/mtail/files/programs/varnishreqstats.mtail
@@ -1,5 +1,9 @@
 counter varnish_requests by status, method
 
 /\thttp_status (?P[0-9][0-9][0-9])\thttp_method (?P[A-Z]+)\t/ {
-varnish_requests[$status][$method]++
+/http_method (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT|PURGE|PATCH)/ 
{
+  varnish_requests[$status][$method]++
+} else {
+  varnish_requests[$status]["invalid"]++
+}
 }
diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index a8f0da0..e5155da 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -7,3 +7,4 @@
 url /wikipedia/commons/5/51/Tang_Shaoyi.jpgcache_status int-front  
http_status 301 http_method GET cache_control - inm -
 url 
/wikipedia/commons/thumb/5/51/Flag_of_North_Korea.svg/250px-Flag_of_North_Korea.svg.png
cache_status hit-front  http_status 200 http_method GET cache_control - inm 
-
 url /wikipedia/en/thumb/f/fd/Portal-puzzle.svg/16px-Portal-puzzle.svg.png  
cache_status hit-front  http_status 200 http_method GET cache_control - inm -
+url /test_for_invalid_method   cache_status hit-front  http_status 200 
http_method FOOZcache_control - inm -
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index c48fb80..0abe36e 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -14,7 +14,7 @@
 def testCacheStatus(self):
 s = self.store.get_samples('varnish_x_cache')
 self.assertIn(('x_cache=int-front', 2), s)
-self.assertIn(('x_cache=hit-front', 7), s)
+self.assertIn(('x_cache=hit-front', 8), s)
 
 
 class VarnishRlsTest(unittest.TestCase):
@@ -79,6 +79,7 @@
 self.assertIn(('status=200,method=GET', 3), s)
 self.assertIn(('status=301,method=GET', 2), s)
 self.assertIn(('status=200,method=HEAD', 2), s)
+self.assertIn(('status=200,method=invalid', 1), s)
 
 
 class VarnishBackendTest(unittest.TestCase):

-- 
To view, visit https://gerrit.wikimedia.org/r/402318
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I27b490f7c2e5acc323053dd1c72f2d75a9c9382b
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Run mtail tests via rake->tox->nose

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402325 )

Change subject: Run mtail tests via rake->tox->nose
..


Run mtail tests via rake->tox->nose

Bug: T181794
Change-Id: I4777d27f942963212f2c7e4c5d7f6fe7b661a991
---
M rake_modules/taskgen.rb
M tox.ini
2 files changed, 10 insertions(+), 0 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, but someone else must approve
  Ema: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/rake_modules/taskgen.rb b/rake_modules/taskgen.rb
index da4b103..c2e2aaa 100644
--- a/rake_modules/taskgen.rb
+++ b/rake_modules/taskgen.rb
@@ -289,6 +289,15 @@
   end
   tasks << 'tox:webperf'
 end
+mtail_files = filter_files_by("modules/mtail/files/**")
+unless mtail_files.empty?
+  desc 'Run tox for mtail'
+  task :mtail do
+res = system("tox -e mtail")
+raise 'Tests for mtail failed!' if !res
+  end
+  tasks << 'tox:mtail'
+end
 tox_files = filter_files_by("*.py")
 unless tox_files.empty?
   desc 'Run flake8 on python files via tox'
diff --git a/tox.ini b/tox.ini
index 62496d1..6c4a4c7 100644
--- a/tox.ini
+++ b/tox.ini
@@ -2,6 +2,7 @@
 minversion = 1.6
 skipsdist = True
 
+# See also rake_modules/taskgen.rb for additional cases where tox gets called 
conditionally
 envlist = webperf, commit-message, admin, apache, pep8, mtail
 
 [flake8]

-- 
To view, visit https://gerrit.wikimedia.org/r/402325
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4777d27f942963212f2c7e4c5d7f6fe7b661a991
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Run mtail tests via tox->rake->nose

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402325 )

Change subject: Run mtail tests via tox->rake->nose
..

Run mtail tests via tox->rake->nose

Bug: T181794
Change-Id: I4777d27f942963212f2c7e4c5d7f6fe7b661a991
---
M rake_modules/taskgen.rb
M tox.ini
2 files changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/25/402325/1

diff --git a/rake_modules/taskgen.rb b/rake_modules/taskgen.rb
index da4b103..c2e2aaa 100644
--- a/rake_modules/taskgen.rb
+++ b/rake_modules/taskgen.rb
@@ -289,6 +289,15 @@
   end
   tasks << 'tox:webperf'
 end
+mtail_files = filter_files_by("modules/mtail/files/**")
+unless mtail_files.empty?
+  desc 'Run tox for mtail'
+  task :mtail do
+res = system("tox -e mtail")
+raise 'Tests for mtail failed!' if !res
+  end
+  tasks << 'tox:mtail'
+end
 tox_files = filter_files_by("*.py")
 unless tox_files.empty?
   desc 'Run flake8 on python files via tox'
diff --git a/tox.ini b/tox.ini
index 62496d1..6c4a4c7 100644
--- a/tox.ini
+++ b/tox.ini
@@ -2,6 +2,7 @@
 minversion = 1.6
 skipsdist = True
 
+# See also rake_modules/taskgen.rb for additional cases where tox gets called 
conditionally
 envlist = webperf, commit-message, admin, apache, pep8, mtail
 
 [flake8]

-- 
To view, visit https://gerrit.wikimedia.org/r/402325
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4777d27f942963212f2c7e4c5d7f6fe7b661a991
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: group invalid methods under a single metric

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402318 )

Change subject: mtail: group invalid methods under a single metric
..

mtail: group invalid methods under a single metric

This field is essentially user input, to avoid cardinality explosions and
general metric clutter group all HTTP methods we consider invalid under a single
metric.

Bug: T183926
Change-Id: I27b490f7c2e5acc323053dd1c72f2d75a9c9382b
---
M modules/mtail/files/programs/varnishreqstats.mtail
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
3 files changed, 8 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/18/402318/1

diff --git a/modules/mtail/files/programs/varnishreqstats.mtail 
b/modules/mtail/files/programs/varnishreqstats.mtail
index 764f0e2..a324f12 100644
--- a/modules/mtail/files/programs/varnishreqstats.mtail
+++ b/modules/mtail/files/programs/varnishreqstats.mtail
@@ -1,5 +1,9 @@
 counter varnish_requests by status, method
 
 /\thttp_status (?P[0-9][0-9][0-9])\thttp_method (?P[A-Z]+)\t/ {
-varnish_requests[$status][$method]++
+/http_method (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT|PURGE|PATCH)/ 
{
+  varnish_requests[$status][$method]++
+} else {
+  varnish_requests[$status]["invalid"]++
+}
 }
diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index a8f0da0..e5155da 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -7,3 +7,4 @@
 url /wikipedia/commons/5/51/Tang_Shaoyi.jpgcache_status int-front  
http_status 301 http_method GET cache_control - inm -
 url 
/wikipedia/commons/thumb/5/51/Flag_of_North_Korea.svg/250px-Flag_of_North_Korea.svg.png
cache_status hit-front  http_status 200 http_method GET cache_control - inm 
-
 url /wikipedia/en/thumb/f/fd/Portal-puzzle.svg/16px-Portal-puzzle.svg.png  
cache_status hit-front  http_status 200 http_method GET cache_control - inm -
+url /test_for_invalid_method   cache_status hit-front  http_status 200 
http_method FOOZcache_control - inm -
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index c48fb80..0abe36e 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -14,7 +14,7 @@
 def testCacheStatus(self):
 s = self.store.get_samples('varnish_x_cache')
 self.assertIn(('x_cache=int-front', 2), s)
-self.assertIn(('x_cache=hit-front', 7), s)
+self.assertIn(('x_cache=hit-front', 8), s)
 
 
 class VarnishRlsTest(unittest.TestCase):
@@ -79,6 +79,7 @@
 self.assertIn(('status=200,method=GET', 3), s)
 self.assertIn(('status=301,method=GET', 2), s)
 self.assertIn(('status=200,method=HEAD', 2), s)
+self.assertIn(('status=200,method=invalid', 1), s)
 
 
 class VarnishBackendTest(unittest.TestCase):

-- 
To view, visit https://gerrit.wikimedia.org/r/402318
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I27b490f7c2e5acc323053dd1c72f2d75a9c9382b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add backend varnish mtail job

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402055 )

Change subject: prometheus: add backend varnish mtail job
..


prometheus: add backend varnish mtail job

Bug: T177199
Change-Id: I8d1271ebf7db7da4ec32483b7b440067215f249a
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 27 insertions(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 8680d6f..0976ddc 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -219,7 +219,7 @@
   },
 ]
 
-# Pull varnish-related metrics generated via mtail
+# Pull varnish-related metrics generated via mtail (frontend)
 prometheus::class_config{ "varnish-canary_mtail_${::site}":
 dest   => "${targets_path}/varnish-canary_mtail_${::site}.yaml",
 site   => $::site,
@@ -245,6 +245,32 @@
 port   => '3903',
 }
 
+# And varnish backend instance stats generated by mtail
+prometheus::class_config{ "varnish-canary_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-canary_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::canary',
+port   => '3904',
+}
+prometheus::class_config{ "varnish-misc_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-misc_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::misc',
+port   => '3904',
+}
+prometheus::class_config{ "varnish-upload_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-upload_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::upload',
+port   => '3904',
+}
+prometheus::class_config{ "varnish-text_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-text_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::text',
+port   => '3904',
+}
+
 # Job definition for memcache_exporter
 $memcached_jobs = [
   {

-- 
To view, visit https://gerrit.wikimedia.org/r/402055
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8d1271ebf7db7da4ec32483b7b440067215f249a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: extend eqiad SMART checking deployment

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402056 )

Change subject: hieradata: extend eqiad SMART checking deployment
..


hieradata: extend eqiad SMART checking deployment

Bug: T86552
Change-Id: If2a6829a76b5eb70328be23130dde5a1044f82a6
---
M hieradata/regex.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Alexandros Kosiaris: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/hieradata/regex.yaml b/hieradata/regex.yaml
index d04a992..537e6b3 100644
--- a/hieradata/regex.yaml
+++ b/hieradata/regex.yaml
@@ -254,7 +254,7 @@
   profile::base::check_smart: true
 
 smart_health_eqiad:
-  __regex: !ruby/regexp 
/^(analytics1028|aqs1004|conf1001|cp1045|db1039|db1001|db1087|db1011|druid1001|elastic1017|es1015|etcd1001|ganeti1001|graphite1001|kafka1001|kafka-jumbo1001|labmon1001|logstash1008|mw1180|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|pc1004|sca1003|scb1001|thumbor1001).eqiad\.wmnet$/
+  __regex: !ruby/regexp 
/^(analytics10[234][0-9]|aqs1004|conf1001|cp1045|db1039|db1001|db1087|db1011|druid1001|elastic1017|es1015|etcd1001|ganeti1001|graphite1.*|kafka1001|kafka-jumbo1001|kube.*|labmon1001|labcontrol1001|labvirt1001|labvirt1010|labsdb1009|labstore1006|lvs1007logstash1008|mw11.*|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|phab1001|poolcounter1001|mwlog1001|netmon1.*|puppetmaster1001|pc1004|sca1003|scb1001|thumbor1.*|tin|tungsten|wdqs1003|wtp1025|xenon).eqiad\.wmnet$/
   profile::base::check_smart: true
 
 labtestpuppetmaster:

-- 
To view, visit https://gerrit.wikimedia.org/r/402056
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If2a6829a76b5eb70328be23130dde5a1044f82a6
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Andrew Bogott 
Gerrit-Reviewer: Arturo Borrero Gonzalez 
Gerrit-Reviewer: Chasemp 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Madhuvishy 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cassandra: switch to using jmx-exporter jar from Debian package

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402070 )

Change subject: cassandra: switch to using jmx-exporter jar from Debian package
..

cassandra: switch to using jmx-exporter jar from Debian package

Bug: T181728
Change-Id: I1c1e5b868684aae5ba29f5a98d18652bdac7622f
---
M modules/cassandra/templates/cassandra-env.sh-2.2.erb
M modules/cassandra/templates/cassandra-env.sh-3.x.erb
2 files changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/70/402070/1

diff --git a/modules/cassandra/templates/cassandra-env.sh-2.2.erb 
b/modules/cassandra/templates/cassandra-env.sh-2.2.erb
index 12267cc..56d29d8 100644
--- a/modules/cassandra/templates/cassandra-env.sh-2.2.erb
+++ b/modules/cassandra/templates/cassandra-env.sh-2.2.erb
@@ -344,5 +344,5 @@
 <% end -%>
 
 <% if @jmx_exporter_enabled -%>
-JVM_OPTS="$JVM_OPTS 
-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=<%=
 @listen_address %>:7800:<%= @config_directory %>/prometheus_jmx_exporter.yaml"
+JVM_OPTS="$JVM_OPTS 
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=<%= 
@listen_address %>:7800:<%= @config_directory %>/prometheus_jmx_exporter.yaml"
 <% end -%>
diff --git a/modules/cassandra/templates/cassandra-env.sh-3.x.erb 
b/modules/cassandra/templates/cassandra-env.sh-3.x.erb
index d05bd5b..7ec619b 100644
--- a/modules/cassandra/templates/cassandra-env.sh-3.x.erb
+++ b/modules/cassandra/templates/cassandra-env.sh-3.x.erb
@@ -302,5 +302,5 @@
 <% end -%>
 
 <% if @jmx_exporter_enabled -%>
-JVM_OPTS="$JVM_OPTS 
-javaagent:/srv/deployment/prometheus/jmx_exporter/lib/jmx_prometheus_javaagent-0.8-20170117.190412-1.jar=<%=
 @listen_address %>:7800:<%= @config_directory %>/prometheus_jmx_exporter.yaml"
+JVM_OPTS="$JVM_OPTS 
-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=<%= 
@listen_address %>:7800:<%= @config_directory %>/prometheus_jmx_exporter.yaml"
 <% end -%>

-- 
To view, visit https://gerrit.wikimedia.org/r/402070
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1c1e5b868684aae5ba29f5a98d18652bdac7622f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cassandra: use prometheus-jmx-exporter Debian package

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402069 )

Change subject: cassandra: use prometheus-jmx-exporter Debian package
..

cassandra: use prometheus-jmx-exporter Debian package

Bug: T181728
Change-Id: I9605139f0f9e2e2d5331f799441f82ea92376bd6
---
M hieradata/labs/deployment-prep/common.yaml
M hieradata/role/common/deployment_server.yaml
M modules/cassandra/manifests/init.pp
M modules/cassandra/manifests/instance.pp
4 files changed, 2 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/69/402069/1

diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 7700115..17e2dc8 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -345,10 +345,6 @@
   cassandra/logstash-logback-encoder:
 repository: operations/software/logstash-logback-encoder
 
-  # Prometheus JMX exporter
-  prometheus/jmx_exporter:
-repository: operations/software/prometheus_jmx_exporter
-
   # Wikimania scholarships app
   scholarships/scholarships:
 repository: wikimedia/wikimania-scholarships
diff --git a/hieradata/role/common/deployment_server.yaml 
b/hieradata/role/common/deployment_server.yaml
index 1921c65..d6017ec 100644
--- a/hieradata/role/common/deployment_server.yaml
+++ b/hieradata/role/common/deployment_server.yaml
@@ -165,9 +165,6 @@
   # Cassandra metrics collection
   cassandra/metrics-collector:
 repository: operations/software/cassandra-metrics-collector
-  # Prometheus JMX exporter
-  prometheus/jmx_exporter:
-repository: operations/software/prometheus_jmx_exporter
   # Librenms software
   librenms/librenms:
 repository: operations/software/librenms
diff --git a/modules/cassandra/manifests/init.pp 
b/modules/cassandra/manifests/init.pp
index cdab081..784762d 100644
--- a/modules/cassandra/manifests/init.pp
+++ b/modules/cassandra/manifests/init.pp
@@ -288,9 +288,4 @@
 mode=> '0755',
 require => Package['cassandra'],
 }
-
-scap::target { 'prometheus/jmx_exporter':
-deploy_user => 'deploy-service',
-manage_user => true,
-}
 }
diff --git a/modules/cassandra/manifests/instance.pp 
b/modules/cassandra/manifests/instance.pp
index 0a31322..b707aba 100644
--- a/modules/cassandra/manifests/instance.pp
+++ b/modules/cassandra/manifests/instance.pp
@@ -425,6 +425,8 @@
 }
 
 if ($jmx_exporter_enabled) {
+require_package('prometheus-jmx-exporter')
+
 $prometheus_target = "${::hostname}-${instance_name}"
 prometheus::jmx_exporter_instance { $prometheus_target:
 hostname => $prometheus_target,

-- 
To view, visit https://gerrit.wikimedia.org/r/402069
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9605139f0f9e2e2d5331f799441f82ea92376bd6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: extend eqiad SMART checking deployment

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402056 )

Change subject: hieradata: extend eqiad SMART checking deployment
..

hieradata: extend eqiad SMART checking deployment

Bug: T86552
Change-Id: If2a6829a76b5eb70328be23130dde5a1044f82a6
---
M hieradata/regex.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/56/402056/1

diff --git a/hieradata/regex.yaml b/hieradata/regex.yaml
index d04a992..7566f6f 100644
--- a/hieradata/regex.yaml
+++ b/hieradata/regex.yaml
@@ -254,7 +254,7 @@
   profile::base::check_smart: true
 
 smart_health_eqiad:
-  __regex: !ruby/regexp 
/^(analytics1028|aqs1004|conf1001|cp1045|db1039|db1001|db1087|db1011|druid1001|elastic1017|es1015|etcd1001|ganeti1001|graphite1001|kafka1001|kafka-jumbo1001|labmon1001|logstash1008|mw1180|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|pc1004|sca1003|scb1001|thumbor1001).eqiad\.wmnet$/
+  __regex: !ruby/regexp 
/^(analytics10[234][0-9]|aqs1004|conf1001|cp1045|db1039|db1001|db1087|db1011|druid1001|elastic1017|es1015|etcd1001|ganeti1001|graphite1.*|kafka1001|kafka-jumbo1001|kube.*|labmon1001|labcontrol1001|labvirt1001|labsdb1009|labstore1006|lvs1007logstash1008|mw11.*|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|phab1001|poolcounter1001|mwlog1001|netmon1.*|puppetmaster1001|pc1004|sca1003|scb1001|thumbor1.*|tin|tungsten|wdqs1003|wtp1025|xenon).eqiad\.wmnet$/
   profile::base::check_smart: true
 
 labtestpuppetmaster:

-- 
To view, visit https://gerrit.wikimedia.org/r/402056
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If2a6829a76b5eb70328be23130dde5a1044f82a6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add backend varnish mtail job

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402055 )

Change subject: prometheus: add backend varnish mtail job
..

prometheus: add backend varnish mtail job

Bug: T177199
Change-Id: I8d1271ebf7db7da4ec32483b7b440067215f249a
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 27 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/55/402055/1

diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 8680d6f..0976ddc 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -219,7 +219,7 @@
   },
 ]
 
-# Pull varnish-related metrics generated via mtail
+# Pull varnish-related metrics generated via mtail (frontend)
 prometheus::class_config{ "varnish-canary_mtail_${::site}":
 dest   => "${targets_path}/varnish-canary_mtail_${::site}.yaml",
 site   => $::site,
@@ -245,6 +245,32 @@
 port   => '3903',
 }
 
+# And varnish backend instance stats generated by mtail
+prometheus::class_config{ "varnish-canary_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-canary_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::canary',
+port   => '3904',
+}
+prometheus::class_config{ "varnish-misc_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-misc_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::misc',
+port   => '3904',
+}
+prometheus::class_config{ "varnish-upload_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-upload_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::upload',
+port   => '3904',
+}
+prometheus::class_config{ "varnish-text_backendmtail_${::site}":
+dest   => 
"${targets_path}/varnish-text_backendmtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::text',
+port   => '3904',
+}
+
 # Job definition for memcache_exporter
 $memcached_jobs = [
   {

-- 
To view, visit https://gerrit.wikimedia.org/r/402055
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I8d1271ebf7db7da4ec32483b7b440067215f249a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: smart: ignore drbd disks

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402024 )

Change subject: smart: ignore drbd disks
..


smart: ignore drbd disks

Bug: T86552
Change-Id: Iad7cab4d2274c2060e63437c8e0417bb97e9b5b9
---
M modules/smart/files/smart-data-dump
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/smart/files/smart-data-dump 
b/modules/smart/files/smart-data-dump
index 372e444..41ba58f 100644
--- a/modules/smart/files/smart-data-dump
+++ b/modules/smart/files/smart-data-dump
@@ -128,6 +128,8 @@
 name, disk_type = line.split(' ', 1)
 if disk_type != 'disk':
 continue
+if name.startswith('drbd'):
+continue
 yield PD(driver='noraid', smart_args=['-d', 'auto', '/dev/%s' % name],
  disk_id=name)
 

-- 
To view, visit https://gerrit.wikimedia.org/r/402024
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iad7cab4d2274c2060e63437c8e0417bb97e9b5b9
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: smart: bump timeout to 60s

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402023 )

Change subject: smart: bump timeout to 60s
..


smart: bump timeout to 60s

In cases with many (and some broken) disks it can take longer that 30s to 
finish running commands.

Bug: T86552
Change-Id: I35d24ae89b87a9ca47a1a0399dab837ef6e7c987
---
M modules/smart/files/smart-data-dump
1 file changed, 4 insertions(+), 4 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/smart/files/smart-data-dump 
b/modules/smart/files/smart-data-dump
index 683b2bc..372e444 100644
--- a/modules/smart/files/smart-data-dump
+++ b/modules/smart/files/smart-data-dump
@@ -82,7 +82,7 @@
 def megaraid_list_pd():
 """List physical disks attached to megaraid controller. Generator to yield 
PD objects."""
 
-raw_output = subprocess.check_output(['/usr/bin/timeout', '30', 
'/usr/sbin/smartctl',
+raw_output = subprocess.check_output(['/usr/bin/timeout', '60', 
'/usr/sbin/smartctl',
   '--scan-open'])
 for line in raw_output.splitlines():
 line = str(line, 'utf8')
@@ -98,7 +98,7 @@
 def hpsa_list_pd():
 """List physical disks attached to hpsa controller. Generator to yield PD 
objects."""
 
-raw_output = subprocess.check_output(['/usr/bin/timeout', '30', 
'/usr/sbin/hpssacli',
+raw_output = subprocess.check_output(['/usr/bin/timeout', '60', 
'/usr/sbin/hpssacli',
   'controller', 'all', 'show', 
'config'])
 in_controller = False
 for line in raw_output.splitlines():
@@ -121,7 +121,7 @@
 """List all physical disks. Generator to yield PD objects."""
 
 # starting with stretch, lsblk has --json but not on trusty/jessie
-raw_output = subprocess.check_output(['/usr/bin/timeout', '30', 
'/bin/lsblk', '--noheadings',
+raw_output = subprocess.check_output(['/usr/bin/timeout', '60', 
'/bin/lsblk', '--noheadings',
   '--output', 'NAME,TYPE', '--raw'])
 for line in raw_output.splitlines():
 line = str(line, 'utf8')
@@ -132,7 +132,7 @@
  disk_id=name)
 
 
-def _run_smartctl(args, timeout=30):
+def _run_smartctl(args, timeout=60):
 try:
 cmd_args = ['/usr/bin/timeout', str(timeout), '/usr/sbin/smartctl']
 cmd_args.extend(args)

-- 
To view, visit https://gerrit.wikimedia.org/r/402023
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I35d24ae89b87a9ca47a1a0399dab837ef6e7c987
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: smart: bump timeout to 60s

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402023 )

Change subject: smart: bump timeout to 60s
..

smart: bump timeout to 60s

In cases with many (and some broken) disks it can take longer that 30s to 
finish running commands.

Bug: T86552
Change-Id: I35d24ae89b87a9ca47a1a0399dab837ef6e7c987
---
M modules/smart/files/smart-data-dump
1 file changed, 4 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/23/402023/1

diff --git a/modules/smart/files/smart-data-dump 
b/modules/smart/files/smart-data-dump
index 683b2bc..372e444 100644
--- a/modules/smart/files/smart-data-dump
+++ b/modules/smart/files/smart-data-dump
@@ -82,7 +82,7 @@
 def megaraid_list_pd():
 """List physical disks attached to megaraid controller. Generator to yield 
PD objects."""
 
-raw_output = subprocess.check_output(['/usr/bin/timeout', '30', 
'/usr/sbin/smartctl',
+raw_output = subprocess.check_output(['/usr/bin/timeout', '60', 
'/usr/sbin/smartctl',
   '--scan-open'])
 for line in raw_output.splitlines():
 line = str(line, 'utf8')
@@ -98,7 +98,7 @@
 def hpsa_list_pd():
 """List physical disks attached to hpsa controller. Generator to yield PD 
objects."""
 
-raw_output = subprocess.check_output(['/usr/bin/timeout', '30', 
'/usr/sbin/hpssacli',
+raw_output = subprocess.check_output(['/usr/bin/timeout', '60', 
'/usr/sbin/hpssacli',
   'controller', 'all', 'show', 
'config'])
 in_controller = False
 for line in raw_output.splitlines():
@@ -121,7 +121,7 @@
 """List all physical disks. Generator to yield PD objects."""
 
 # starting with stretch, lsblk has --json but not on trusty/jessie
-raw_output = subprocess.check_output(['/usr/bin/timeout', '30', 
'/bin/lsblk', '--noheadings',
+raw_output = subprocess.check_output(['/usr/bin/timeout', '60', 
'/bin/lsblk', '--noheadings',
   '--output', 'NAME,TYPE', '--raw'])
 for line in raw_output.splitlines():
 line = str(line, 'utf8')
@@ -132,7 +132,7 @@
  disk_id=name)
 
 
-def _run_smartctl(args, timeout=30):
+def _run_smartctl(args, timeout=60):
 try:
 cmd_args = ['/usr/bin/timeout', str(timeout), '/usr/sbin/smartctl']
 cmd_args.extend(args)

-- 
To view, visit https://gerrit.wikimedia.org/r/402023
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I35d24ae89b87a9ca47a1a0399dab837ef6e7c987
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: smart: ignore drbd disks

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402024 )

Change subject: smart: ignore drbd disks
..

smart: ignore drbd disks

Bug: T86552
Change-Id: Iad7cab4d2274c2060e63437c8e0417bb97e9b5b9
---
M modules/smart/files/smart-data-dump
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/24/402024/1

diff --git a/modules/smart/files/smart-data-dump 
b/modules/smart/files/smart-data-dump
index 372e444..41ba58f 100644
--- a/modules/smart/files/smart-data-dump
+++ b/modules/smart/files/smart-data-dump
@@ -128,6 +128,8 @@
 name, disk_type = line.split(' ', 1)
 if disk_type != 'disk':
 continue
+if name.startswith('drbd'):
+continue
 yield PD(driver='noraid', smart_args=['-d', 'auto', '/dev/%s' % name],
  disk_id=name)
 

-- 
To view, visit https://gerrit.wikimedia.org/r/402024
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iad7cab4d2274c2060e63437c8e0417bb97e9b5b9
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: graphite: cleanup stale ORES metrics

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401917 )

Change subject: graphite: cleanup stale ORES metrics
..

graphite: cleanup stale ORES metrics

Also group cleanups into a separate class.

Bug: T169969
Change-Id: I01b37944e614a3e2e5a310094282381b67ade6dc
---
A modules/role/manifests/graphite/cleanup.pp
M modules/role/manifests/graphite/production.pp
2 files changed, 40 insertions(+), 32 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/17/401917/1

diff --git a/modules/role/manifests/graphite/cleanup.pp 
b/modules/role/manifests/graphite/cleanup.pp
new file mode 100644
index 000..efa6be0
--- /dev/null
+++ b/modules/role/manifests/graphite/cleanup.pp
@@ -0,0 +1,39 @@
+class role::graphite::cleanup {
+# Cleanup stale labs instances data - T143405
+graphite::whisper_cleanup { 'graphite-labs-instances':
+directory => "${storage_dir}/whisper/instances",
+}
+
+# Cleanup eventstreams rdkafka stale data - T160644
+graphite::whisper_cleanup { 'graphite-eventstreams':
+directory => "${storage_dir}/whisper/eventstreams/rdkafka",
+keep_days => 5,
+}
+
+# Cleanup zuul data
+graphite::whisper_cleanup { 'graphite-zuul':
+directory => "${storage_dir}/whisper/zuul",
+}
+# Zuul also generates metrics related to Gerrit
+graphite::whisper_cleanup { 'graphite-zuul-gerrit':
+directory => "${storage_dir}/whisper/gerrit",
+}
+
+# Nodepool, which has several metrics for each of the Jenkins jobs
+graphite::whisper_cleanup { 'graphite-nodepool':
+directory => "${storage_dir}/whisper/nodepool",
+keep_days => 15,
+}
+
+# Cassandra metrics - T179057
+graphite::whisper_cleanup { 'graphite-cassandra':
+directory => "${storage_dir}/whisper/cassandra",
+keep_days => 182,
+}
+
+# ORES metrics - T169969
+graphite::whisper_cleanup { 'graphite-ores':
+directory => "${storage_dir}/whisper/ores",
+keep_days => 30,
+}
+}
diff --git a/modules/role/manifests/graphite/production.pp 
b/modules/role/manifests/graphite/production.pp
index b0b33b0..e96906f 100644
--- a/modules/role/manifests/graphite/production.pp
+++ b/modules/role/manifests/graphite/production.pp
@@ -9,6 +9,7 @@
 
 include ::standard
 include ::base::firewall
+include ::role::graphite::cleanup
 
 class { 'role::graphite::base':
 storage_dir  => $storage_dir,
@@ -30,38 +31,6 @@
 ['^MediaWiki\.wanobjectcache\.[a-zA-Z0-9]{32}', 'blackhole'],
 ]
 }
-}
-
-# Cleanup stale labs instances data - T143405
-graphite::whisper_cleanup { 'graphite-labs-instances':
-directory => "${storage_dir}/whisper/instances",
-}
-
-# Cleanup eventstreams rdkafka stale data - T160644
-graphite::whisper_cleanup { 'graphite-eventstreams':
-directory => "${storage_dir}/whisper/eventstreams/rdkafka",
-keep_days => 5,
-}
-
-# Cleanup zuul data
-graphite::whisper_cleanup { 'graphite-zuul':
-directory => "${storage_dir}/whisper/zuul",
-}
-# Zuul also generates metrics related to Gerrit
-graphite::whisper_cleanup { 'graphite-zuul-gerrit':
-directory => "${storage_dir}/whisper/gerrit",
-}
-
-# Nodepool, which has several metrics for each of the Jenkins jobs
-graphite::whisper_cleanup { 'graphite-nodepool':
-directory => "${storage_dir}/whisper/nodepool",
-keep_days => 15,
-}
-
-# Cassandra metrics - T179057
-graphite::whisper_cleanup { 'graphite-cassandra':
-directory => "${storage_dir}/whisper/cassandra",
-keep_days => 182,
 }
 
 $graphite_hosts = [

-- 
To view, visit https://gerrit.wikimedia.org/r/401917
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I01b37944e614a3e2e5a310094282381b67ade6dc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-jmx-exporter[master]: debian: force maven repo directory

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401758 )

Change subject: debian: force maven repo directory
..

debian: force maven repo directory

When running in cowbuilder $HOME is set to /nonexistent, and maven will fail to
create $HOME/.m2

Change-Id: Ia730eff34793f591b4d5985052b8d572dc09e1eb
---
M debian/rules
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-jmx-exporter 
refs/changes/58/401758/1

diff --git a/debian/rules b/debian/rules
index 7e8aeb7..bcd19c2 100755
--- a/debian/rules
+++ b/debian/rules
@@ -11,7 +11,7 @@
 
 
 override_dh_auto_build:
-   mvn package -DskipTests
+   mvn package -DskipTests -Dmaven.repo.local=/tmp/.m2
dh_auto_build
 
 override_dh_install:

-- 
To view, visit https://gerrit.wikimedia.org/r/401758
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia730eff34793f591b4d5985052b8d572dc09e1eb
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-jmx-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-jmx-exporter[master]: debian: tweak gbp config

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401759 )

Change subject: debian: tweak gbp config
..

debian: tweak gbp config

Remove default and not needed flags

Change-Id: I65a59d9d0dfde5b773a678c888f3d870aaa452c4
---
M debian/gbp.conf
1 file changed, 0 insertions(+), 4 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-jmx-exporter 
refs/changes/59/401759/1

diff --git a/debian/gbp.conf b/debian/gbp.conf
index 2795046..9cf0067 100644
--- a/debian/gbp.conf
+++ b/debian/gbp.conf
@@ -1,7 +1,3 @@
 [buildpackage]
-upstream-tree=tag
-upstream-branch=master
 debian-branch=debian
 upstream-tag=parent-%(version)s
-debian-tag=debian/%(version)s
-builder=GIT_PBUILDER_AUTOCONF=no git-pbuilder -sa

-- 
To view, visit https://gerrit.wikimedia.org/r/401759
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I65a59d9d0dfde5b773a678c888f3d870aaa452c4
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-jmx-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: smart: pin smartmontools to backports on Debian

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401707 )

Change subject: smart: pin smartmontools to backports on Debian
..


smart: pin smartmontools to backports on Debian

We are not enabling backports on (deprecated) Trusty.

Bug: T86552
Change-Id: I4510bd0d538502d158833560bb2e375cba23a9b2
---
M modules/smart/manifests/init.pp
1 file changed, 11 insertions(+), 4 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/smart/manifests/init.pp b/modules/smart/manifests/init.pp
index 90d9dab..0e70724 100644
--- a/modules/smart/manifests/init.pp
+++ b/modules/smart/manifests/init.pp
@@ -9,11 +9,18 @@
 fail('smart module is not supported on virtual hosts')
 }
 
-# Prefer smartmontools version from backports (if any) because of newer
-# smart drivedb.
+# Prefer smartmontools version from backports (on Debian, if any) because 
of newer smart
+# drivedb.
+if os_version('debian >= jessie') {
+apt::pin { 'smartmontools_backports':
+pin  => "release a=${::lsbdistcodename}-backports",
+priority => '1001',
+before   => Package['smartmontools'],
+}
+}
+
 package { 'smartmontools':
-ensure  => $ensure,
-install_options => ['-t', "${::lsbdistcodename}-backports"],
+ensure => $ensure,
 }
 
 # Make sure we send smart alerts from smartd via syslog and not email.

-- 
To view, visit https://gerrit.wikimedia.org/r/401707
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4510bd0d538502d158833560bb2e375cba23a9b2
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Jcrespo 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...python-thumbor-wikimedia[master]: Upgrade to 1.8

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401725 )

Change subject: Upgrade to 1.8
..


Upgrade to 1.8

Bug: T183907
Change-Id: Ie6123058b098f17f61fcb69505471ab70693cc5a
---
M debian/changelog
M setup.py
A tests/integration/originals/Cape_Town_under_the_clouds.webm
A 
tests/integration/originals/Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm
M tests/integration/test_video.py
A tests/integration/thumbnails/640px--Cape_Town_under_the_clouds.webm.jpg
A 
tests/integration/thumbnails/640px--Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm.jpg
M tox.ini
M wikimedia_thumbor/engine/stl/stl.py
M wikimedia_thumbor/loader/video/__init__.py
M wikimedia_thumbor/logging/filter/context/context.py
M wikimedia_thumbor/logging/filter/http404/http404.py
12 files changed, 55 insertions(+), 8 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/debian/changelog b/debian/changelog
index 0a00260..59b5c76 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+python-thumbor-wikimedia (1.8-1) jessie-wikimedia; urgency=low
+
+  * New upstream release
+
+ -- Gilles Dubuc   Wed, 3 Jan 2018 12:46:00 +
+
 python-thumbor-wikimedia (1.7-1) jessie-wikimedia; urgency=low
 
   * New upstream release
diff --git a/setup.py b/setup.py
index 57716a5..36d 100644
--- a/setup.py
+++ b/setup.py
@@ -12,7 +12,7 @@
 
 setup(
 name='wikimedia_thumbor',
-version='1.7',
+version='1.8',
 url='https://phabricator.wikimedia.org/diffusion/THMBREXT/',
 license='MIT',
 author='Gilles Dubuc, Wikimedia Foundation',
diff --git a/tests/integration/originals/Cape_Town_under_the_clouds.webm 
b/tests/integration/originals/Cape_Town_under_the_clouds.webm
new file mode 100644
index 000..82e16fc
--- /dev/null
+++ b/tests/integration/originals/Cape_Town_under_the_clouds.webm
Binary files differ
diff --git 
a/tests/integration/originals/Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm
 
b/tests/integration/originals/Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm
new file mode 100644
index 000..71b9428
--- /dev/null
+++ 
b/tests/integration/originals/Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm
Binary files differ
diff --git a/tests/integration/test_video.py b/tests/integration/test_video.py
index f5d5d91..8da8d73 100644
--- a/tests/integration/test_video.py
+++ b/tests/integration/test_video.py
@@ -38,6 +38,32 @@
 1.0
 )
 
+path = os.path.join(
+os.path.dirname(__file__),
+'originals',
+'Cape_Town_under_the_clouds.webm'
+)
+
+self.run_and_check_ssim_and_size(
+'thumbor/unsafe/640x/' + path,
+'640px--Cape_Town_under_the_clouds.webm.jpg',
+0.98,
+1.0
+)
+
+path = os.path.join(
+os.path.dirname(__file__),
+'originals',
+
'Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm'
+)
+
+self.run_and_check_ssim_and_size(
+'thumbor/unsafe/640x/' + path,
+
'640px--Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm.jpg',
+0.99,
+1.0
+)
+
 def test_webm_with_seek(self):
 path = os.path.join(
 os.path.dirname(__file__),
diff --git 
a/tests/integration/thumbnails/640px--Cape_Town_under_the_clouds.webm.jpg 
b/tests/integration/thumbnails/640px--Cape_Town_under_the_clouds.webm.jpg
new file mode 100644
index 000..0cfa53a
--- /dev/null
+++ b/tests/integration/thumbnails/640px--Cape_Town_under_the_clouds.webm.jpg
Binary files differ
diff --git 
a/tests/integration/thumbnails/640px--Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm.jpg
 
b/tests/integration/thumbnails/640px--Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm.jpg
new file mode 100644
index 000..4c4fecd
--- /dev/null
+++ 
b/tests/integration/thumbnails/640px--Debris_flow_-_22_juillet_2013_-_Crue_torrentielle_a_Saint_Julien_Montdenis.webm.jpg
Binary files differ
diff --git a/tox.ini b/tox.ini
index 157253d..2fca023 100644
--- a/tox.ini
+++ b/tox.ini
@@ -6,6 +6,7 @@
 [testenv]
 setenv = VIRTUAL_ENV={envdir}
 deps = -r{toxinidir}/requirements.txt
+commands = python -c 'print("done!")'
 
 [testenv:flake8]
 commands = flake8 {posargs}
diff --git a/wikimedia_thumbor/engine/stl/stl.py 
b/wikimedia_thumbor/engine/stl/stl.py
index 9dd3d47..a7cf8ec 100644
--- a/wikimedia_thumbor/engine/stl/stl.py
+++ b/wikimedia_thumbor/engine/stl/stl.py
@@ -18,6 +18,7 @@
 
 from wikimedia_thumbor.engine import BaseWikimediaEngine
 from 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: smart: special case for backports on Ubuntu

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401707 )

Change subject: smart: special case for backports on Ubuntu
..

smart: special case for backports on Ubuntu

We don't ship trusty-backports, thus special case the install on Ubuntu.

Bug: T86552
Change-Id: I4510bd0d538502d158833560bb2e375cba23a9b2
---
M modules/smart/manifests/init.pp
1 file changed, 10 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/07/401707/1

diff --git a/modules/smart/manifests/init.pp b/modules/smart/manifests/init.pp
index 90d9dab..7129480 100644
--- a/modules/smart/manifests/init.pp
+++ b/modules/smart/manifests/init.pp
@@ -9,11 +9,17 @@
 fail('smart module is not supported on virtual hosts')
 }
 
-# Prefer smartmontools version from backports (if any) because of newer
+# Prefer smartmontools version from backports (on Debian, if any) because 
of newer
 # smart drivedb.
-package { 'smartmontools':
-ensure  => $ensure,
-install_options => ['-t', "${::lsbdistcodename}-backports"],
+if os_version('ubuntu >= trusty') {
+package { 'smartmontools':
+ensure => $ensure,
+}
+} else {
+package { 'smartmontools':
+ensure  => $ensure,
+install_options => ['-t', "${::lsbdistcodename}-backports"],
+}
 }
 
 # Make sure we send smart alerts from smartd via syslog and not email.

-- 
To view, visit https://gerrit.wikimedia.org/r/401707
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4510bd0d538502d158833560bb2e375cba23a9b2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: partial eqiad SMART metrics rollout

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401491 )

Change subject: hieradata: partial eqiad SMART metrics rollout
..


hieradata: partial eqiad SMART metrics rollout

Bug: T86552
Change-Id: I436fff25a02b55a86b3fe468f2dd9154b6da89c2
---
M hieradata/regex.yaml
1 file changed, 4 insertions(+), 0 deletions(-)

Approvals:
  Marostegui: Looks good to me, but someone else must approve
  Ottomata: Looks good to me, but someone else must approve
  Elukey: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Jcrespo: Looks good to me, but someone else must approve
  Filippo Giunchedi: Looks good to me, approved
  Gehel: Looks good to me, but someone else must approve



diff --git a/hieradata/regex.yaml b/hieradata/regex.yaml
index c715acb..d04a992 100644
--- a/hieradata/regex.yaml
+++ b/hieradata/regex.yaml
@@ -253,6 +253,10 @@
   __regex: !ruby/regexp /^.*\.ulsfo\.wmnet$/
   profile::base::check_smart: true
 
+smart_health_eqiad:
+  __regex: !ruby/regexp 
/^(analytics1028|aqs1004|conf1001|cp1045|db1039|db1001|db1087|db1011|druid1001|elastic1017|es1015|etcd1001|ganeti1001|graphite1001|kafka1001|kafka-jumbo1001|labmon1001|logstash1008|mw1180|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|pc1004|sca1003|scb1001|thumbor1001).eqiad\.wmnet$/
+  profile::base::check_smart: true
+
 labtestpuppetmaster:
   __regex: !ruby/regexp /^labtestpuppetmaster.*\.wikimedia\.org$/
   puppetmaster: "puppetmaster2001.codfw.wmnet"

-- 
To view, visit https://gerrit.wikimedia.org/r/401491
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I436fff25a02b55a86b3fe468f2dd9154b6da89c2
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Chasemp 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Jcrespo 
Gerrit-Reviewer: Marostegui 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: admin: add yubikey for ema

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401702 )

Change subject: admin: add yubikey for ema
..

admin: add yubikey for ema

Change-Id: I73a6eed15fdd239fc6cd3fe4311593549f7fc542
---
M modules/admin/data/data.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/02/401702/1

diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index 73cdcb2..41f751d 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -2061,6 +2061,7 @@
 realname: Emanuele Rocca
 ssh_keys:
   - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCfoCT4+pXcLRk3jsZsxFsAtVP38Rs4pfdxSS05+hkMcN1yxi2R1ZeNFkQkCPo6s25Gq863ru5E/vxSjy9mDS4HeaxNSJlPPWD0iEmGdyCg+wEGAZ01n1muFDyb+oiSNy5JcgOk0ZqAPNPYlISKVp+vLro5txdEK8cySy4BIbiM5ygKIjmq1xP+dM9zVdd/UbVQPPBFS2UDBzLX3AeYen4biMx2GkHBpZ/nJLqtb9JE4LLzQAtrqi4+J9jKQGepeTemVQ/n3lbheiGmTDZbRsUx7GTXmdIOPqN4k6FEXEgMIe5plxV8YeAoWqMsGpln4J4bz9HCtM1dL5mdLsa7tkq/
 ema@orion
+  - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCZ4V90QtsYJ4QuykOcVJQPJ64S5wkQdus7N4W8km9LChzvD/dgu/K2NbvRpUSVki+JGGIi8rPyUpGi+ELYd0E6ul3izSnIsMfyTSHqM/craBe8CPmoiFEHKIkXmOGvl1HHczaeziotFVkBm6D5e7MEuz2QJc83XT95trwjZW+e8Cz6f6xW+SGcWDCqOceI6n3wc0PwnWUKZVum2nU71FFZlcCg/fL5srLPis3GWU3zfTmZpvHmXnBE5fFEd6wzQOoSNPHVpehtbgIIrFh5KU5j5HhksHyRZAez+LM19VuZyYOeqUkmIiGq/3hHn+npCOR/r51Lf1Smua+GPyEQNqbn
 ema@newbikey
 uid: 4565
 email: ero...@wikimedia.org
   elukey:

-- 
To view, visit https://gerrit.wikimedia.org/r/401702
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I73a6eed15fdd239fc6cd3fe4311593549f7fc542
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add program to count varnish backend metrics

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401535 )

Change subject: mtail: add program to count varnish backend metrics
..

mtail: add program to count varnish backend metrics

The histogram buckets will likely need some tuning.

Bug: T177199
Change-Id: Ia2fa39674be38ef340a7785e8f10722bd04373bf
---
A modules/mtail/files/programs/varnish_backend.mtail
1 file changed, 56 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/35/401535/1

diff --git a/modules/mtail/files/programs/varnish_backend.mtail 
b/modules/mtail/files/programs/varnish_backend.mtail
new file mode 100644
index 000..360df7f
--- /dev/null
+++ b/modules/mtail/files/programs/varnish_backend.mtail
@@ -0,0 +1,56 @@
+# Three counters to implement Prometheus histograms
+counter varnish_backend_requests_seconds_bucket by le, status, method, backend
+counter varnish_backend_requests_seconds_sum by status, method, backend
+counter varnish_backend_requests_seconds_count by status, method, backend
+
+/http_status (?P[0-9][0-9][0-9])\thttp_method 
(?P[A-Z]+)\tbackend (?P\S+)\tttfb (?P[0-9]+)/ {
+  varnish_backend_requests_seconds_count[$status][$method][$backend]++
+  varnish_backend_requests_seconds_sum[$status][$method][$backend] += $ttfb
+
+  # Latency histogram buckets
+  $ttfb <= 0.005 {
+http_request_duration_seconds_bucket["0.005"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 0.010 {
+http_request_duration_seconds_bucket["0.010"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 0.050 {
+http_request_duration_seconds_bucket["0.050"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 0.100 {
+http_request_duration_seconds_bucket["0.100"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 0.120 {
+http_request_duration_seconds_bucket["0.100"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 0.150 {
+http_request_duration_seconds_bucket["0.150"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 0.200 {
+http_request_duration_seconds_bucket["0.200"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 0.500 {
+http_request_duration_seconds_bucket["0.500"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 1.000 {
+http_request_duration_seconds_bucket["1.000"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 5.000 {
+http_request_duration_seconds_bucket["5.000"][$status][$method][$backend]++
+  }
+
+  $ttfb <= 10.000 {
+
http_request_duration_seconds_bucket["10.000"][$status][$method][$backend]++
+  }
+
+  http_request_duration_seconds_bucket["+Inf"][$status][$method][$backend]++
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/401535
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia2fa39674be38ef340a7785e8f10722bd04373bf
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: stop sending metrics to graphite

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401502 )

Change subject: mtail: stop sending metrics to graphite
..


mtail: stop sending metrics to graphite

Prometheus is running production now, also lessen the load on Graphite when 
emitting high
cardinality metrics.

Change-Id: I7021322cce9af95cee975b81af2a49de57df8000
---
M modules/mtail/manifests/init.pp
M modules/mtail/templates/default.erb
2 files changed, 1 insertion(+), 8 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/mtail/manifests/init.pp b/modules/mtail/manifests/init.pp
index fc3fd1d..7bb8497 100644
--- a/modules/mtail/manifests/init.pp
+++ b/modules/mtail/manifests/init.pp
@@ -10,23 +10,17 @@
 # [*port*]
 #   TCP port to listen to for Prometheus-style metrics
 #
-# [*graphite_hostport*]
-#   Also send metrics via graphite line-oriented protocol to this host:port.
-#
 # [*ensure*]
 #   Whether mtail should be running or stopped.
 
 class mtail (
   $logs = ['/var/log/syslog'],
   $port = '3903',
-  $graphite_hostport = 'graphite-in.eqiad.wmnet:2003',
-  $graphite_prefix = "mtail.${::hostname}.",
   $ensure = 'running',
   $group = 'root',
 ) {
 validate_array($logs)
 validate_re($port, '^[0-9]+$')
-validate_string($graphite_hostport)
 validate_re($ensure, '^(running|stopped)$')
 
 require_package('mtail')
diff --git a/modules/mtail/templates/default.erb 
b/modules/mtail/templates/default.erb
index e26726f..8c0ca64 100644
--- a/modules/mtail/templates/default.erb
+++ b/modules/mtail/templates/default.erb
@@ -1,6 +1,5 @@
 # Arguments used by SysV init script, systemd service file uses only EXTRA_ARGS
-#GRAPHITE_HOSTPORT=<%= @graphite_hostport %>
 #PORT=<%= @port %>
 #LOGS=<%= @logs.join(',') %>
 
-EXTRA_ARGS="-graphite_prefix <%= @graphite_prefix %> -graphite_host_port <%= 
@graphite_hostport %> -logs <%= @logs.join(',') %> -port <%= @port %> "
+EXTRA_ARGS="-logs <%= @logs.join(',') %> -port <%= @port %>"

-- 
To view, visit https://gerrit.wikimedia.org/r/401502
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7021322cce9af95cee975b81af2a49de57df8000
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add mtail to varnish-text job

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401516 )

Change subject: prometheus: add mtail to varnish-text job
..


prometheus: add mtail to varnish-text job

Bug: T177199
Change-Id: I1c1e09f29d5390f0670b481ea05d3a9ff167b625
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 6 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 1373e1e..ab4e3e3 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -238,6 +238,12 @@
 class_name => 'role::cache::upload',
 port   => '3903',
 }
+prometheus::class_config{ "varnish-text_mtail_${::site}":
+dest   => "${targets_path}/varnish-text_mtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::text',
+port   => '3903',
+}
 
 # Job definition for memcache_exporter
 $memcached_jobs = [

-- 
To view, visit https://gerrit.wikimedia.org/r/401516
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1c1e09f29d5390f0670b481ea05d3a9ff167b625
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add mtail to varnish-text job

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401516 )

Change subject: prometheus: add mtail to varnish-text job
..

prometheus: add mtail to varnish-text job

Bug: T177199
Change-Id: I1c1e09f29d5390f0670b481ea05d3a9ff167b625
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 6 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/16/401516/1

diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 1373e1e..ab4e3e3 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -238,6 +238,12 @@
 class_name => 'role::cache::upload',
 port   => '3903',
 }
+prometheus::class_config{ "varnish-text_mtail_${::site}":
+dest   => "${targets_path}/varnish-text_mtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::text',
+port   => '3903',
+}
 
 # Job definition for memcache_exporter
 $memcached_jobs = [

-- 
To view, visit https://gerrit.wikimedia.org/r/401516
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1c1e09f29d5390f0670b481ea05d3a9ff167b625
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: secrets: create digicert 2017 empty certs

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401504 )

Change subject: secrets: create digicert 2017 empty certs
..


secrets: create digicert 2017 empty certs

Change-Id: I891af50b1a5e9c1cf05ab9bd86fc20b5af408bc9
---
A modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key
A modules/secret/secrets/ssl/digicert-2017-rsa-unified.key
2 files changed, 0 insertions(+), 0 deletions(-)

Approvals:
  Filippo Giunchedi: Verified; Looks good to me, approved



diff --git a/modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key 
b/modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key
new file mode 100644
index 000..e69de29
--- /dev/null
+++ b/modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key
diff --git a/modules/secret/secrets/ssl/digicert-2017-rsa-unified.key 
b/modules/secret/secrets/ssl/digicert-2017-rsa-unified.key
new file mode 100644
index 000..e69de29
--- /dev/null
+++ b/modules/secret/secrets/ssl/digicert-2017-rsa-unified.key

-- 
To view, visit https://gerrit.wikimedia.org/r/401504
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I891af50b1a5e9c1cf05ab9bd86fc20b5af408bc9
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] labs/private[master]: secrets: create digicert 2017 empty certs

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401504 )

Change subject: secrets: create digicert 2017 empty certs
..

secrets: create digicert 2017 empty certs

Change-Id: I891af50b1a5e9c1cf05ab9bd86fc20b5af408bc9
---
A modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key
A modules/secret/secrets/ssl/digicert-2017-rsa-unified.key
2 files changed, 0 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/labs/private 
refs/changes/04/401504/1

diff --git a/modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key 
b/modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key
new file mode 100644
index 000..e69de29
--- /dev/null
+++ b/modules/secret/secrets/ssl/digicert-2017-ecdsa-unified.key
diff --git a/modules/secret/secrets/ssl/digicert-2017-rsa-unified.key 
b/modules/secret/secrets/ssl/digicert-2017-rsa-unified.key
new file mode 100644
index 000..e69de29
--- /dev/null
+++ b/modules/secret/secrets/ssl/digicert-2017-rsa-unified.key

-- 
To view, visit https://gerrit.wikimedia.org/r/401504
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I891af50b1a5e9c1cf05ab9bd86fc20b5af408bc9
Gerrit-PatchSet: 1
Gerrit-Project: labs/private
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: stop sending metrics to graphite

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401502 )

Change subject: mtail: stop sending metrics to graphite
..

mtail: stop sending metrics to graphite

Prometheus is running production now, also lessen the load on Graphite when 
emitting high
cardinality metrics.

Change-Id: I7021322cce9af95cee975b81af2a49de57df8000
---
M modules/mtail/manifests/init.pp
M modules/mtail/templates/default.erb
2 files changed, 1 insertion(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/02/401502/1

diff --git a/modules/mtail/manifests/init.pp b/modules/mtail/manifests/init.pp
index fc3fd1d..7bb8497 100644
--- a/modules/mtail/manifests/init.pp
+++ b/modules/mtail/manifests/init.pp
@@ -10,23 +10,17 @@
 # [*port*]
 #   TCP port to listen to for Prometheus-style metrics
 #
-# [*graphite_hostport*]
-#   Also send metrics via graphite line-oriented protocol to this host:port.
-#
 # [*ensure*]
 #   Whether mtail should be running or stopped.
 
 class mtail (
   $logs = ['/var/log/syslog'],
   $port = '3903',
-  $graphite_hostport = 'graphite-in.eqiad.wmnet:2003',
-  $graphite_prefix = "mtail.${::hostname}.",
   $ensure = 'running',
   $group = 'root',
 ) {
 validate_array($logs)
 validate_re($port, '^[0-9]+$')
-validate_string($graphite_hostport)
 validate_re($ensure, '^(running|stopped)$')
 
 require_package('mtail')
diff --git a/modules/mtail/templates/default.erb 
b/modules/mtail/templates/default.erb
index e26726f..8c0ca64 100644
--- a/modules/mtail/templates/default.erb
+++ b/modules/mtail/templates/default.erb
@@ -1,6 +1,5 @@
 # Arguments used by SysV init script, systemd service file uses only EXTRA_ARGS
-#GRAPHITE_HOSTPORT=<%= @graphite_hostport %>
 #PORT=<%= @port %>
 #LOGS=<%= @logs.join(',') %>
 
-EXTRA_ARGS="-graphite_prefix <%= @graphite_prefix %> -graphite_host_port <%= 
@graphite_hostport %> -logs <%= @logs.join(',') %> -port <%= @port %> "
+EXTRA_ARGS="-logs <%= @logs.join(',') %> -port <%= @port %>"

-- 
To view, visit https://gerrit.wikimedia.org/r/401502
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7021322cce9af95cee975b81af2a49de57df8000
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: partial eqiad SMART metrics rollout

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401491 )

Change subject: hieradata: partial eqiad SMART metrics rollout
..

hieradata: partial eqiad SMART metrics rollout

Bug: T86552
Change-Id: I436fff25a02b55a86b3fe468f2dd9154b6da89c2
---
M hieradata/regex.yaml
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/91/401491/1

diff --git a/hieradata/regex.yaml b/hieradata/regex.yaml
index c715acb..43ed71b 100644
--- a/hieradata/regex.yaml
+++ b/hieradata/regex.yaml
@@ -253,6 +253,10 @@
   __regex: !ruby/regexp /^.*\.ulsfo\.wmnet$/
   profile::base::check_smart: true
 
+smart_health_eqiad:
+  __regex: !ruby/regexp 
/^(analytics1028|aqs1004|conf1001|cp1045|db1039|db1001|db1087|db1011|druid1001|elastic1017|es1011|etcd1001|ganeti1001|graphite1001|kafka1001|kafka-jumbo1001|labmon1001|logstash1008|mw1180|maps1001|mc1019|ms-be1013|ms-be1039|ores1001|oresrdb1002|pc1004|sca1003|scb1001|thumbor1001).eqiad\.wmnet$/
+  profile::base::check_smart: true
+
 labtestpuppetmaster:
   __regex: !ruby/regexp /^labtestpuppetmaster.*\.wikimedia\.org$/
   puppetmaster: "puppetmaster2001.codfw.wmnet"

-- 
To view, visit https://gerrit.wikimedia.org/r/401491
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I436fff25a02b55a86b3fe468f2dd9154b6da89c2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: labs: split pdns-rec jobs

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399598 )

Change subject: labs: split pdns-rec jobs
..


labs: split pdns-rec jobs

Change-Id: I7c148540bfc55002c6370ea35d74065f881ab4e6
---
M modules/role/manifests/labs/prometheus.pp
1 file changed, 3 insertions(+), 3 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/role/manifests/labs/prometheus.pp 
b/modules/role/manifests/labs/prometheus.pp
index 2891056..96c1f8c 100644
--- a/modules/role/manifests/labs/prometheus.pp
+++ b/modules/role/manifests/labs/prometheus.pp
@@ -55,7 +55,7 @@
 'job_name'=> 'pdns_rec',
 'scheme'  => 'http',
 'file_sd_configs' => [
-{ 'files' => [ "${targets_path}/pdns_*.yaml" ] }
+{ 'files' => [ "${targets_path}/pdns-rec_*.yaml" ] }
 ],
 },
 ]
@@ -85,8 +85,8 @@
 port   => '9192',
 }
 
-prometheus::class_config{ "pdns_rec_${::site}":
-dest   => "${targets_path}/pdns_rec_${::site}.yaml",
+prometheus::class_config{ "pdns-rec_${::site}":
+dest   => "${targets_path}/pdns-rec_${::site}.yaml",
 site   => $::site,
 class_name => 'role::wmcs::openstack::main::services_primary',
 port   => '9199',

-- 
To view, visit https://gerrit.wikimedia.org/r/399598
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7c148540bfc55002c6370ea35d74065f881ab4e6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: labs: split pdns-rec jobs

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399598 )

Change subject: labs: split pdns-rec jobs
..

labs: split pdns-rec jobs

Change-Id: I7c148540bfc55002c6370ea35d74065f881ab4e6
---
M modules/role/manifests/labs/prometheus.pp
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/98/399598/1

diff --git a/modules/role/manifests/labs/prometheus.pp 
b/modules/role/manifests/labs/prometheus.pp
index 2891056..96c1f8c 100644
--- a/modules/role/manifests/labs/prometheus.pp
+++ b/modules/role/manifests/labs/prometheus.pp
@@ -55,7 +55,7 @@
 'job_name'=> 'pdns_rec',
 'scheme'  => 'http',
 'file_sd_configs' => [
-{ 'files' => [ "${targets_path}/pdns_*.yaml" ] }
+{ 'files' => [ "${targets_path}/pdns-rec_*.yaml" ] }
 ],
 },
 ]
@@ -85,8 +85,8 @@
 port   => '9192',
 }
 
-prometheus::class_config{ "pdns_rec_${::site}":
-dest   => "${targets_path}/pdns_rec_${::site}.yaml",
+prometheus::class_config{ "pdns-rec_${::site}":
+dest   => "${targets_path}/pdns-rec_${::site}.yaml",
 site   => $::site,
 class_name => 'role::wmcs::openstack::main::services_primary',
 port   => '9199',

-- 
To view, visit https://gerrit.wikimedia.org/r/399598
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7c148540bfc55002c6370ea35d74065f881ab4e6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add PowerDNS Recursor scraper config on labmon1001

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399422 )

Change subject: Add PowerDNS Recursor scraper config on labmon1001
..


Add PowerDNS Recursor scraper config on labmon1001

Change-Id: I098d0290b5f55d5ffd1acc9333cf618d1a93b4a3
---
M modules/role/manifests/labs/prometheus.pp
1 file changed, 18 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/role/manifests/labs/prometheus.pp 
b/modules/role/manifests/labs/prometheus.pp
index c54b34e..2891056 100644
--- a/modules/role/manifests/labs/prometheus.pp
+++ b/modules/role/manifests/labs/prometheus.pp
@@ -50,6 +50,16 @@
 },
 ]
 
+$pdns_rec_jobs = [
+{
+'job_name'=> 'pdns_rec',
+'scheme'  => 'http',
+'file_sd_configs' => [
+{ 'files' => [ "${targets_path}/pdns_*.yaml" ] }
+],
+},
+]
+
 file { "${targets_path}/blackbox_http_keystone.yaml":
   content => ordered_yaml([{
 'targets' => ['labcontrol1001.wikimedia.org:5000/v3', # keystone
@@ -75,6 +85,13 @@
 port   => '9192',
 }
 
+prometheus::class_config{ "pdns_rec_${::site}":
+dest   => "${targets_path}/pdns_rec_${::site}.yaml",
+site   => $::site,
+class_name => 'role::wmcs::openstack::main::services_primary',
+port   => '9199',
+}
+
 prometheus::server { 'labs':
 storage_encoding  => '2',
 listen_address=> ':9900',
@@ -83,6 +100,7 @@
 memory_chunks => $memory_chunks,
 scrape_configs_extra  => array_concat(
 $blackbox_jobs, $rabbitmq_jobs, $pdns_jobs,
+$pdns_rec_jobs,
 ),
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/399422
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I098d0290b5f55d5ffd1acc9333cf618d1a93b4a3
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Muehlenhoff 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Use custom profile for PDNS exporter on labservices1001

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399582 )

Change subject: Use custom profile for PDNS exporter on labservices1001
..


Use custom profile for PDNS exporter on labservices1001

Change-Id: Ia96c300212be7dc139ff309a7334b67d361eaaf9
---
A modules/profile/manifests/prometheus/pdns_rec_exporter_wmcs.pp
M modules/role/manifests/wmcs/openstack/main/services_primary.pp
2 files changed, 15 insertions(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/profile/manifests/prometheus/pdns_rec_exporter_wmcs.pp 
b/modules/profile/manifests/prometheus/pdns_rec_exporter_wmcs.pp
new file mode 100644
index 000..3d9c779
--- /dev/null
+++ b/modules/profile/manifests/prometheus/pdns_rec_exporter_wmcs.pp
@@ -0,0 +1,14 @@
+class profile::prometheus::pdns_rec_exporter_wmcs (
+) {
+require_package('prometheus-pdns-rec-exporter')
+
+service { 'prometheus-pdns-rec-exporter':
+ensure  => running,
+}
+
+ferm::service { 'prometheus-pdns-rec-exporter':
+proto  => 'tcp',
+port   => '9199',
+srange => '@resolve(labmon1001.eqiad.wmnet)', # Should be properly 
defined via Hiera for WMCS
+}
+}
diff --git a/modules/role/manifests/wmcs/openstack/main/services_primary.pp 
b/modules/role/manifests/wmcs/openstack/main/services_primary.pp
index 754ce0b..eb4b1ec 100644
--- a/modules/role/manifests/wmcs/openstack/main/services_primary.pp
+++ b/modules/role/manifests/wmcs/openstack/main/services_primary.pp
@@ -7,5 +7,5 @@
 include ::profile::openstack::main::designate::service
 include ::profile::openstack::main::pdns::dns_floating_ip_updater
 include ::profile::prometheus::pdns_exporter
-include ::profile::prometheus::pdns_rec_exporter
+include ::profile::prometheus::pdns_rec_exporter_wmcs
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/399582
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia96c300212be7dc139ff309a7334b67d361eaaf9
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Muehlenhoff 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: WIP allow labmon1001 to contact pdns exporters

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399439 )

Change subject: WIP allow labmon1001 to contact pdns exporters
..

WIP allow labmon1001 to contact pdns exporters

Change-Id: Ifad13cccf4351a6954c500632cda032e467d0ed1
---
A hieradata/common/profile/openstack/main/pdns/auth/service.yaml
A hieradata/common/profile/openstack/main/pdns/recursor/primary.yaml
M modules/profile/manifests/prometheus/pdns_exporter.pp
M modules/profile/manifests/prometheus/pdns_rec_exporter.pp
4 files changed, 12 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/39/399439/1

diff --git a/hieradata/common/profile/openstack/main/pdns/auth/service.yaml 
b/hieradata/common/profile/openstack/main/pdns/auth/service.yaml
new file mode 100644
index 000..aabeac6
--- /dev/null
+++ b/hieradata/common/profile/openstack/main/pdns/auth/service.yaml
@@ -0,0 +1,2 @@
+prometheus_nodes_extra:
+  - labmon1001.eqiad.wmnet
diff --git a/hieradata/common/profile/openstack/main/pdns/recursor/primary.yaml 
b/hieradata/common/profile/openstack/main/pdns/recursor/primary.yaml
new file mode 100644
index 000..aabeac6
--- /dev/null
+++ b/hieradata/common/profile/openstack/main/pdns/recursor/primary.yaml
@@ -0,0 +1,2 @@
+prometheus_nodes_extra:
+  - labmon1001.eqiad.wmnet
diff --git a/modules/profile/manifests/prometheus/pdns_exporter.pp 
b/modules/profile/manifests/prometheus/pdns_exporter.pp
index 292d06e..5525365 100644
--- a/modules/profile/manifests/prometheus/pdns_exporter.pp
+++ b/modules/profile/manifests/prometheus/pdns_exporter.pp
@@ -1,5 +1,10 @@
 class profile::prometheus::pdns_exporter (
+$prometheus_nodes = hiera('prometheus_nodes'),
+$prometheus_nodes_extra = hiera('prometheus_nodes_extra', []),
 ) {
+$prometheus_ferm_nodes = join($prometheus_nodes + $prometheus_nodes_extra, 
' ')
+$ferm_srange = "(@resolve((${prometheus_ferm_nodes})) 
@resolve((${prometheus_ferm_nodes}), ))"
+
 require_package('prometheus-pdns-exporter')
 
 service { 'prometheus-pdns-exporter':
@@ -9,6 +14,6 @@
 ferm::service { 'prometheus-pdns-exporter':
 proto  => 'tcp',
 port   => '9192',
-srange => '@resolve(labmon1001.eqiad.wmnet)', # Should be properly 
defined via Hiera for WMCS
+srange => $ferm_srange,
 }
 }
diff --git a/modules/profile/manifests/prometheus/pdns_rec_exporter.pp 
b/modules/profile/manifests/prometheus/pdns_rec_exporter.pp
index 0902ab2..582393a 100644
--- a/modules/profile/manifests/prometheus/pdns_rec_exporter.pp
+++ b/modules/profile/manifests/prometheus/pdns_rec_exporter.pp
@@ -1,7 +1,8 @@
 class profile::prometheus::pdns_rec_exporter (
 $prometheus_nodes = hiera('prometheus_nodes'),
+$prometheus_nodes_extra = hiera('prometheus_nodes_extra', []),
 ) {
-$prometheus_ferm_nodes = join($prometheus_nodes, ' ')
+$prometheus_ferm_nodes = join($prometheus_nodes + $prometheus_nodes_extra, 
' ')
 $ferm_srange = "(@resolve((${prometheus_ferm_nodes})) 
@resolve((${prometheus_ferm_nodes}), ))"
 
 require_package('prometheus-pdns-rec-exporter')

-- 
To view, visit https://gerrit.wikimedia.org/r/399439
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifad13cccf4351a6954c500632cda032e467d0ed1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add nutcracker job

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399163 )

Change subject: prometheus: add nutcracker job
..


prometheus: add nutcracker job

Bug: T181995
Change-Id: I33dc71707d85173d6c03384bfe4f48a09e81992d
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 17 insertions(+), 1 deletion(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 7af9500..5807e26 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -759,6 +759,22 @@
 port   => '9108',
 }
 
+$nutcracker_jobs = [
+{
+'job_name'=> 'nutcracker',
+'scheme'  => 'http',
+'file_sd_configs' => [
+{ 'files' => [ "${targets_path}/nutcracker_*.yaml" ]}
+],
+},
+]
+prometheus::class_config { "nutcracker_${::site}":
+dest   => "${targets_path}/nutcracker_${::site}.yaml",
+site   => $::site,
+class_name => 'profile::prometheus::nutcracker_exporter',
+port   => '9191',
+}
+
 prometheus::server { 'ops':
 storage_encoding  => '2',
 listen_address=> '127.0.0.1:9900',
@@ -771,7 +787,7 @@
 $nginx_jobs, $pybal_jobs, $blackbox_jobs, $jmx_exporter_jobs,
 $redis_jobs, $mtail_jobs, $ldap_jobs, $ircd_jobs, $pdns_rec_jobs,
 $etherpad_jobs, $elasticsearch_jobs, $wdqs_updater_jobs,
-$blazegraph_jobs,
+$blazegraph_jobs, $nutcracker_jobs,
 ),
 global_config_extra   => $config_extra,
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/399163
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I33dc71707d85173d6c03384bfe4f48a09e81992d
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add nutcracker job

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399163 )

Change subject: prometheus: add nutcracker job
..

prometheus: add nutcracker job

Bug: T181995
Change-Id: I33dc71707d85173d6c03384bfe4f48a09e81992d
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 17 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/63/399163/1

diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index bc2b7b3..50a58a5 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -725,6 +725,22 @@
 port   => '9108',
 }
 
+$nutcracker_jobs = [
+{
+'job_name'=> 'nutcracker',
+'scheme'  => 'http',
+'file_sd_configs' => [
+{ 'files' => [ "${targets_path}/nutcracker*.yaml" ]}
+],
+},
+]
+prometheus::class_config { "nutcracker_${::site}":
+dest   => "${targets_path}/nutcracker_${::site}.yaml",
+site   => $::site,
+class_name => 'profile::prometheus::nutcracker_exporter',
+port   => '9191',
+}
+
 prometheus::server { 'ops':
 storage_encoding  => '2',
 listen_address=> '127.0.0.1:9900',
@@ -736,7 +752,7 @@
 $apache_jobs, $etcd_jobs, $etcdmirror_jobs, $pdu_jobs,
 $nginx_jobs, $pybal_jobs, $blackbox_jobs, $jmx_exporter_jobs,
 $redis_jobs, $mtail_jobs, $ldap_jobs, $ircd_jobs, $pdns_rec_jobs,
-$etherpad_jobs, $elasticsearch_jobs,
+$etherpad_jobs, $elasticsearch_jobs, $nutcracker_jobs,
 ),
 global_config_extra   => $config_extra,
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/399163
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I33dc71707d85173d6c03384bfe4f48a09e81992d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-nutcracker-exporter[master]: Fix nutcracker metrics fetching

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399154 )

Change subject: Fix nutcracker metrics fetching
..


Fix nutcracker metrics fetching

Bug: T181995
Change-Id: I127e0cc057e5f5ff3156e11db0cd52424cc01ffd
---
M debian/changelog
M prometheus-nutcracker-exporter
2 files changed, 32 insertions(+), 15 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/debian/changelog b/debian/changelog
index eeb235f..9e0fc5a 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+prometheus-nutcracker-exporter (0.2) UNRELEASED; urgency=medium
+
+  * Fix nutcracker metrics fetching
+
+ -- Filippo Giunchedi   Tue, 19 Dec 2017 11:23:48 +0100
+
 prometheus-nutcracker-exporter (0.1) jessie-wikimedia; urgency=medium
 
   * First upload.
diff --git a/prometheus-nutcracker-exporter b/prometheus-nutcracker-exporter
index c505573..dcf0ae0 100755
--- a/prometheus-nutcracker-exporter
+++ b/prometheus-nutcracker-exporter
@@ -20,8 +20,9 @@
 import re
 import sys
 import time
-import urllib2
+import socket
 
+from contextlib import closing
 from prometheus_client import start_http_server, Summary
 from prometheus_client.core import (CounterMetricFamily, GaugeMetricFamily,
 REGISTRY)
@@ -33,11 +34,22 @@
 scrape_duration = Summary(
 'nutcracker_scrape_duration_seconds', 'Nutcracker exporter scrape 
duration')
 
-def __init__(self, stats_url='http://localhost:2'):
-self.stats_url = stats_url
+def __init__(self, address, port):
+self.address = address
+self.port = port
 
-def _load_stats(self, url):
-fixed_json = re.sub(r'"("[^"]+")"(?=:)', r'\1', url.read())
+def _load_stats(self, host, port):
+sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+
+with closing(sock):
+sock.connect((host, int(port)))
+sock.settimeout(1)
+raw_data = sock.recv(65536).decode('utf-8', errors='ignore')
+sock.shutdown(socket.SHUT_RDWR)
+
+# does not properly escape quotation marks in server aliases.
+# https://github.com/twitter/twemproxy/issues/532
+fixed_json = re.sub(r'"("[^"]+")"(?=:)', r'\1', raw_data.strip())
 return json.loads(fixed_json)
 
 def _pool_metrics(self, stats):
@@ -83,13 +95,11 @@
 
 up = GaugeMetricFamily('nutcracker_up', 'Nutcracker is running')
 try:
-url = urllib2.urlopen(self.stats_url)
-if url.code == 200:
-stats = self._load_stats(url)
-up.add_metric([], 1)
+stats = self._load_stats(self.address, self.port)
+up.add_metric([], 1)
 yield up
-except urllib2.URLError:
-log.error('Could not connect to Nutcracker stats URL')
+except socket.error as e:
+log.error('Could not fetch Nutcracker stats', e)
 up.add_metric([], 0)
 yield up
 return
@@ -129,9 +139,9 @@
 help='Listen on this address', default=':9191')
 parser.add_argument('-d', '--debug', action='store_true',
 help='Enable debug logging')
-parser.add_argument('-u', '--url', metavar='URL',
-help='Fetch pdns-rec stats from this url',
-default='http://localhost:2')
+parser.add_argument('-s', '--stats', metavar='ADDRESS',
+help='Fetch nutcracker stats from this address',
+default=':2')
 args = parser.parse_args()
 
 if args.debug:
@@ -143,7 +153,8 @@
 
 log.info('Starting nutcracker_exporter on %s:%s', address, port)
 
-REGISTRY.register(PrometheusNutcrackerCollector(args.url))
+stats_address, stats_port = args.stats.split(':', 1)
+REGISTRY.register(PrometheusNutcrackerCollector(stats_address, stats_port))
 start_http_server(int(port), addr=address)
 
 try:

-- 
To view, visit https://gerrit.wikimedia.org/r/399154
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I127e0cc057e5f5ff3156e11db0cd52424cc01ffd
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-nutcracker-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-nutcracker-exporter[master]: Fix nutcracker metrics fetching

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399154 )

Change subject: Fix nutcracker metrics fetching
..

Fix nutcracker metrics fetching

Bug: T181995
Change-Id: I127e0cc057e5f5ff3156e11db0cd52424cc01ffd
---
M debian/changelog
M prometheus-nutcracker-exporter
2 files changed, 32 insertions(+), 15 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-nutcracker-exporter 
refs/changes/54/399154/1

diff --git a/debian/changelog b/debian/changelog
index eeb235f..9e0fc5a 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+prometheus-nutcracker-exporter (0.2) UNRELEASED; urgency=medium
+
+  * Fix nutcracker metrics fetching
+
+ -- Filippo Giunchedi   Tue, 19 Dec 2017 11:23:48 +0100
+
 prometheus-nutcracker-exporter (0.1) jessie-wikimedia; urgency=medium
 
   * First upload.
diff --git a/prometheus-nutcracker-exporter b/prometheus-nutcracker-exporter
index c505573..dcf0ae0 100755
--- a/prometheus-nutcracker-exporter
+++ b/prometheus-nutcracker-exporter
@@ -20,8 +20,9 @@
 import re
 import sys
 import time
-import urllib2
+import socket
 
+from contextlib import closing
 from prometheus_client import start_http_server, Summary
 from prometheus_client.core import (CounterMetricFamily, GaugeMetricFamily,
 REGISTRY)
@@ -33,11 +34,22 @@
 scrape_duration = Summary(
 'nutcracker_scrape_duration_seconds', 'Nutcracker exporter scrape 
duration')
 
-def __init__(self, stats_url='http://localhost:2'):
-self.stats_url = stats_url
+def __init__(self, address, port):
+self.address = address
+self.port = port
 
-def _load_stats(self, url):
-fixed_json = re.sub(r'"("[^"]+")"(?=:)', r'\1', url.read())
+def _load_stats(self, host, port):
+sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+
+with closing(sock):
+sock.connect((host, int(port)))
+sock.settimeout(1)
+raw_data = sock.recv(65536).decode('utf-8', errors='ignore')
+sock.shutdown(socket.SHUT_RDWR)
+
+# does not properly escape quotation marks in server aliases.
+# https://github.com/twitter/twemproxy/issues/532
+fixed_json = re.sub(r'"("[^"]+")"(?=:)', r'\1', raw_data.strip())
 return json.loads(fixed_json)
 
 def _pool_metrics(self, stats):
@@ -83,13 +95,11 @@
 
 up = GaugeMetricFamily('nutcracker_up', 'Nutcracker is running')
 try:
-url = urllib2.urlopen(self.stats_url)
-if url.code == 200:
-stats = self._load_stats(url)
-up.add_metric([], 1)
+stats = self._load_stats(self.address, self.port)
+up.add_metric([], 1)
 yield up
-except urllib2.URLError:
-log.error('Could not connect to Nutcracker stats URL')
+except socket.error as e:
+log.error('Could not fetch Nutcracker stats', e)
 up.add_metric([], 0)
 yield up
 return
@@ -129,9 +139,9 @@
 help='Listen on this address', default=':9191')
 parser.add_argument('-d', '--debug', action='store_true',
 help='Enable debug logging')
-parser.add_argument('-u', '--url', metavar='URL',
-help='Fetch pdns-rec stats from this url',
-default='http://localhost:2')
+parser.add_argument('-s', '--stats', metavar='ADDRESS',
+help='Fetch nutcracker stats from this address',
+default=':2')
 args = parser.parse_args()
 
 if args.debug:
@@ -143,7 +153,8 @@
 
 log.info('Starting nutcracker_exporter on %s:%s', address, port)
 
-REGISTRY.register(PrometheusNutcrackerCollector(args.url))
+stats_address, stats_port = args.stats.split(':', 1)
+REGISTRY.register(PrometheusNutcrackerCollector(stats_address, stats_port))
 start_http_server(int(port), addr=address)
 
 try:

-- 
To view, visit https://gerrit.wikimedia.org/r/399154
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I127e0cc057e5f5ff3156e11db0cd52424cc01ffd
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-nutcracker-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add nutcracker_exporter profile

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398847 )

Change subject: Add nutcracker_exporter profile
..


Add nutcracker_exporter profile

Bug: T181995
Change-Id: Ie44705e78fd349e80556e88964c589e44057838d
---
M modules/profile/manifests/mediawiki/nutcracker.pp
M modules/profile/manifests/nutcracker.pp
M modules/profile/manifests/openstack/base/wikitech/service.pp
A modules/profile/manifests/prometheus/nutcracker_exporter.pp
M modules/role/manifests/thumbor/mediawiki.pp
5 files changed, 22 insertions(+), 0 deletions(-)

Approvals:
  Andrew Bogott: Looks good to me, but someone else must approve
  Muehlenhoff: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/profile/manifests/mediawiki/nutcracker.pp 
b/modules/profile/manifests/mediawiki/nutcracker.pp
index 1f885a3..eae2656 100644
--- a/modules/profile/manifests/mediawiki/nutcracker.pp
+++ b/modules/profile/manifests/mediawiki/nutcracker.pp
@@ -8,6 +8,7 @@
 ) {
 $redis_servers = $redis_shards['sessions']
 include ::passwords::redis
+include ::profile::prometheus::nutcracker_exporter
 
 $pools = {
 'memcached' => {
diff --git a/modules/profile/manifests/nutcracker.pp 
b/modules/profile/manifests/nutcracker.pp
index bb40052..5aee9c2 100644
--- a/modules/profile/manifests/nutcracker.pp
+++ b/modules/profile/manifests/nutcracker.pp
@@ -41,6 +41,7 @@
 pools => $pools,
 }
 
+include ::profile::prometheus::nutcracker_exporter
 class { '::nutcracker::monitoring':
 port => $monitor_port,
 }
diff --git a/modules/profile/manifests/openstack/base/wikitech/service.pp 
b/modules/profile/manifests/openstack/base/wikitech/service.pp
index ae51f73..0493042 100644
--- a/modules/profile/manifests/openstack/base/wikitech/service.pp
+++ b/modules/profile/manifests/openstack/base/wikitech/service.pp
@@ -8,6 +8,7 @@
 ) {
 
 include ::nutcracker::monitoring
+include ::profile::prometheus::nutcracker_exporter
 include ::mediawiki::packages::php5
 include ::mediawiki::packages::math
 include ::mediawiki::packages::tex
diff --git a/modules/profile/manifests/prometheus/nutcracker_exporter.pp 
b/modules/profile/manifests/prometheus/nutcracker_exporter.pp
new file mode 100644
index 000..2530548
--- /dev/null
+++ b/modules/profile/manifests/prometheus/nutcracker_exporter.pp
@@ -0,0 +1,18 @@
+class profile::prometheus::nutcracker_exporter (
+$prometheus_nodes = hiera('prometheus_nodes'),
+) {
+$prometheus_ferm_nodes = join($prometheus_nodes, ' ')
+$ferm_srange = "(@resolve((${prometheus_ferm_nodes})) 
@resolve((${prometheus_ferm_nodes}), ))"
+
+require_package('prometheus-nutcracker-exporter')
+
+service { 'prometheus-nutcracker-exporter':
+ensure  => running,
+}
+
+ferm::service { 'prometheus-nutcracker-exporter':
+proto  => 'tcp',
+port   => '9191',
+srange => $ferm_srange,
+}
+}
diff --git a/modules/role/manifests/thumbor/mediawiki.pp 
b/modules/role/manifests/thumbor/mediawiki.pp
index 2cfb847..6214b4c 100644
--- a/modules/role/manifests/thumbor/mediawiki.pp
+++ b/modules/role/manifests/thumbor/mediawiki.pp
@@ -9,6 +9,7 @@
 include ::base::firewall
 include ::mediawiki::packages::fonts
 include role::statsite
+include ::profile::prometheus::nutcracker_exporter
 
 class { '::thumbor::nutcracker':
 thumbor_memcached_servers => 
hiera('thumbor_memcached_servers_nutcracker')

-- 
To view, visit https://gerrit.wikimedia.org/r/398847
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie44705e78fd349e80556e88964c589e44057838d
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Andrew Bogott 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Gilles 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: recording rules for redis

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398871 )

Change subject: prometheus: recording rules for redis
..


prometheus: recording rules for redis

Used to speed up dashboards.

Bug: T148637
Change-Id: I2561d8a1230d5ce8445b5fdb06a64a27a8025eed
---
M modules/role/files/prometheus/rules_ops.conf
1 file changed, 4 insertions(+), 0 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index c7a6f8f..b6bbede 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -177,3 +177,7 @@
 instance:node_ipvs_backend_connections_active:rate5m = sum by (instance) 
(rate(node_ipvs_backend_connections_active[5m]))
 local_port_local_address_instance:node_ipvs_backend_connections_active:rate5m 
= sum by (local_port,local_address,instance) 
(rate(node_ipvs_backend_connections_active[5m]))
 
local_port_local_address_instance:node_ipvs_backend_connections_inactive:rate5m 
= sum by (local_port,local_address,instance) 
(rate(node_ipvs_backend_connections_inactive[5m]))
+
+# Redis aggregation
+cmd_job:redis_command_call_duration_seconds_count:rate5m = sum by (cmd, job) 
(rate(redis_command_call_duration_seconds_count[5m]))
+cmd_job:redis_command_call_duration_seconds_sum:rate5m = sum by (cmd, job) 
(rate(redis_command_call_duration_seconds_sum[5m]))

-- 
To view, visit https://gerrit.wikimedia.org/r/398871
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2561d8a1230d5ce8445b5fdb06a64a27a8025eed
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: recording rules for redis

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398871 )

Change subject: prometheus: recording rules for redis
..

prometheus: recording rules for redis

Used to speed up dashboards.

Bug: T148637
Change-Id: I2561d8a1230d5ce8445b5fdb06a64a27a8025eed
---
M modules/role/files/prometheus/rules_ops.conf
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/71/398871/1

diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index c7a6f8f..b6bbede 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -177,3 +177,7 @@
 instance:node_ipvs_backend_connections_active:rate5m = sum by (instance) 
(rate(node_ipvs_backend_connections_active[5m]))
 local_port_local_address_instance:node_ipvs_backend_connections_active:rate5m 
= sum by (local_port,local_address,instance) 
(rate(node_ipvs_backend_connections_active[5m]))
 
local_port_local_address_instance:node_ipvs_backend_connections_inactive:rate5m 
= sum by (local_port,local_address,instance) 
(rate(node_ipvs_backend_connections_inactive[5m]))
+
+# Redis aggregation
+cmd_job:redis_command_call_duration_seconds_count:rate5m = sum by (cmd, job) 
(rate(redis_command_call_duration_seconds_count[5m]))
+cmd_job:redis_command_call_duration_seconds_sum:rate5m = sum by (cmd, job) 
(rate(redis_command_call_duration_seconds_sum[5m]))

-- 
To view, visit https://gerrit.wikimedia.org/r/398871
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2561d8a1230d5ce8445b5fdb06a64a27a8025eed
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add nutcracker_exporter profile

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398847 )

Change subject: Add nutcracker_exporter profile
..

Add nutcracker_exporter profile

Bug: T181995
Change-Id: Ie44705e78fd349e80556e88964c589e44057838d
---
M modules/profile/manifests/mediawiki/nutcracker.pp
M modules/profile/manifests/nutcracker.pp
M modules/profile/manifests/openstack/base/wikitech/service.pp
A modules/profile/manifests/prometheus/nutcracker_exporter.pp
M modules/role/manifests/thumbor/mediawiki.pp
5 files changed, 22 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/47/398847/1

diff --git a/modules/profile/manifests/mediawiki/nutcracker.pp 
b/modules/profile/manifests/mediawiki/nutcracker.pp
index 1f885a3..eae2656 100644
--- a/modules/profile/manifests/mediawiki/nutcracker.pp
+++ b/modules/profile/manifests/mediawiki/nutcracker.pp
@@ -8,6 +8,7 @@
 ) {
 $redis_servers = $redis_shards['sessions']
 include ::passwords::redis
+include ::profile::prometheus::nutcracker_exporter
 
 $pools = {
 'memcached' => {
diff --git a/modules/profile/manifests/nutcracker.pp 
b/modules/profile/manifests/nutcracker.pp
index bb40052..5aee9c2 100644
--- a/modules/profile/manifests/nutcracker.pp
+++ b/modules/profile/manifests/nutcracker.pp
@@ -41,6 +41,7 @@
 pools => $pools,
 }
 
+include ::profile::prometheus::nutcracker_exporter
 class { '::nutcracker::monitoring':
 port => $monitor_port,
 }
diff --git a/modules/profile/manifests/openstack/base/wikitech/service.pp 
b/modules/profile/manifests/openstack/base/wikitech/service.pp
index ae51f73..0493042 100644
--- a/modules/profile/manifests/openstack/base/wikitech/service.pp
+++ b/modules/profile/manifests/openstack/base/wikitech/service.pp
@@ -8,6 +8,7 @@
 ) {
 
 include ::nutcracker::monitoring
+include ::profile::prometheus::nutcracker_exporter
 include ::mediawiki::packages::php5
 include ::mediawiki::packages::math
 include ::mediawiki::packages::tex
diff --git a/modules/profile/manifests/prometheus/nutcracker_exporter.pp 
b/modules/profile/manifests/prometheus/nutcracker_exporter.pp
new file mode 100644
index 000..2530548
--- /dev/null
+++ b/modules/profile/manifests/prometheus/nutcracker_exporter.pp
@@ -0,0 +1,18 @@
+class profile::prometheus::nutcracker_exporter (
+$prometheus_nodes = hiera('prometheus_nodes'),
+) {
+$prometheus_ferm_nodes = join($prometheus_nodes, ' ')
+$ferm_srange = "(@resolve((${prometheus_ferm_nodes})) 
@resolve((${prometheus_ferm_nodes}), ))"
+
+require_package('prometheus-nutcracker-exporter')
+
+service { 'prometheus-nutcracker-exporter':
+ensure  => running,
+}
+
+ferm::service { 'prometheus-nutcracker-exporter':
+proto  => 'tcp',
+port   => '9191',
+srange => $ferm_srange,
+}
+}
diff --git a/modules/role/manifests/thumbor/mediawiki.pp 
b/modules/role/manifests/thumbor/mediawiki.pp
index 2cfb847..6214b4c 100644
--- a/modules/role/manifests/thumbor/mediawiki.pp
+++ b/modules/role/manifests/thumbor/mediawiki.pp
@@ -9,6 +9,7 @@
 include ::base::firewall
 include ::mediawiki::packages::fonts
 include role::statsite
+include ::profile::prometheus::nutcracker_exporter
 
 class { '::thumbor::nutcracker':
 thumbor_memcached_servers => 
hiera('thumbor_memcached_servers_nutcracker')

-- 
To view, visit https://gerrit.wikimedia.org/r/398847
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie44705e78fd349e80556e88964c589e44057838d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...prometheus-nutcracker-exporter[master]: Add debian/ and .gitreview

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398839 )

Change subject: Add debian/ and .gitreview
..


Add debian/ and .gitreview

Bug: T181995
Change-Id: Icfb9a2c02b5b95a48eed22ac6f9e86d3bddb4132
---
A .gitreview
A debian/changelog
A debian/compat
A debian/control
A debian/copyright
A debian/dirs
A debian/install
A debian/postinst
A debian/rules
A debian/service
10 files changed, 88 insertions(+), 0 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/.gitreview b/.gitreview
new file mode 100644
index 000..ce381ea
--- /dev/null
+++ b/.gitreview
@@ -0,0 +1,5 @@
+[gerrit]
+host=gerrit.wikimedia.org
+port=29418
+project=operations/debs/prometheus-nutcracker-exporter.git
+defaultbranch=master
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 000..eeb235f
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+prometheus-nutcracker-exporter (0.1) jessie-wikimedia; urgency=medium
+
+  * First upload.
+
+ -- Filippo Giunchedi   Mon, 18 Dec 2017 14:46:33 +0100
diff --git a/debian/compat b/debian/compat
new file mode 100644
index 000..ec63514
--- /dev/null
+++ b/debian/compat
@@ -0,0 +1 @@
+9
diff --git a/debian/control b/debian/control
new file mode 100644
index 000..bc5f096
--- /dev/null
+++ b/debian/control
@@ -0,0 +1,12 @@
+Source: prometheus-nutcracker-exporter
+Section: net
+Priority: extra
+Maintainer: Filippo Giunchedi 
+Build-Depends: debhelper (>= 9), dh-systemd (>= 1.5)
+Standards-Version: 3.9.8
+
+Package: prometheus-nutcracker-exporter
+Architecture: all
+Depends: python-prometheus-client, ${misc:Depends}
+Description: Prometheus exporter for Nutcracker
+ Prometheus exporter for Nutcracker metrics.
diff --git a/debian/copyright b/debian/copyright
new file mode 100644
index 000..3825b2a
--- /dev/null
+++ b/debian/copyright
@@ -0,0 +1,21 @@
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+
+Files: *
+Copyright: 2017 Moritz Muehlenhoff , Filippo Giunchedi 
, Wikimedia Foundation
+License: Apache-2.0
+
+License: Apache-2.0
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ .
+ On Debian systems, the full text of the Apache License version 2 can be found
+ in the file `/usr/share/common-licenses/Apache-2.0'.
diff --git a/debian/dirs b/debian/dirs
new file mode 100644
index 000..e772481
--- /dev/null
+++ b/debian/dirs
@@ -0,0 +1 @@
+usr/bin
diff --git a/debian/install b/debian/install
new file mode 100644
index 000..a9b40d0
--- /dev/null
+++ b/debian/install
@@ -0,0 +1 @@
+prometheus-nutcracker-exporter usr/bin
diff --git a/debian/postinst b/debian/postinst
new file mode 100644
index 000..a43ff27
--- /dev/null
+++ b/debian/postinst
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+set -e
+
+case "$1" in
+  configure)
+# Add prometheus user
+if ! getent passwd prometheus > /dev/null; then
+adduser --quiet --system --no-create-home \
+--group --gecos "Prometheus daemon" prometheus || true
+fi
+
+  ;;
+
+  abort-upgrade|abort-remove|abort-deconfigure)
+:
+  ;;
+
+  *)
+echo "postinst called with unknown argument \`$1'" >&2
+exit 1
+  ;;
+esac
+
+#DEBHELPER#
+
+exit 0
diff --git a/debian/rules b/debian/rules
new file mode 100755
index 000..305c822
--- /dev/null
+++ b/debian/rules
@@ -0,0 +1,4 @@
+#!/usr/bin/make -f
+
+%:
+   dh $@ --with systemd
diff --git a/debian/service b/debian/service
new file mode 100644
index 000..62a77a4
--- /dev/null
+++ b/debian/service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Prometheus Nutcracker exporter
+
+[Service]
+Restart=always
+User=prometheus
+EnvironmentFile=-/etc/default/prometheus-nutcracker-exporter
+ExecStart=/usr/bin/prometheus-nutcracker-exporter $ARGS
+
+[Install]
+WantedBy=multi-user.target

-- 
To view, visit https://gerrit.wikimedia.org/r/398839
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icfb9a2c02b5b95a48eed22ac6f9e86d3bddb4132
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-nutcracker-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: 

[MediaWiki-commits] [Gerrit] operations...prometheus-nutcracker-exporter[master]: Add debian/ and .gitreview

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398839 )

Change subject: Add debian/ and .gitreview
..

Add debian/ and .gitreview

Bug: T181995
Change-Id: Icfb9a2c02b5b95a48eed22ac6f9e86d3bddb4132
---
A .gitreview
A debian/changelog
A debian/compat
A debian/control
A debian/copyright
A debian/dirs
A debian/install
A debian/postinst
A debian/rules
A debian/service
10 files changed, 88 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-nutcracker-exporter 
refs/changes/39/398839/1

diff --git a/.gitreview b/.gitreview
new file mode 100644
index 000..ce381ea
--- /dev/null
+++ b/.gitreview
@@ -0,0 +1,5 @@
+[gerrit]
+host=gerrit.wikimedia.org
+port=29418
+project=operations/debs/prometheus-nutcracker-exporter.git
+defaultbranch=master
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 000..eeb235f
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+prometheus-nutcracker-exporter (0.1) jessie-wikimedia; urgency=medium
+
+  * First upload.
+
+ -- Filippo Giunchedi   Mon, 18 Dec 2017 14:46:33 +0100
diff --git a/debian/compat b/debian/compat
new file mode 100644
index 000..ec63514
--- /dev/null
+++ b/debian/compat
@@ -0,0 +1 @@
+9
diff --git a/debian/control b/debian/control
new file mode 100644
index 000..bc5f096
--- /dev/null
+++ b/debian/control
@@ -0,0 +1,12 @@
+Source: prometheus-nutcracker-exporter
+Section: net
+Priority: extra
+Maintainer: Filippo Giunchedi 
+Build-Depends: debhelper (>= 9), dh-systemd (>= 1.5)
+Standards-Version: 3.9.8
+
+Package: prometheus-nutcracker-exporter
+Architecture: all
+Depends: python-prometheus-client, ${misc:Depends}
+Description: Prometheus exporter for Nutcracker
+ Prometheus exporter for Nutcracker metrics.
diff --git a/debian/copyright b/debian/copyright
new file mode 100644
index 000..3825b2a
--- /dev/null
+++ b/debian/copyright
@@ -0,0 +1,21 @@
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+
+Files: *
+Copyright: 2017 Moritz Muehlenhoff , Filippo Giunchedi 
, Wikimedia Foundation
+License: Apache-2.0
+
+License: Apache-2.0
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ .
+ On Debian systems, the full text of the Apache License version 2 can be found
+ in the file `/usr/share/common-licenses/Apache-2.0'.
diff --git a/debian/dirs b/debian/dirs
new file mode 100644
index 000..e772481
--- /dev/null
+++ b/debian/dirs
@@ -0,0 +1 @@
+usr/bin
diff --git a/debian/install b/debian/install
new file mode 100644
index 000..a9b40d0
--- /dev/null
+++ b/debian/install
@@ -0,0 +1 @@
+prometheus-nutcracker-exporter usr/bin
diff --git a/debian/postinst b/debian/postinst
new file mode 100644
index 000..a43ff27
--- /dev/null
+++ b/debian/postinst
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+set -e
+
+case "$1" in
+  configure)
+# Add prometheus user
+if ! getent passwd prometheus > /dev/null; then
+adduser --quiet --system --no-create-home \
+--group --gecos "Prometheus daemon" prometheus || true
+fi
+
+  ;;
+
+  abort-upgrade|abort-remove|abort-deconfigure)
+:
+  ;;
+
+  *)
+echo "postinst called with unknown argument \`$1'" >&2
+exit 1
+  ;;
+esac
+
+#DEBHELPER#
+
+exit 0
diff --git a/debian/rules b/debian/rules
new file mode 100755
index 000..305c822
--- /dev/null
+++ b/debian/rules
@@ -0,0 +1,4 @@
+#!/usr/bin/make -f
+
+%:
+   dh $@ --with systemd
diff --git a/debian/service b/debian/service
new file mode 100644
index 000..62a77a4
--- /dev/null
+++ b/debian/service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Prometheus Nutcracker exporter
+
+[Service]
+Restart=always
+User=prometheus
+EnvironmentFile=-/etc/default/prometheus-nutcracker-exporter
+ExecStart=/usr/bin/prometheus-nutcracker-exporter $ARGS
+
+[Install]
+WantedBy=multi-user.target

-- 
To view, visit https://gerrit.wikimedia.org/r/398839
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icfb9a2c02b5b95a48eed22ac6f9e86d3bddb4132
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/prometheus-nutcracker-exporter
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org

[MediaWiki-commits] [Gerrit] operations...prometheus-nutcracker-exporter[master]: First version

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398505 )

Change subject: First version
..


First version

Still missing backend servers aggregated metrics.

Bug: T181995
Change-Id: Ib8455100f0d2c435384fc9d88bc5c9ba2158e3f7
---
A prometheus-nutcracker-exporter
1 file changed, 157 insertions(+), 0 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, but someone else must approve
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/prometheus-nutcracker-exporter b/prometheus-nutcracker-exporter
new file mode 100755
index 000..c505573
--- /dev/null
+++ b/prometheus-nutcracker-exporter
@@ -0,0 +1,157 @@
+#!/usr/bin/python
+# Copyright 2017 Moritz Muehlenhoff
+#Filippo Giunchedi
+#Wikimedia Foundation
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import logging
+import re
+import sys
+import time
+import urllib2
+
+from prometheus_client import start_http_server, Summary
+from prometheus_client.core import (CounterMetricFamily, GaugeMetricFamily,
+REGISTRY)
+
+log = logging.getLogger(__name__)
+
+
+class PrometheusNutcrackerCollector(object):
+scrape_duration = Summary(
+'nutcracker_scrape_duration_seconds', 'Nutcracker exporter scrape 
duration')
+
+def __init__(self, stats_url='http://localhost:2'):
+self.stats_url = stats_url
+
+def _load_stats(self, url):
+fixed_json = re.sub(r'"("[^"]+")"(?=:)', r'\1', url.read())
+return json.loads(fixed_json)
+
+def _pool_metrics(self, stats):
+metrics = {
+'client_eof': CounterMetricFamily('nutcracker_pool_client_eof',
+  'EOF on client connections',
+  labels=['pool']),
+'client_err': CounterMetricFamily('nutcracker_pool_client_err',
+  'Errors on client connections',
+  labels=['pool']),
+'client_connections': 
GaugeMetricFamily('nutcracker_pool_client_connections',
+  'Active client 
connections',
+  labels=['pool']),
+'server_ejects': 
CounterMetricFamily('nutcracker_pool_server_ejects',
+ 'Times backend server was 
ejected',
+ labels=['pool']),
+'forward_error': 
CounterMetricFamily('nutcracker_pool_forward_error',
+ 'Times we encountered a 
forwarding error',
+ labels=['pool']),
+'fragments': CounterMetricFamily('nutcracker_pool_fragments',
+ 'Fragments created from a 
multi-vector request',
+ labels=['pool']),
+}
+
+for pool_name, pool_stats in stats.iteritems():
+if not isinstance(pool_stats, dict):
+continue
+for stat, value in pool_stats.iteritems():
+if isinstance(value, dict):
+# XXX aggregate server stats
+continue
+if stat not in metrics:
+continue
+metrics[stat].add_metric([pool_name], value)
+
+for metric in metrics.values():
+yield metric
+
+
+@scrape_duration.time()
+def collect(self):
+stats = ''
+
+up = GaugeMetricFamily('nutcracker_up', 'Nutcracker is running')
+try:
+url = urllib2.urlopen(self.stats_url)
+if url.code == 200:
+stats = self._load_stats(url)
+up.add_metric([], 1)
+yield up
+except urllib2.URLError:
+log.error('Could not connect to Nutcracker stats URL')
+up.add_metric([], 0)
+yield up
+return
+
+metrics = {
+'version': CounterMetricFamily('nutcracker_info', 'Version 
information', labels=['version']),
+'timestamp': CounterMetricFamily('nutcracker_start_time_seconds',
+ 

[MediaWiki-commits] [Gerrit] operations...prometheus-nutcracker-exporter[master]: First version

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398505 )

Change subject: First version
..

First version

Still missing backend servers aggregated metrics.

Bug: T181995
Change-Id: Ib8455100f0d2c435384fc9d88bc5c9ba2158e3f7
---
A prometheus-nutcracker-exporter
1 file changed, 138 insertions(+), 0 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-nutcracker-exporter 
refs/changes/05/398505/1

diff --git a/prometheus-nutcracker-exporter b/prometheus-nutcracker-exporter
new file mode 100755
index 000..90ccb28
--- /dev/null
+++ b/prometheus-nutcracker-exporter
@@ -0,0 +1,138 @@
+#!/usr/bin/python
+# Copyright 2017 Moritz Muehlenhoff
+#Filippo Giunchedi
+#Wikimedia Foundation
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import logging
+import re
+import sys
+import time
+import urllib2
+
+from prometheus_client import start_http_server, Summary
+from prometheus_client.core import (CounterMetricFamily, GaugeMetricFamily,
+REGISTRY)
+
+log = logging.getLogger(__name__)
+
+
+class PrometheusNutcrackerCollector(object):
+scrape_duration = Summary(
+'nutcracker_scrape_duration_seconds', 'Nutcracker exporter scrape 
duration')
+
+stats_url = 'http://localhost:2'
+
+def _load_stats(self, url):
+fixed_json = re.sub(r'"("[^"]+")"(?=:)', r'\1', url.read())
+return json.loads(fixed_json)
+
+def _pool_metrics(self, stats):
+metrics = {
+'client_eof': CounterMetricFamily('nutcracker_pool_client_eof', 
'', labels=['pool']),
+'client_err': CounterMetricFamily('nutcracker_pool_client_err', 
'', labels=['pool']),
+'client_connections': 
CounterMetricFamily('nutcracker_pool_client_connections', '', labels=['pool']),
+'server_ejects': 
CounterMetricFamily('nutcracker_pool_server_ejects', '', labels=['pool']),
+'forward_error': 
CounterMetricFamily('nutcracker_pool_forward_error', '', labels=['pool']),
+'fragments': CounterMetricFamily('nutcracker_pool_fragments', '', 
labels=['pool']),
+}
+
+for pool_name, pool_stats in stats.iteritems():
+if not isinstance(pool_stats, dict):
+continue
+for stat, value in pool_stats.iteritems():
+if isinstance(value, dict):
+# XXX aggregate server stats
+continue
+if stat not in metrics:
+continue
+metrics[stat].add_metric([pool_name], value)
+
+for metric in metrics.values():
+yield metric
+
+
+@scrape_duration.time()
+def collect(self):
+stats = ''
+
+up = GaugeMetricFamily('nutcracker_up', 'Nutcracker is running')
+try:
+url = urllib2.urlopen(self.stats_url)
+if url.code == 200:
+stats = self._load_stats(url)
+up.add_metric([], 1)
+yield up
+except urllib2.URLError:
+log.error('Could not connect to Nutcracker stats URL')
+up.add_metric([], 0)
+yield up
+return
+
+metrics = {
+'version': CounterMetricFamily('nutcracker_info', '', 
labels=['version']),
+'timestamp': CounterMetricFamily('nutcracker_start_time_seconds', 
''),
+'total_connections': 
CounterMetricFamily('nutcracker_total_connections', ''),
+'curr_connections': 
CounterMetricFamily('nutcracker_curr_connections', ''),
+}
+
+for name, family in metrics.iteritems():
+if name not in stats:
+continue
+
+if name == 'version':
+family.add_metric([stats.get(name)], 1)
+else:
+try:
+family.add_metric([], float(stats.get(name)))
+except ValueError:
+family.add_metric([], float('nan'))
+
+for metric in metrics.values():
+yield metric
+
+for metric in self._pool_metrics(stats):
+yield metric
+
+
+def main():
+parser = argparse.ArgumentParser()
+parser.add_argument('-l', '--listen', metavar='ADDRESS',
+help='Listen on this address', default=':9198')
+

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: restructure tests

[Filippo Giunchedi (Code Review)]

Filippo Giunchedi has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397889 )

Change subject: mtail: restructure tests
..


mtail: restructure tests

Get rid of MtailMetric class and return samples in a fashion similar to 
Prometheus' python client.

Bug: T177199
Change-Id: I7bc5e375e355153c8d5cce9e496cb648deb24cb2
---
M modules/mtail/files/test/exim_test.py
M modules/mtail/files/test/mtail_store.py
2 files changed, 36 insertions(+), 42 deletions(-)

Approvals:
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, approved



diff --git a/modules/mtail/files/test/exim_test.py 
b/modules/mtail/files/test/exim_test.py
index a0de96c..345a9a4 100644
--- a/modules/mtail/files/test/exim_test.py
+++ b/modules/mtail/files/test/exim_test.py
@@ -12,39 +12,40 @@
 os.path.join(test_dir, 'logs/exim.test'))
 
 def testEximMessages(self):
-m = self.store.get_metric('exim_messages_total')
-self.assertEqual(3, m._value)
-self.assertIn('status=out', m._labelpairs)
+s = self.store.get_samples('exim_messages_total')
+self.assertIn(('status=out', 3), s)
 
-m = self.store.get_metric('exim_messages_bytes')
-self.assertEqual(183084, m._value)
-self.assertIn('status=out', m._labelpairs)
+s = self.store.get_samples('exim_messages_bytes')
+self.assertIn(('status=out', 183084), s)
 
 def testDKIM(self):
-m = self.store.get_metric('exim_dkim_failure_total')
-self.assertEqual(2, m._value)
+s = self.store.get_samples('exim_dkim_failure_total')
+self.assertIn(('', 2), s)
 
-m = self.store.get_metric('exim_dkim_success_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_dkim_success_total')
+self.assertIn(('', 1), s)
 
 def testMiscErrors(self):
-m = self.store.get_metric('exim_smtp_errors_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_smtp_errors_total')
+self.assertIn(('', 1), s)
 
-m = self.store.get_metric('exim_rejected_rcpt_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_smtp_errors_total')
+self.assertIn(('', 1), s)
 
-m = self.store.get_metric('exim_tls_errors_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_rejected_rcpt_total')
+self.assertIn(('', 1), s)
 
-m = self.store.get_metric('exim_sender_verify_fail_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_tls_errors_total')
+self.assertIn(('', 1), s)
 
-m = self.store.get_metric('exim_sender_verify_defer_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_sender_verify_fail_total')
+self.assertIn(('', 1), s)
 
-m = self.store.get_metric('exim_too_many_connections_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_sender_verify_defer_total')
+self.assertIn(('', 1), s)
 
-m = self.store.get_metric('exim_rejected_helo_total')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('exim_too_many_connections_total')
+self.assertIn(('', 1), s)
+
+s = self.store.get_samples('exim_rejected_helo_total')
+self.assertIn(('', 1), s)
diff --git a/modules/mtail/files/test/mtail_store.py 
b/modules/mtail/files/test/mtail_store.py
index 527e315..360611c 100644
--- a/modules/mtail/files/test/mtail_store.py
+++ b/modules/mtail/files/test/mtail_store.py
@@ -29,23 +29,16 @@
 
 self._store = json.loads(''.join(metrics_store))
 
-def get_metric(self, name):
+def get_samples(self, name):
+"""Return all samples for metric name as a list of samples.
+   Each sample is in this form: ("k1=v1,k2=v2", value)"""
+samples = []
 if name not in self._store:
 raise ValueError('metric %s not found in store', name)
-return MtailMetric(self._store[name][0].get('Keys', []),
-   
self._store[name][0]['LabelValues'][0].get('Labels', []),
-   
self._store[name][0]['LabelValues'][0]['Value']['Value'])
-
-
-class MtailMetric(object):
-def __init__(self, keys, labels, value):
-self._keys = keys
-self._labels = labels
-self._value = value
-self._labelpairs = self.get_labelpairs(keys, labels)
-
-def get_labelpairs(self, keys, labels):
-res = []
-for k, v in zip(keys, labels):
-res.append('%s=%s' % (k, v))
-return res
+for metric in self._store[name][0]['LabelValues']:
+label_names = self._store[name][0].get('Keys', [])
+label_values = metric.get('Labels', [])
+value = metric['Value']['Value']
+labelpairs =