Ottomata has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/364427 )
Change subject: Prep for stat100[56]
......................................................................
Prep for stat100[56]
This mostly fixes some comments and adds these boxes to list of allowed rsync
hosts
Bug: T152712
Change-Id: Ifc9fe9073ce8de47d79820dff3daf0f3d44478d5
---
M hieradata/common.yaml
M hieradata/role/common/dumps/server.yaml
M modules/admin/data/data.yaml
M modules/dataset/files/rsync/rsyncd.conf.media
M modules/role/manifests/logging/mediawiki/udp2log.pp
M modules/role/manifests/statistics/private.pp
M modules/udp2log/manifests/rsyncd.pp
7 files changed, 17 insertions(+), 10 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/27/364427/1
diff --git a/hieradata/common.yaml b/hieradata/common.yaml
index 32363be..ff20792 100644
--- a/hieradata/common.yaml
+++ b/hieradata/common.yaml
@@ -292,6 +292,9 @@
statistics_servers:
- stat1002.eqiad.wmnet
- stat1003.eqiad.wmnet
+ - stat1004.eqiad.wmnet
+ - stat1005.eqiad.wmnet
+ - stat1006.eqiad.wmnet
- dataset1001.wikimedia.org
- thorium.eqiad.wmnet
dataset_clients_snapshots:
@@ -302,6 +305,8 @@
dataset_clients_other:
- stat1002.eqiad.wmnet
- stat1003.eqiad.wmnet
+ - stat1005.eqiad.wmnet
+ - stat1006.eqiad.wmnet
# Schemas names that match this regex
# will not be produced to the eventlogging-valid-mixed
diff --git a/hieradata/role/common/dumps/server.yaml
b/hieradata/role/common/dumps/server.yaml
index b28aa25..696b7ec 100644
--- a/hieradata/role/common/dumps/server.yaml
+++ b/hieradata/role/common/dumps/server.yaml
@@ -6,6 +6,8 @@
- ms1001.wikimedia.org
- stat1002.eqiad.wmnet
- stat1003.eqiad.wmnet
+ - stat1005.eqiad.wmnet
+ - stat1006.eqiad.wmnet
- sagres.c3sl.ufpr.br
- odysseus.fi.muni.cz
- odysseus.linux.cz
diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index dc262af..202c4d1 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -182,7 +182,7 @@
description: Access to stat boxes that host private data, including
sampled webrequest logs. This does not (necessarily)
grant access to Hadoop. If someone wants access to
- stat1002 to access webrequest logs in the Hadoop cluster
+ access webrequest logs in the Hadoop cluster
you should put them in the analytics-privatedata-users
group.
members: [ezachte, milimetric, dartar, halfak, awight, dr0ptp4kt, nuria,
@@ -220,13 +220,13 @@
analytics-users:
gid: 7080
description: Gives generic client access to the Analytics (Hadoop) cluster.
- This will grant shell access on Hadoop client nodes (stat1002) and on
+ This will grant shell access on Hadoop client nodes and on
Hadoop NameNodes.
members: [debt, goransm, jdittrich]
analytics-privatedata-users:
gid: 731
description: Gives access to the Analytics (Hadoop) cluster as well as
private data within.
- This will grant shell access on Hadoop client nodes
(stat1002) and on
+ This will grant shell access on Hadoop client nodes and on
Hadoop NameNodes. Some files in HDFS have sensitive data in
them.
Those files are group readable by the
analytics-privatedata-users group.
members: [dartar, milimetric, dfoy, aude, jforrester, nithum, shrlak,
diff --git a/modules/dataset/files/rsync/rsyncd.conf.media
b/modules/dataset/files/rsync/rsyncd.conf.media
index 68c321b..b4fc009 100644
--- a/modules/dataset/files/rsync/rsyncd.conf.media
+++ b/modules/dataset/files/rsync/rsyncd.conf.media
@@ -10,5 +10,5 @@
gid = wikidev
read only = false
path = /data/xmldatadumps/public/other/media
-hosts allow = stat1002.eqiad.wmnet stat1003.eqiad.wmnet
+hosts allow = stat1002.eqiad.wmnet stat1003.eqiad.wmnet stat1005.eqiad.wmnet
stat1006.eqiad.wmnet
diff --git a/modules/role/manifests/logging/mediawiki/udp2log.pp
b/modules/role/manifests/logging/mediawiki/udp2log.pp
index 45a1f15..7bd12fa 100644
--- a/modules/role/manifests/logging/mediawiki/udp2log.pp
+++ b/modules/role/manifests/logging/mediawiki/udp2log.pp
@@ -86,7 +86,8 @@
# Allow rsyncing of udp2log generated files to
# analysis hosts.
class { 'udp2log::rsyncd':
- path => $log_directory,
+ path => $log_directory,
+ hosts_allow => hiera('statistics_servers', 'stat1005.eqiad.wmnet')
}
cron { 'mw-log-cleanup':
diff --git a/modules/role/manifests/statistics/private.pp
b/modules/role/manifests/statistics/private.pp
index c2294db..b75e097 100644
--- a/modules/role/manifests/statistics/private.pp
+++ b/modules/role/manifests/statistics/private.pp
@@ -35,7 +35,7 @@
# This is so that users in the statistics-privatedata-users
# group who want to access the research slave dbs do not
# have to be in the research group, which is not included
- # in the private role (stat1002).
+ # in the private role.
mysql::config::client { 'statistics-private':
user => $::passwords::mysql::research::user,
pass => $::passwords::mysql::research::pass,
@@ -44,13 +44,13 @@
}
# Set up reportupdater to be executed on this machine.
- # Reportupdater on stat1002 launches Hadoop jobs, and
+ # Reportupdater launches Hadoop jobs, and
# the 'hdfs' user is the only 'system' user that has
# access to required files in Hadoop.
class { 'reportupdater':
base_path => "${::statistics::working_path}/reportupdater",
user => 'hdfs',
- # We know that this is included on stat1002, but unfortunetly
+ # We know that this is included, but unfortunetly
# it is done so outside of this role. Perhaps
# reportupdater should have its own role!
require => Class['cdh::hadoop'],
diff --git a/modules/udp2log/manifests/rsyncd.pp
b/modules/udp2log/manifests/rsyncd.pp
index 3749378..1c75025 100644
--- a/modules/udp2log/manifests/rsyncd.pp
+++ b/modules/udp2log/manifests/rsyncd.pp
@@ -26,9 +26,8 @@
hosts_allow => $hosts_allow;
}
- ferm::service { 'udp2log_rsyncd':
+ ferm::service { 'rsyncd':
proto => 'tcp',
port => '873',
- srange => '@resolve(stat1002.eqiad.wmnet)',
}
}
--
To view, visit https://gerrit.wikimedia.org/r/364427
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifc9fe9073ce8de47d79820dff3daf0f3d44478d5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
