Filippo Giunchedi has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/370098 )
Change subject: Reshape RESTBase Cassandra production cluster; Provision new
3.x cluster
......................................................................
Reshape RESTBase Cassandra production cluster; Provision new 3.x cluster
Basic steps:
- 9T of storage will be freed
- restbase2001.codfw.wmnet is decommissioned
- remove/depool restbase2001 from LVS
- this changeset is merged
- restbase2001.codfw.wmnet is re-imaged
- restbase2001.codfw.wmnet is bootstrapped
- new changeset to comment/uncomment instances is merged
- lather
- rinse
- repeat
Bug: T169939
Change-Id: Ife09538ec52e8f48b8d0899d81b1c961a8476b94
---
M hieradata/role/codfw/restbase/production.yaml
M hieradata/role/common/restbase/production.yaml
A hieradata/role/common/restbase/production_ng.yaml
M hieradata/role/eqiad/restbase/production.yaml
M manifests/site.pp
M modules/profile/manifests/restbase.pp
A modules/role/manifests/restbase/production_ng.pp
7 files changed, 167 insertions(+), 30 deletions(-)
Approvals:
jenkins-bot: Verified
Filippo Giunchedi: Looks good to me, approved
diff --git a/hieradata/role/codfw/restbase/production.yaml
b/hieradata/role/codfw/restbase/production.yaml
index 507267a..3357158 100644
--- a/hieradata/role/codfw/restbase/production.yaml
+++ b/hieradata/role/codfw/restbase/production.yaml
@@ -1,13 +1,12 @@
profile::restbase::seeds:
- - restbase2001-a.codfw.wmnet
- - restbase2001-b.codfw.wmnet
- - restbase2001-c.codfw.wmnet
- - restbase2002-a.codfw.wmnet
- - restbase2002-b.codfw.wmnet
- - restbase2002-c.codfw.wmnet
+ - restbase2002-a.codfw.wmnet
+ - restbase2002-b.codfw.wmnet
+ - restbase2002-c.codfw.wmnet
+ - restbase2007-a.codfw.wmnet
+ - restbase2007-b.codfw.wmnet
+ - restbase2007-c.codfw.wmnet
# used for rate limiting
profile::restbase::hosts:
- - restbase2001.codfw.wmnet
- restbase2002.codfw.wmnet
- restbase2003.codfw.wmnet
- restbase2004.codfw.wmnet
diff --git a/hieradata/role/common/restbase/production.yaml
b/hieradata/role/common/restbase/production.yaml
index 82f3b2b..ec7ff16 100644
--- a/hieradata/role/common/restbase/production.yaml
+++ b/hieradata/role/common/restbase/production.yaml
@@ -23,6 +23,11 @@
profile::restbase::monitor_domain: en.wikipedia.org
profile::restbase::cassandra_tls:
ca: /dev/null
+profile::restbase::seeds_ng:
+ - restbase2001-a.codfw.wmnet
+ # - restbase2001-b.codfw.wmnet
+ # - restbase2001-c.codfw.wmnet
+
#
# Cassandra
#
@@ -148,13 +153,13 @@
listen_address: 10.64.48.99
c:
listen_address: 10.64.48.100
- "restbase2001.codfw.wmnet":
- a:
- listen_address: 10.192.16.162
- b:
- listen_address: 10.192.16.163
- c:
- listen_address: 10.192.16.164
+ # "restbase2001.codfw.wmnet":
+ # a:
+ # listen_address: 10.192.16.162
+ # b:
+ # listen_address: 10.192.16.163
+ # c:
+ # listen_address: 10.192.16.164
"restbase2002.codfw.wmnet":
a:
listen_address: 10.192.16.165
diff --git a/hieradata/role/common/restbase/production_ng.yaml
b/hieradata/role/common/restbase/production_ng.yaml
new file mode 100644
index 0000000..72e5241
--- /dev/null
+++ b/hieradata/role/common/restbase/production_ng.yaml
@@ -0,0 +1,114 @@
+cluster: restbase
+
+#
+# Cassandra
+#
+graphite_host: 'graphite1003.eqiad.wmnet'
+profile::cassandra::allow_analytics: false
+
+profile::cassandra::metrics_whitelist:
+ -
.*\.metrics\.ColumnFamily\.local_group_.*\.meta\.CoordinatorReadLatency\..*$
+ -
.*\.metrics\.ColumnFamily\.local_group_.*\.meta\.CoordinatorScanLatency\..*$
+ -
.*\.metrics\.ColumnFamily\.local_group_.*\.meta\.EstimatedColumnCountHistogram\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.EstimatedRowCount\..*$
+ -
.*\.metrics\.ColumnFamily\.local_group_.*\.meta\.EstimatedRowSizeHistogram\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.LiveDiskSpaceUsed\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.LiveSSTableCount\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.LiveScannedHistogram\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.MaxRowSize\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.MeanRowSize\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.MinRowSize\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.PendingCompactions\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.PendingFlushes\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.RangeLatency\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.ReadLatency\..*$
+ -
.*\.metrics\.ColumnFamily\.local_group_.*\.meta\.SSTablesPerReadHistogram\..*$
+ -
.*\.metrics\.ColumnFamily\.local_group_.*\.meta\.TombstoneScannedHistogram\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.TotalDiskSpaceUsed\..*$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\.WriteLatency\..*$
+profile::cassandra::metrics_blacklist:
+ - .*\.15MinuteRate$
+ - .*\.5MinuteRate$
+ - .*\.98percentile$
+ - .*\.999percentile$
+ - .*\.mean$
+ - .*\.meanRate$
+ - .*\.min$
+ - .*\.stddev$
+ - .*\.metrics\.ColumnFamily\.local_group_.*\.meta\..*$
+ - .*\.metrics\.Keyspace\..*$
+ - .*\.metrics\.Client\..*$
+ - .*\.metrics\.Table\..*$
+## Instances
+profile::cassandra::instances:
+ # "restbase1007.eqiad.wmnet":
+ # a:
+ # listen_address: 10.64.0.230
+ # b:
+ # listen_address: 10.64.0.231
+ # c:
+ # listen_address: 10.64.0.232
+ # "restbase1008.eqiad.wmnet":
+ # a:
+ # listen_address: 10.64.32.187
+ # b:
+ # listen_address: 10.64.32.195
+ # c:
+ # listen_address: 10.64.32.196
+ # "restbase1009.eqiad.wmnet":
+ # a:
+ # listen_address: 10.64.48.120
+ # b:
+ # listen_address: 10.64.48.130
+ # c:
+ # listen_address: 10.64.48.131
+ "restbase2001.codfw.wmnet":
+ a:
+ listen_address: 10.192.16.162
+ b:
+ listen_address: 10.192.16.163
+ c:
+ listen_address: 10.192.16.164
+ # "restbase2003.codfw.wmnet":
+ # a:
+ # listen_address: 10.192.32.134
+ # b:
+ # listen_address: 10.192.32.135
+ # c:
+ # listen_address: 10.192.32.136
+ # "restbase2005.codfw.wmnet":
+ # a:
+ # listen_address: 10.192.48.46
+ # b:
+ # listen_address: 10.192.48.47
+ # c:
+ # listen_address: 10.192.48.48
+
+
+profile::cassandra::settings:
+ start_rpc: 'false'
+ max_heap_size: 12g
+ # 1/4 heap size, no more than 100m/thread
+ heap_newsize: 2048m
+ compaction_throughput_mb_per_sec: 20
+ concurrent_compactors: 10
+ concurrent_writes: 18
+ concurrent_reads: 18
+ tls_cluster_name: "restbase"
+ internode_encryption: dc
+ dc: "%{::site}"
+ cluster_name: "restbase"
+ application_username: restb
+ application_password: "%{passwords::cassandra::restbase}"
+ target_version: '3.x'
+ permissions_validity_in_ms: 600000
+ trickle_fsync: false
+ client_encryption_enabled: true
+
+
+#
+# General
+#
+admin::groups:
+ - restbase-roots
+ - restbase-admins
diff --git a/hieradata/role/eqiad/restbase/production.yaml
b/hieradata/role/eqiad/restbase/production.yaml
index 5d2d041..7ba16e0 100644
--- a/hieradata/role/eqiad/restbase/production.yaml
+++ b/hieradata/role/eqiad/restbase/production.yaml
@@ -1,18 +1,18 @@
profile::restbase::seeds:
- - restbase1007-a.eqiad.wmnet
- - restbase1007-b.eqiad.wmnet
- - restbase1008-a.eqiad.wmnet
- - restbase1008-b.eqiad.wmnet
- - restbase1009-a.eqiad.wmnet
- - restbase1009-b.eqiad.wmnet
- - restbase1010-a.eqiad.wmnet
- - restbase1010-b.eqiad.wmnet
- - restbase1011-a.eqiad.wmnet
- - restbase1011-b.eqiad.wmnet
- - restbase1012-a.eqiad.wmnet
- - restbase1012-b.eqiad.wmnet
- - restbase1013-a.eqiad.wmnet
- - restbase1013-b.eqiad.wmnet
+ - restbase1007-a.eqiad.wmnet
+ - restbase1007-b.eqiad.wmnet
+ - restbase1008-a.eqiad.wmnet
+ - restbase1008-b.eqiad.wmnet
+ - restbase1009-a.eqiad.wmnet
+ - restbase1009-b.eqiad.wmnet
+ - restbase1010-a.eqiad.wmnet
+ - restbase1010-b.eqiad.wmnet
+ - restbase1011-a.eqiad.wmnet
+ - restbase1011-b.eqiad.wmnet
+ - restbase1012-a.eqiad.wmnet
+ - restbase1012-b.eqiad.wmnet
+ - restbase1013-a.eqiad.wmnet
+ - restbase1013-b.eqiad.wmnet
# used for rate limiting
profile::restbase::hosts:
- restbase1007.eqiad.wmnet
diff --git a/manifests/site.pp b/manifests/site.pp
index 64f9d64..d39abfb 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2027,15 +2027,19 @@
}
# restbase eqiad cluster
-node /^restbase10[01][0-9]\.eqiad\.wmnet$/ {
+node /^restbase10(0[789]|1[012345678])\.eqiad\.wmnet$/ {
role(restbase::production)
}
# restbase codfw cluster
-node /^restbase20[01][0-9]\.codfw\.wmnet$/ {
+node /^restbase20(0[23456789]|1[012])\.codfw\.wmnet$/ {
role(restbase::production)
}
+node /^restbase2001\.codfw\.wmnet$/ {
+ role(restbase::production_ng)
+}
+
# cassandra multi-dc temporary test T111382
node /^restbase-test200[1-3]\.codfw\.wmnet$/ {
role(restbase::test_cluster)
diff --git a/modules/profile/manifests/restbase.pp
b/modules/profile/manifests/restbase.pp
index 1df7ce3..37d4a38 100644
--- a/modules/profile/manifests/restbase.pp
+++ b/modules/profile/manifests/restbase.pp
@@ -13,6 +13,9 @@
# [*seeds*]
# Array of cassandra hosts (IP or host names) to contact.
#
+# [*seeds_ng*]
+# Array of cassandra hosts (IP or host names) to contact (next-gen storage
module).
+#
# [*cassandra_local_dc*]
# Which DC should be considered local.
#
@@ -73,6 +76,7 @@
$cassandra_user = hiera('profile::restbase::cassandra_user'),
$cassandra_password = hiera('profile::restbase::cassandra_password'),
$seeds = hiera('profile::restbase::seeds'),
+ $seeds_ng = hiera('profile::restbase::seeds_ng', []),
$hosts = hiera('profile::restbase::hosts'),
$cassandra_local_dc = hiera('profile::restbase::cassandra_local_dc'),
$cassandra_datacenters = hiera('profile::restbase::cassandra_datacenters'),
@@ -113,6 +117,7 @@
ipaddress => $::ipaddress,
rl_seeds => reject(reject($hosts, $::hostname),
$::ipaddress),
seeds => $seeds,
+ seeds_ng => $seeds_ng,
cassandra_local_dc => $cassandra_local_dc,
cassandra_datacenters => $cassandra_datacenters,
cassandra_user => $cassandra_user,
diff --git a/modules/role/manifests/restbase/production_ng.pp
b/modules/role/manifests/restbase/production_ng.pp
new file mode 100644
index 0000000..7e6a20c
--- /dev/null
+++ b/modules/role/manifests/restbase/production_ng.pp
@@ -0,0 +1,10 @@
+# == Class role::restbase::production_ng
+#
+# Configures the production cluster (next-gen)
+class role::restbase::production_ng {
+ include ::passwords::cassandra
+ include ::base::firewall
+ include ::standard
+ include ::profile::cassandra
+ system::role { 'restbase': description => 'Restbase (Cassandra 3.x-only)' }
+}
--
To view, visit https://gerrit.wikimedia.org/r/370098
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ife09538ec52e8f48b8d0899d81b1c961a8476b94
Gerrit-PatchSet: 18
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Eevans <[email protected]>
Gerrit-Reviewer: Dzahn <[email protected]>
Gerrit-Reviewer: Eevans <[email protected]>
Gerrit-Reviewer: Filippo Giunchedi <[email protected]>
Gerrit-Reviewer: Giuseppe Lavagetto <[email protected]>
Gerrit-Reviewer: Mobrovac <[email protected]>
Gerrit-Reviewer: Ppchelko <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
