Alexandros Kosiaris has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/350421 )
Change subject: ores: Add twemproxy support
......................................................................
ores: Add twemproxy support
Implement a sharding policy for ORES's redis databases. This is expected
to increase throughput as well as redundancy. The former is clear why,
the latter is due to the way ORES handles the redis databases where data
loss due to a redis db being down, while bad, is not a SPOF, at least
not directly. That is, it might lead to an ORES overload and thus to an
indirect outage, but not a direct outage. That, if needed can be solved,
for the current traffic at least, by adding more redis databases.
Twemproxy listens over unix sockets in this setup to minimize the
number of network connections required. This is due to the permissions
admittedly a bit less secure in a multi-tenant environment like scb and
needs to be revisited.
* Add a profile class profile::ores::twemproxy
* Use it on the scb role
* Populate the correct hiera data for the new profile
* Move the redis password one level up in the hierarchy so that it is
clear that it is shared by more than one profile class
Bug: T122676
Change-Id: I97f535a5e7c8eb34b9c0555d3c84d55c59703ea5
---
A hieradata/role/codfw/scb.yaml
M hieradata/role/common/ores/redis.yaml
A hieradata/role/eqiad/scb.yaml
M modules/profile/manifests/ores/redis.pp
A modules/profile/manifests/ores/twemproxy.pp
M modules/role/manifests/scb.pp
M utils/hiera_lookup
7 files changed, 58 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/21/350421/1
diff --git a/hieradata/role/codfw/scb.yaml b/hieradata/role/codfw/scb.yaml
new file mode 100644
index 0000000..14d7af1
--- /dev/null
+++ b/hieradata/role/codfw/scb.yaml
@@ -0,0 +1,6 @@
+profile::ores::twemproxy::queue_servers:
+ - 10.192.16.31:6379:1 # oresrdb2001
+ - 10.192.32.21:6379:1 # oresrdb2002
+profile::ores::twemproxy::cache_servers:
+ - 10.192.16.31:6380:1 # oresrdb2001
+ - 10.192.32.21:6380:1 # oresrdb2002
diff --git a/hieradata/role/common/ores/redis.yaml
b/hieradata/role/common/ores/redis.yaml
index cd7eba2..8267571 100644
--- a/hieradata/role/common/ores/redis.yaml
+++ b/hieradata/role/common/ores/redis.yaml
@@ -1,6 +1,6 @@
ores::redis::queue_maxmemory: 1G
ores::redis::cache_maxmemory: 6G
-#profile::ores::redis::password: actually in the private repo
+#profile::ores::redis_password: actually in the private repo
debdeploy::grains:
debdeploy-oresrdb:
value: standard
diff --git a/hieradata/role/eqiad/scb.yaml b/hieradata/role/eqiad/scb.yaml
new file mode 100644
index 0000000..4391587
--- /dev/null
+++ b/hieradata/role/eqiad/scb.yaml
@@ -0,0 +1,6 @@
+profile::ores::twemproxy::queue_servers:
+ - 10.64.48.129:6379:1 # oresrdb1001
+ - 10.64.0.10:6379:1 # oresrdb1002
+profile::ores::twemproxy::cache_servers:
+ - 10.64.48.129:6380:1 # oresrdb1001
+ - 10.64.0.10:6380:1 # oresrdb1002
diff --git a/modules/profile/manifests/ores/redis.pp
b/modules/profile/manifests/ores/redis.pp
index 8da21a2..e6bd11a 100644
--- a/modules/profile/manifests/ores/redis.pp
+++ b/modules/profile/manifests/ores/redis.pp
@@ -1,7 +1,9 @@
# Setting up ORES Redis database in a replicated way in order to facilitate
# failover if required
+# Note that the password is looked up at namespace one level up in the
hierarchy
+# as it is being reused in another profile class in that namespace
class profile::ores::redis(
- $password = hiera('profile::ores::redis::password'),
+ $password = hiera('profile::ores::redis_password'),
$redis_clients = hiera('profile::ores::redis::client_hosts'),
$slaveof = hiera('profile::ores::redis::slaveof', undef),
){
diff --git a/modules/profile/manifests/ores/twemproxy.pp
b/modules/profile/manifests/ores/twemproxy.pp
new file mode 100644
index 0000000..521f599
--- /dev/null
+++ b/modules/profile/manifests/ores/twemproxy.pp
@@ -0,0 +1,40 @@
+# Note that the password is looked up at namespace one level up in the
hierarchy
+# as it is being reused in another profile class in that namespace
+
+class profile::ores::twemproxy (
+ $password = hiera('profile::ores::redis_password'),
+ $queue_servers = hiera('profile::ores::twemproxy::queue_servers'),
+ $cache_servers = hiera('profile::ores::twemproxy::cache_servers'),
+){
+ $pools = {
+ 'queue' => {
+ auto_eject_hosts => true,
+ distribution => 'ketama',
+ redis => true,
+ redis_auth => $password,
+ hash => 'md5',
+ listen => '/var/run/nutcracker/ores_queue.sock 0666',
+ server_connections => 1,
+ server_failure_limit => 3,
+ server_retry_timeout => to_milliseconds('30s'),
+ timeout => 1000,
+ servers => $queue_servers,
+ },
+ 'cache' => {
+ auto_eject_hosts => true,
+ distribution => 'ketama',
+ redis => true,
+ redis_auth => $password,
+ hash => 'md5',
+ listen => '/var/run/nutcracker/ores_cache.sock 0666',
+ server_connections => 1,
+ server_failure_limit => 3,
+ server_retry_timeout => to_milliseconds('30s'),
+ timeout => 1000,
+ servers => $cache_servers,
+ }
+ }
+ class { '::nutcracker':
+ pools => $pools,
+ }
+}
diff --git a/modules/role/manifests/scb.pp b/modules/role/manifests/scb.pp
index 585cf34..4936a54 100644
--- a/modules/role/manifests/scb.pp
+++ b/modules/role/manifests/scb.pp
@@ -3,6 +3,7 @@
class role::scb {
include ::profile::ores::worker
include ::profile::ores::web
+ include ::profile::ores::twemproxy
include role::mobileapps
include role::mathoid
diff --git a/utils/hiera_lookup b/utils/hiera_lookup
index b37c89b..87a3273 100755
--- a/utils/hiera_lookup
+++ b/utils/hiera_lookup
@@ -76,7 +76,7 @@
# Transform roles in the form puppet expects
if scope['::roles']
- scope['_roles'] = Hash[scope['::roles'].split(',').map { |role| [role,
true] }]
+ scope['_roles'] = Hash[scope['::roles'].split(',').map { |role|
[role.gsub('role::',''), true] }]
scope.delete('::roles')
end
scope
--
To view, visit https://gerrit.wikimedia.org/r/350421
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I97f535a5e7c8eb34b9c0555d3c84d55c59703ea5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Alexandros Kosiaris <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
