[MediaWiki-commits] [Gerrit] operations/puppet[production]: contint: Lower caching length on doc.wikimedia.org

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403401 )

Change subject: contint: Lower caching length on doc.wikimedia.org
..


contint: Lower caching length on doc.wikimedia.org

These files are auto-generated and published by Jenkins, but often older
versions remain cached by varnish.

Apache will now send a Cache-Control header to tell varnish to only cache the
content for an hour. Since these are mostly static HTML/JS/CSS with a few PHP
scripts, there shouldn't be a performance impact.

Bug: T184255
Change-Id: I4092bb26007cba53157d29557fc1902fd746c055
---
M modules/contint/templates/apache/doc.wikimedia.org.erb
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  Ema: Looks good to me, but someone else must approve
  Hashar: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/contint/templates/apache/doc.wikimedia.org.erb 
b/modules/contint/templates/apache/doc.wikimedia.org.erb
index 889e3ea..d811431 100644
--- a/modules/contint/templates/apache/doc.wikimedia.org.erb
+++ b/modules/contint/templates/apache/doc.wikimedia.org.erb
@@ -20,6 +20,9 @@
 RewriteCond %{REQUEST_FILENAME} -d
 RewriteRule ^(.+[^/])$ https://doc.wikimedia.org/$1/ [R=301,QSA]
 
+# Lower caching length (T184255)
+Header set Cache-Control "s-maxage=3600, must-revalidate, max-age=0"
+
 # Back-compat for T73060: Redirect mediawiki-core/master/php/html/ to 
mediawiki-core/master/php/
 RewriteRule ^mediawiki-core/master/php/html/(.*)$ 
https://doc.wikimedia.org/mediawiki-core/master/php/$1 [L,QSA]
 

-- 
To view, visit https://gerrit.wikimedia.org/r/403401
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4092bb26007cba53157d29557fc1902fd746c055
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Legoktm 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Hashar 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...conftool[master]: Refactor conftool.action

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405303 )

Change subject: Refactor conftool.action
..

Refactor conftool.action

Split the Action class, which had grown out of proportion, to a separate
list of smaller Action classes.

Change-Id: I6cccb9e62887aa9151ac2ede50e923689cfe57b2
---
M conftool/action.py
M conftool/cli/tool.py
M conftool/tests/unit/test_action.py
3 files changed, 77 insertions(+), 63 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/software/conftool 
refs/changes/03/405303/1

diff --git a/conftool/action.py b/conftool/action.py
index 1870435..76d3c84 100644
--- a/conftool/action.py
+++ b/conftool/action.py
@@ -12,30 +12,60 @@
 pass
 
 
-class Action(object):
+def get_action(obj, act):
+prefix = act[:3]
+if prefix == 'get':
+return GetAction(obj)
+elif act == 'delete':
+return DelAction(obj)
+elif prefix == 'set':
+return SetAction(obj, act[4:])
+else:
+raise ActionError("Could not parse action %s" % act)
+
+
+class GetAction(object):
+"""Action to perform when a get request is involved"""
+def __init__(self, obj):
+self.entity = obj
+
+def run(self):
+self.entity.fetch()
+if self.entity.exists:
+return str(self.entity)
+else:
+return "%s not found" % self.entity.name
+
+
+class DelAction(GetAction):
+"""Action to perform when deleting an object"""
+def run(self):
+self.entity.delete()
+entity_type = self.entity.__class__.__name__,
+return "Deleted %s %s." % (entity_type,
+   self.entity.name)
+
+
+class SetAction(object):
 
 def __init__(self, obj, act):
+"""Action to perform when editing an object"""
 self.entity = obj
-self.action, self.args = self._parse_action(act)
+if not self.entity.exists:
+raise ActionError("Entity %s doesn't exist" % self.entity.name)
+
+self.args = self._parse_action(act)
 self.description = ""
 
-def _parse_action(self, act):
-# TODO: Move this to the cli.tool submodule
+def _parse_action(self, arg):
 # TODO: make the parsing of the argument a bit more formal
-if act.startswith('get'):
-return ('get', None)
-elif act.startswith('delete'):
-return ('delete', None)
-elif not act.startswith('set/'):
-raise ActionError("Cannot parse action %s" % act)
-set_arg = act.replace('set/', '', 1)
-if set_arg.startswith('@'):
-return ('set', self._from_file(set_arg))
+if arg.startswith('@'):
+return self._from_file(arg)
 try:
-values = dict((el.strip().split('=')) for el in set_arg.split(':'))
+values = dict((el.strip().split('=')) for el in arg.split(':'))
 except Exception:
-raise ActionError("Could not parse set instructions: %s" % set_arg)
-return ('set', self._from_cli(values))
+raise ActionError("Could not parse set instructions: %s" % arg)
+return self._from_cli(values)
 
 def _from_cli(self, values):
 for k, v in values.items():
@@ -70,33 +100,19 @@
 return values
 
 def run(self):
-if self.action == 'get':
-self.entity.fetch()
-if self.entity.exists:
-return str(self.entity)
-else:
-return "%s not found" % self.entity.name
-elif self.action == 'delete':
-self.entity.delete()
-entity_type = self.entity.__class__.__name__,
-return "Deleted %s %s." % (entity_type,
-   self.entity.name)
-elif self.action == 'set':
-if not self.entity.exists:
-raise ActionError("Entity %s doesn't exist" % self.entity.name)
-# Validate the new data *before* updating the object
-try:
-self.entity.validate(self.args)
-except Exception as e:
-raise ActionValidationError("The provided data is not valid: 
%s" % e)
+# Validate the new data *before* updating the object
+try:
+self.entity.validate(self.args)
+except Exception as e:
+raise ActionValidationError("The provided data is not valid: %s" % 
e)
 
-desc = []
-for (k, v) in self.args.items():
-curval = getattr(self.entity, k)
-if v != curval:
-msg = "%s: %s changed %s => %s" % (
-self.entity.name, k,
-curval, v)
-desc.append(msg)
-self.entity.update(self.args)
-return "\n".join(desc)
+desc = []
+for (k, v) in self.args.items():
+

[MediaWiki-commits] [Gerrit] operations...conftool[master]: cli.tool: drop the "find" interface

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405301 )

Change subject: cli.tool: drop the "find" interface
..

cli.tool: drop the "find" interface

Since its function is fully covered by 'select', which ends up being
both more powerful and equally efficient, let's drop it.

Change-Id: I62633d7e180efd34d7f468c4b71e3ed7220a0f35
---
M conftool/cli/tool.py
M conftool/tests/integration/test_tool.py
M conftool/tests/unit/test_cli_tool.py
M setup.py
4 files changed, 54 insertions(+), 108 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/software/conftool 
refs/changes/01/405301/1

diff --git a/conftool/cli/tool.py b/conftool/cli/tool.py
index 18f5e01..32abf34 100644
--- a/conftool/cli/tool.py
+++ b/conftool/cli/tool.py
@@ -27,13 +27,22 @@
 pass
 
 
-class ToolCli(object):
+class ToolCliBase(object):
 
 def __init__(self, args):
 self.args = args
 self._load_schema()
-self._tags = self.args.taglist.split(',')
 self.irc = logging.getLogger('conftool.announce')
+
+def tags(self):
+return []
+
+def announce(self):
+if self._action != 'get' and not self.args.quiet:
+self.irc.warning(
+"conftool action : %s; selector: %s", self._action,
+self._namedef
+)
 
 def _load_schema(self):
 self._schema = loader.Schema.from_file(self.args.schema)
@@ -50,6 +59,42 @@
 c = configuration.get(self.args.config)
 KVObject.setup(c)
 setup_irc(c)
+
+def _run_action(self):
+fail = False
+for obj in self.host_list():
+try:
+a = action.Action(obj, self._action)
+msg = a.run()
+except action.ActionError as e:
+fail = True
+_log.error("Invalid action, reason: %s", str(e))
+except BackendError as e:
+fail = True
+_log.error("Error when trying to %s on %s", self._action,
+   self._namedef)
+_log.error("Failure writing to the kvstore: %s", str(e))
+except Exception as e:
+fail = True
+_log.error("Error when trying to %s on %s", self._action,
+   self._namedef)
+_log.exception("Generic action failure: %s", str(e))
+else:
+if sys.version_info[0] == 2:  # Python 2
+msg = msg.decode('utf-8')
+print(msg)
+if not fail:
+self.announce()
+return True
+else:
+return False
+
+
+class ToolCli(ToolCliBase):
+
+def __init__(self, args):
+super(ToolCli, self).__init__(args)
+self._tags = self.args.taglist.split(',')
 
 @property
 def tags(self):
@@ -112,35 +157,6 @@
 self._action, self._namedef = unit
 return self._run_action()
 
-def _run_action(self):
-fail = False
-for obj in self.host_list():
-try:
-a = action.Action(obj, self._action)
-msg = a.run()
-except action.ActionError as e:
-fail = True
-_log.error("Invalid action, reason: %s", str(e))
-except BackendError as e:
-fail = True
-_log.error("Error when trying to %s on %s", self._action,
-   self._namedef)
-_log.error("Failure writing to the kvstore: %s", str(e))
-except Exception as e:
-fail = True
-_log.error("Error when trying to %s on %s", self._action,
-   self._namedef)
-_log.exception("Generic action failure: %s", str(e))
-else:
-if sys.version_info[0] == 2:  # Python 2
-msg = msg.decode('utf-8')
-print(msg)
-if not fail:
-self.announce()
-return True
-else:
-return False
-
 @staticmethod
 def raise_warning():
 if not sys.stdin.isatty() or not sys.stdout.isatty():
@@ -158,30 +174,7 @@
 sys.exit(1)
 
 
-class ToolCliFind(ToolCli):
-"""Subclass used for the --find mode"""
-def __init__(self, args):
-self.args = args
-self._load_schema()
-self.irc = logging.getLogger('conftool.announce')
-
-@property
-def tags(self):
-return []
-
-def host_list(self):
-for o in self.entity.find(self._namedef):
-yield o
-
-def announce(self):
-if self._action != 'get' and not self.args.quiet:
-self.irc.warning(
-"conftool action : %s; selector: %s", self._action,
-self._namedef
-)
-
-
-class ToolCliByLabel(ToolCliFind):
+class ToolCliByLabel(ToolCliBase):
 

[MediaWiki-commits] [Gerrit] operations...conftool[master]: Add preemptive validation.

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405302 )

Change subject: Add preemptive validation.
..

Add preemptive validation.

When writing to the backend, let's check that all the new values are
valid before writing.

Bug: T185080
Change-Id: I17c8e5cebd71369e27f78f05db1aaac31165467c
---
M conftool/action.py
M conftool/cli/syncer.py
M conftool/kvobject.py
M conftool/tests/integration/__init__.py
M conftool/tests/unit/test_kvobject.py
5 files changed, 33 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/software/conftool 
refs/changes/02/405302/1

diff --git a/conftool/action.py b/conftool/action.py
index 92313ae..1870435 100644
--- a/conftool/action.py
+++ b/conftool/action.py
@@ -8,6 +8,10 @@
 pass
 
 
+class ActionValidationError(ActionError):
+pass
+
+
 class Action(object):
 
 def __init__(self, obj, act):
@@ -80,6 +84,12 @@
 elif self.action == 'set':
 if not self.entity.exists:
 raise ActionError("Entity %s doesn't exist" % self.entity.name)
+# Validate the new data *before* updating the object
+try:
+self.entity.validate(self.args)
+except Exception as e:
+raise ActionValidationError("The provided data is not valid: 
%s" % e)
+
 desc = []
 for (k, v) in self.args.items():
 curval = getattr(self.entity, k)
diff --git a/conftool/cli/syncer.py b/conftool/cli/syncer.py
index 2debe94..fc3b430 100644
--- a/conftool/cli/syncer.py
+++ b/conftool/cli/syncer.py
@@ -135,6 +135,7 @@
 obj = self.cls(*tags)
 if obj.static_values:
 _log.info("Syncing static object %s:%s", self.entity, key)
+obj.validate(self.data[key])
 obj.from_net(self.data[key])
 else:
 if obj.exists:
diff --git a/conftool/kvobject.py b/conftool/kvobject.py
index 45ab060..e45ec5f 100644
--- a/conftool/kvobject.py
+++ b/conftool/kvobject.py
@@ -121,6 +121,20 @@
 self._set_value(k, self._schema[k], {k: v}, set_defaults=False)
 self.write()
 
+def validate(self, values):
+"""
+Validate a set of proposed values against the schema.
+Returns True on success, raises an exception otherwise
+"""
+for k, v in values.items():
+if k not in self._schema:
+if self.strict_schema:
+raise TypeError("Key %s not in the schema" % k)
+continue
+validator = self._schema[k]
+validator(v)
+return True
+
 @classmethod
 def from_yaml(cls, data):
 if cls.static_values:
@@ -232,6 +246,7 @@
 General-purpose entity with a strict schema
 """
 depends = []
+strict_schema = True
 
 def __init__(self, *tags):
 if len(tags) != (len(self._tags) + 1):
@@ -266,6 +281,7 @@
 
 
 class FreeSchemaEntity(Entity):
+strict_schema = False
 
 def __init__(self, *tags, **kwargs):
 self._schemaless = kwargs
diff --git a/conftool/tests/integration/__init__.py 
b/conftool/tests/integration/__init__.py
index a207875..4dc3a8b 100644
--- a/conftool/tests/integration/__init__.py
+++ b/conftool/tests/integration/__init__.py
@@ -60,6 +60,7 @@
 def stop(self):
 self.proc.kill()
 self.proc = None
+time.sleep(2)
 
 
 class IntegrationTestBase(unittest.TestCase):
diff --git a/conftool/tests/unit/test_kvobject.py 
b/conftool/tests/unit/test_kvobject.py
index 494c810..d503ab6 100644
--- a/conftool/tests/unit/test_kvobject.py
+++ b/conftool/tests/unit/test_kvobject.py
@@ -133,6 +133,11 @@
   set_defaults=False)
 self.entity.write.assert_called_with()
 
+def test_validate(self):
+self.assertTrue(self.entity.validate({'a': 1, 'b': 'testtest'}))
+self.assertRaises(TypeError, self.entity.validate, {'a': 1, 'b': 
'testtest', 'c': True})
+self.assertRaises(ValueError, self.entity.validate, {'a': 'test'})
+
 def test_to_net(self):
 self.entity.a = 100
 self.entity.b = 'meoow'

-- 
To view, visit https://gerrit.wikimedia.org/r/405302
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I17c8e5cebd71369e27f78f05db1aaac31165467c
Gerrit-PatchSet: 1
Gerrit-Project: operations/software/conftool
Gerrit-Branch: master
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: decommission mw1209-1220

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404500 )

Change subject: site.pp: decommission mw1209-1220
..


site.pp: decommission mw1209-1220

Bug: T185004
Change-Id: Icbe8b6201f476f3e2b0e4fcae610e6179639128e
---
M conftool-data/node/eqiad.yaml
M hieradata/common/scap/dsh.yaml
M manifests/site.pp
3 files changed, 3 insertions(+), 14 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index d94acc1..c6705e2 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -73,17 +73,6 @@
 mw1347.eqiad.wmnet: [apache2,nginx]
 mw1348.eqiad.wmnet: [apache2,nginx]
   appserver:
-mw1209.eqiad.wmnet: [apache2,nginx]
-mw1210.eqiad.wmnet: [apache2,nginx]
-mw1211.eqiad.wmnet: [apache2,nginx]
-mw1212.eqiad.wmnet: [apache2,nginx]
-mw1213.eqiad.wmnet: [apache2,nginx]
-mw1214.eqiad.wmnet: [apache2,nginx]
-mw1215.eqiad.wmnet: [apache2,nginx]
-mw1216.eqiad.wmnet: [apache2,nginx]
-mw1218.eqiad.wmnet: [apache2,nginx]
-mw1219.eqiad.wmnet: [apache2,nginx]
-mw1220.eqiad.wmnet: [apache2,nginx]
 mw1238.eqiad.wmnet: [apache2,nginx]
 mw1239.eqiad.wmnet: [apache2,nginx]
 mw1240.eqiad.wmnet: [apache2,nginx]
diff --git a/hieradata/common/scap/dsh.yaml b/hieradata/common/scap/dsh.yaml
index 3626e3b..0db4625 100644
--- a/hieradata/common/scap/dsh.yaml
+++ b/hieradata/common/scap/dsh.yaml
@@ -2,8 +2,8 @@
 #
 scap::dsh::scap_proxies:
   - "mw1280.eqiad.wmnet" # A7 eqiad
-  - "mw1211.eqiad.wmnet" # B7 eqiad
-  - "mw1216.eqiad.wmnet" # B8 eqiad
+  - "mw1284.eqiad.wmnet" # B7 eqiad
+  - "mw1313.eqiad.wmnet" # B8 eqiad
   - "mw1319.eqiad.wmnet" # C6 eqiad
   - "mw1250.eqiad.wmnet" # D5 eqiad
   - "mw2117.codfw.wmnet" # B3 codfw
diff --git a/manifests/site.pp b/manifests/site.pp
index 49ac858..3a27719 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1451,7 +1451,7 @@
 
 # mw1209-1216, 1218-1220 are apaches
 node /^mw12(09|1[012345689]|20)\.eqiad\.wmnet$/ {
-role(mediawiki::appserver)
+role(spare::system)
 }
 
 # mw1259-60 are videoscalers

-- 
To view, visit https://gerrit.wikimedia.org/r/404500
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icbe8b6201f476f3e2b0e4fcae610e6179639128e
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: decommission mw1201-1208

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404499 )

Change subject: site.pp: decommission mw1201-1208
..


site.pp: decommission mw1201-1208

Bug: T185004
Change-Id: I2a3afd3fb6e820076d3f2531f641a7861c335a7a
---
M conftool-data/node/eqiad.yaml
M hieradata/common/scap/dsh.yaml
M manifests/site.pp
3 files changed, 2 insertions(+), 10 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  Elukey: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index fcac9ea..d94acc1 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -26,14 +26,6 @@
 mw1336.eqiad.wmnet: [apache2,nginx]
 mw1337.eqiad.wmnet: [apache2,nginx]
   api_appserver:
-mw1201.eqiad.wmnet: [apache2,nginx]
-mw1202.eqiad.wmnet: [apache2,nginx]
-mw1203.eqiad.wmnet: [apache2,nginx]
-mw1204.eqiad.wmnet: [apache2,nginx]
-mw1205.eqiad.wmnet: [apache2,nginx]
-mw1206.eqiad.wmnet: [apache2,nginx]
-mw1207.eqiad.wmnet: [apache2,nginx]
-mw1208.eqiad.wmnet: [apache2,nginx]
 mw1221.eqiad.wmnet: [apache2,nginx]
 mw1222.eqiad.wmnet: [apache2,nginx]
 mw1223.eqiad.wmnet: [apache2,nginx]
diff --git a/hieradata/common/scap/dsh.yaml b/hieradata/common/scap/dsh.yaml
index aecfa1a..3626e3b 100644
--- a/hieradata/common/scap/dsh.yaml
+++ b/hieradata/common/scap/dsh.yaml
@@ -5,7 +5,7 @@
   - "mw1211.eqiad.wmnet" # B7 eqiad
   - "mw1216.eqiad.wmnet" # B8 eqiad
   - "mw1319.eqiad.wmnet" # C6 eqiad
-  - "mw1201.eqiad.wmnet" # D5 eqiad
+  - "mw1250.eqiad.wmnet" # D5 eqiad
   - "mw2117.codfw.wmnet" # B3 codfw
   - "mw2187.codfw.wmnet" # C4 codfw
   - "mw2215.codfw.wmnet" # A3 codfw
diff --git a/manifests/site.pp b/manifests/site.pp
index 459042e..49ac858 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1446,7 +1446,7 @@
 
 # mw1201-1208 are api apaches
 node /^mw120[1-8]\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::api)
+role(spare::system)
 }
 
 # mw1209-1216, 1218-1220 are apaches

-- 
To view, visit https://gerrit.wikimedia.org/r/404499
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2a3afd3fb6e820076d3f2531f641a7861c335a7a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: reorganize appservers in eqiad by function/row

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404453 )

Change subject: site.pp: reorganize appservers in eqiad by function/row
..


site.pp: reorganize appservers in eqiad by function/row

This should make it easier for users to find out which machines have
what role, and where they're located.

Change-Id: I073c00c1ac749ac5dc1239e0853e560044c6c177
---
M manifests/site.pp
1 file changed, 92 insertions(+), 49 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  Elukey: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index d42c43d..459042e 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1432,11 +1432,17 @@
 role(swift::storage)
 }
 
-# mwdebug servers are for mediawiki testing/debugging
-# They replace mw1017 and mw1099
+
+## MEDIAWIKI APPLICATION SERVERS
+
+## DATACENTER: EQIAD
+
+# Debug servers
 node /^mwdebug100[12]\.eqiad\.wmnet$/ {
 role(mediawiki::canary_appserver)
 }
+
+# Hosts to decommission (if any)
 
 # mw1201-1208 are api apaches
 node /^mw120[1-8]\.eqiad\.wmnet$/ {
@@ -1448,93 +1454,128 @@
 role(mediawiki::appserver)
 }
 
-#mw1221-mw1235 are api apaches
-node /^mw12(2[1-9]|3[0-5])\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::api)
-}
-
-#mw1238-mw1258 are apaches
-node /^mw12(3[8-9]|4[0-9]|5[0-8])\.eqiad\.wmnet$/ {
-role(mediawiki::appserver)
-}
-
-#mw1259-60 are videoscalers
+# mw1259-60 are videoscalers
 node /^mw12(59|60)\.eqiad\.wmnet/ {
 role(mediawiki::videoscaler)
 }
 
-# ROW A eqiad appservers
-#mw1261 - mw1275
+# Appservers (serving normal website traffic)
 
+# Row A
+
+# mw1261 - mw1275 are in rack A7
 node /^mw126[1-5]\.eqiad\.wmnet$/ {
 role(mediawiki::canary_appserver)
 }
-
 node /^mw12(6[6-9]|7[0-5])\.eqiad\.wmnet$/ {
 role(mediawiki::appserver)
 }
 
-# ROW A eqiad api appserver
-# mw1276 - mw1290
-node /^mw127[6-9]\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::canary_api)
-}
+# Row C
 
-node /^mw12(8[0-9]|90)\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::api)
-}
-
-# ROW A eqiad imagescalers
-node /^mw129[3-8]\.eqiad\.wmnet$/ {
-role(mediawiki::imagescaler)
-}
-
-# ROW A eqiad jobrunners
-node /^mw1(299|30[0-6])\.eqiad\.wmnet$/ {
-role(mediawiki::jobrunner)
-}
-
-# ROW C eqiad jobrunners
-node /^mw133[4-7]\.eqiad\.wmnet$/ {
-role(mediawiki::jobrunner)
-}
-
-# T165519
-# ROW C eqiad appservers
+# mw1319-33 are in rack C6
 node /^mw13(19|2[0-9]|3[0-3])\.eqiad\.wmnet$/ {
 role(mediawiki::appserver)
 }
 
-# ROW B eqiad api-appservers
-node /^mw13(1[2-7])\.eqiad\.wmnet$/ {
+# Row D
+
+#mw1238-mw1258 are in rack D5
+node /^mw12(3[8-9]|4[0-9]|5[0-8])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver)
+}
+
+# API (serving api traffic)
+
+# Row A
+
+# mw1276 - mw1283 are in rack A7
+node /^mw127[6-9]\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::canary_api)
+}
+node /^mw128[0-3]\.eqiad\.wmnet$/ {
 role(mediawiki::appserver::api)
 }
 
-# ROW C api-appservers
+# mw1312 is in rack A6
+node 'mw1312.eqiad.wmnet' {
+role(mediawiki::appserver::api)
+}
+
+# Row B
+
+# mw1284-1290 are in rack B6
+node /^mw12(8[4-9]|90)\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+# mw1313-17 are in rack B7
+node /^mw13(1[3-7])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+# Row C
+
+# mw1339-48 are in rack C6
 node /^mw13(39|4[0-8])\.eqiad\.wmnet$/ {
 role(mediawiki::appserver::api)
 }
 
-# ROW A eqiad jobrunners
+# Row D
+# mw1221-mw1235 are in rack D5
+node /^mw12(2[1-9]|3[0-5])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+
+# Imagescalers (mostly obsolete functionality, replaced by thumbor)
+
+# Row B (B6)
+node /^mw129[3-8]\.eqiad\.wmnet$/ {
+role(mediawiki::imagescaler)
+}
+
+# Jobrunners (now mostly used via changepropagation as a LVS endpoint)
+
+# Row A
+
+# mw1308-mw1311 are in rack A6
 node /^mw13(0[89]|1[01])\.eqiad\.wmnet$/ {
 role(mediawiki::jobrunner)
 }
 
-# ROW A videoscaler
+# Row B
+
+# mw1299-mw1306 are in rack B6
+node /^mw1(299|30[0-6])\.eqiad\.wmnet$/ {
+role(mediawiki::jobrunner)
+}
+
+# Row C
+
+# mw1334-mw1337 are in rack C6
+node /^mw133[4-7]\.eqiad\.wmnet$/ {
+role(mediawiki::jobrunner)
+}
+
+# Videoscalers
+
+# Row A (A6)
 node 'mw1307.eqiad.wmnet' {
 role(mediawiki::videoscaler)
 }
 
-# ROW B videoscaler
+# Row B (B7)
 node 'mw1318.eqiad.wmnet' {
 role(mediawiki::videoscaler)
 }
 
-# ROW C videoscaler
+# Row C (C6)
 node 'mw1338.eqiad.wmnet' {
 role(mediawiki::videoscaler)
 }
 
+## DATACENTER: CODFW
 
 # ROW A codfw appservers: mw2017, mw2075-mw2079, and mw2215-2250
 
@@ -1624,6 +1665,8 @@
 role(mediawiki::appserver)
 }
 
+## END MEDIAWIKI APPLICATION SERVERS
+
 # mw logging host codfw
 node 'mwlog2001.codfw.wmnet' {
 role(logging::mediawiki::udp2log)

-- 
To view, visit 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: decommission mw1209-1220

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404500 )

Change subject: site.pp: decommission mw1209-1220
..

site.pp: decommission mw1209-1220

Bug: T185004
Change-Id: Icbe8b6201f476f3e2b0e4fcae610e6179639128e
---
M conftool-data/node/eqiad.yaml
M hieradata/common/scap/dsh.yaml
M manifests/site.pp
3 files changed, 3 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/00/404500/1

diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index d94acc1..c6705e2 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -73,17 +73,6 @@
 mw1347.eqiad.wmnet: [apache2,nginx]
 mw1348.eqiad.wmnet: [apache2,nginx]
   appserver:
-mw1209.eqiad.wmnet: [apache2,nginx]
-mw1210.eqiad.wmnet: [apache2,nginx]
-mw1211.eqiad.wmnet: [apache2,nginx]
-mw1212.eqiad.wmnet: [apache2,nginx]
-mw1213.eqiad.wmnet: [apache2,nginx]
-mw1214.eqiad.wmnet: [apache2,nginx]
-mw1215.eqiad.wmnet: [apache2,nginx]
-mw1216.eqiad.wmnet: [apache2,nginx]
-mw1218.eqiad.wmnet: [apache2,nginx]
-mw1219.eqiad.wmnet: [apache2,nginx]
-mw1220.eqiad.wmnet: [apache2,nginx]
 mw1238.eqiad.wmnet: [apache2,nginx]
 mw1239.eqiad.wmnet: [apache2,nginx]
 mw1240.eqiad.wmnet: [apache2,nginx]
diff --git a/hieradata/common/scap/dsh.yaml b/hieradata/common/scap/dsh.yaml
index 3626e3b..0db4625 100644
--- a/hieradata/common/scap/dsh.yaml
+++ b/hieradata/common/scap/dsh.yaml
@@ -2,8 +2,8 @@
 #
 scap::dsh::scap_proxies:
   - "mw1280.eqiad.wmnet" # A7 eqiad
-  - "mw1211.eqiad.wmnet" # B7 eqiad
-  - "mw1216.eqiad.wmnet" # B8 eqiad
+  - "mw1284.eqiad.wmnet" # B7 eqiad
+  - "mw1313.eqiad.wmnet" # B8 eqiad
   - "mw1319.eqiad.wmnet" # C6 eqiad
   - "mw1250.eqiad.wmnet" # D5 eqiad
   - "mw2117.codfw.wmnet" # B3 codfw
diff --git a/manifests/site.pp b/manifests/site.pp
index 71f3be6..309bc04 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1451,7 +1451,7 @@
 
 # mw1209-1216, 1218-1220 are apaches
 node /^mw12(09|1[012345689]|20)\.eqiad\.wmnet$/ {
-role(mediawiki::appserver)
+role(spare::system)
 }
 
 #mw1259-60 are videoscalers

-- 
To view, visit https://gerrit.wikimedia.org/r/404500
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icbe8b6201f476f3e2b0e4fcae610e6179639128e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: reorganize MediaWiki appservers in codfw for role/row

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404498 )

Change subject: site.pp: reorganize MediaWiki appservers in codfw for role/row
..

site.pp: reorganize MediaWiki appservers in codfw for role/row

Change-Id: Id21d8d99ea9d8d4be11b55c4b21cff7d921abab4
---
M manifests/site.pp
1 file changed, 92 insertions(+), 63 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/98/404498/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 23053b1..b3013e5 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1579,92 +1579,121 @@
 
 ## DATACENTER: CODFW
 
-# ROW A codfw appservers: mw2017, mw2075-mw2079, and mw2215-2250
+# Debug
 
 # mw2017/mw2099 are codfw test appservers
 node /^mw20(17|99)\.codfw\.wmnet$/ {
 role(mediawiki::canary_appserver)
 }
 
-#mw2097, mw2100-mw2117 are appservers
+# Hosts to decommission (if any)
+# TODO: mw2099-2134 should be decommissioned as soon as we have the new systems
+
+# Appservers
+
+# Row A
+
+# mw2224-38 are in rack A3
+# mw2239-42 are in rack A4
+node /^mw22(2[4-9]|3[0-9]|4[0-2])\.codfw\.wmnet$/ {
+role(mediawiki::appserver)
+}
+
+# Row B
+
+# mw2097, mw2100-mw2117 are in rack B3
 node /^mw2(097|10[0-9]|11[0-7])\.codfw\.wmnet$/ {
 role(mediawiki::appserver)
 include ::base::firewall
 }
+#mw2254-2258 are in rack B3
+node /^mw225[4-8]\.codfw\.wmnet$/ {
+role(mediawiki::appserver)
+}
 
-#mw2120-2147 are api appservers
+# Row C
+
+# mw2163-mw2186 are in rack C3
+# mw2187-mw2199 are in rack C4
+node /^mw21(6[3-9]|[6-9][0-9])\.codfw\.wmnet$/ {
+role(mediawiki::appserver)
+}
+
+# Api
+
+# Row A
+
+# mw2215-2223 are in rack A3
+node /^mw22(1[5-9]|2[0123])\.codfw\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+# mw2251-2253 are rack A4
+node /^mw225[1-3]\.codfw\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+# Row B
+
+# mw2120-2147 are in rack B4
 node /^mw21([2-3][0-9]|4[0-7])\.codfw\.wmnet$/ {
 role(mediawiki::appserver::api)
 }
 
-# ROW B codfw appservers
+# Row C
+
+# mw2200-2214 are in rack C4
+node /^mw22(0[0-9]|1[0-4])\.codfw\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+
+# Imagescalers
+
+# Row C (C3)
+node /^mw215[01]\.codfw\.wmnet$/ {
+role(mediawiki::imagescaler)
+}
+
+# Row A (A4)
+node /^mw224[45]\.codfw\.wmnet$/ {
+role(mediawiki::imagescaler)
+}
+
+# Jobrunners
+
+# Row A
+
+# mw2243, mw22477-mw2250 are in rack A4
+node /^mw22(4[3789]|50)\.codfw\.wmnet$/ {
+role(mediawiki::jobrunner)
+}
+
+# Row C
+
+# mw2153-62 are in rack C3
+node /^mw21(5[3-9]|6[0-2])\.codfw\.wmnet$/ {
+role(mediawiki::jobrunner)
+}
+
+# Videoscalers
+
+# Row A (A4)
+
+node 'mw2246.codfw.wmnet' {
+role(mediawiki::videoscaler)
+}
+
+# Row B (B3-B4)
 
 node /^mw211[89]\.codfw\.wmnet$/ {
 role(mediawiki::videoscaler)
 }
 
-# ROW C codfw appservers: mw2150-mw2234
+# Row C (C3)
 
-#mw2150-mw2151 are imagescalers
-node /^mw215[01]\.codfw\.wmnet$/ {
-role(mediawiki::imagescaler)
-}
-
-#mw2152 is a videoscaler
 node 'mw2152.codfw.wmnet' {
 role(mediawiki::videoscaler)
-}
-
-#mw2153-62 are jobrunners
-node /^mw21(5[3-9]|6[0-2])\.codfw\.wmnet$/ {
-role(mediawiki::jobrunner)
-}
-
-#mw2163-mw2199 are appservers
-node /^mw21(6[3-9]|[6-9][0-9])\.codfw\.wmnet$/ {
-role(mediawiki::appserver)
-}
-
-#mw2200-2214 are api appservers
-node /^mw22(0[0-9]|1[0-4])\.codfw\.wmnet$/ {
-role(mediawiki::appserver::api)
-}
-
-# New Appservers, in row A3/A4
-
-#mw2215-2223 are api appservers
-node /^mw22(1[5-9]|2[0123])\.codfw\.wmnet$/ {
-role(mediawiki::appserver::api)
-}
-
-# mw2224-42 are appservers
-node /^mw22(2[4-9]|3[0-9]|4[0-2])\.codfw\.wmnet$/ {
-role(mediawiki::appserver)
-}
-
-#mw2244-mw2245 are imagescalers
-node /^mw224[45]\.codfw\.wmnet$/ {
-role(mediawiki::imagescaler)
-}
-
-# mw2246 is a videoscaler
-node 'mw2246.codfw.wmnet' {
-role(mediawiki::videoscaler)
-}
-
-# mw2247-2250 are jobrunners
-node /^mw22(4[3789]|50)\.codfw\.wmnet$/ {
-role(mediawiki::jobrunner)
-}
-
-#mw2251-2253 are api-appservers
-node /^mw225[1-3]\.codfw\.wmnet$/ {
-role(mediawiki::appserver::api)
-}
-
-#mw2254-2258 are appservers
-node /^mw225[4-8]\.codfw\.wmnet$/ {
-role(mediawiki::appserver)
 }
 
 ## END MEDIAWIKI APPLICATION SERVERS

-- 
To view, visit https://gerrit.wikimedia.org/r/404498
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id21d8d99ea9d8d4be11b55c4b21cff7d921abab4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: decommission mw1201-1208

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404499 )

Change subject: site.pp: decommission mw1201-1208
..

site.pp: decommission mw1201-1208

Bug: T185004
Change-Id: I2a3afd3fb6e820076d3f2531f641a7861c335a7a
---
M conftool-data/node/eqiad.yaml
M hieradata/common/scap/dsh.yaml
M manifests/site.pp
3 files changed, 2 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/99/404499/1

diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index fcac9ea..d94acc1 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -26,14 +26,6 @@
 mw1336.eqiad.wmnet: [apache2,nginx]
 mw1337.eqiad.wmnet: [apache2,nginx]
   api_appserver:
-mw1201.eqiad.wmnet: [apache2,nginx]
-mw1202.eqiad.wmnet: [apache2,nginx]
-mw1203.eqiad.wmnet: [apache2,nginx]
-mw1204.eqiad.wmnet: [apache2,nginx]
-mw1205.eqiad.wmnet: [apache2,nginx]
-mw1206.eqiad.wmnet: [apache2,nginx]
-mw1207.eqiad.wmnet: [apache2,nginx]
-mw1208.eqiad.wmnet: [apache2,nginx]
 mw1221.eqiad.wmnet: [apache2,nginx]
 mw1222.eqiad.wmnet: [apache2,nginx]
 mw1223.eqiad.wmnet: [apache2,nginx]
diff --git a/hieradata/common/scap/dsh.yaml b/hieradata/common/scap/dsh.yaml
index aecfa1a..3626e3b 100644
--- a/hieradata/common/scap/dsh.yaml
+++ b/hieradata/common/scap/dsh.yaml
@@ -5,7 +5,7 @@
   - "mw1211.eqiad.wmnet" # B7 eqiad
   - "mw1216.eqiad.wmnet" # B8 eqiad
   - "mw1319.eqiad.wmnet" # C6 eqiad
-  - "mw1201.eqiad.wmnet" # D5 eqiad
+  - "mw1250.eqiad.wmnet" # D5 eqiad
   - "mw2117.codfw.wmnet" # B3 codfw
   - "mw2187.codfw.wmnet" # C4 codfw
   - "mw2215.codfw.wmnet" # A3 codfw
diff --git a/manifests/site.pp b/manifests/site.pp
index b3013e5..71f3be6 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1446,7 +1446,7 @@
 
 # mw1201-1208 are api apaches
 node /^mw120[1-8]\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::api)
+role(spare::system)
 }
 
 # mw1209-1216, 1218-1220 are apaches

-- 
To view, visit https://gerrit.wikimedia.org/r/404499
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2a3afd3fb6e820076d3f2531f641a7861c335a7a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: reorganize appservers in eqiad by function/row

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404453 )

Change subject: site.pp: reorganize appservers in eqiad by function/row
..

site.pp: reorganize appservers in eqiad by function/row

This should make it easier for users to find out which machines have
what role, and where they're located.

Change-Id: I073c00c1ac749ac5dc1239e0853e560044c6c177
---
M manifests/site.pp
1 file changed, 92 insertions(+), 47 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/53/404453/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 57b6948..23053b1 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1432,11 +1432,17 @@
 role(swift::storage)
 }
 
-# mwdebug servers are for mediawiki testing/debugging
-# They replace mw1017 and mw1099
+
+## MEDIAWIKI APPLICATION SERVERS
+
+## DATACENTER: EQIAD
+
+# Debug servers
 node /^mwdebug100[12]\.eqiad\.wmnet$/ {
 role(mediawiki::canary_appserver)
 }
+
+# Hosts to decommission (if any)
 
 # mw1201-1208 are api apaches
 node /^mw120[1-8]\.eqiad\.wmnet$/ {
@@ -1448,93 +1454,130 @@
 role(mediawiki::appserver)
 }
 
-#mw1221-mw1235 are api apaches
-node /^mw12(2[1-9]|3[0-5])\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::api)
-}
-
-#mw1238-mw1258 are apaches
-node /^mw12(3[8-9]|4[0-9]|5[0-8])\.eqiad\.wmnet$/ {
-role(mediawiki::appserver)
-}
-
 #mw1259-60 are videoscalers
 node /^mw12(59|60)\.eqiad\.wmnet/ {
 role(mediawiki::videoscaler)
 }
 
-# ROW A eqiad appservers
-#mw1261 - mw1275
+# Appservers (serving normal website traffic)
 
+# Row A
+
+#mw1261 - mw1275 are in rack A7
 node /^mw126[1-5]\.eqiad\.wmnet$/ {
 role(mediawiki::canary_appserver)
 }
-
 node /^mw12(6[6-9]|7[0-5])\.eqiad\.wmnet$/ {
 role(mediawiki::appserver)
 }
 
-# ROW A eqiad api appserver
-# mw1276 - mw1290
-node /^mw127[6-9]\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::canary_api)
-}
+# Row B
 
-node /^mw12(8[0-9]|90)\.eqiad\.wmnet$/ {
-role(mediawiki::appserver::api)
-}
+# Row C
 
-# ROW A eqiad imagescalers
-node /^mw129[3-8]\.eqiad\.wmnet$/ {
-role(mediawiki::imagescaler)
-}
-
-# ROW A eqiad jobrunners
-node /^mw1(299|30[0-6])\.eqiad\.wmnet$/ {
-role(mediawiki::jobrunner)
-}
-
-# ROW C eqiad jobrunners
-node /^mw133[4-7]\.eqiad\.wmnet$/ {
-role(mediawiki::jobrunner)
-}
-
-# T165519
-# ROW C eqiad appservers
+# mw1319-33 are in rack C6
 node /^mw13(19|2[0-9]|3[0-3])\.eqiad\.wmnet$/ {
 role(mediawiki::appserver)
 }
 
-# ROW B eqiad api-appservers
-node /^mw13(1[2-7])\.eqiad\.wmnet$/ {
+# Row D
+
+#mw1238-mw1258 are in rack D5
+node /^mw12(3[8-9]|4[0-9]|5[0-8])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver)
+}
+
+# API (serving api traffic)
+
+# Row A
+
+# mw1276 - mw1283 are in rack A7
+node /^mw127[6-9]\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::canary_api)
+}
+node /^mw128[0-3]\.eqiad\.wmnet$/ {
 role(mediawiki::appserver::api)
 }
 
-# ROW C api-appservers
+# mw1312 is in rack A6
+node 'mw1312.eqiad.wmnet' {
+role(mediawiki::appserver::api)
+}
+
+# Row B
+
+# mw1284-1290 are in rack B6
+node /^mw12(8[4-9]|90)\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+# mw1313-18 are in ROW B7
+node /^mw13(1[3-7])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+# Row C
+
+# mw1339-48 are in rack C6
 node /^mw13(39|4[0-8])\.eqiad\.wmnet$/ {
 role(mediawiki::appserver::api)
 }
 
-# ROW A eqiad jobrunners
+# Row D
+#mw1221-mw1235 are in rack D5
+node /^mw12(2[1-9]|3[0-5])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+
+# Imagescalers (mostly obsolete functionality, replaced by thumbor)
+
+# Row B (B6)
+node /^mw129[3-8]\.eqiad\.wmnet$/ {
+role(mediawiki::imagescaler)
+}
+
+# Jobrunners (now mostly used via changepropagation as a LVS endpoint)
+
+# Row A
+
+# mw1308-mw1311 are in rack A6
 node /^mw13(0[89]|1[01])\.eqiad\.wmnet$/ {
 role(mediawiki::jobrunner)
 }
 
-# ROW A videoscaler
+# Row B
+
+# mw1299-mw1306 are in rack B6
+node /^mw1(299|30[0-6])\.eqiad\.wmnet$/ {
+role(mediawiki::jobrunner)
+}
+
+# Row C
+
+# mw1334-mw1337 are in rack C6
+node /^mw133[4-7]\.eqiad\.wmnet$/ {
+role(mediawiki::jobrunner)
+}
+
+# Videoscalers
+
+# Row A (A6)
 node 'mw1307.eqiad.wmnet' {
 role(mediawiki::videoscaler)
 }
 
-# ROW B videoscaler
+# Row B (B7)
 node 'mw1318.eqiad.wmnet' {
 role(mediawiki::videoscaler)
 }
 
-# ROW C videoscaler
+# Row C (C6)
 node 'mw1338.eqiad.wmnet' {
 role(mediawiki::videoscaler)
 }
 
+## DATACENTER: CODFW
 
 # ROW A codfw appservers: mw2017, mw2075-mw2079, and mw2215-2250
 
@@ -1624,6 +1667,8 @@
 role(mediawiki::appserver)
 }
 
+## END MEDIAWIKI APPLICATION SERVERS
+
 # mw logging host codfw
 node 'mwlog2001.codfw.wmnet' {
 role(logging::mediawiki::udp2log)

-- 
To view, visit https://gerrit.wikimedia.org/r/404453
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mediawiki::scap: fetch mediawiki after mwdeploy has its sudo...

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404431 )

Change subject: mediawiki::scap: fetch mediawiki after mwdeploy has its sudo 
rules
..


mediawiki::scap: fetch mediawiki after mwdeploy has its sudo rules

This causes a resource order problem causing scap pull to fail during
the first puppet run.

Change-Id: I69e96b8e1c5ae819ddd4e6c627104ea15a2c52b7
---
M modules/mediawiki/manifests/scap.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified
  Volans: Looks good to me, but someone else must approve



diff --git a/modules/mediawiki/manifests/scap.pp 
b/modules/mediawiki/manifests/scap.pp
index 73d72fd..903b773 100644
--- a/modules/mediawiki/manifests/scap.pp
+++ b/modules/mediawiki/manifests/scap.pp
@@ -43,7 +43,7 @@
 exec { 'fetch_mediawiki':
 command => "${scap_bin_dir}/scap pull",
 creates => "${mediawiki_deployment_dir}/docroot",
-require => [ File[$mediawiki_deployment_dir], Package['scap'] ],
+require => [ File[$mediawiki_deployment_dir], Package['scap'], 
Sudo::User['mwdeploy'] ],
 timeout => 30 * 60,  # 30 minutes
 user=> 'mwdeploy',
 group   => 'mwdeploy',

-- 
To view, visit https://gerrit.wikimedia.org/r/404431
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I69e96b8e1c5ae819ddd4e6c627104ea15a2c52b7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Alex Monk 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mediawiki::scap: fetch mediawiki after mwdeploy has its sudo...

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404431 )

Change subject: mediawiki::scap: fetch mediawiki after mwdeploy has its sudo 
rules
..

mediawiki::scap: fetch mediawiki after mwdeploy has its sudo rules

This causes a resource order problem causing scap pull to fail during
the first puppet run.

Change-Id: I69e96b8e1c5ae819ddd4e6c627104ea15a2c52b7
---
M modules/mediawiki/manifests/scap.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/31/404431/1

diff --git a/modules/mediawiki/manifests/scap.pp 
b/modules/mediawiki/manifests/scap.pp
index 73d72fd..903b773 100644
--- a/modules/mediawiki/manifests/scap.pp
+++ b/modules/mediawiki/manifests/scap.pp
@@ -43,7 +43,7 @@
 exec { 'fetch_mediawiki':
 command => "${scap_bin_dir}/scap pull",
 creates => "${mediawiki_deployment_dir}/docroot",
-require => [ File[$mediawiki_deployment_dir], Package['scap'] ],
+require => [ File[$mediawiki_deployment_dir], Package['scap'], 
Sudo::User['mwdeploy'] ],
 timeout => 30 * 60,  # 30 minutes
 user=> 'mwdeploy',
 group   => 'mwdeploy',

-- 
To view, visit https://gerrit.wikimedia.org/r/404431
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I69e96b8e1c5ae819ddd4e6c627104ea15a2c52b7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: apt: make apt::conf happen before removing apt.conf

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404307 )

Change subject: apt: make apt::conf happen before removing apt.conf
..


apt: make apt::conf happen before removing apt.conf

Else, we might end up with a period where packages are installed with an
invalid configuration and can fail to install, or even be installed at
wrong versions.

Change-Id: I321f86d185e642e810e72d21cbcdc964162b6f6e
---
M modules/apt/manifests/init.pp
1 file changed, 14 insertions(+), 8 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified
  Volans: Looks good to me, but someone else must approve



diff --git a/modules/apt/manifests/init.pp b/modules/apt/manifests/init.pp
index ed35d23..9f376a9 100644
--- a/modules/apt/manifests/init.pp
+++ b/modules/apt/manifests/init.pp
@@ -49,14 +49,6 @@
 purge   => $purge_sources,
 }
 
-# This will munge /etc/apt/apt.conf that get's created during installation
-# process (either labs vmbuilder or d-i). Given the ones below exist, it is
-# no longer needed after the installation is over
-file { '/etc/apt/apt.conf':
-ensure => absent,
-notify => Exec['apt-get update'],
-}
-
 if $use_proxy {
 $http_proxy = "http://webproxy.${::site}.wmnet:8080;
 
@@ -66,12 +58,14 @@
 priority => '80',
 key  => 'Acquire::http::Proxy::security.debian.org',
 value=> $http_proxy,
+before   => File['/etc/apt/apt.conf'],
 }
 apt::conf { 'security-cdn-debian-proxy':
 ensure   => present,
 priority => '80',
 key  => 'Acquire::http::Proxy::security-cdn.debian.org',
 value=> $http_proxy,
+before   => File['/etc/apt/apt.conf']
 }
 } elsif $::operatingsystem == 'Ubuntu' {
 apt::conf { 'security-ubuntu-proxy':
@@ -79,6 +73,7 @@
 priority => '80',
 key  => 'Acquire::http::Proxy::security.ubuntu.com',
 value=> $http_proxy,
+before   => File['/etc/apt/apt.conf']
 }
 
 apt::conf { 'ubuntu-cloud-archive-proxy':
@@ -86,6 +81,7 @@
 priority => '80',
 key  => 
'Acquire::http::Proxy::ubuntu-cloud.archive.canonical.com',
 value=> $http_proxy,
+before   => File['/etc/apt/apt.conf']
 }
 
 apt::conf { 'old-releases-proxy':
@@ -93,6 +89,7 @@
 priority => '80',
 key  => 'Acquire::http::Proxy::old-releases.ubuntu.com',
 value=> $http_proxy,
+before   => File['/etc/apt/apt.conf']
 }
 } else {
 fail("Unknown operating system '${::operatingsystem}'.")
@@ -150,5 +147,14 @@
 priority => '90',
 key  => 'APT::Install-Recommends',
 value=> '0',
+before   => File['/etc/apt/apt.conf'],
+}
+
+# This will munge /etc/apt/apt.conf that get's created during installation
+# process (either labs vmbuilder or d-i). Given the ones below exist, it is
+# no longer needed after the installation is over
+file { '/etc/apt/apt.conf':
+ensure => absent,
+notify => Exec['apt-get update'],
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/404307
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I321f86d185e642e810e72d21cbcdc964162b6f6e
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: apt: make apt::conf and apt::pin configs happen before remov...

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404307 )

Change subject: apt: make apt::conf and apt::pin configs happen before removing 
apt.conf
..

apt: make apt::conf and apt::pin configs happen before removing apt.conf

Else, we might end up with a period where packages are installed with an
invalid configuration and can fail to install, or even be installed at
wrong versions.

Change-Id: I321f86d185e642e810e72d21cbcdc964162b6f6e
---
M modules/apt/manifests/init.pp
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/07/404307/1

diff --git a/modules/apt/manifests/init.pp b/modules/apt/manifests/init.pp
index ed35d23..169c92f 100644
--- a/modules/apt/manifests/init.pp
+++ b/modules/apt/manifests/init.pp
@@ -56,6 +56,8 @@
 ensure => absent,
 notify => Exec['apt-get update'],
 }
+Apt::Conf <| |> -> File['/etc/apt/apt.conf']
+Apt::Pin <| |> -> File['/etc/apt/apt.conf']
 
 if $use_proxy {
 $http_proxy = "http://webproxy.${::site}.wmnet:8080;

-- 
To view, visit https://gerrit.wikimedia.org/r/404307
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I321f86d185e642e810e72d21cbcdc964162b6f6e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::base: run the apt configuration before anything else

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404305 )

Change subject: profile::base: run the apt configuration before anything else
..

profile::base: run the apt configuration before anything else

To this end, introduce a stage => 'apt-config', that's supposed to
include only the apt class.

Change-Id: I21bf60687864e1a796c72fb293b99350bc09c64f
---
M manifests/realm.pp
M modules/profile/manifests/base.pp
2 files changed, 8 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/05/404305/1

diff --git a/manifests/realm.pp b/manifests/realm.pp
index 1652589..5c6425b 100644
--- a/manifests/realm.pp
+++ b/manifests/realm.pp
@@ -22,6 +22,11 @@
 $realm = hiera('realm', 'production')
 }
 
+stage { 'apt-config':
+before => Stage['main']
+}
+
+
 if $realm == 'labs' {
 # Pull the project name from the certname.
 # Labs certs are ...wmflabs
diff --git a/modules/profile/manifests/base.pp 
b/modules/profile/manifests/base.pp
index fbdaa84..51b4ff5 100644
--- a/modules/profile/manifests/base.pp
+++ b/modules/profile/manifests/base.pp
@@ -25,9 +25,12 @@
 $overlayfs = hiera('profile::base::overlayfs', false),
 ) {
 require ::profile::base::certificates
+
+# Apt configuration needs to happen before anything else happens.
 class { '::apt':
 use_proxy => $use_apt_proxy,
 purge_sources => $purge_apt_sources,
+stage => 'apt-config'
 }
 
 file { ['/usr/local/sbin', '/usr/local/share/bash']:

-- 
To view, visit https://gerrit.wikimedia.org/r/404305
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I21bf60687864e1a796c72fb293b99350bc09c64f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: profile::base: configure apt before installing any package

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404304 )

Change subject: profile::base: configure apt before installing any package
..

profile::base: configure apt before installing any package

Change-Id: Ia05fac12eac6b5e25ff8f90dc15d1a345cccfced
---
M modules/profile/manifests/base.pp
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/04/404304/1

diff --git a/modules/profile/manifests/base.pp 
b/modules/profile/manifests/base.pp
index fbdaa84..66c8123 100644
--- a/modules/profile/manifests/base.pp
+++ b/modules/profile/manifests/base.pp
@@ -29,6 +29,8 @@
 use_proxy => $use_apt_proxy,
 purge_sources => $purge_apt_sources,
 }
+# Class apt must be applied before any package is added.
+Class['apt'] -> Package <| |>
 
 file { ['/usr/local/sbin', '/usr/local/share/bash']:
 ensure => directory,

-- 
To view, visit https://gerrit.wikimedia.org/r/404304
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia05fac12eac6b5e25ff8f90dc15d1a345cccfced
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: add mw1338->48

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403928 )

Change subject: site.pp: add mw1338->48
..


site.pp: add mw1338->48

The site.pp mw hosts ordering has been changed to follow the rest
of site.pp (number increasing).

Bug: T165519
Change-Id: I688a90b1cb671f8b964d2541908ce2ef833512f7
---
M conftool-data/node/eqiad.yaml
M manifests/site.pp
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
3 files changed, 23 insertions(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index 48d4950..fcac9ea 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -7,6 +7,7 @@
 mw1260.eqiad.wmnet: [apache2]
 mw1307.eqiad.wmnet: [apache2]
 mw1318.eqiad.wmnet: [apache2]
+mw1338.eqiad.wmnet: [apache2]
   jobrunner:
 mw1299.eqiad.wmnet: [apache2,nginx]
 mw1300.eqiad.wmnet: [apache2,nginx]
@@ -69,6 +70,16 @@
 mw1315.eqiad.wmnet: [apache2,nginx]
 mw1316.eqiad.wmnet: [apache2,nginx]
 mw1317.eqiad.wmnet: [apache2,nginx]
+mw1339.eqiad.wmnet: [apache2,nginx]
+mw1340.eqiad.wmnet: [apache2,nginx]
+mw1341.eqiad.wmnet: [apache2,nginx]
+mw1342.eqiad.wmnet: [apache2,nginx]
+mw1343.eqiad.wmnet: [apache2,nginx]
+mw1344.eqiad.wmnet: [apache2,nginx]
+mw1345.eqiad.wmnet: [apache2,nginx]
+mw1346.eqiad.wmnet: [apache2,nginx]
+mw1347.eqiad.wmnet: [apache2,nginx]
+mw1348.eqiad.wmnet: [apache2,nginx]
   appserver:
 mw1209.eqiad.wmnet: [apache2,nginx]
 mw1210.eqiad.wmnet: [apache2,nginx]
diff --git a/manifests/site.pp b/manifests/site.pp
index 967ed6a..576a0ac 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1499,7 +1499,6 @@
 role(mediawiki::jobrunner)
 }
 
-
 # T165519
 # ROW C eqiad appservers
 node /^mw13(19|2[0-9]|3[0-3])\.eqiad\.wmnet$/ {
@@ -1508,6 +1507,11 @@
 
 # ROW B eqiad api-appservers
 node /^mw13(1[2-7])\.eqiad\.wmnet$/ {
+role(mediawiki::appserver::api)
+}
+
+# ROW C api-appservers
+node /^mw13(39|4[0-8])\.eqiad\.wmnet$/ {
 role(mediawiki::appserver::api)
 }
 
@@ -1526,6 +1530,11 @@
 role(mediawiki::videoscaler)
 }
 
+# ROW C videoscaler
+node 'mw1338.eqiad.wmnet' {
+role(mediawiki::videoscaler)
+}
+
 
 # ROW A codfw appservers: mw2017, mw2075-mw2079, and mw2215-2250
 
diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200 
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index dd88d5c..65e8ff6 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -4764,6 +4764,8 @@
 host mw1338 {
 hardware ethernet 18:66:DA:99:52:38;
 fixed-address mw1338.eqiad.wmnet;
+option pxelinux.pathprefix "stretch-installer/";
+filename "stretch-installer/debian-installer/amd64/pxelinux.0";
 }
 
 host mw1339 {

-- 
To view, visit https://gerrit.wikimedia.org/r/403928
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I688a90b1cb671f8b964d2541908ce2ef833512f7
Gerrit-PatchSet: 6
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wmflib: simplify the role() function, convert to the new API

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402345 )

Change subject: wmflib: simplify the role() function, convert to the new API
..


wmflib: simplify the role() function, convert to the new API

Converting to the new API makes some things possible, namely:
- Clean access to the top scope
- Cleanly require the function is called exactly once
- Use call_function for 'include'

but on the other hand makes it impossible to verify if we're calling the
function from the node scope. Please note that this happens for a reason
- messing with the scope in function can cause undefined behaviour and
indeed I found edge cases where this can happen, for instance when
trying to access the variables before they're defined can make them null.

Change-Id: I32631c84a7f6340a0d7de2ae57dfbef87015c46b
---
A modules/wmflib/lib/puppet/functions/role.rb
D modules/wmflib/lib/puppet/parser/functions/role.rb
M modules/wmflib/spec/functions/role_spec.rb
3 files changed, 73 insertions(+), 101 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/wmflib/lib/puppet/functions/role.rb 
b/modules/wmflib/lib/puppet/functions/role.rb
new file mode 100644
index 000..877624e
--- /dev/null
+++ b/modules/wmflib/lib/puppet/functions/role.rb
@@ -0,0 +1,54 @@
+# == Function: role ( string $role_name [, string $... ] )
+#
+# Declare the _roles variable (or add keys to it), and if the role class
+# role::$role_name is in the scope, include it. This is roughly a
+# shortcut for
+#
+# $::_roles[foo] = true
+# include role::foo
+#
+# and has the notable advantage over that the syntax is shorter. Also,
+# this function  will refuse to run anywhere but at the node scope,
+# thus making any additional role added to a node explicit.
+#
+# If you have more than one role to declare, you MUST do that in one
+# single role stanza, or you would encounter unexpected behaviour. If
+# you do, an exception will be raised.
+#
+# This function is very useful with our "role" hiera backend if you
+# have global configs that are role-based
+#
+# === Example
+#
+# node /^www\d+/ {
+# role mediawiki::appserver  # this will load the 
role::mediawiki::appserver class
+# include ::standard  #this class will use hiera lookups defined for the 
role.
+# }
+#
+# node monitoring.local {
+# role icinga, ganglia::collector #GOOD
+# }
+#
+# node monitoring2.local {
+# role icinga
+# role ganglia::collector #BAD, issues a warning
+# }
+Puppet::Functions.create_function(:role) do
+  dispatch :main do
+param 'String', :main_role
+  end
+  def main(main_role)
+scope = closure_scope
+# Check if the function has already been called
+if scope.include? '_role'
+  raise Puppet::ParseError, "role() can only be called once per node"
+end
+role = main_role.gsub(/^::/, '')
+# Backwards compat
+scope['_roles'] = { role => true }
+# This is where we're going in the future
+scope['_role'] = role
+rolename = 'role::' + role
+call_function('include', rolename)
+  end
+end
diff --git a/modules/wmflib/lib/puppet/parser/functions/role.rb 
b/modules/wmflib/lib/puppet/parser/functions/role.rb
deleted file mode 100644
index 971782b..000
--- a/modules/wmflib/lib/puppet/parser/functions/role.rb
+++ /dev/null
@@ -1,85 +0,0 @@
-# == Function: role ( string $role_name [, string $... ] )
-#
-# Declare the _roles variable (or add keys to it), and if the role class
-# role::$role_name is in the scope, include it. This is roughly a
-# shortcut for
-#
-# $::_roles[foo] = true
-# include role::foo
-#
-# and has the notable advantage over that the syntax is shorter. Also,
-# this function  will refuse to run anywhere but at the node scope,
-# thus making any additional role added to a node explicit.
-#
-# If you have more than one role to declare, you MUST do that in one
-# single role stanza, or you would encounter unexpected behaviour. If
-# you do, an exception will be raised.
-#
-# This function is very useful with our "role" hiera backend if you
-# have global configs that are role-based
-#
-# === Example
-#
-# node /^www\d+/ {
-# role mediawiki::appserver  # this will load the 
role::mediawiki::appserver class
-# include ::standard  #this class will use hiera lookups defined for the 
role.
-# }
-#
-# node monitoring.local {
-# role icinga, ganglia::collector #GOOD
-# }
-#
-# node monitoring2.local {
-# role icinga
-# role ganglia::collector #BAD, issues a warning
-# }
-
-module Puppet::Parser::Functions
-  newfunction(:role, :arity => -2) do |args|
-# This will add to the catalog, and to the node specifically:
-# - A global 'role' hash with the 'role::#{arg}' key set to true;
-# if the variable is present, append to it
-# - Include class role::#{arg} if present
-
-# Prevent use outside of node definitions
- 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base::resolving: explicitly pass arguments

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403440 )

Change subject: base::resolving: explicitly pass arguments
..


base::resolving: explicitly pass arguments

Instead of relying on top-scope variable overriding each other, do the
following:
* Remove the conditional on $nameserver_override in the resolv.conf
  template
* Add $nameservers as a local variable, which defaults to $::nameservers
* Pass the value from profile::base in case it's defined in hiera, and
  have care to exclude the IP of the current node from the list to avoid
  self-dependencies
* Remove all the $nameservers_override from the node definitions and add
  those to per-site, per-role hiera

Change-Id: Ib0926a8966db2066d87a9ddea4265ed741c07437
---
M hieradata/role/codfw/lvs/balancer.yaml
M hieradata/role/codfw/recursor.yaml
M hieradata/role/eqiad/lvs/balancer.yaml
M hieradata/role/eqiad/recursor.yaml
M hieradata/role/esams/lvs/balancer.yaml
M hieradata/role/ulsfo/lvs/balancer.yaml
M manifests/site.pp
M modules/base/manifests/resolving.pp
M modules/base/spec/classes/resolving_spec.rb
M modules/base/templates/resolv.conf.erb
M modules/profile/manifests/base.pp
11 files changed, 51 insertions(+), 53 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/codfw/lvs/balancer.yaml 
b/hieradata/role/codfw/lvs/balancer.yaml
index 5fd35e0..19d968c 100644
--- a/hieradata/role/codfw/lvs/balancer.yaml
+++ b/hieradata/role/codfw/lvs/balancer.yaml
@@ -8,3 +8,11 @@
   - public1-b-codfw
   - public1-c-codfw
   - public1-d-codfw
+# lvs200[25] are LVS balancers for the codfw recursive DNS IP,
+#   so they need to use the recursive DNS backends directly
+#   (acamar and achernar) with fallback to eqiad
+# (doing this for all lvs for now, see T103921)
+profile::base::nameservers:
+  - '208.80.153.12' # acamar
+  - '208.80.153.42' # achenar
+  - '208.80.154.254' # eqiad lvs
diff --git a/hieradata/role/codfw/recursor.yaml 
b/hieradata/role/codfw/recursor.yaml
index d3fb55e..7f5238b 100644
--- a/hieradata/role/codfw/recursor.yaml
+++ b/hieradata/role/codfw/recursor.yaml
@@ -1,3 +1,8 @@
 profile::bird::neighbors_list:
   - 208.80.153.192 # cr1-codfw loopback
   - 208.80.153.193 # cr2-codfw loopback
+profile::base::nameservers:
+#  - '208.80.153.12' # acamar
+#  - '208.80.153.42' # achenar
+  - '208.80.153.254' # codfw lvs
+  - '208.80.154.254' # eqiad lvs
diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index 4b1764e..b14b9ba 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -8,3 +8,11 @@
   - public1-b-eqiad
   - public1-c-eqiad
   - public1-d-eqiad
+# lvs100[25] are LVS balancers for the eqiad recursive DNS IP,
+#   so they need to use the recursive DNS backends directly
+#   (chromium and hydrogen) with fallback to codfw
+# (doing this for all lvs for now, see T103921)
+profile::base::nameservers:
+  - '208.80.154.50' # hydrogen
+  - '208.80.154.157' # chromium
+  - '208.80.153.254' # codfw lvs
diff --git a/hieradata/role/eqiad/recursor.yaml 
b/hieradata/role/eqiad/recursor.yaml
index 446f990..d560dc0 100644
--- a/hieradata/role/eqiad/recursor.yaml
+++ b/hieradata/role/eqiad/recursor.yaml
@@ -1,3 +1,8 @@
 profile::bird::neighbors_list:
   - 208.80.154.196 # cr1-eqiad loopback
   - 208.80.154.197 # cr2-eqiad loopback
+profile::base::nameservers:
+#  - '208.80.154.50' # hydrogen
+#  - '208.80.154.157' # chromium
+  - '208.80.154.254' # eqiad lvs
+  - '208.80.153.254' # codfw lvs
diff --git a/hieradata/role/esams/lvs/balancer.yaml 
b/hieradata/role/esams/lvs/balancer.yaml
index c4e9150..3b1d667 100644
--- a/hieradata/role/esams/lvs/balancer.yaml
+++ b/hieradata/role/esams/lvs/balancer.yaml
@@ -1,3 +1,11 @@
 profile::pybal::config_host: conf1003.eqiad.wmnet
 profile::lvs::tagged_subnets:
   - public1-esams
+# lvs300[24] are LVS balancers for the esams recursive DNS IP,
+#   so they need to use the recursive DNS backends directly
+#   (nescio and maerlant) with fallback to eqiad
+# (doing this for all lvs for now, see T103921)
+profile::base::nameservers:
+  - '91.198.174.106' # nescio
+  - '91.198.174.122' # maerlant
+  - '208.80.154.254' # eqiad lvs
diff --git a/hieradata/role/ulsfo/lvs/balancer.yaml 
b/hieradata/role/ulsfo/lvs/balancer.yaml
index ada9df4..d4976fb 100644
--- a/hieradata/role/ulsfo/lvs/balancer.yaml
+++ b/hieradata/role/ulsfo/lvs/balancer.yaml
@@ -1,2 +1,7 @@
 profile::pybal::config_host: conf2003.codfw.wmnet
 profile::lvs::tagged_subnets: []
+# ns override for all lvs for now, see T103921
+profile::base::nameservers:
+  - '208.80.153.12' # acamar
+  - '208.80.153.42' # achenar
+  - '208.80.154.254' # eqiad lvs
diff --git a/manifests/site.pp b/manifests/site.pp
index b5b4d43..8803b67 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -12,17 +12,11 @@
 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base::resolving: remove useless "else" clause

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403439 )

Change subject: base::resolving: remove useless "else" clause
..


base::resolving: remove useless "else" clause

Change-Id: I3ca4358f02835e89864ab15b550e9346fdc3f0b5
---
M modules/base/manifests/resolving.pp
1 file changed, 34 insertions(+), 35 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified
  Volans: Looks good to me, but someone else must approve



diff --git a/modules/base/manifests/resolving.pp 
b/modules/base/manifests/resolving.pp
index 5501a5b..86fccc3 100644
--- a/modules/base/manifests/resolving.pp
+++ b/modules/base/manifests/resolving.pp
@@ -5,44 +5,43 @@
 if ! $::nameservers {
 fail('Variable $::nameservers is not defined!')
 }
-else {
-if $::realm == 'labs' {
-$labs_tld = hiera('labs_tld')
-# Thanks to dhcp, resolvconf is constantly messing with our 
resolv.conf.  Disable it.
-file { '/sbin/resolvconf':
-owner  => 'root',
-group  => 'root',
-mode   => '0555',
-source => 'puppet:///modules/base/resolv/resolvconf.dummy',
-}
 
-file { '/etc/dhcp/dhclient-enter-hooks.d':
-ensure => 'directory',
-}
+if $::realm == 'labs' {
+$labs_tld = hiera('labs_tld')
+# Thanks to dhcp, resolvconf is constantly messing with our 
resolv.conf.  Disable it.
+file { '/sbin/resolvconf':
+owner  => 'root',
+group  => 'root',
+mode   => '0555',
+source => 'puppet:///modules/base/resolv/resolvconf.dummy',
+}
 
-# also stop dhclient from updating resolv.conf.
-file { '/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate':
-owner   => 'root',
-group   => 'root',
-mode=> '0555',
-source  => 'puppet:///modules/base/resolv/nodnsupdate',
-require => File['/etc/dhcp/dhclient-enter-hooks.d'],
-}
+file { '/etc/dhcp/dhclient-enter-hooks.d':
+ensure => 'directory',
+}
 
-# Now, finally, we can just puppetize the damn file
-file { '/etc/resolv.conf':
-owner   => 'root',
-group   => 'root',
-mode=> '0444',
-content => template('base/resolv.conf.labs.erb'),
-}
-} else {
-file { '/etc/resolv.conf':
-owner   => 'root',
-group   => 'root',
-mode=> '0444',
-content => template('base/resolv.conf.erb'),
-}
+# also stop dhclient from updating resolv.conf.
+file { '/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate':
+owner   => 'root',
+group   => 'root',
+mode=> '0555',
+source  => 'puppet:///modules/base/resolv/nodnsupdate',
+require => File['/etc/dhcp/dhclient-enter-hooks.d'],
+}
+
+# Now, finally, we can just puppetize the damn file
+file { '/etc/resolv.conf':
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+content => template('base/resolv.conf.labs.erb'),
+}
+} else {
+file { '/etc/resolv.conf':
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+content => template('base/resolv.conf.erb'),
 }
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/403439
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3ca4358f02835e89864ab15b550e9346fdc3f0b5
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base::resolving: properly extend the tests

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403612 )

Change subject: base::resolving: properly extend the tests
..


base::resolving: properly extend the tests

Change-Id: I29393de4ffba08ab634126984b8f5c61ad7bbc58
---
M modules/base/spec/classes/resolving_spec.rb
M modules/base/spec/fixtures/hieradata/labs.yaml
2 files changed, 41 insertions(+), 0 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/base/spec/classes/resolving_spec.rb 
b/modules/base/spec/classes/resolving_spec.rb
index 301e525..24754d4 100644
--- a/modules/base/spec/classes/resolving_spec.rb
+++ b/modules/base/spec/classes/resolving_spec.rb
@@ -5,4 +5,44 @@
 should compile.and_raise_error(
 /Variable \$::nameservers is not defined!/)
 end
+context "when nameservers are defined" do
+  let(:facts) { {'domain' => 'example.com'} }
+  let(:node_params){ {'nameservers' => ['1.2.3.4'], 'realm' => 
'production'}}
+  it { is_expected.to compile.with_all_deps }
+  it "contains a correct resolv.conf"  do
+is_expected.to contain_file('/etc/resolv.conf')
+ .with_owner('root')
+ .with_group('root')
+ .with_mode('0444')
+content = catalogue.resource('file', 
'/etc/resolv.conf').send(:parameters)[:content]
+
+expect(content).to match(/search example.com\noptions timeout:1 
attempts:3\nnameserver 1.2.3.4\n/)
+  end
+  context "when nameservers are overridden" do
+let(:node_params){ super().merge({'nameservers_override' => 
['2.2.2.2'] })}
+it "nameserver is changed in resolv.conf" do
+  content = catalogue.resource('file', 
'/etc/resolv.conf').send(:parameters)[:content]
+
+  expect(content).to match(/search example.com\noptions timeout:1 
attempts:3\nnameserver 2.2.2.2\n/)
+end
+  end
+end
+context "when realm is labs" do
+  let(:facts) { {'domain' => 'example.com', 'labsproject' => 'fun' } }
+  let(:node_params){
+{
+  'nameservers' => ['1.2.3.4'],
+  'realm' => 'labs',
+  'site' => 'testdc',
+}
+  }
+  it { is_expected.to compile.with_all_deps }
+  it { is_expected.to 
contain_file('/sbin/resolvconf').with_source('puppet:///modules/base/resolv/resolvconf.dummy')
 }
+  it { is_expected.to 
contain_file('/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate').with_source('puppet:///modules/base/resolv/nodnsupdate')}
+  it "contains a correct resolv.conf" do
+content = catalogue.resource('file', 
'/etc/resolv.conf').send(:parameters)[:content]
+
+expect(content).to match(/search fun.testdc.foo.bar testdc.foo.bar 
\nnameserver 1.2.3.4\noptions timeout:2 ndots:2/)
+  end
+end
 end
diff --git a/modules/base/spec/fixtures/hieradata/labs.yaml 
b/modules/base/spec/fixtures/hieradata/labs.yaml
index 11cd7c2..5d900e9 100644
--- a/modules/base/spec/fixtures/hieradata/labs.yaml
+++ b/modules/base/spec/fixtures/hieradata/labs.yaml
@@ -1 +1,2 @@
 auto_puppetmaster_switching: true
+labs_tld: "foo.bar"

-- 
To view, visit https://gerrit.wikimedia.org/r/403612
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I29393de4ffba08ab634126984b8f5c61ad7bbc58
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base::resolving: properly extend the tests

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403612 )

Change subject: base::resolving: properly extend the tests
..

base::resolving: properly extend the tests

Change-Id: I29393de4ffba08ab634126984b8f5c61ad7bbc58
---
M modules/base/spec/classes/resolving_spec.rb
M modules/base/spec/fixtures/hieradata/labs.yaml
2 files changed, 41 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/12/403612/1

diff --git a/modules/base/spec/classes/resolving_spec.rb 
b/modules/base/spec/classes/resolving_spec.rb
index 301e525..0db9ef2 100644
--- a/modules/base/spec/classes/resolving_spec.rb
+++ b/modules/base/spec/classes/resolving_spec.rb
@@ -5,4 +5,44 @@
 should compile.and_raise_error(
 /Variable \$::nameservers is not defined!/)
 end
+context "when nameservers are defined" do
+  let(:facts) { {'domain' => 'example.com'} }
+  let(:node_params){ {'nameservers' => ['1.2.3.4'], 'realm' => 
'production'}}
+  it { is_expected.to compile.with_all_deps }
+  it "contains a correct resolv.conf"  do
+is_expected.to contain_file('/etc/resolv.conf')
+ .with_owner('root')
+ .with_group('root')
+ .with_mode('0444')
+content = catalogue.resource('file', 
'/etc/resolv.conf').send(:parameters)[:content]
+
+expect(content).to match(/search example.com\noptions timeout:1 
attempts:3\nnameserver 1.2.3.4\n/)
+  end
+  context "when nameservers are overridden" do
+let(:node_params){ super().merge({'nameservers_override' => 
['2.2.2.2'] })}
+it "nameserver is changed in resolv.conf" do
+  content = catalogue.resource('file', 
'/etc/resolv.conf').send(:parameters)[:content]
+
+  expect(content).to match(/search example.com\noptions timeout:1 
attempts:3\nnameserver 2.2.2.2\n/)
+end
+  end
+end
+context "when realm is labs" do
+  let(:facts) { {'domain' => 'example.com', 'labsproject' => 'fun' } }
+  let(:node_params){
+{
+  'nameservers' => ['1.2.3.4'],
+  'realm' => 'labs',
+  'site'  => 'testdc',
+}
+  }
+  it { is_expected.to compile.with_all_deps }
+  it { is_expected.to 
contain_file('/sbin/resolvconf').with_source('puppet:///modules/base/resolv/resolvconf.dummy')
 }
+  it { is_expected.to 
contain_file('/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate').with_source('puppet:///modules/base/resolv/nodnsupdate')}
+  it "contains a correct resolv.conf" do
+content = catalogue.resource('file', 
'/etc/resolv.conf').send(:parameters)[:content]
+
+expect(content).to match(/search fun.testdc.foo.bar testdc.foo.bar 
\nnameserver 1.2.3.4\noptions timeout:2 ndots:2/)
+  end
+end
 end
diff --git a/modules/base/spec/fixtures/hieradata/labs.yaml 
b/modules/base/spec/fixtures/hieradata/labs.yaml
index 11cd7c2..5d900e9 100644
--- a/modules/base/spec/fixtures/hieradata/labs.yaml
+++ b/modules/base/spec/fixtures/hieradata/labs.yaml
@@ -1 +1,2 @@
 auto_puppetmaster_switching: true
+labs_tld: "foo.bar"

-- 
To view, visit https://gerrit.wikimedia.org/r/403612
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I29393de4ffba08ab634126984b8f5c61ad7bbc58
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: puppetdb: fix hiera key

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403600 )

Change subject: puppetdb: fix hiera key
..


puppetdb: fix hiera key

Change-Id: I0a0ebcf2d31b4a71ab3b676b6334f8f3da96868a
---
M hieradata/role/common/puppetmaster/puppetdb.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/puppetmaster/puppetdb.yaml 
b/hieradata/role/common/puppetmaster/puppetdb.yaml
index b3be4d0..0340f0c 100644
--- a/hieradata/role/common/puppetmaster/puppetdb.yaml
+++ b/hieradata/role/common/puppetmaster/puppetdb.yaml
@@ -10,4 +10,4 @@
 cidr: 10.192.16.184/32
 profile::puppetdb::master: nitrogen.eqiad.wmnet
 profile::puppetdb::slaves: [nihal.codfw.wmnet]
-puppetmaster::puppetdb::jvm_opts: '-Xmx6g'
+profile::puppetdb::jvm_opts: '-Xmx6g'

-- 
To view, visit https://gerrit.wikimedia.org/r/403600
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0a0ebcf2d31b4a71ab3b676b6334f8f3da96868a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: puppetdb: fix hiera key

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403600 )

Change subject: puppetdb: fix hiera key
..

puppetdb: fix hiera key

Change-Id: I0a0ebcf2d31b4a71ab3b676b6334f8f3da96868a
---
M hieradata/role/common/puppetmaster/puppetdb.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/00/403600/1

diff --git a/hieradata/role/common/puppetmaster/puppetdb.yaml 
b/hieradata/role/common/puppetmaster/puppetdb.yaml
index b3be4d0..0340f0c 100644
--- a/hieradata/role/common/puppetmaster/puppetdb.yaml
+++ b/hieradata/role/common/puppetmaster/puppetdb.yaml
@@ -10,4 +10,4 @@
 cidr: 10.192.16.184/32
 profile::puppetdb::master: nitrogen.eqiad.wmnet
 profile::puppetdb::slaves: [nihal.codfw.wmnet]
-puppetmaster::puppetdb::jvm_opts: '-Xmx6g'
+profile::puppetdb::jvm_opts: '-Xmx6g'

-- 
To view, visit https://gerrit.wikimedia.org/r/403600
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0a0ebcf2d31b4a71ab3b676b6334f8f3da96868a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::puppetmaster::puppetdb: add Prometheus monitoring for ...

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394966 )

Change subject: role::puppetmaster::puppetdb: add Prometheus monitoring for 
puppetdb
..


role::puppetmaster::puppetdb: add Prometheus monitoring for puppetdb

This change adds only a subset of the Mbeans available since using
the JMX agent's whitelist turned out to be more perfomant.

The puppetdb's jvm options are now configurable via hiera to allow
a more friendly labs deployment.

Change-Id: I58f036e85edb98ef4170580d093c42f0bc8ef786
---
M hieradata/role/common/puppetmaster/puppetdb.yaml
A 
modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml
M modules/profile/manifests/puppetdb.pp
M modules/puppetdb/manifests/app.pp
M modules/puppetdb/templates/puppetdb.service.erb
M modules/puppetmaster/manifests/puppetdb.pp
M modules/role/manifests/puppetmaster/puppetdb.pp
7 files changed, 36 insertions(+), 10 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/puppetmaster/puppetdb.yaml 
b/hieradata/role/common/puppetmaster/puppetdb.yaml
index fd8c9a8..b3be4d0 100644
--- a/hieradata/role/common/puppetmaster/puppetdb.yaml
+++ b/hieradata/role/common/puppetmaster/puppetdb.yaml
@@ -10,3 +10,4 @@
 cidr: 10.192.16.184/32
 profile::puppetdb::master: nitrogen.eqiad.wmnet
 profile::puppetdb::slaves: [nihal.codfw.wmnet]
+puppetmaster::puppetdb::jvm_opts: '-Xmx6g'
diff --git 
a/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml
 
b/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml
new file mode 100644
index 000..6ea2bc3
--- /dev/null
+++ 
b/modules/profile/files/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml
@@ -0,0 +1,8 @@
+---
+lowercaseOutputLabelNames: true
+lowercaseOutputName: false
+whitelistObjectNames:
+  - 'com.puppetlabs.puppetdb.command:type=global,name=*'
+  - 'com.puppetlabs.puppetdb.command:type=replace facts.3,name=*'
+  - 'com.puppetlabs.puppetdb.http.server:type=/v3/commands,name=*'
+  - 'com.puppetlabs.puppetdb.http.server:type=/v3/nodes,name=*'
\ No newline at end of file
diff --git a/modules/profile/manifests/puppetdb.pp 
b/modules/profile/manifests/puppetdb.pp
index b8717af..7c9bea8 100644
--- a/modules/profile/manifests/puppetdb.pp
+++ b/modules/profile/manifests/puppetdb.pp
@@ -1,13 +1,32 @@
 class profile::puppetdb(
 $master = hiera('profile::puppetdb::master'),
-$puppetmasters = hiera('puppetmaster::servers')
+$puppetmasters = hiera('puppetmaster::servers'),
+$jvm_opts = hiera('profile::puppetdb::jvm_opts', '-Xmx4G'),
+$prometheus_nodes = hiera('prometheus_nodes'),
 ) {
+# Prometheus JMX agent for the Puppetdb's JVM
+$jmx_exporter_config_file = 
'/etc/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml'
+$prometheus_jmx_exporter_port = 9400
+$prometheus_java_opts = 
"-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent.jar=${::ipaddress}:${prometheus_jmx_exporter_port}:${jmx_exporter_config_file}"
 # The JVM heap size has been raised to 6G for T170740
 class { '::puppetmaster::puppetdb':
-master=> $master,
-heap_size => '6G',
+master   => $master,
+jvm_opts => "${jvm_opts} ${prometheus_java_opts}",
 }
 
+
+# Export JMX metrics to prometheus
+profile::prometheus::jmx_exporter { "puppetdb_${::hostname}":
+hostname => $::hostname,
+port => $prometheus_jmx_exporter_port,
+prometheus_nodes => $prometheus_nodes,
+config_file  => $jmx_exporter_config_file,
+source   => 
'puppet:///modules/profile/puppetmaster/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml',
+}
+
+
+# Firewall rules
+
 # Only the TLS-terminating nginx proxy will be exposed
 $puppetmasters_ferm = inline_template('<%= 
@puppetmasters.values.flatten(1).map { |p| p[\'worker\'] }.sort.join(\' \')%>')
 
diff --git a/modules/puppetdb/manifests/app.pp 
b/modules/puppetdb/manifests/app.pp
index a012ee1..1b64d57 100644
--- a/modules/puppetdb/manifests/app.pp
+++ b/modules/puppetdb/manifests/app.pp
@@ -11,7 +11,7 @@
 $db_user='puppetdb',
 $db_password=undef,
 $perform_gc=false,
-$heap_size='4G',
+$jvm_opts='-Xmx4G',
 $bind_ip=undef,
 $ssldir=puppet_ssldir(),
 $command_processing_threads=16,
diff --git a/modules/puppetdb/templates/puppetdb.service.erb 
b/modules/puppetdb/templates/puppetdb.service.erb
index cef26bd..3bcd7ba 100644
--- a/modules/puppetdb/templates/puppetdb.service.erb
+++ b/modules/puppetdb/templates/puppetdb.service.erb
@@ -6,8 +6,8 @@
 Group=puppetdb
 Environment=CONFIG=/etc/puppetdb/conf.d
 ExecStartPre=/bin/bash -c "test -e /var/log/puppetdb/puppetdb-oom.hprof && mv 
/var/log/puppetdb/puppetdb-oom.hprof 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cumin: stop cross-referencing the puppetdb master

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403590 )

Change subject: cumin: stop cross-referencing the puppetdb master
..


cumin: stop cross-referencing the puppetdb master

Also, it might make sense to use something different in the future.

Change-Id: I770e6f6793ee379538f35750dea284b9f005e1a1
---
M hieradata/role/common/cluster/management.yaml
M modules/profile/manifests/cumin/master.pp
2 files changed, 3 insertions(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/cluster/management.yaml 
b/hieradata/role/common/cluster/management.yaml
index c9f0069..71632dd 100644
--- a/hieradata/role/common/cluster/management.yaml
+++ b/hieradata/role/common/cluster/management.yaml
@@ -1,3 +1,5 @@
 admin::groups:
   - datacenter-ops
 monitor_screens: false
+# TODO: use nihal in codfw?
+profile::cumin::master::puppetdb_host: nitrogen.eqiad.wmnet
diff --git a/modules/profile/manifests/cumin/master.pp 
b/modules/profile/manifests/cumin/master.pp
index 1f0502d..dcf11aa 100644
--- a/modules/profile/manifests/cumin/master.pp
+++ b/modules/profile/manifests/cumin/master.pp
@@ -1,5 +1,5 @@
 class profile::cumin::master (
-$puppetdb_host  = hiera('puppetmaster::puppetdb::master'),
+$puppetdb_host  = hiera('profile::cumin::master::puppetdb_host'),
 $datacenters= hiera('datacenters'),
 ) {
 include passwords::phabricator

-- 
To view, visit https://gerrit.wikimedia.org/r/403590
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I770e6f6793ee379538f35750dea284b9f005e1a1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cumin: stop cross-referencing the puppetdb master

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403590 )

Change subject: cumin: stop cross-referencing the puppetdb master
..

cumin: stop cross-referencing the puppetdb master

Also, it might make sense to use something different in the future.

Change-Id: I770e6f6793ee379538f35750dea284b9f005e1a1
---
M hieradata/role/common/cluster/management.yaml
M modules/profile/manifests/cumin/master.pp
2 files changed, 3 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/90/403590/1

diff --git a/hieradata/role/common/cluster/management.yaml 
b/hieradata/role/common/cluster/management.yaml
index c9f0069..71632dd 100644
--- a/hieradata/role/common/cluster/management.yaml
+++ b/hieradata/role/common/cluster/management.yaml
@@ -1,3 +1,5 @@
 admin::groups:
   - datacenter-ops
 monitor_screens: false
+# TODO: use nihal in codfw?
+profile::cumin::master::puppetdb_host: nitrogen.eqiad.wmnet
diff --git a/modules/profile/manifests/cumin/master.pp 
b/modules/profile/manifests/cumin/master.pp
index 1f0502d..dcf11aa 100644
--- a/modules/profile/manifests/cumin/master.pp
+++ b/modules/profile/manifests/cumin/master.pp
@@ -1,5 +1,5 @@
 class profile::cumin::master (
-$puppetdb_host  = hiera('puppetmaster::puppetdb::master'),
+$puppetdb_host  = hiera('profile::cumin::master::puppetdb_host'),
 $datacenters= hiera('datacenters'),
 ) {
 include passwords::phabricator

-- 
To view, visit https://gerrit.wikimedia.org/r/403590
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I770e6f6793ee379538f35750dea284b9f005e1a1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: puppetdb: refactor to role/profile

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403388 )

Change subject: puppetdb: refactor to role/profile
..


puppetdb: refactor to role/profile

* Split the content of the puppetdb role in two profiles: one for the
  puppetdb application, one for the corresponding postgresql database,
  that can be used separately
* Rework puppetmaster::puppetdb::database, that was a mess.
* Move monitoring to the profile for the database, move db tuning to the
  module class itself
* Added type checking to the parameters of puppetmaster::puppetdb::database

Change-Id: I93b02c19398e4fe16818910dfd3ed6f0653aa3a6
---
D hieradata/common/puppetmaster/puppetdb.yaml
A hieradata/role/common/puppetmaster/puppetdb.yaml
A modules/profile/manifests/puppetdb.pp
A modules/profile/manifests/puppetdb/database.pp
M modules/puppetmaster/manifests/puppetdb/database.pp
R modules/puppetmaster/templates/puppetdb/tuning.conf.erb
M modules/role/manifests/puppetmaster/puppetdb.pp
7 files changed, 152 insertions(+), 114 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/common/puppetmaster/puppetdb.yaml 
b/hieradata/common/puppetmaster/puppetdb.yaml
deleted file mode 100644
index 4de1c15..000
--- a/hieradata/common/puppetmaster/puppetdb.yaml
+++ /dev/null
@@ -1,14 +0,0 @@
-puppetmaster::puppetdb::postgres_users:
-  replication@nihal-v4:
-user: replication
-database: all
-password: "%{::puppetmaster::puppetdb::database::replication_pass}"
-cidr: 10.192.16.184/32
-attrs: REPLICATION
-  puppetdb@nihal-v4:
-user: puppetdb
-database: puppetdb
-password: "%{::puppetmaster::puppetdb::database::puppetdb_pass}"
-cidr: 10.192.16.184/32
-puppetmaster::puppetdb::master: nitrogen.eqiad.wmnet
-puppetmaster::puppetdb::slaves: [nihal.codfw.wmnet]
diff --git a/hieradata/role/common/puppetmaster/puppetdb.yaml 
b/hieradata/role/common/puppetmaster/puppetdb.yaml
new file mode 100644
index 000..fd8c9a8
--- /dev/null
+++ b/hieradata/role/common/puppetmaster/puppetdb.yaml
@@ -0,0 +1,12 @@
+profile::puppetdb::database::users:
+  replication@nihal-v4:
+user: replication
+database: all
+cidr: 10.192.16.184/32
+attrs: REPLICATION
+  puppetdb@nihal-v4:
+user: puppetdb
+database: puppetdb
+cidr: 10.192.16.184/32
+profile::puppetdb::master: nitrogen.eqiad.wmnet
+profile::puppetdb::slaves: [nihal.codfw.wmnet]
diff --git a/modules/profile/manifests/puppetdb.pp 
b/modules/profile/manifests/puppetdb.pp
new file mode 100644
index 000..b8717af
--- /dev/null
+++ b/modules/profile/manifests/puppetdb.pp
@@ -0,0 +1,26 @@
+class profile::puppetdb(
+$master = hiera('profile::puppetdb::master'),
+$puppetmasters = hiera('puppetmaster::servers')
+) {
+# The JVM heap size has been raised to 6G for T170740
+class { '::puppetmaster::puppetdb':
+master=> $master,
+heap_size => '6G',
+}
+
+# Only the TLS-terminating nginx proxy will be exposed
+$puppetmasters_ferm = inline_template('<%= 
@puppetmasters.values.flatten(1).map { |p| p[\'worker\'] }.sort.join(\' \')%>')
+
+ferm::service { 'puppetdb':
+proto   => 'tcp',
+port=> 443,
+notrack => true,
+srange  => "@resolve((${puppetmasters_ferm}))",
+}
+
+ferm::service { 'puppetdb-cumin':
+proto  => 'tcp',
+port   => 443,
+srange => '$CUMIN_MASTERS',
+}
+}
diff --git a/modules/profile/manifests/puppetdb/database.pp 
b/modules/profile/manifests/puppetdb/database.pp
new file mode 100644
index 000..81e3af7
--- /dev/null
+++ b/modules/profile/manifests/puppetdb/database.pp
@@ -0,0 +1,53 @@
+# == Class profile::puppetdb::database
+#
+# Sets up a puppetdb postgresql database.
+#
+class profile::puppetdb::database(
+$master = hiera('profile::puppetdb::master'),
+$slaves = hiera('profile::puppetdb::slaves'),
+$shared_buffers = hiera('profile::puppetdb::database::shared_buffers', 
'7680MB'),
+$replication_password = hiera('puppetdb::password::replication'),
+$puppetdb_password =  hiera('puppetdb::password::rw'),
+$users = hiera('profile::puppetdb::database::users', {}),
+) {
+include ::passwords::postgres
+
+$pgversion = $::lsbdistcodename ? {
+'stretch' => '9.6',
+'jessie'  => '9.4',
+}
+$slave_range = join($slaves, ' ')
+
+$role = $master ? {
+$::fqdn => 'master',
+default => 'slave',
+}
+
+class { '::puppetmaster::puppetdb::database':
+master   => $master,
+pgversion=> $pgversion,
+shared_buffers   => $shared_buffers,
+replication_pass => $replication_password,
+puppetdb_pass=> $puppetdb_password,
+puppetdb_users   => $users,
+}
+
+# Monitoring
+class { '::prometheus::postgres_exporter': }
+
+if $role 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base::resolving: remove useless "else" clause

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403439 )

Change subject: base::resolving: remove useless "else" clause
..

base::resolving: remove useless "else" clause

Change-Id: I3ca4358f02835e89864ab15b550e9346fdc3f0b5
---
M modules/base/manifests/resolving.pp
1 file changed, 34 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/39/403439/1

diff --git a/modules/base/manifests/resolving.pp 
b/modules/base/manifests/resolving.pp
index 5501a5b..86fccc3 100644
--- a/modules/base/manifests/resolving.pp
+++ b/modules/base/manifests/resolving.pp
@@ -5,44 +5,43 @@
 if ! $::nameservers {
 fail('Variable $::nameservers is not defined!')
 }
-else {
-if $::realm == 'labs' {
-$labs_tld = hiera('labs_tld')
-# Thanks to dhcp, resolvconf is constantly messing with our 
resolv.conf.  Disable it.
-file { '/sbin/resolvconf':
-owner  => 'root',
-group  => 'root',
-mode   => '0555',
-source => 'puppet:///modules/base/resolv/resolvconf.dummy',
-}
 
-file { '/etc/dhcp/dhclient-enter-hooks.d':
-ensure => 'directory',
-}
+if $::realm == 'labs' {
+$labs_tld = hiera('labs_tld')
+# Thanks to dhcp, resolvconf is constantly messing with our 
resolv.conf.  Disable it.
+file { '/sbin/resolvconf':
+owner  => 'root',
+group  => 'root',
+mode   => '0555',
+source => 'puppet:///modules/base/resolv/resolvconf.dummy',
+}
 
-# also stop dhclient from updating resolv.conf.
-file { '/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate':
-owner   => 'root',
-group   => 'root',
-mode=> '0555',
-source  => 'puppet:///modules/base/resolv/nodnsupdate',
-require => File['/etc/dhcp/dhclient-enter-hooks.d'],
-}
+file { '/etc/dhcp/dhclient-enter-hooks.d':
+ensure => 'directory',
+}
 
-# Now, finally, we can just puppetize the damn file
-file { '/etc/resolv.conf':
-owner   => 'root',
-group   => 'root',
-mode=> '0444',
-content => template('base/resolv.conf.labs.erb'),
-}
-} else {
-file { '/etc/resolv.conf':
-owner   => 'root',
-group   => 'root',
-mode=> '0444',
-content => template('base/resolv.conf.erb'),
-}
+# also stop dhclient from updating resolv.conf.
+file { '/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate':
+owner   => 'root',
+group   => 'root',
+mode=> '0555',
+source  => 'puppet:///modules/base/resolv/nodnsupdate',
+require => File['/etc/dhcp/dhclient-enter-hooks.d'],
+}
+
+# Now, finally, we can just puppetize the damn file
+file { '/etc/resolv.conf':
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+content => template('base/resolv.conf.labs.erb'),
+}
+} else {
+file { '/etc/resolv.conf':
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+content => template('base/resolv.conf.erb'),
 }
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/403439
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3ca4358f02835e89864ab15b550e9346fdc3f0b5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base::resolving: explicitly pass arguments

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403440 )

Change subject: base::resolving: explicitly pass arguments
..

base::resolving: explicitly pass arguments

Instead of relying on top-scope variable overriding each other, do the
following:
* Remove the conditional on $nameserver_override in the resolv.conf
  template
* Add $nameservers as a local variable, which defaults to $::nameservers
* Pass the value from profile::base in case it's defined in hiera, and
  have care to exclude the IP of the current node from the list to avoid
  self-dependencies
* Remove all the $nameservers_override from the node definitions and add
  those to per-site, per-role hiera

Change-Id: Ib0926a8966db2066d87a9ddea4265ed741c07437
---
M hieradata/role/codfw/lvs/balancer.yaml
M hieradata/role/codfw/recursor.yaml
M hieradata/role/eqiad/lvs/balancer.yaml
M hieradata/role/eqiad/recursor.yaml
M hieradata/role/esams/lvs/balancer.yaml
M hieradata/role/ulsfo/lvs/balancer.yaml
M manifests/site.pp
M modules/base/manifests/resolving.pp
M modules/base/templates/resolv.conf.erb
M modules/profile/manifests/base.pp
10 files changed, 45 insertions(+), 49 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/40/403440/1

diff --git a/hieradata/role/codfw/lvs/balancer.yaml 
b/hieradata/role/codfw/lvs/balancer.yaml
index 5fd35e0..19d968c 100644
--- a/hieradata/role/codfw/lvs/balancer.yaml
+++ b/hieradata/role/codfw/lvs/balancer.yaml
@@ -8,3 +8,11 @@
   - public1-b-codfw
   - public1-c-codfw
   - public1-d-codfw
+# lvs200[25] are LVS balancers for the codfw recursive DNS IP,
+#   so they need to use the recursive DNS backends directly
+#   (acamar and achernar) with fallback to eqiad
+# (doing this for all lvs for now, see T103921)
+profile::base::nameservers:
+  - '208.80.153.12' # acamar
+  - '208.80.153.42' # achenar
+  - '208.80.154.254' # eqiad lvs
diff --git a/hieradata/role/codfw/recursor.yaml 
b/hieradata/role/codfw/recursor.yaml
index d3fb55e..1b56bf8 100644
--- a/hieradata/role/codfw/recursor.yaml
+++ b/hieradata/role/codfw/recursor.yaml
@@ -1,3 +1,7 @@
 profile::bird::neighbors_list:
   - 208.80.153.192 # cr1-codfw loopback
   - 208.80.153.193 # cr2-codfw loopback
+profile::base::nameservers:
+  - '208.80.153.12' # acamar
+  - '208.80.153.42' # achenar
+  - '208.80.154.254' # eqiad lvs
diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index 4b1764e..b14b9ba 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -8,3 +8,11 @@
   - public1-b-eqiad
   - public1-c-eqiad
   - public1-d-eqiad
+# lvs100[25] are LVS balancers for the eqiad recursive DNS IP,
+#   so they need to use the recursive DNS backends directly
+#   (chromium and hydrogen) with fallback to codfw
+# (doing this for all lvs for now, see T103921)
+profile::base::nameservers:
+  - '208.80.154.50' # hydrogen
+  - '208.80.154.157' # chromium
+  - '208.80.153.254' # codfw lvs
diff --git a/hieradata/role/eqiad/recursor.yaml 
b/hieradata/role/eqiad/recursor.yaml
index 446f990..043a099 100644
--- a/hieradata/role/eqiad/recursor.yaml
+++ b/hieradata/role/eqiad/recursor.yaml
@@ -1,3 +1,7 @@
 profile::bird::neighbors_list:
   - 208.80.154.196 # cr1-eqiad loopback
   - 208.80.154.197 # cr2-eqiad loopback
+profile::base::nameservers:
+  - '208.80.154.50' # hydrogen
+  - '208.80.154.157' # chromium
+  - '208.80.153.254' # codfw lvs
diff --git a/hieradata/role/esams/lvs/balancer.yaml 
b/hieradata/role/esams/lvs/balancer.yaml
index c4e9150..3b1d667 100644
--- a/hieradata/role/esams/lvs/balancer.yaml
+++ b/hieradata/role/esams/lvs/balancer.yaml
@@ -1,3 +1,11 @@
 profile::pybal::config_host: conf1003.eqiad.wmnet
 profile::lvs::tagged_subnets:
   - public1-esams
+# lvs300[24] are LVS balancers for the esams recursive DNS IP,
+#   so they need to use the recursive DNS backends directly
+#   (nescio and maerlant) with fallback to eqiad
+# (doing this for all lvs for now, see T103921)
+profile::base::nameservers:
+  - '91.198.174.106' # nescio
+  - '91.198.174.122' # maerlant
+  - '208.80.154.254' # eqiad lvs
diff --git a/hieradata/role/ulsfo/lvs/balancer.yaml 
b/hieradata/role/ulsfo/lvs/balancer.yaml
index ada9df4..d4976fb 100644
--- a/hieradata/role/ulsfo/lvs/balancer.yaml
+++ b/hieradata/role/ulsfo/lvs/balancer.yaml
@@ -1,2 +1,7 @@
 profile::pybal::config_host: conf2003.codfw.wmnet
 profile::lvs::tagged_subnets: []
+# ns override for all lvs for now, see T103921
+profile::base::nameservers:
+  - '208.80.153.12' # acamar
+  - '208.80.153.42' # achenar
+  - '208.80.154.254' # eqiad lvs
diff --git a/manifests/site.pp b/manifests/site.pp
index 020e122..00eae02 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -12,17 +12,11 @@
 node 'acamar.wikimedia.org' {
 role(recursor)
 
-# use achernar (directly) + eqiad LVS (avoid self-dep)
-

[MediaWiki-commits] [Gerrit] operations/puppet[production]: puppetdb: refactor to role/profile

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403388 )

Change subject: puppetdb: refactor to role/profile
..

puppetdb: refactor to role/profile

* Split the content of the puppetdb role in two profiles: one for the
  puppetdb application, one for the corresponding postgresql database,
  that can be used separately
* Rework puppetmaster::puppetdb::database, that was a mess.
* Move monitoring to the profile for the database, move db tuning to the
  module class itself
* Added type checking to the parameters of puppetmaster::puppetdb::database

Change-Id: I93b02c19398e4fe16818910dfd3ed6f0653aa3a6
---
R hieradata/role/common/puppetmaster/puppetdb.yaml
A modules/profile/manifests/puppetdb.pp
A modules/profile/manifests/puppetdb/database.pp
M modules/puppetmaster/manifests/puppetdb/database.pp
R modules/puppetmaster/templates/puppetdb/tuning.conf.erb
M modules/role/manifests/puppetmaster/puppetdb.pp
6 files changed, 132 insertions(+), 102 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/88/403388/1

diff --git a/hieradata/common/puppetmaster/puppetdb.yaml 
b/hieradata/role/common/puppetmaster/puppetdb.yaml
similarity index 63%
rename from hieradata/common/puppetmaster/puppetdb.yaml
rename to hieradata/role/common/puppetmaster/puppetdb.yaml
index 4de1c15..6b56a8a 100644
--- a/hieradata/common/puppetmaster/puppetdb.yaml
+++ b/hieradata/role/common/puppetmaster/puppetdb.yaml
@@ -1,14 +1,14 @@
-puppetmaster::puppetdb::postgres_users:
+profile::puppetdb::postgres_users:
   replication@nihal-v4:
 user: replication
 database: all
-password: "%{::puppetmaster::puppetdb::database::replication_pass}"
+password: "%{::profile::puppetdb::database::replication_password}"
 cidr: 10.192.16.184/32
 attrs: REPLICATION
   puppetdb@nihal-v4:
 user: puppetdb
 database: puppetdb
-password: "%{::puppetmaster::puppetdb::database::puppetdb_pass}"
+password: "%{::profile::puppetdb::database::puppetdb_password}"
 cidr: 10.192.16.184/32
 puppetmaster::puppetdb::master: nitrogen.eqiad.wmnet
 puppetmaster::puppetdb::slaves: [nihal.codfw.wmnet]
diff --git a/modules/profile/manifests/puppetdb.pp 
b/modules/profile/manifests/puppetdb.pp
new file mode 100644
index 000..b8717af
--- /dev/null
+++ b/modules/profile/manifests/puppetdb.pp
@@ -0,0 +1,26 @@
+class profile::puppetdb(
+$master = hiera('profile::puppetdb::master'),
+$puppetmasters = hiera('puppetmaster::servers')
+) {
+# The JVM heap size has been raised to 6G for T170740
+class { '::puppetmaster::puppetdb':
+master=> $master,
+heap_size => '6G',
+}
+
+# Only the TLS-terminating nginx proxy will be exposed
+$puppetmasters_ferm = inline_template('<%= 
@puppetmasters.values.flatten(1).map { |p| p[\'worker\'] }.sort.join(\' \')%>')
+
+ferm::service { 'puppetdb':
+proto   => 'tcp',
+port=> 443,
+notrack => true,
+srange  => "@resolve((${puppetmasters_ferm}))",
+}
+
+ferm::service { 'puppetdb-cumin':
+proto  => 'tcp',
+port   => 443,
+srange => '$CUMIN_MASTERS',
+}
+}
diff --git a/modules/profile/manifests/puppetdb/database.pp 
b/modules/profile/manifests/puppetdb/database.pp
new file mode 100644
index 000..13b5407
--- /dev/null
+++ b/modules/profile/manifests/puppetdb/database.pp
@@ -0,0 +1,52 @@
+# == Class profile::puppetdb::database
+#
+# Sets up a puppetdb postgresql database.
+#
+class profile::puppetdb::database(
+$master = hiera('profile::puppetdb::master'),
+$slaves = hiera('profile::puppetdb::slaves'),
+$shared_buffers = hiera('profile::puppetdb::database::shared_buffers', 
'7680MB'),
+$replication_password = hiera('puppetdb::password::replication'),
+$puppetdb_password =  hiera('puppetdb::password::rw'),
+$users = hiera('profile::puppetdb::database::users', {}),
+) {
+include ::passwords::postgres
+
+$pgversion = $::lsbdistcodename ? {
+'stretch' => '9.6',
+'jessie'  => '9.4',
+}
+$slave_range = join($slaves, ' ')
+
+$role = $master ? {
+$::fqdn => 'master',
+default => 'slave',
+}
+
+class { '::puppetmaster::puppetdb::database':
+master   => $master,
+pgversion=> $pgversion,
+replication_pass => $replication_password,
+puppetdb_pass=> $puppetdb_password,
+puppetdb_users   => $users,
+}
+
+# Monitoring
+class { '::prometheus::postgres_exporter': }
+
+if $role == 'slave' {
+class { 'postgresql::slave::monitoring':
+pg_master   => $master,
+pg_user => 'replication',
+pg_password => $replication_password,
+}
+}
+
+# Firewall rules
+# Allow connections from all the slaves
+ferm::service { 'postgresql_puppetdb':
+proto  => 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix Style/NegatedIf RuboCop offense across the tree

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359484 )

Change subject: Fix Style/NegatedIf RuboCop offense across the tree
..


Fix Style/NegatedIf RuboCop offense across the tree

Basically replacing "if !foo" with "unless foo".

Change-Id: Ifdce182962937533ba4b7c99fe8095f00687a7f4
---
M .rubocop_todo.yml
M Rakefile
M modules/base/files/monitoring/check_puppetrun
M modules/network/lib/puppet/parser/functions/slice_network_constants.rb
M modules/puppetmaster/lib/puppet/reports/servermon.rb
M modules/strongswan/Rakefile
M modules/wmflib/lib/puppet/parser/functions/role.rb
M modules/wmflib/lib/puppet/parser/functions/ssl_ciphersuite.rb
M rake_modules/specdeps.rb
M rake_modules/taskgen.rb
M utils/hiera_lookup
11 files changed, 16 insertions(+), 19 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 03bb06f..c783ae3 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -26,9 +26,6 @@
 Layout/IndentationWidth:
   Enabled: false
 
-Style/NegatedIf:
-  Enabled: false
-
 Style/RescueModifier:
   Exclude:
 - 'modules/wmflib/lib/puppet/parser/functions/require_package.rb'
diff --git a/Rakefile b/Rakefile
index 1e50b08..d13815e 100644
--- a/Rakefile
+++ b/Rakefile
@@ -69,7 +69,7 @@
   module_name = Regexp.last_match(1)
   task module_name do
 spec_result = system("cd 'modules/#{module_name}' && rake spec")
-spec_failed << module_name if !spec_result
+spec_failed << module_name unless spec_result
   end
   spec_tasks << "spec:#{module_name}"
 end
diff --git a/modules/base/files/monitoring/check_puppetrun 
b/modules/base/files/monitoring/check_puppetrun
index 7ad4a4d..0dc6fb5 100755
--- a/modules/base/files/monitoring/check_puppetrun
+++ b/modules/base/files/monitoring/check_puppetrun
@@ -70,7 +70,7 @@
 if File.exists?(summaryfile)
 begin
 summary = YAML.load_file(summaryfile)
-if !enabled
+unless enabled
 summary["time"]["last_disabled_check"] = Time.now.to_i
 File.open(summaryfile, 'w') { |f| f.write(YAML.dump summary) }
 end
@@ -134,7 +134,7 @@
 exit 0
 end
 
-if !enabled
+unless enabled
 puts "WARNING: Puppet is currently disabled, message: #{disabled_message}, 
last run #{human_time_since_last_run} ago with #{failcount} failures"
 exit 1
 end
diff --git 
a/modules/network/lib/puppet/parser/functions/slice_network_constants.rb 
b/modules/network/lib/puppet/parser/functions/slice_network_constants.rb
index d4c6b33..322dd07 100644
--- a/modules/network/lib/puppet/parser/functions/slice_network_constants.rb
+++ b/modules/network/lib/puppet/parser/functions/slice_network_constants.rb
@@ -47,7 +47,7 @@
 af = options['af'] if options
 description = options['description'] if options
 
-if !all_network_subnets.key?(realm)
+unless all_network_subnets.key?(realm)
 fail ArgumentError, 'slice_network_constants() realm non existant in 
network::subnets'
 end
 if requested_site && (!all_network_subnets[realm].key? requested_site)
diff --git a/modules/puppetmaster/lib/puppet/reports/servermon.rb 
b/modules/puppetmaster/lib/puppet/reports/servermon.rb
index 2d55a84..47ea7a1 100644
--- a/modules/puppetmaster/lib/puppet/reports/servermon.rb
+++ b/modules/puppetmaster/lib/puppet/reports/servermon.rb
@@ -54,7 +54,7 @@
 end
 # We failed to get all of our configs, let's bail early to avoid 
causing
 # puppet issues
-if !(dbserver && dbuser && dbpassword && log_level)
+unless dbserver && dbuser && dbpassword && log_level
   return true
 end
 begin
diff --git a/modules/strongswan/Rakefile b/modules/strongswan/Rakefile
index 0cc35f8..622ce5a 100644
--- a/modules/strongswan/Rakefile
+++ b/modules/strongswan/Rakefile
@@ -14,7 +14,7 @@
 task :setup do
   FileUtils.mkdir_p(format('spec/fixtures/modules/%s', modulename))
   symlinks.each do |x|
-if !File.exist?(x[0])
+unless File.exist?(x[0])
   FileUtils.ln_s(x[1], x[0])
 end
   end
diff --git a/modules/wmflib/lib/puppet/parser/functions/role.rb 
b/modules/wmflib/lib/puppet/parser/functions/role.rb
index 1fae3c2..971782b 100644
--- a/modules/wmflib/lib/puppet/parser/functions/role.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/role.rb
@@ -42,7 +42,7 @@
 # - Include class role::#{arg} if present
 
 # Prevent use outside of node definitions
-if !is_nodescope?
+unless is_nodescope?
   raise Puppet::ParseError,
 "role can only be used in node scope, while you are in scope 
#{self}"
 end
diff --git a/modules/wmflib/lib/puppet/parser/functions/ssl_ciphersuite.rb 
b/modules/wmflib/lib/puppet/parser/functions/ssl_ciphersuite.rb
index e12ced2..4556fe9 100644
--- 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base: fix spec tests for puppet 4

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403188 )

Change subject: base: fix spec tests for puppet 4
..


base: fix spec tests for puppet 4

Change-Id: I70175736de9cbf44ddb44b5c373b7467e6626fd6
---
M modules/base/spec/classes/puppet_spec.rb
M modules/base/spec/defines/expose_puppet_certs_spec.rb
A modules/base/spec/fixtures/hiera.yaml
A modules/base/spec/fixtures/hieradata/common.yaml
A modules/base/spec/fixtures/hieradata/labs.yaml
A modules/base/spec/fixtures/hieradata/some_realm.yaml
M modules/base/spec/functions/pick_initscript_spec.rb
M modules/base/spec/spec_helper.rb
8 files changed, 48 insertions(+), 44 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/base/spec/classes/puppet_spec.rb 
b/modules/base/spec/classes/puppet_spec.rb
index 0c66cb2..0570f47 100644
--- a/modules/base/spec/classes/puppet_spec.rb
+++ b/modules/base/spec/classes/puppet_spec.rb
@@ -1,49 +1,47 @@
 require 'spec_helper'
+test_on = {
+  supported_os: [
+{
+  'operatingsystem'=> 'Debian',
+  'operatingsystemrelease' => ['8', '9'],
+}
+  ]
+}
 
 describe 'base::puppet' do
-let(:pre_condition) {
-'''
-class passwords::puppet::database {}
-'''
-}
-before(:each) do
-Puppet::Parser::Functions.newfunction(:os_version, :type => :rvalue) 
do |_args|
-TRUE
-end
-end
-it { should compile }
+  let(:pre_condition) {
+['class passwords::puppet::database {}']
+  }
+  on_supported_os(test_on).each do |os, facts|
+context "On #{os}" do
+  let(:facts) { facts}
+  it { should compile }
 
-context 'when auto_puppetmaster_switching is enabled' do
-before(:each) {
-Puppet::Parser::Functions.newfunction(:hiera, :type => :rvalue) do 
|args|
-case args[0]
-when 'auto_puppetmaster_switching'
-return TRUE
-else
-return args[1]
-end
-end
-}
+  context 'when auto_puppetmaster_switching is enabled' do
 context 'on labs' do
-let(:facts) { { :realm => 'labs' } }
-it { should compile }
-context 'on a standalone puppetmaster' do
-let(:pre_condition) {
-super().concat('''
-class role::puppetmaster::standalone {}
-include role::puppetmaster::standalone
-''')
-}
-it 'should fail' do
-should compile.and_raise_error(/should only be applied on 
puppet clients/)
-end
+  let(:node_params) { { :realm => 'labs' } }
+  it { should compile }
+  context 'on a standalone puppetmaster' do
+let(:pre_condition) {
+  super().concat(
+[
+  'class role::puppetmaster::standalone {}',
+  'require role::puppetmaster::standalone'
+]
+  )
+}
+it 'should fail' do
+  should compile.and_raise_error(/should only be applied on puppet 
clients/)
 end
+  end
 end
 context 'on other realms' do
-let(:facts) { { :realm => 'some realm' } }
-it 'auto_puppetmaster_switching must not be enableable' do
-should compile.and_raise_error(/auto_puppetmaster_switching 
should never.*/)
-end
+  let(:node_params) {{:realm => 'some_realm'}}
+  it 'auto_puppetmaster_switching must not be enableable' do
+should compile.and_raise_error(/auto_puppetmaster_switching should 
never.*/)
+  end
 end
+  end
 end
+  end
 end
diff --git a/modules/base/spec/defines/expose_puppet_certs_spec.rb 
b/modules/base/spec/defines/expose_puppet_certs_spec.rb
index 0d12490..bab24f9 100644
--- a/modules/base/spec/defines/expose_puppet_certs_spec.rb
+++ b/modules/base/spec/defines/expose_puppet_certs_spec.rb
@@ -37,7 +37,7 @@
   describe 'all files are removed when ensure => absent' do
 let(:params) { { :ensure => 'absent' } }
 
-it { should contain_file('/my/ssl/dir/ssl/').with({ 'ensure' => 'absent' 
}) }
+it { should contain_file('/my/ssl/dir/ssl').with({ 'ensure' => 'absent' }) 
}
 it { should contain_file('/my/ssl/dir/ssl/cert.pem').with({ 'ensure' => 
'absent' }) }
 it { should contain_file('/my/ssl/dir/ssl/server.key').with({ 'ensure' => 
'absent' }) }
   end
diff --git a/modules/base/spec/fixtures/hiera.yaml 
b/modules/base/spec/fixtures/hiera.yaml
new file mode 100644
index 000..6aac931
--- /dev/null
+++ b/modules/base/spec/fixtures/hiera.yaml
@@ -0,0 +1,6 @@
+:backends:
+  - yaml
+:yaml:
+  :datadir: 'spec/fixtures/hieradata'
+:hierarchy:
+ 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: service: properly fix specs for puppet 4

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403189 )

Change subject: service: properly fix specs for puppet 4
..


service: properly fix specs for puppet 4

Change-Id: I2f206c4719d94176eb2291917411bf8f14c9104b
---
M modules/service/spec/defines/service_node_spec.rb
M modules/service/spec/spec_helper.rb
2 files changed, 33 insertions(+), 18 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/service/spec/defines/service_node_spec.rb 
b/modules/service/spec/defines/service_node_spec.rb
index 7d2fd92..3c6ebaa 100644
--- a/modules/service/spec/defines/service_node_spec.rb
+++ b/modules/service/spec/defines/service_node_spec.rb
@@ -1,26 +1,39 @@
 require 'spec_helper'
+test_on = {
+  supported_os: [
+{
+  'operatingsystem'=> 'Debian',
+  'operatingsystemrelease' => ['8', '9'],
+}
+  ]
+}
 
 describe 'service::node', :type => :define do
-  let(:pre_condition) {
-['class passwords::etcd { $accounts = {}}',
- 'include ::passwords::etcd',
- 'class profile::base {
-  $notifications_enabled = "1"
-}
-include ::profile::base'
-]
-  }
-  let(:title) { 'my_service_name' }
-  let(:facts) { { :initsystem => 'systemd' } }
-  let(:node_params) { {'cluster' => 'test', 'site' => 'eqiad', 'realm' => 
'production'} }
-  context 'when only port is given' do
-let(:params) { { :port => 1234 } }
+  on_supported_os(test_on).each do |os, facts|
+context "On #{os}" do
+  let(:facts) do
+facts.merge({initsystem: 'systemd'})
+  end
 
-it { is_expected.to compile }
+  let(:pre_condition) {
+['class passwords::etcd { $accounts = {}}',
+ 'include ::passwords::etcd',
+ 'class profile::base { $notifications_enabled = "1"}',
+ 'include ::profile::base'
+]
+  }
+  let(:title) { 'my_service_name' }
+  let(:node_params) { {'cluster' => 'test', 'site' => 'eqiad', 'realm' => 
'production'} }
+  context 'when only port is given' do
+let(:params) { { :port => 1234 } }
 
-it 'create the appropriate scap target' do
-  is_expected.to contain_scap__target('my_service_name/deploy')
-   .with_service_name('my_service_name')
+it { is_expected.to compile }
+
+it 'create the appropriate scap target' do
+  is_expected.to contain_scap__target('my_service_name/deploy')
+   .with_service_name('my_service_name')
+end
+  end
 end
   end
 end
diff --git a/modules/service/spec/spec_helper.rb 
b/modules/service/spec/spec_helper.rb
index 26af1be..d44ced8 100644
--- a/modules/service/spec/spec_helper.rb
+++ b/modules/service/spec/spec_helper.rb
@@ -1,5 +1,7 @@
 require 'puppetlabs_spec_helper/module_spec_helper'
+require 'rspec-puppet-facts'
 
+include RspecPuppetFacts
 fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures'))
 
 RSpec.configure do |c|

-- 
To view, visit https://gerrit.wikimedia.org/r/403189
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2f206c4719d94176eb2291917411bf8f14c9104b
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Mobrovac 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: service: properly fix specs for puppet 4

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403189 )

Change subject: service: properly fix specs for puppet 4
..

service: properly fix specs for puppet 4

Change-Id: I2f206c4719d94176eb2291917411bf8f14c9104b
---
M modules/service/spec/defines/service_node_spec.rb
M modules/service/spec/spec_helper.rb
2 files changed, 33 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/89/403189/1

diff --git a/modules/service/spec/defines/service_node_spec.rb 
b/modules/service/spec/defines/service_node_spec.rb
index 7d2fd92..3c6ebaa 100644
--- a/modules/service/spec/defines/service_node_spec.rb
+++ b/modules/service/spec/defines/service_node_spec.rb
@@ -1,26 +1,39 @@
 require 'spec_helper'
+test_on = {
+  supported_os: [
+{
+  'operatingsystem'=> 'Debian',
+  'operatingsystemrelease' => ['8', '9'],
+}
+  ]
+}
 
 describe 'service::node', :type => :define do
-  let(:pre_condition) {
-['class passwords::etcd { $accounts = {}}',
- 'include ::passwords::etcd',
- 'class profile::base {
-  $notifications_enabled = "1"
-}
-include ::profile::base'
-]
-  }
-  let(:title) { 'my_service_name' }
-  let(:facts) { { :initsystem => 'systemd' } }
-  let(:node_params) { {'cluster' => 'test', 'site' => 'eqiad', 'realm' => 
'production'} }
-  context 'when only port is given' do
-let(:params) { { :port => 1234 } }
+  on_supported_os(test_on).each do |os, facts|
+context "On #{os}" do
+  let(:facts) do
+facts.merge({initsystem: 'systemd'})
+  end
 
-it { is_expected.to compile }
+  let(:pre_condition) {
+['class passwords::etcd { $accounts = {}}',
+ 'include ::passwords::etcd',
+ 'class profile::base { $notifications_enabled = "1"}',
+ 'include ::profile::base'
+]
+  }
+  let(:title) { 'my_service_name' }
+  let(:node_params) { {'cluster' => 'test', 'site' => 'eqiad', 'realm' => 
'production'} }
+  context 'when only port is given' do
+let(:params) { { :port => 1234 } }
 
-it 'create the appropriate scap target' do
-  is_expected.to contain_scap__target('my_service_name/deploy')
-   .with_service_name('my_service_name')
+it { is_expected.to compile }
+
+it 'create the appropriate scap target' do
+  is_expected.to contain_scap__target('my_service_name/deploy')
+   .with_service_name('my_service_name')
+end
+  end
 end
   end
 end
diff --git a/modules/service/spec/spec_helper.rb 
b/modules/service/spec/spec_helper.rb
index 26af1be..d44ced8 100644
--- a/modules/service/spec/spec_helper.rb
+++ b/modules/service/spec/spec_helper.rb
@@ -1,5 +1,7 @@
 require 'puppetlabs_spec_helper/module_spec_helper'
+require 'rspec-puppet-facts'
 
+include RspecPuppetFacts
 fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures'))
 
 RSpec.configure do |c|

-- 
To view, visit https://gerrit.wikimedia.org/r/403189
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2f206c4719d94176eb2291917411bf8f14c9104b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base: fix spec tests for puppet 4

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403188 )

Change subject: base: fix spec tests for puppet 4
..

base: fix spec tests for puppet 4

Change-Id: I70175736de9cbf44ddb44b5c373b7467e6626fd6
---
M modules/base/spec/classes/puppet_spec.rb
M modules/base/spec/defines/expose_puppet_certs_spec.rb
A modules/base/spec/fixtures/hiera.yaml
A modules/base/spec/fixtures/hieradata/common.yaml
A modules/base/spec/fixtures/hieradata/labs.yaml
A modules/base/spec/fixtures/hieradata/some_realm.yaml
M modules/base/spec/functions/pick_initscript_spec.rb
M modules/base/spec/spec_helper.rb
8 files changed, 49 insertions(+), 43 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/88/403188/1

diff --git a/modules/base/spec/classes/puppet_spec.rb 
b/modules/base/spec/classes/puppet_spec.rb
index 0c66cb2..f737c93 100644
--- a/modules/base/spec/classes/puppet_spec.rb
+++ b/modules/base/spec/classes/puppet_spec.rb
@@ -1,49 +1,48 @@
 require 'spec_helper'
+test_on = {
+  supported_os: [
+{
+  'operatingsystem'=> 'Debian',
+  'operatingsystemrelease' => ['8', '9'],
+}
+  ]
+}
 
 describe 'base::puppet' do
-let(:pre_condition) {
-'''
-class passwords::puppet::database {}
-'''
-}
-before(:each) do
-Puppet::Parser::Functions.newfunction(:os_version, :type => :rvalue) 
do |_args|
-TRUE
-end
-end
-it { should compile }
+  let(:pre_condition) {
+['class passwords::puppet::database {}']
+  }
+  on_supported_os(test_on).each do |os, facts|
+context "On #{os}" do
+  let(:facts) { facts}
+  it { should compile }
 
-context 'when auto_puppetmaster_switching is enabled' do
-before(:each) {
-Puppet::Parser::Functions.newfunction(:hiera, :type => :rvalue) do 
|args|
-case args[0]
-when 'auto_puppetmaster_switching'
-return TRUE
-else
-return args[1]
-end
-end
-}
+  context 'when auto_puppetmaster_switching is enabled' do
+
 context 'on labs' do
-let(:facts) { { :realm => 'labs' } }
-it { should compile }
-context 'on a standalone puppetmaster' do
-let(:pre_condition) {
-super().concat('''
-class role::puppetmaster::standalone {}
-include role::puppetmaster::standalone
-''')
-}
-it 'should fail' do
-should compile.and_raise_error(/should only be applied on 
puppet clients/)
-end
+  let(:node_params) { { :realm => 'labs' } }
+  it { should compile }
+  context 'on a standalone puppetmaster' do
+let(:pre_condition) {
+  super().concat(
+[
+  'class role::puppetmaster::standalone {}',
+  'require role::puppetmaster::standalone'
+]
+  )
+}
+it 'should fail' do
+  should compile.and_raise_error(/should only be applied on puppet 
clients/)
 end
+  end
 end
 context 'on other realms' do
-let(:facts) { { :realm => 'some realm' } }
-it 'auto_puppetmaster_switching must not be enableable' do
-should compile.and_raise_error(/auto_puppetmaster_switching 
should never.*/)
-end
+  let(:node_params) {{:realm => 'some_realm'}}
+  it 'auto_puppetmaster_switching must not be enableable' do
+should compile.and_raise_error(/auto_puppetmaster_switching should 
never.*/)
+  end
 end
+  end
 end
+  end
 end
diff --git a/modules/base/spec/defines/expose_puppet_certs_spec.rb 
b/modules/base/spec/defines/expose_puppet_certs_spec.rb
index 0d12490..bab24f9 100644
--- a/modules/base/spec/defines/expose_puppet_certs_spec.rb
+++ b/modules/base/spec/defines/expose_puppet_certs_spec.rb
@@ -37,7 +37,7 @@
   describe 'all files are removed when ensure => absent' do
 let(:params) { { :ensure => 'absent' } }
 
-it { should contain_file('/my/ssl/dir/ssl/').with({ 'ensure' => 'absent' 
}) }
+it { should contain_file('/my/ssl/dir/ssl').with({ 'ensure' => 'absent' }) 
}
 it { should contain_file('/my/ssl/dir/ssl/cert.pem').with({ 'ensure' => 
'absent' }) }
 it { should contain_file('/my/ssl/dir/ssl/server.key').with({ 'ensure' => 
'absent' }) }
   end
diff --git a/modules/base/spec/fixtures/hiera.yaml 
b/modules/base/spec/fixtures/hiera.yaml
new file mode 100644
index 000..6aac931
--- /dev/null
+++ b/modules/base/spec/fixtures/hiera.yaml
@@ -0,0 +1,6 @@
+:backends:
+  - yaml
+:yaml:
+  :datadir: 'spec/fixtures/hieradata'
+:hierarchy:
+  - 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: utils/expanderrb.rb: fix Style/SpecialGlobalVars

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359483 )

Change subject: utils/expanderrb.rb: fix Style/SpecialGlobalVars
..


utils/expanderrb.rb: fix Style/SpecialGlobalVars

Change-Id: I52960ceacad36e5f2d3ced92e7ec67ac3a9d9213
---
M .rubocop_todo.yml
M utils/expanderb.rb
2 files changed, 3 insertions(+), 6 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index c39ac895..03bb06f 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -36,10 +36,6 @@
 Layout/SpaceInsideHashLiteralBraces:
   Enabled: false
 
-Style/SpecialGlobalVars:
-  Exclude:
-- 'utils/expanderb.rb'
-
 Style/StringLiterals:
   Enabled: false
 
diff --git a/utils/expanderb.rb b/utils/expanderb.rb
index 431c26a..cb2882f 100755
--- a/utils/expanderb.rb
+++ b/utils/expanderb.rb
@@ -12,6 +12,7 @@
 require 'optparse'
 require 'erb'
 require 'ostruct'
+require 'English'
 
 # Filename of the ERB template we are going to expand
 $filename = nil
@@ -41,7 +42,7 @@
   end
 rescue
   # Catch all
-  puts $!.to_s
+  puts $ERROR_INFO.to_s
   puts optparse
   exit
 end
@@ -65,5 +66,5 @@
   puts render_erb(File.read($filename), template_values)
 rescue
   p "Something went wrong, usually because you are missing a variable."
-  p $!.to_s
+  p $ERROR_INFO.to_s
 end

-- 
To view, visit https://gerrit.wikimedia.org/r/359483
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I52960ceacad36e5f2d3ced92e7ec67ac3a9d9213
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon Liambotis 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: base: fix RuboCop MethodCallWithoutArgsParentheses

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359482 )

Change subject: base: fix RuboCop MethodCallWithoutArgsParentheses
..


base: fix RuboCop MethodCallWithoutArgsParentheses

Fix a service_unit_spec.rb Style/MethodCallWithoutArgsParentheses, and
remove the ignore entirely, as the other two ignores are obsolete.

Change-Id: Ie73f028afe7967a60f4e29fabc4c1a1cc5bf7ff4
---
M .rubocop_todo.yml
M modules/base/spec/defines/service_unit_spec.rb
2 files changed, 1 insertion(+), 7 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index c463b8a..c39ac895 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -26,12 +26,6 @@
 Layout/IndentationWidth:
   Enabled: false
 
-Style/MethodCallWithoutArgsParentheses:
-  Exclude:
-- 'modules/base/spec/defines/service_unit_spec.rb'
-- 
'modules/install_server/spec/classes/install_server_preseed_server_spec.rb'
-- 'modules/squid3/spec/classes/squid3_spec.rb'
-
 Style/NegatedIf:
   Enabled: false
 
diff --git a/modules/base/spec/defines/service_unit_spec.rb 
b/modules/base/spec/defines/service_unit_spec.rb
index 2c79374..3aa3914 100644
--- a/modules/base/spec/defines/service_unit_spec.rb
+++ b/modules/base/spec/defines/service_unit_spec.rb
@@ -43,7 +43,7 @@
 it 'should not refresh service' do
   expect {
 should 
contain_file('/etc/init.d/nginx').that_notifies('Service[nginx]')
-  }.to raise_error()
+  }.to raise_error
 end
   end
 

-- 
To view, visit https://gerrit.wikimedia.org/r/359482
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie73f028afe7967a60f4e29fabc4c1a1cc5bf7ff4
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon Liambotis 
Gerrit-Reviewer: Gehel 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wmflib, admin: fix RuboCop Style/For offenses

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359481 )

Change subject: wmflib, admin: fix RuboCop Style/For offenses
..


wmflib, admin: fix RuboCop Style/For offenses

Replace "for" with "each" as that more Ruby-ish. Also, while at it,
cleanup unique_users a tiny little bit and fix another issue with it
(Style/EmptyLiteral).

Change-Id: Icdca0cf24ed990c4afbd9eafd748f7ee207b333c
---
M .rubocop_todo.yml
M modules/admin/lib/puppet/parser/functions/unique_users.rb
M modules/wmflib/lib/puppet/parser/functions/cron_splay.rb
3 files changed, 10 insertions(+), 16 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 0b503bf..c463b8a 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -12,13 +12,6 @@
   Exclude:
 - 'modules/labstore/lib/puppet/parser/functions/mount_nfs_volume.rb'
 
-Style/EmptyLiteral:
-  Exclude:
-- 'modules/admin/lib/puppet/parser/functions/unique_users.rb'
-
-Style/For:
-  Enabled: false
-
 Style/GlobalVars:
   Exclude:
 - 'utils/expanderb.rb'
diff --git a/modules/admin/lib/puppet/parser/functions/unique_users.rb 
b/modules/admin/lib/puppet/parser/functions/unique_users.rb
index 3c98778..77fff67 100644
--- a/modules/admin/lib/puppet/parser/functions/unique_users.rb
+++ b/modules/admin/lib/puppet/parser/functions/unique_users.rb
@@ -1,13 +1,14 @@
 module Puppet::Parser::Functions
-newfunction(:unique_users, :type => :rvalue) do |args|
-myhash = args[0]
-applied_groups = args[1]
-users = Array.new
-for group in applied_groups
+newfunction(:unique_users, :arity => 2, :type => :rvalue) do |args|
+myhash, applied_groups = args
+
+users = []
+applied_groups.each do |group|
 if myhash['groups'].key?(group)
 users.push(myhash['groups'][group]['members'])
 end
 end
-return users.flatten(2).uniq
+
+users.flatten(2).uniq
 end
 end
diff --git a/modules/wmflib/lib/puppet/parser/functions/cron_splay.rb 
b/modules/wmflib/lib/puppet/parser/functions/cron_splay.rb
index 88df67d..e13e403 100644
--- a/modules/wmflib/lib/puppet/parser/functions/cron_splay.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/cron_splay.rb
@@ -79,7 +79,7 @@
 
 # split hosts into N lists based the first digit of //, defaulting to 
zero
 sublists = [[], [], [], [], [], [], [], [], [], []]
-for h in hosts
+hosts.each do |h|
   match = /([1-9])[0-9]{3}/.match(h)
   if match
 sublists[match[1].to_i].push(h)
@@ -89,7 +89,7 @@
 end
 
 # sort each sublist into a determinstic order based on seed
-for s in sublists
+sublists.each do |s|
   s.sort_by! { |x| Digest::MD5.hexdigest(seed + x) }
 end
 
@@ -127,7 +127,7 @@
   output['weekday'] = '*'
 end
 
-return output
+output
   end
 end
 

-- 
To view, visit https://gerrit.wikimedia.org/r/359481
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icdca0cf24ed990c4afbd9eafd748f7ee207b333c
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon Liambotis 
Gerrit-Reviewer: Alex Monk 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wmflib: fix another couple minor RuboCop offenses

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359480 )

Change subject: wmflib: fix another couple minor RuboCop offenses
..


wmflib: fix another couple minor RuboCop offenses

Change-Id: I44083947cd2e739d334b98840ae9246896ab5c40
---
M .rubocop_todo.yml
M modules/wmflib/lib/puppet/parser/functions/array_concat.rb
M modules/wmflib/lib/puppet/parser/functions/validate_array_re.rb
3 files changed, 2 insertions(+), 7 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 8c5ff72..0b503bf 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -2,10 +2,6 @@
   Exclude:
 - 'modules/wmflib/lib/puppet/parser/functions/require_package.rb'
 
-Lint/UselessAssignment:
-  Exclude:
-- 'modules/wmflib/lib/puppet/parser/functions/validate_array_re.rb'
-
 Metrics/LineLength:
   Max: 159
 
@@ -19,7 +15,6 @@
 Style/EmptyLiteral:
   Exclude:
 - 'modules/admin/lib/puppet/parser/functions/unique_users.rb'
-- 'modules/wmflib/lib/puppet/parser/functions/array_concat.rb'
 
 Style/For:
   Enabled: false
diff --git a/modules/wmflib/lib/puppet/parser/functions/array_concat.rb 
b/modules/wmflib/lib/puppet/parser/functions/array_concat.rb
index 89b9212..0fa1c98 100644
--- a/modules/wmflib/lib/puppet/parser/functions/array_concat.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/array_concat.rb
@@ -15,7 +15,7 @@
 #
 module Puppet::Parser::Functions
   newfunction(:array_concat, :type => :rvalue) do |args|
-retval = Array.new
+retval = []
 args.each do |arg|
 if arg.is_a? Array
 retval += arg
diff --git a/modules/wmflib/lib/puppet/parser/functions/validate_array_re.rb 
b/modules/wmflib/lib/puppet/parser/functions/validate_array_re.rb
index 22d1e19..52a8ce6 100644
--- a/modules/wmflib/lib/puppet/parser/functions/validate_array_re.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/validate_array_re.rb
@@ -15,7 +15,7 @@
   newfunction(:validate_array_re, :arity => 2) do |args|
 items, re = args
 re = Regexp.new(re)
-invalid = args.first.find { |item| item.to_s !~ re }
+invalid = items.find { |item| item.to_s !~ re }
 unless invalid.nil?
   fail(Puppet::ParseError, "Array element \"#{invalid}\" does not match 
regular expression \"#{re.source}\".")
 end

-- 
To view, visit https://gerrit.wikimedia.org/r/359480
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I44083947cd2e739d334b98840ae9246896ab5c40
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon Liambotis 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix Style/RegexpLiteral RuboCop offenses

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359479 )

Change subject: Fix Style/RegexpLiteral RuboCop offenses
..


Fix Style/RegexpLiteral RuboCop offenses

Mostly in bacula but also a couple in servermon.

Change-Id: I123ac7fb80d9bcbc0ecf0a55bed1b2f93e908134
---
M .rubocop_todo.yml
M modules/bacula/spec/classes/bacula_client_spec.rb
M modules/bacula/spec/classes/bacula_director_spec.rb
M modules/bacula/spec/classes/bacula_storage_spec.rb
M modules/bacula/spec/defines/fileset_spec.rb
M modules/bacula/spec/defines/storage_device_spec.rb
M modules/servermon/spec/classes/servermon_app_spec.rb
7 files changed, 15 insertions(+), 24 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 681b2a0..8c5ff72 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -47,15 +47,6 @@
 Style/NegatedIf:
   Enabled: false
 
-Style/RegexpLiteral:
-  Exclude:
-- 'modules/bacula/spec/classes/bacula_client_spec.rb'
-- 'modules/bacula/spec/classes/bacula_director_spec.rb'
-- 'modules/bacula/spec/classes/bacula_storage_spec.rb'
-- 'modules/bacula/spec/defines/fileset_spec.rb'
-- 'modules/bacula/spec/defines/storage_device_spec.rb'
-- 'modules/servermon/spec/classes/servermon_app_spec.rb'
-
 Style/RescueModifier:
   Exclude:
 - 'modules/wmflib/lib/puppet/parser/functions/require_package.rb'
diff --git a/modules/bacula/spec/classes/bacula_client_spec.rb 
b/modules/bacula/spec/classes/bacula_client_spec.rb
index 2d01eb4..ebec576 100644
--- a/modules/bacula/spec/classes/bacula_client_spec.rb
+++ b/modules/bacula/spec/classes/bacula_client_spec.rb
@@ -23,10 +23,10 @@
 }) \
 .with_content(/Name = "testdirector"/) \
 .with_content(/Password = "testdirectorpass"/) \
-.with_content(/TLS Certificate = "\/etc\/bacula\/ssl\/cert.pem"/) \
-.with_content(/TLS Key = "\/etc\/bacula\/ssl\/server.key"/) \
+.with_content(%r{TLS Certificate = "/etc/bacula/ssl/cert.pem"}) \
+.with_content(%r{TLS Key = "/etc/bacula/ssl/server.key"}) \
 .with_content(/Name = "testhost.example.com-fd"/) \
 .with_content(/FDport = 2000/) \
-.with_content(/PKI Keypair = "\/etc\/bacula\/ssl\/server-keypair.pem"/)
+.with_content(%r{PKI Keypair = "/etc/bacula/ssl/server-keypair.pem"})
 end
 end
diff --git a/modules/bacula/spec/classes/bacula_director_spec.rb 
b/modules/bacula/spec/classes/bacula_director_spec.rb
index da9f398..c62f930 100644
--- a/modules/bacula/spec/classes/bacula_director_spec.rb
+++ b/modules/bacula/spec/classes/bacula_director_spec.rb
@@ -65,8 +65,8 @@
 }) \
 .with_content(/Name = "testhost.example.com"/) \
 .with_content(/Password = "bconsolepass"/) \
-.with_content(/TLS Certificate = 
"\/etc\/bacula\/director\/ssl\/cert.pem"/) \
-.with_content(/TLS Key = "\/etc\/bacula\/director\/ssl\/server.key"/) \
+.with_content(%r{TLS Certificate = 
"/etc/bacula/director/ssl/cert.pem"}) \
+.with_content(%r{TLS Key = "/etc/bacula/director/ssl/server.key"}) \
 .with_content(/DIRport = 9900/) \
 .with_content(/Maximum Concurrent Jobs = 10/)
 end
diff --git a/modules/bacula/spec/classes/bacula_storage_spec.rb 
b/modules/bacula/spec/classes/bacula_storage_spec.rb
index 32a8320..67c97c1 100644
--- a/modules/bacula/spec/classes/bacula_storage_spec.rb
+++ b/modules/bacula/spec/classes/bacula_storage_spec.rb
@@ -33,8 +33,8 @@
 }) \
 .with_content(/Name = "testdirector"/) \
 .with_content(/Password = "testdirectorpass"/) \
-.with_content(/TLS Certificate = "\/etc\/bacula\/sd\/ssl\/cert.pem"/) \
-.with_content(/TLS Key = "\/etc\/bacula\/sd\/ssl\/server.key"/) \
+.with_content(%r{TLS Certificate = "/etc/bacula/sd/ssl/cert.pem"}) \
+.with_content(%r{TLS Key = "/etc/bacula/sd/ssl/server.key"}) \
 .with_content(/Name = "testhost.example.com-fd"/) \
 .with_content(/SDport = 9000/) \
 .with_content(/Maximum Concurrent Jobs = 10/)
diff --git a/modules/bacula/spec/defines/fileset_spec.rb 
b/modules/bacula/spec/defines/fileset_spec.rb
index 7bea6cd..08fd77a 100644
--- a/modules/bacula/spec/defines/fileset_spec.rb
+++ b/modules/bacula/spec/defines/fileset_spec.rb
@@ -16,8 +16,8 @@
 context 'without excludes' do
 it 'should create valid content for 
/etc/bacula/conf.d/fileset-something.conf' do
 should contain_file('/etc/bacula/conf.d/fileset-something.conf') \
-.with_content(/File = \//) \
-.with_content(/File = \/var/)
+.with_content(%r{File = /}) \
+.with_content(%r{File = /var})
 end
 end
 
@@ -29,9 +29,9 @@
 }
 it 'should create valid content for 
/etc/bacula/conf.d/fileset-something.conf' 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix more whitespace-related RuboCop across the tree

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359478 )

Change subject: Fix more whitespace-related RuboCop across the tree
..


Fix more whitespace-related RuboCop across the tree

- Layout/LeadingCommentSpace
- Layout/SpaceAroundEqualsInParameterDefault
- Layout/SpaceInsideBrackets
- Layout/DotPosition

Change-Id: Idd4fd19daa55ba01dbe96b0f913bc9e48103
---
M .rubocop_todo.yml
M modules/bacula/spec/defines/fileset_spec.rb
M modules/base/spec/defines/service_unit_spec.rb
M modules/postgresql/spec/classes/postgresql_master_spec.rb
M modules/postgresql/spec/classes/postgresql_slave_spec.rb
M modules/raid/lib/facter/raid.rb
M modules/scap/lib/puppet/provider/package/scap3.rb
M modules/scap/spec/types/package/scap3_spec.rb
M modules/wmflib/lib/hiera/backend/proxy_backend.rb
M modules/wmflib/lib/puppet/parser/functions/array_concat.rb
M modules/wmflib/lib/puppet/parser/functions/cron_splay.rb
M modules/wmflib/lib/puppet/parser/functions/hash_deselect_re.rb
M modules/wmflib/lib/puppet/parser/functions/hash_select_re.rb
M modules/wmflib/lib/puppet/parser/functions/htpasswd.rb
M rake_modules/taskgen.rb
M utils/expanderb.rb
M utils/hiera_lookup
17 files changed, 51 insertions(+), 68 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 56f72db..681b2a0 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -16,9 +16,6 @@
   Exclude:
 - 'modules/labstore/lib/puppet/parser/functions/mount_nfs_volume.rb'
 
-Layout/DotPosition:
-  Enabled: false
-
 Style/EmptyLiteral:
   Exclude:
 - 'modules/admin/lib/puppet/parser/functions/unique_users.rb'
@@ -40,10 +37,6 @@
 
 Layout/IndentationWidth:
   Enabled: false
-
-Layout/LeadingCommentSpace:
-  Exclude:
-- 'modules/base/spec/defines/service_unit_spec.rb'
 
 Style/MethodCallWithoutArgsParentheses:
   Exclude:
@@ -67,18 +60,8 @@
   Exclude:
 - 'modules/wmflib/lib/puppet/parser/functions/require_package.rb'
 
-Layout/SpaceAroundEqualsInParameterDefault:
-  Enabled: false
-
-Layout/SpaceInsideBrackets:
-  Enabled: false
-
 Layout/SpaceInsideHashLiteralBraces:
   Enabled: false
-
-Layout/SpaceInsideParens:
-  Exclude:
-- 'utils/expanderb.rb'
 
 Style/SpecialGlobalVars:
   Exclude:
diff --git a/modules/bacula/spec/defines/fileset_spec.rb 
b/modules/bacula/spec/defines/fileset_spec.rb
index c069c27..7bea6cd 100644
--- a/modules/bacula/spec/defines/fileset_spec.rb
+++ b/modules/bacula/spec/defines/fileset_spec.rb
@@ -2,7 +2,7 @@
 
 describe 'bacula::director::fileset', :type => :define do
 let(:title) { 'something' }
-let(:params) { { :includes => [ "/", "/var", ], } }
+let(:params) { { :includes => ["/", "/var",], } }
 
 it 'should create /etc/bacula/conf.d/fileset-something.conf' do
 should contain_file('/etc/bacula/conf.d/fileset-something.conf').with({
@@ -23,8 +23,8 @@
 
 context 'with excludes' do
 let(:params) { {
-:includes=> [ "/", "/var",],
-:excludes=> [ "/tmp", ],
+:includes=> ["/", "/var",],
+:excludes=> ["/tmp",],
 }
 }
 it 'should create valid content for 
/etc/bacula/conf.d/fileset-something.conf' do
diff --git a/modules/base/spec/defines/service_unit_spec.rb 
b/modules/base/spec/defines/service_unit_spec.rb
index 0800e9c..2c79374 100644
--- a/modules/base/spec/defines/service_unit_spec.rb
+++ b/modules/base/spec/defines/service_unit_spec.rb
@@ -85,6 +85,6 @@
   end
 end
 
-#MARK
+# MARK
   end
 end
diff --git a/modules/postgresql/spec/classes/postgresql_master_spec.rb 
b/modules/postgresql/spec/classes/postgresql_master_spec.rb
index 6b693e3..06ba903 100644
--- a/modules/postgresql/spec/classes/postgresql_master_spec.rb
+++ b/modules/postgresql/spec/classes/postgresql_master_spec.rb
@@ -11,16 +11,16 @@
 context 'ensure present' do
 it { should contain_class('postgresql::server') }
 it do
-should contain_file('/etc/postgresql/9.4/main/postgresql.conf').
-with_ensure('present').
-with_content(/include 'master.conf'/)
+should contain_file('/etc/postgresql/9.4/main/postgresql.conf')
+.with_ensure('present')
+.with_content(/include 'master.conf'/)
 end
 it do
-should contain_file('/etc/postgresql/9.4/main/master.conf').
-with_ensure('present').
-with_content(/max_wal_senders = 5/).
-with_content(/checkpoint_segments = 64/).
-with_content(/wal_keep_segments = 128/)
+should contain_file('/etc/postgresql/9.4/main/master.conf')
+.with_ensure('present')
+.with_content(/max_wal_senders = 5/)
+.with_content(/checkpoint_segments = 64/)

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wmflib: fix RuboCop infractions in serializers

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359453 )

Change subject: wmflib: fix RuboCop infractions in serializers
..


wmflib: fix RuboCop infractions in serializers

- Layout/SpaceAfterComma
- Layout/SpaceAroundEqualsInParameterDefault
- Layout/SpaceAroundOperators
- Style/RescueModifier
- Style/Semicolon

Change-Id: I743615bcc90c73ffa902995b73e25f8f4fb89c75
---
M .rubocop_todo.yml
M modules/librenms/lib/puppet/parser/functions/phpdump.rb
M modules/wmflib/lib/puppet/parser/functions/ini.rb
M modules/wmflib/lib/puppet/parser/functions/ordered_json.rb
M modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb
M modules/wmflib/lib/puppet/parser/functions/php_ini.rb
6 files changed, 32 insertions(+), 30 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index faf59a1..56f72db 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -65,27 +65,10 @@
 
 Style/RescueModifier:
   Exclude:
-- 'modules/librenms/lib/puppet/parser/functions/phpdump.rb'
-- 'modules/wmflib/lib/puppet/parser/functions/ini.rb'
-- 'modules/wmflib/lib/puppet/parser/functions/ordered_json.rb'
-- 'modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb'
-- 'modules/wmflib/lib/puppet/parser/functions/php_ini.rb'
 - 'modules/wmflib/lib/puppet/parser/functions/require_package.rb'
-
-Style/Semicolon:
-  Exclude:
-- 'modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb'
-
-Layout/SpaceAfterComma:
-  Exclude:
-- 'modules/wmflib/lib/puppet/parser/functions/ini.rb'
 
 Layout/SpaceAroundEqualsInParameterDefault:
   Enabled: false
-
-Layout/SpaceAroundOperators:
-  Exclude:
-- 'modules/librenms/lib/puppet/parser/functions/phpdump.rb'
 
 Layout/SpaceInsideBrackets:
   Enabled: false
diff --git a/modules/librenms/lib/puppet/parser/functions/phpdump.rb 
b/modules/librenms/lib/puppet/parser/functions/phpdump.rb
index 03e3f2b..743c98f 100644
--- a/modules/librenms/lib/puppet/parser/functions/phpdump.rb
+++ b/modules/librenms/lib/puppet/parser/functions/phpdump.rb
@@ -3,20 +3,20 @@
 # Serialize a hash into PHP array with lexicographically sorted keys.
 #
 
-def phpdump(o, level=1)
-  indent = " "*4
+def phpdump(o, level = 1)
+  indent = " " * 4
 
   case o
   when Hash
 contents = ''
 o.sort.each do |k, v|
-  contents += indent*level
-  contents += k.to_pson + " => " + phpdump(v, level+1)
+  contents += indent * level
+  contents += k.to_pson + " => " + phpdump(v, level + 1)
   contents += ",\n"
 end
-"array(\n" + contents + indent*(level-1) + ")"
+"array(\n" + contents + indent * (level - 1) + ")"
   when Array
-"array(" + o.map { |x| phpdump(x, level+1) }.join(', ') + ")"
+"array(" + o.map { |x| phpdump(x, level + 1) }.join(', ') + ")"
   when TrueClass
 "TRUE"
   when FalseClass
@@ -24,7 +24,11 @@
   when nil
 "NULL"
   else
-o.include?('.') ? Float(o).to_s : Integer(o).to_s rescue o.to_pson
+begin
+  o.include?('.') ? Float(o).to_s : Integer(o).to_s
+rescue
+  o.to_pson
+end
   end
 end
 
diff --git a/modules/wmflib/lib/puppet/parser/functions/ini.rb 
b/modules/wmflib/lib/puppet/parser/functions/ini.rb
index 19a0d28..ce1fba2 100644
--- a/modules/wmflib/lib/puppet/parser/functions/ini.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/ini.rb
@@ -22,7 +22,9 @@
 end
 
 def ini_cast(v)
-  v.include?('.') ? Float(v) : Integer(v) rescue v
+v.include?('.') ? Float(v) : Integer(v)
+  rescue
+v
 end
 
 module Puppet::Parser::Functions
@@ -30,7 +32,7 @@
 if args.map(&:class).uniq != [Hash]
   fail(ArgumentError, 'ini(): hash arguments required')
 end
-args.reduce(&:merge).map do |section,items|
+args.reduce(&:merge).map do |section, items|
   ini_flatten(items).map do |k, vs|
 case vs
 when Array then vs.map { |v| "#{k}[#{v}] = #{ini_cast(v)}" }
diff --git a/modules/wmflib/lib/puppet/parser/functions/ordered_json.rb 
b/modules/wmflib/lib/puppet/parser/functions/ordered_json.rb
index 74a4e2c..f6a9cdb 100644
--- a/modules/wmflib/lib/puppet/parser/functions/ordered_json.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/ordered_json.rb
@@ -25,7 +25,11 @@
   when Hash
 '{' + o.sort.map { |k, v| k.to_pson + ': ' + ordered_json(v) }.join(', ') 
+ '}'
   else
-o.include?('.') ? Float(o).to_s : Integer(o).to_s rescue o.to_pson
+begin
+  o.include?('.') ? Float(o).to_s : Integer(o).to_s
+rescue
+  o.to_pson
+end
   end
 end
 
diff --git a/modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb 
b/modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb
index 8a459d1..737b1cc 100644
--- a/modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/ordered_yaml.rb
@@ -24,13 +24,20 @@
 def 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix Style/FormatString RuboCop across all Rakefiles

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359452 )

Change subject: Fix Style/FormatString RuboCop across all Rakefiles
..


Fix Style/FormatString RuboCop across all Rakefiles

Only base and strongswan were really affected, but we had ignores for a
bunch more. Fix and remove the TODO.

Change-Id: Ib0d870f9096ef78ee3089cf05f4e1c65595f373d
---
M .rubocop_todo.yml
M modules/base/Rakefile
M modules/strongswan/Rakefile
3 files changed, 19 insertions(+), 33 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 8fd94f6..faf59a1 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -4,7 +4,6 @@
 
 Lint/UselessAssignment:
   Exclude:
-- 'modules/service/Rakefile'
 - 'modules/wmflib/lib/puppet/parser/functions/validate_array_re.rb'
 
 Metrics/LineLength:
@@ -27,21 +26,6 @@
 
 Style/For:
   Enabled: false
-
-Style/FormatString:
-  Exclude:
-- 'modules/backup/Rakefile'
-- 'modules/bacula/Rakefile'
-- 'modules/base/Rakefile'
-- 'modules/install_server/Rakefile'
-- 'modules/mirrors/Rakefile'
-- 'modules/nrpe/Rakefile'
-- 'modules/osm/Rakefile'
-- 'modules/postgresql/Rakefile'
-- 'modules/servermon/Rakefile'
-- 'modules/squid3/Rakefile'
-- 'modules/strongswan/Rakefile'
-- 'modules/wmflib/Rakefile'
 
 Style/GlobalVars:
   Exclude:
diff --git a/modules/base/Rakefile b/modules/base/Rakefile
index 42f6ed4..3437c0f 100644
--- a/modules/base/Rakefile
+++ b/modules/base/Rakefile
@@ -5,22 +5,23 @@
 
 # Note: The nginx puppet module is a git submodule right now, so don't forget 
to
 # checkout submodules before testing
-extra_modules = [ 'wmflib' ]
+extra_modules = ['wmflib']
 
 modulename = File.basename(File.expand_path(File.dirname(__FILE__)))
 
-symlinks = { 'spec/fixtures/modules/%s/files' % modulename => 
'../../../../files',
-  'spec/fixtures/modules/%s/manifests' % modulename => '../../../../manifests',
-  'spec/fixtures/modules/%s/templates' % modulename => '../../../../templates',
+symlinks = {
+  format('spec/fixtures/modules/%s/files', modulename) => 
'../../../../files',
+  format('spec/fixtures/modules/%s/manifests', modulename) => 
'../../../../manifests',
+  format('spec/fixtures/modules/%s/templates', modulename) => 
'../../../../templates',
 }
 
 task :setup do
-  FileUtils.mkdir_p('spec/fixtures/modules/%s' % modulename)
+  FileUtils.mkdir_p(format('spec/fixtures/modules/%s', modulename))
   extra_modules.each do |x|
-  FileUtils.ln_s('../../../../%s' % x, 'spec/fixtures/modules/%s' % x)
+  FileUtils.ln_s(format('../../../../%s', x), 
format('spec/fixtures/modules/%s', x))
   end
   symlinks.each do |x|
-if !File.exist?(x[0])
+unless File.exist?(x[0])
   FileUtils.ln_s(x[1], x[0])
 end
   end
@@ -28,9 +29,9 @@
 
 task :teardown do
   symlinks.each { |x| FileUtils.rm(x[0], :force => true) }
-  FileUtils.rmdir('spec/fixtures/modules/%s' % modulename)
+  FileUtils.rmdir(format('spec/fixtures/modules/%s', modulename))
   extra_modules.each do |x|
-  FileUtils.rm('spec/fixtures/modules/%s' % x, :force => true)
+  FileUtils.rm(format('spec/fixtures/modules/%s', x), :force => true)
   end
   FileUtils.rmdir('spec/fixtures/modules')
 end
@@ -40,7 +41,7 @@
   t.pattern = 'spec/*/*_spec.rb'
 end
 
-task :spec => [ :setup, :realspec, :teardown]
+task :spec => [:setup, :realspec, :teardown]
 
 task :default => :spec do
 end
diff --git a/modules/strongswan/Rakefile b/modules/strongswan/Rakefile
index 401ca3d..0cc35f8 100644
--- a/modules/strongswan/Rakefile
+++ b/modules/strongswan/Rakefile
@@ -5,13 +5,14 @@
 
 modulename = File.basename(File.expand_path(File.dirname(__FILE__)))
 
-symlinks = { 'spec/fixtures/modules/%s/files' % modulename => 
'../../../../files',
- 'spec/fixtures/modules/%s/manifests' % modulename => 
'../../../../manifests',
- 'spec/fixtures/modules/%s/templates' % modulename => 
'../../../../templates',
-   }
+symlinks = {
+  format('spec/fixtures/modules/%s/files', modulename) => 
'../../../../files',
+  format('spec/fixtures/modules/%s/manifests', modulename) => 
'../../../../manifests',
+  format('spec/fixtures/modules/%s/templates', modulename) => 
'../../../../templates',
+}
 
 task :setup do
-  FileUtils.mkdir_p('spec/fixtures/modules/%s' % modulename)
+  FileUtils.mkdir_p(format('spec/fixtures/modules/%s', modulename))
   symlinks.each do |x|
 if !File.exist?(x[0])
   FileUtils.ln_s(x[1], x[0])
@@ -21,7 +22,7 @@
 
 task :teardown do
   symlinks.each { |x| FileUtils.rm(x[0], :force => true) }
-  FileUtils.rmdir('spec/fixtures/modules/%s' % modulename)
+  FileUtils.rmdir(format('spec/fixtures/modules/%s', modulename))
   FileUtils.rmdir('spec/fixtures/modules')
 end
 
@@ -30,7 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: graphite: cleanup configparser_format a little bit

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359451 )

Change subject: graphite: cleanup configparser_format a little bit
..


graphite: cleanup configparser_format a little bit

Fix indentation and Rubocop infractions:
- Style/NestedTernaryOperator
- Style/RedundantReturn
- Layout/SpaceAfterComma
- Layout/SpaceInsideParens

Still needs a little bit of cleanup work, though.

Change-Id: Idda59e236e3bbc82bd55cacd03c79f57f47237db
---
M .rubocop_todo.yml
M modules/graphite/lib/puppet/parser/functions/configparser_format.rb
2 files changed, 26 insertions(+), 31 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 1c50dc5..8fd94f6 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -70,14 +70,6 @@
 Style/NegatedIf:
   Enabled: false
 
-Style/NestedTernaryOperator:
-  Exclude:
-- 'modules/graphite/lib/puppet/parser/functions/configparser_format.rb'
-
-Style/RedundantReturn:
-  Exclude:
-- 'modules/graphite/lib/puppet/parser/functions/configparser_format.rb'
-
 Style/RegexpLiteral:
   Exclude:
 - 'modules/bacula/spec/classes/bacula_client_spec.rb'
@@ -102,7 +94,6 @@
 
 Layout/SpaceAfterComma:
   Exclude:
-- 'modules/graphite/lib/puppet/parser/functions/configparser_format.rb'
 - 'modules/wmflib/lib/puppet/parser/functions/ini.rb'
 
 Layout/SpaceAroundEqualsInParameterDefault:
@@ -120,7 +111,6 @@
 
 Layout/SpaceInsideParens:
   Exclude:
-- 'modules/graphite/lib/puppet/parser/functions/configparser_format.rb'
 - 'utils/expanderb.rb'
 
 Style/SpecialGlobalVars:
diff --git 
a/modules/graphite/lib/puppet/parser/functions/configparser_format.rb 
b/modules/graphite/lib/puppet/parser/functions/configparser_format.rb
index 32e0bc0..f475277 100644
--- a/modules/graphite/lib/puppet/parser/functions/configparser_format.rb
+++ b/modules/graphite/lib/puppet/parser/functions/configparser_format.rb
@@ -4,31 +4,36 @@
 # See 
 #
 def rmerge(*args)
-# Recursively merge hashes.
-merged = args.shift.clone
-args.each { |hash|
-merged.merge!(hash) { |k, old, new|
-merged[k] = old.is_a?(Hash) && new.is_a?(Hash) ? rmerge(old, new) 
: new
-}
-}
-return merged
+  # Recursively merge hashes.
+  merged = args.shift.clone
+  args.each do |hash|
+merged.merge!(hash) do |k, old, new|
+  merged[k] = old.is_a?(Hash) && new.is_a?(Hash) ? rmerge(old, new) : new
+end
+  end
+  merged
 end
 
 def configparser_format(config)
-# Serialize a hash to Python ConfigParser format.
-config.sort.map { |section,items|
-["[#{section}]"].concat items.sort.map { |k,v|
-v = v.is_a?(Array) ? v.join(',') : ( v == :undef ? '' : v )
-"#{k} = #{v}"
-}.push []
-}.join("\n")
+  # Serialize a hash to Python ConfigParser format.
+  config.sort.map { |section, items|
+["[#{section}]"].concat items.sort.map { |k, v|
+  if v.is_a?(Array)
+v = v.join(',')
+  else
+v = v == :undef ? '' : v
+  end
+
+  "#{k} = #{v}"
+}.push []
+  }.join("\n")
 end
 
 module Puppet::Parser::Functions
-newfunction(:configparser_format, :type => :rvalue) { |args|
-unless args.any? && args.all? { |a| a.is_a? Hash }
-fail 'configparser_format() requires one or more hash arguments'
-end
-configparser_format(rmerge(*args))
-}
+  newfunction(:configparser_format, :type => :rvalue) do |args|
+unless args.any? && args.all? { |a| a.is_a? Hash }
+  fail 'configparser_format() requires one or more hash arguments'
+end
+configparser_format(rmerge(*args))
+  end
 end

-- 
To view, visit https://gerrit.wikimedia.org/r/359451
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Idda59e236e3bbc82bd55cacd03c79f57f47237db
Gerrit-PatchSet: 6
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon Liambotis 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Hashar 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wmflib: cleanup secret.rb a little bit

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/359449 )

Change subject: wmflib: cleanup secret.rb a little bit
..


wmflib: cleanup secret.rb a little bit

These are really minor fixes for Lint/AssignmentInCondition,
Style/MethodCallWithoutArgsParentheses and Style/NegatedIf, but given
the prevalence and importance of secret.rb, this needs a separate and
very careful review.

Change-Id: Ib5a1f0629766d1dd33bea2be383ee802e190adb7
---
M .rubocop_todo.yml
M modules/wmflib/lib/puppet/parser/functions/secret.rb
2 files changed, 6 insertions(+), 12 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
index 636bb27..1c50dc5 100644
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@@ -1,7 +1,3 @@
-Lint/AssignmentInCondition:
-  Exclude:
-- 'modules/wmflib/lib/puppet/parser/functions/secret.rb'
-
 Lint/HandleExceptions:
   Exclude:
 - 'modules/wmflib/lib/puppet/parser/functions/require_package.rb'
@@ -70,7 +66,6 @@
 - 'modules/base/spec/defines/service_unit_spec.rb'
 - 
'modules/install_server/spec/classes/install_server_preseed_server_spec.rb'
 - 'modules/squid3/spec/classes/squid3_spec.rb'
-- 'modules/wmflib/lib/puppet/parser/functions/secret.rb'
 
 Style/NegatedIf:
   Enabled: false
diff --git a/modules/wmflib/lib/puppet/parser/functions/secret.rb 
b/modules/wmflib/lib/puppet/parser/functions/secret.rb
index fc66a02..06970d0 100644
--- a/modules/wmflib/lib/puppet/parser/functions/secret.rb
+++ b/modules/wmflib/lib/puppet/parser/functions/secret.rb
@@ -10,20 +10,19 @@
 end
 in_path = args.first
 
-if mod = Puppet::Module.find(mod_name)
-   mod_path = mod.path()
-else
+mod = Puppet::Module.find(mod_name)
+unless mod
   fail("secret(): Module #{mod_name} not found")
 end
 
-sec_path = mod_path + secs_subdir + in_path
-final_path = Pathname.new(sec_path).cleanpath()
+sec_path = mod.path + secs_subdir + in_path
+final_path = Pathname.new(sec_path).cleanpath
 
 # Bail early if it's not a regular, readable file
-if !final_path.file? || !final_path.readable?
+unless final_path.file? && final_path.readable?
   fail(ArgumentError, "secret(): invalid secret #{in_path}")
 end
 
-return final_path.read()
+final_path.read
   end
 end

-- 
To view, visit https://gerrit.wikimedia.org/r/359449
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib5a1f0629766d1dd33bea2be383ee802e190adb7
Gerrit-PatchSet: 7
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Faidon Liambotis 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: secret: add spec

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403121 )

Change subject: secret: add spec
..


secret: add spec

This function is important enough that breaking any of its functionality
would be severely harmful to us. So, let's add a tiny amount of spec
tests to ensure we don't break it.

Change-Id: I7cbe1454f793c5de0567b6c2edb5372269386029
---
A modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt
A modules/wmflib/spec/functions/secret_spec.rb
2 files changed, 18 insertions(+), 0 deletions(-)

Approvals:
  Giuseppe Lavagetto: Verified; Looks good to me, approved



diff --git a/modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt 
b/modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt
new file mode 100644
index 000..d81cc07
--- /dev/null
+++ b/modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt
@@ -0,0 +1 @@
+42
diff --git a/modules/wmflib/spec/functions/secret_spec.rb 
b/modules/wmflib/spec/functions/secret_spec.rb
new file mode 100644
index 000..87fde1c
--- /dev/null
+++ b/modules/wmflib/spec/functions/secret_spec.rb
@@ -0,0 +1,17 @@
+require 'spec_helper'
+
+describe 'secret' do
+  it 'should throw an error if the file is not existent' do
+is_expected.to run.with_params('test/text1').and_raise_error(ArgumentError)
+  end
+
+  it 'should run correctly with a good filename' do
+is_expected.to run.with_params('test/test.txt').and_return("42\n")
+  end
+
+  it 'should throw an error if the number of arguments is wrong' do
+is_expected.to run.with_params('a', 'b').and_raise_error(ArgumentError)
+is_expected.to run.with_params(12).and_raise_error(ArgumentError)
+is_expected.to run.and_raise_error(ArgumentError)
+  end
+end

-- 
To view, visit https://gerrit.wikimedia.org/r/403121
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7cbe1454f793c5de0567b6c2edb5372269386029
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: secret: add spec

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403121 )

Change subject: secret: add spec
..

secret: add spec

This function is important enough that breaking any of its functionality
would be severely harmful to us. So, let's add a tiny amount of spec
tests to ensure we don't break it.

Change-Id: I7cbe1454f793c5de0567b6c2edb5372269386029
---
A modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt
A modules/wmflib/spec/functions/secret_spec.rb
2 files changed, 18 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/21/403121/1

diff --git a/modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt 
b/modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt
new file mode 100644
index 000..d81cc07
--- /dev/null
+++ b/modules/wmflib/spec/fixtures/modules/secret/secrets/test/test.txt
@@ -0,0 +1 @@
+42
diff --git a/modules/wmflib/spec/functions/secret_spec.rb 
b/modules/wmflib/spec/functions/secret_spec.rb
new file mode 100644
index 000..87fde1c
--- /dev/null
+++ b/modules/wmflib/spec/functions/secret_spec.rb
@@ -0,0 +1,17 @@
+require 'spec_helper'
+
+describe 'secret' do
+  it 'should throw an error if the file is not existent' do
+is_expected.to run.with_params('test/text1').and_raise_error(ArgumentError)
+  end
+
+  it 'should run correctly with a good filename' do
+is_expected.to run.with_params('test/test.txt').and_return("42\n")
+  end
+
+  it 'should throw an error if the number of arguments is wrong' do
+is_expected.to run.with_params('a', 'b').and_raise_error(ArgumentError)
+is_expected.to run.with_params(12).and_raise_error(ArgumentError)
+is_expected.to run.and_raise_error(ArgumentError)
+  end
+end

-- 
To view, visit https://gerrit.wikimedia.org/r/403121
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7cbe1454f793c5de0567b6c2edb5372269386029
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role per node called with role()

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403112 )

Change subject: site.pp: one role per node called with role()
..


site.pp: one role per node called with role()

Also, a role for the netmon machines is created.

Change-Id: Id7c42905293b7752f74b1e636406ec6bcb0b0f00
---
R hieradata/role/common/netmon.yaml
M manifests/site.pp
A modules/role/manifests/netmon.pp
3 files changed, 18 insertions(+), 3 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/librenms.yaml 
b/hieradata/role/common/netmon.yaml
similarity index 100%
rename from hieradata/role/common/librenms.yaml
rename to hieradata/role/common/netmon.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 4397bd6..27c7d68 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1721,13 +1721,13 @@
 
 # network monitoring tools, stretch (T125020, T166180)
 node /^netmon(1002|2001)\.wikimedia\.org$/ {
-role(network::monitor, librenms, rancid, smokeping, netbox)
-interface::add_ip6_mapped { 'main': }
+role(netmon)
 }
 
 # jessie VM for servermon until it supports stretch (T170653)
 node 'netmon1003.wikimedia.org' {
-role(network::monitor, servermon)
+role(servermon)
+include ::role::network::monitor
 interface::add_ip6_mapped { 'main': }
 }
 
diff --git a/modules/role/manifests/netmon.pp b/modules/role/manifests/netmon.pp
new file mode 100644
index 000..3165daf0
--- /dev/null
+++ b/modules/role/manifests/netmon.pp
@@ -0,0 +1,15 @@
+class role::netmon {
+system::role { 'netmon':
+description => 'Network monitoring and management'
+}
+# Basic boilerplate for network-related servers
+require ::role::network::monitor
+include ::profile::backup::host
+include ::profile::librenms
+include ::profile::rancid
+include ::profile::smokeping
+include ::profile::netbox
+include ::profile::prometheus::postgres_exporter
+
+interface::add_ip6_mapped { 'main': }
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/403112
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id7c42905293b7752f74b1e636406ec6bcb0b0f00
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role per node called with role()

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403112 )

Change subject: site.pp: one role per node called with role()
..

site.pp: one role per node called with role()

Also, a role for the netmon machines is created.

Change-Id: Id7c42905293b7752f74b1e636406ec6bcb0b0f00
---
R hieradata/role/common/netmon.yaml
M manifests/site.pp
A modules/role/manifests/netmon.pp
3 files changed, 17 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/12/403112/1

diff --git a/hieradata/role/common/librenms.yaml 
b/hieradata/role/common/netmon.yaml
similarity index 100%
rename from hieradata/role/common/librenms.yaml
rename to hieradata/role/common/netmon.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 4397bd6..27c7d68 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1721,13 +1721,13 @@
 
 # network monitoring tools, stretch (T125020, T166180)
 node /^netmon(1002|2001)\.wikimedia\.org$/ {
-role(network::monitor, librenms, rancid, smokeping, netbox)
-interface::add_ip6_mapped { 'main': }
+role(netmon)
 }
 
 # jessie VM for servermon until it supports stretch (T170653)
 node 'netmon1003.wikimedia.org' {
-role(network::monitor, servermon)
+role(servermon)
+include ::role::network::monitor
 interface::add_ip6_mapped { 'main': }
 }
 
diff --git a/modules/role/manifests/netmon.pp b/modules/role/manifests/netmon.pp
new file mode 100644
index 000..913ed9b
--- /dev/null
+++ b/modules/role/manifests/netmon.pp
@@ -0,0 +1,14 @@
+class role::netmon {
+system::role { 'netmon':
+description => 'Network monitoring and management'
+}
+# Basic boilerplate for network-related servers
+require ::role::network::monitor
+include ::profile::librenms
+include ::profile::rancid
+include ::profile::smokeping
+include ::profile::netbox
+include ::profile::prometheus::postgres_exporter
+
+interface::add_ip6_mapped { 'main': }
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/403112
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id7c42905293b7752f74b1e636406ec6bcb0b0f00
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: bastionhost::twofa: re-add the new install access keystone

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402848 )

Change subject: bastionhost::twofa: re-add the new install access keystone
..


bastionhost::twofa: re-add the new install access keystone

They were needed after all, now that is clarified in a comment.

Change-Id: I0cc7a2da4e836ab46b6eb7a2b6f69ea54d5b8f0d
---
M modules/role/manifests/bastionhost/twofa.pp
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  Andrew Bogott: Looks good to me, but someone else must approve
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/bastionhost/twofa.pp 
b/modules/role/manifests/bastionhost/twofa.pp
index 2d679cc..93cf156 100644
--- a/modules/role/manifests/bastionhost/twofa.pp
+++ b/modules/role/manifests/bastionhost/twofa.pp
@@ -7,6 +7,9 @@
 include ::standard
 include ::profile::base::firewall
 include ::profile::backup::host
+
+# Needed to allow installation of servers in the labs subnet
+include ::profile::access_new_install
 include ::passwords::yubiauth
 
 backup::set {'home': }

-- 
To view, visit https://gerrit.wikimedia.org/r/402848
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0cc7a2da4e836ab46b6eb7a2b6f69ea54d5b8f0d
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Andrew Bogott 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: bastionhost::twofa: re-add the new install access keystone

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402848 )

Change subject: bastionhost::twofa: re-add the new install access keystone
..

bastionhost::twofa: re-add the new install access keystone

They were needed after all, now that is clarified in a comment.

Change-Id: I0cc7a2da4e836ab46b6eb7a2b6f69ea54d5b8f0d
---
M modules/role/manifests/bastionhost/twofa.pp
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/48/402848/1

diff --git a/modules/role/manifests/bastionhost/twofa.pp 
b/modules/role/manifests/bastionhost/twofa.pp
index 2d679cc..93cf156 100644
--- a/modules/role/manifests/bastionhost/twofa.pp
+++ b/modules/role/manifests/bastionhost/twofa.pp
@@ -7,6 +7,9 @@
 include ::standard
 include ::profile::base::firewall
 include ::profile::backup::host
+
+# Needed to allow installation of servers in the labs subnet
+include ::profile::access_new_install
 include ::passwords::yubiauth
 
 backup::set {'home': }

-- 
To view, visit https://gerrit.wikimedia.org/r/402848
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0cc7a2da4e836ab46b6eb7a2b6f69ea54d5b8f0d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role called with role() for stat1005

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402841 )

Change subject: site.pp: one role called with role() for stat1005
..


site.pp: one role called with role() for stat1005

I didn't bother creating a compound role for a machine that looks
definitely a one-off unicorn.

Change-Id: I01ddda470bf38391d687d57ee759fb4fbaf00692
---
M manifests/site.pp
1 file changed, 23 insertions(+), 23 deletions(-)

Approvals:
  Giuseppe Lavagetto: Verified; Looks good to me, approved
  Ottomata: Looks good to me, but someone else must approve



diff --git a/manifests/site.pp b/manifests/site.pp
index 3fafc42..81992cf 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2031,35 +2031,35 @@
 
 # WIP: stat1002 replacement (T152712)
 node 'stat1005.eqiad.wmnet' {
-role(statistics::private,
-# This is a Hadoop client, and should
-# have any special analytics system users on it
-# for interacting with HDFS.
-analytics_cluster::users,
+role(statistics::private)
+# This is a Hadoop client, and should
+# have any special analytics system users on it
+# for interacting with HDFS.
+include ::role::analytics_cluster::users
 
-# Include Hadoop and other analytics cluster
-# clients so that analysts can access Hadoop
-# from here.
-analytics_cluster::client,
+# Include Hadoop and other analytics cluster
+# clients so that analysts can access Hadoop
+# from here.
+include ::role::analytics_cluster::client
 
-# Include analytics/refinery deployment target.
-analytics_cluster::refinery,
+# Include analytics/refinery deployment target.
+include ::role::analytics_cluster::refinery
 
-# Include analytics/refinery checks that send email about
-# webrequest partitions faultyness.
-analytics_cluster::refinery::job::data_check,
+# Include analytics/refinery checks that send email about
+# webrequest partitions faultyness.
+include ::role::analytics_cluster::refinery::job::data_check
 
-# Include analytics/refinery/source guard checks
-# Disabled due to T166937
-# analytics_cluster::refinery::job::guard,
+# Include analytics/refinery/source guard checks
+# Disabled due to T166937
+# analytics_cluster::refinery::job::guard,
 
-# Set up a read only rsync module to allow access
-# to public data generated by the Analytics Cluster.
-analytics_cluster::rsyncd,
+# Set up a read only rsync module to allow access
+# to public data generated by the Analytics Cluster.
+include ::role::analytics_cluster::rsyncd
 
-# Deploy wikimedia/discovery/analytics repository
-# to this node.
-elasticsearch::analytics)
+# Deploy wikimedia/discovery/analytics repository
+# to this node.
+include ::role::elasticsearch::analytics
 }
 
 # stat1006 is a general purpose number cruncher for

-- 
To view, visit https://gerrit.wikimedia.org/r/402841
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I01ddda470bf38391d687d57ee759fb4fbaf00692
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role() call in ruthenium, tungsten

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402840 )

Change subject: site.pp: one role() call in ruthenium, tungsten
..


site.pp: one role() call in ruthenium, tungsten

Change-Id: I9be3214609e84741116834cea1f7eac8eea9e886
---
M manifests/site.pp
1 file changed, 7 insertions(+), 4 deletions(-)

Approvals:
  Giuseppe Lavagetto: Verified; Looks good to me, approved



diff --git a/manifests/site.pp b/manifests/site.pp
index 9706cb2..3fafc42 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1943,9 +1943,11 @@
 # Right now, both rt-server and rt-clients run on the same node
 # But, we are likely going to split them into different boxes soon.
 node 'ruthenium.eqiad.wmnet' {
-role(test,
-parsoid::testing, parsoid::rt_server, parsoid::rt_client,
-parsoid::vd_server, parsoid::vd_client, parsoid::diffserver)
+role(parsoid::testing)
+include ::role::test
+include ::role::parsoid::rt_server, ::role::parsoid::vd_server
+include ::role::parsoid::rt_client, ::role::parsoid::vd_client
+include ::role::parsoid::diffserver
 }
 
 # cluster management (cumin master)
@@ -2112,7 +2114,8 @@
 
 # test system for performance team (T117888)
 node 'tungsten.eqiad.wmnet' {
-role(test, xhgui::app)
+role(xhgui::app)
+include ::role::test
 }
 
 # replaced magnesium (RT) (T119112 T123713)

-- 
To view, visit https://gerrit.wikimedia.org/r/402840
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9be3214609e84741116834cea1f7eac8eea9e886
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role called with role() for stat1005

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402841 )

Change subject: site.pp: one role called with role() for stat1005
..

site.pp: one role called with role() for stat1005

I didn't bother creating a compound role for a machine that looks
definitely a one-off unicorn.

Change-Id: I01ddda470bf38391d687d57ee759fb4fbaf00692
---
M manifests/site.pp
1 file changed, 23 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/41/402841/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 3fafc42..81992cf 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2031,35 +2031,35 @@
 
 # WIP: stat1002 replacement (T152712)
 node 'stat1005.eqiad.wmnet' {
-role(statistics::private,
-# This is a Hadoop client, and should
-# have any special analytics system users on it
-# for interacting with HDFS.
-analytics_cluster::users,
+role(statistics::private)
+# This is a Hadoop client, and should
+# have any special analytics system users on it
+# for interacting with HDFS.
+include ::role::analytics_cluster::users
 
-# Include Hadoop and other analytics cluster
-# clients so that analysts can access Hadoop
-# from here.
-analytics_cluster::client,
+# Include Hadoop and other analytics cluster
+# clients so that analysts can access Hadoop
+# from here.
+include ::role::analytics_cluster::client
 
-# Include analytics/refinery deployment target.
-analytics_cluster::refinery,
+# Include analytics/refinery deployment target.
+include ::role::analytics_cluster::refinery
 
-# Include analytics/refinery checks that send email about
-# webrequest partitions faultyness.
-analytics_cluster::refinery::job::data_check,
+# Include analytics/refinery checks that send email about
+# webrequest partitions faultyness.
+include ::role::analytics_cluster::refinery::job::data_check
 
-# Include analytics/refinery/source guard checks
-# Disabled due to T166937
-# analytics_cluster::refinery::job::guard,
+# Include analytics/refinery/source guard checks
+# Disabled due to T166937
+# analytics_cluster::refinery::job::guard,
 
-# Set up a read only rsync module to allow access
-# to public data generated by the Analytics Cluster.
-analytics_cluster::rsyncd,
+# Set up a read only rsync module to allow access
+# to public data generated by the Analytics Cluster.
+include ::role::analytics_cluster::rsyncd
 
-# Deploy wikimedia/discovery/analytics repository
-# to this node.
-elasticsearch::analytics)
+# Deploy wikimedia/discovery/analytics repository
+# to this node.
+include ::role::elasticsearch::analytics
 }
 
 # stat1006 is a general purpose number cruncher for

-- 
To view, visit https://gerrit.wikimedia.org/r/402841
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I01ddda470bf38391d687d57ee759fb4fbaf00692
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role() call in ruthenium, tungsten

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402840 )

Change subject: site.pp: one role() call in ruthenium, tungsten
..

site.pp: one role() call in ruthenium, tungsten

Change-Id: I9be3214609e84741116834cea1f7eac8eea9e886
---
M manifests/site.pp
1 file changed, 7 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/40/402840/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 9706cb2..3fafc42 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1943,9 +1943,11 @@
 # Right now, both rt-server and rt-clients run on the same node
 # But, we are likely going to split them into different boxes soon.
 node 'ruthenium.eqiad.wmnet' {
-role(test,
-parsoid::testing, parsoid::rt_server, parsoid::rt_client,
-parsoid::vd_server, parsoid::vd_client, parsoid::diffserver)
+role(parsoid::testing)
+include ::role::test
+include ::role::parsoid::rt_server, ::role::parsoid::vd_server
+include ::role::parsoid::rt_client, ::role::parsoid::vd_client
+include ::role::parsoid::diffserver
 }
 
 # cluster management (cumin master)
@@ -2112,7 +2114,8 @@
 
 # test system for performance team (T117888)
 node 'tungsten.eqiad.wmnet' {
-role(test, xhgui::app)
+role(xhgui::app)
+include ::role::test
 }
 
 # replaced magnesium (RT) (T119112 T123713)

-- 
To view, visit https://gerrit.wikimedia.org/r/402840
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9be3214609e84741116834cea1f7eac8eea9e886
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: fixup for Icd70ef861dcadeeae7df0415a5c2779679c5e144

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402836 )

Change subject: site.pp: fixup for Icd70ef861dcadeeae7df0415a5c2779679c5e144
..


site.pp: fixup for Icd70ef861dcadeeae7df0415a5c2779679c5e144

Change-Id: I0094e1b007f5b9db0cce2db1c1e0b405d9989239
---
M hieradata/role/codfw/prometheus.yaml
A hieradata/role/common/prometheus.yaml
M modules/role/manifests/bastionhost/pop.pp
M modules/role/manifests/cluster/management.pp
4 files changed, 16 insertions(+), 10 deletions(-)

Approvals:
  Giuseppe Lavagetto: Verified; Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/codfw/prometheus.yaml 
b/hieradata/role/codfw/prometheus.yaml
index b3e0545..d45f9e3 100644
--- a/hieradata/role/codfw/prometheus.yaml
+++ b/hieradata/role/codfw/prometheus.yaml
@@ -1,7 +1,6 @@
-profile::base::domain_search:
-  - wikimedia.org
-  - eqiad.wmnet
-  - codfw.wmnet
-  - esams.wmnet
-  - ulsfo.wmnet
-  - eqsin.wmnet
+prometheus::server::memory_chunks: '3145728'
+prometheus::server::max_chunks_to_persist: '1048576'
+lvs::realserver::realserver_ips:
+ - 10.2.1.25
+profile::base::remote_syslog_tls: ['syslog.eqiad.wmnet:6514', 
'syslog.codfw.wmnet:6514']
+profile::base::remote_syslog: []
diff --git a/hieradata/role/common/prometheus.yaml 
b/hieradata/role/common/prometheus.yaml
new file mode 100644
index 000..b3e0545
--- /dev/null
+++ b/hieradata/role/common/prometheus.yaml
@@ -0,0 +1,7 @@
+profile::base::domain_search:
+  - wikimedia.org
+  - eqiad.wmnet
+  - codfw.wmnet
+  - esams.wmnet
+  - ulsfo.wmnet
+  - eqsin.wmnet
diff --git a/modules/role/manifests/bastionhost/pop.pp 
b/modules/role/manifests/bastionhost/pop.pp
index ade59e5..c568b03 100644
--- a/modules/role/manifests/bastionhost/pop.pp
+++ b/modules/role/manifests/bastionhost/pop.pp
@@ -6,5 +6,5 @@
 require ::role::bastionhost::general
 require ::role::installserver::tftp
 require ::role::prometheus::ops
-require ::role::ipmi::mgmt
+require ::profile::ipmi::mgmt
 }
diff --git a/modules/role/manifests/cluster/management.pp 
b/modules/role/manifests/cluster/management.pp
index f267542..ab6974c 100644
--- a/modules/role/manifests/cluster/management.pp
+++ b/modules/role/manifests/cluster/management.pp
@@ -11,8 +11,8 @@
 
 include ::profile::mariadb::wmf_root_client
 include ::role::cumin::master
-include ::role::ipmi::mgmt
-include ::role::access_new_install
+include ::profile::ipmi::mgmt
+include ::profile::access_new_install
 include ::profile::conftool::client
 include ::profile::switchdc
 include ::profile::debdeploy

-- 
To view, visit https://gerrit.wikimedia.org/r/402836
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0094e1b007f5b9db0cce2db1c1e0b405d9989239
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: fixup for Icd70ef861dcadeeae7df0415a5c2779679c5e144

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402836 )

Change subject: site.pp: fixup for Icd70ef861dcadeeae7df0415a5c2779679c5e144
..

site.pp: fixup for Icd70ef861dcadeeae7df0415a5c2779679c5e144

Change-Id: I0094e1b007f5b9db0cce2db1c1e0b405d9989239
---
M hieradata/role/codfw/prometheus.yaml
A hieradata/role/common/prometheus.yaml
M modules/role/manifests/bastionhost/pop.pp
3 files changed, 12 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/36/402836/1

diff --git a/hieradata/role/codfw/prometheus.yaml 
b/hieradata/role/codfw/prometheus.yaml
index b3e0545..279b590 100644
--- a/hieradata/role/codfw/prometheus.yaml
+++ b/hieradata/role/codfw/prometheus.yaml
@@ -1,7 +1,4 @@
-profile::base::domain_search:
-  - wikimedia.org
-  - eqiad.wmnet
-  - codfw.wmnet
-  - esams.wmnet
-  - ulsfo.wmnet
-  - eqsin.wmnet
+prometheus::server::memory_chunks: '3145728'
+prometheus::server::max_chunks_to_persist: '1048576'
+lvs::realserver::realserver_ips:
+ - 10.2.2.25
diff --git a/hieradata/role/common/prometheus.yaml 
b/hieradata/role/common/prometheus.yaml
new file mode 100644
index 000..b3e0545
--- /dev/null
+++ b/hieradata/role/common/prometheus.yaml
@@ -0,0 +1,7 @@
+profile::base::domain_search:
+  - wikimedia.org
+  - eqiad.wmnet
+  - codfw.wmnet
+  - esams.wmnet
+  - ulsfo.wmnet
+  - eqsin.wmnet
diff --git a/modules/role/manifests/bastionhost/pop.pp 
b/modules/role/manifests/bastionhost/pop.pp
index ade59e5..c568b03 100644
--- a/modules/role/manifests/bastionhost/pop.pp
+++ b/modules/role/manifests/bastionhost/pop.pp
@@ -6,5 +6,5 @@
 require ::role::bastionhost::general
 require ::role::installserver::tftp
 require ::role::prometheus::ops
-require ::role::ipmi::mgmt
+require ::profile::ipmi::mgmt
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/402836
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0094e1b007f5b9db0cce2db1c1e0b405d9989239
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: rationalize prometheus, puppetmaster frontends

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402789 )

Change subject: site.pp: rationalize prometheus, puppetmaster frontends
..


site.pp: rationalize prometheus, puppetmaster frontends

* Created a compound role for prometheus
* Incorporated the additional roles inside role::puppetmaster::frontend

Change-Id: Icd70ef861dcadeeae7df0415a5c2779679c5e144
---
R hieradata/role/codfw/prometheus.yaml
R hieradata/role/eqiad/prometheus.yaml
M manifests/site.pp
R modules/profile/manifests/access_new_install.pp
A modules/profile/manifests/ipmi/mgmt.pp
D modules/role/manifests/ipmi/mgmt.pp
A modules/role/manifests/prometheus.pp
M modules/role/manifests/puppetmaster/frontend.pp
8 files changed, 29 insertions(+), 31 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/prometheus/ops.yaml 
b/hieradata/role/codfw/prometheus.yaml
similarity index 100%
rename from hieradata/role/common/prometheus/ops.yaml
rename to hieradata/role/codfw/prometheus.yaml
diff --git a/hieradata/role/eqiad/prometheus/ops.yaml 
b/hieradata/role/eqiad/prometheus.yaml
similarity index 100%
rename from hieradata/role/eqiad/prometheus/ops.yaml
rename to hieradata/role/eqiad/prometheus.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 812fe74..9706cb2 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1056,7 +1056,6 @@
 }
 interface::add_ip6_mapped { 'main': }
 role(bastionhost::twofa)
-include ::role::access_new_install
 }
 
 # The host is going to be decom as part of T181518
@@ -1825,14 +1824,8 @@
 }
 
 node /^prometheus200[34]\.codfw\.wmnet$/ {
-role(prometheus::ops, prometheus::global, prometheus::services, 
prometheus::analytics,
-  prometheus::k8s)
-
+role(prometheus)
 include ::base::firewall
-include ::standard
-include ::lvs::realserver
-
-interface::add_ip6_mapped { 'main': }
 }
 
 node 'puppetcompiler1001.eqiad.wmnet' {
@@ -1841,21 +1834,11 @@
 }
 
 node /^prometheus100[34]\.eqiad\.wmnet$/ {
-role(prometheus::ops, prometheus::global, prometheus::services, 
prometheus::analytics,
-  prometheus::k8s)
-
-include ::lvs::realserver
-
-interface::add_ip6_mapped { 'main': }
+role(prometheus)
 }
 
 node /^puppetmaster[12]001\.(codfw|eqiad)\.wmnet$/ {
-role(
-ipmi::mgmt, access_new_install,
-puppetmaster::frontend,
-)
-include ::standard
-interface::add_ip6_mapped { 'main': }
+role(puppetmaster::frontend)
 }
 
 node /^puppetmaster[12]002\.(codfw|eqiad)\.wmnet$/ {
diff --git a/modules/role/manifests/access_new_install.pp 
b/modules/profile/manifests/access_new_install.pp
similarity index 96%
rename from modules/role/manifests/access_new_install.pp
rename to modules/profile/manifests/access_new_install.pp
index 4a80193..af6339f 100644
--- a/modules/role/manifests/access_new_install.pp
+++ b/modules/profile/manifests/access_new_install.pp
@@ -2,7 +2,7 @@
 #  to set up the initial puppet run.
 # This key is dangerous, do not deploy widely!
 # Also install a convenience script to ssh in using this key
-class role::access_new_install {
+class profile::access_new_install {
 file { '/root/.ssh/new_install':
 owner => 'root',
 group => 'root',
diff --git a/modules/profile/manifests/ipmi/mgmt.pp 
b/modules/profile/manifests/ipmi/mgmt.pp
new file mode 100644
index 000..0c9b1a3
--- /dev/null
+++ b/modules/profile/manifests/ipmi/mgmt.pp
@@ -0,0 +1,4 @@
+# IPMItool mgmt hosts
+class profile::ipmi::mgmt {
+class { '::ipmi::mgmt': }
+}
diff --git a/modules/role/manifests/ipmi/mgmt.pp 
b/modules/role/manifests/ipmi/mgmt.pp
deleted file mode 100644
index 176ab45..000
--- a/modules/role/manifests/ipmi/mgmt.pp
+++ /dev/null
@@ -1,10 +0,0 @@
-# IPMItool mgmt hosts
-class role::ipmi::mgmt {
-
-system::role { 'ipmi::mgmt':
-description => 'IPMI Management'
-}
-
-include ::ipmi::mgmt
-
-}
diff --git a/modules/role/manifests/prometheus.pp 
b/modules/role/manifests/prometheus.pp
new file mode 100644
index 000..f475c21
--- /dev/null
+++ b/modules/role/manifests/prometheus.pp
@@ -0,0 +1,14 @@
+class role::prometheus {
+system::role { 'prometheus::server':  }
+include ::role::prometheus::ops
+include ::role::prometheus::global
+include ::role::prometheus::services
+include ::role::prometheus::analytics
+include ::role::prometheus::k8s
+
+include ::standard
+# TODO: use role::lvs::realserver instead
+include ::lvs::realserver
+
+interface::add_ip6_mapped { 'main': }
+}
diff --git a/modules/role/manifests/puppetmaster/frontend.pp 
b/modules/role/manifests/puppetmaster/frontend.pp
index f6ee73d..33b6be6 100644
--- a/modules/role/manifests/puppetmaster/frontend.pp
+++ b/modules/role/manifests/puppetmaster/frontend.pp
@@ -5,6 +5,7 @@
 description => 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: fix more cases of multiple roles being declared

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402788 )

Change subject: site.pp: fix more cases of multiple roles being declared
..


site.pp: fix more cases of multiple roles being declared

Change-Id: Ifdc95d6c1f20e013b997b443e55f24e4aedcfa39
---
R hieradata/role/common/paws_internal.yaml
M manifests/site.pp
A modules/role/manifests/paws_internal.pp
3 files changed, 18 insertions(+), 8 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/paws_internal/jupyterhub.yaml 
b/hieradata/role/common/paws_internal.yaml
similarity index 100%
rename from hieradata/role/common/paws_internal/jupyterhub.yaml
rename to hieradata/role/common/paws_internal.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 8c5f1a0..812fe74 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1429,7 +1429,8 @@
 }
 
 node /^ms-fe1005\.eqiad\.wmnet$/ {
-role(swift::proxy, swift::stats_reporter)
+role(swift::proxy)
+include ::role::swift::stats_reporter
 include ::lvs::realserver
 }
 
@@ -1456,7 +1457,8 @@
 }
 
 node /^ms-fe2005\.codfw\.wmnet$/ {
-role(swift::proxy, swift::stats_reporter)
+role(swift::proxy)
+include ::role::swift::stats_reporter
 include ::lvs::realserver
 }
 
@@ -1673,12 +1675,14 @@
 
 # mw logging host codfw
 node 'mwlog2001.codfw.wmnet' {
-role(xenon, logging::mediawiki::udp2log)
+role(logging::mediawiki::udp2log)
+include ::role::xenon
 }
 
 # mw logging host eqiad
 node 'mwlog1001.eqiad.wmnet' {
-role(xenon, logging::mediawiki::udp2log)
+role(logging::mediawiki::udp2log)
+include ::role::xenon
 }
 
 node 'mx1001.wikimedia.org' {
@@ -1705,14 +1709,13 @@
 
 # Experimental Jupyter notebook servers
 node 'notebook1001.eqiad.wmnet' {
-role(paws_internal::jupyterhub, analytics_cluster::client, 
paws_internal::mysql_access)
-include ::standard
+role(paws_internal)
 }
 
 # cluster management (cumin master) + other management tools
 node 'neodymium.eqiad.wmnet' {
-role(mgmt::drac_ilo, cluster::management)
-
+role(cluster::management)
+include ::role::mgmt::drac_ilo
 interface::add_ip6_mapped { 'main': }
 }
 
diff --git a/modules/role/manifests/paws_internal.pp 
b/modules/role/manifests/paws_internal.pp
new file mode 100644
index 000..b5741d4
--- /dev/null
+++ b/modules/role/manifests/paws_internal.pp
@@ -0,0 +1,7 @@
+class role::paws_internal {
+system::role { 'PAWS internal': }
+include ::standard
+include ::role::paws_internal::jupyterhub
+include ::role::analytics_cluster::client
+include ::role::paws_internal::mysql_access
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402788
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifdc95d6c1f20e013b997b443e55f24e4aedcfa39
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: create compound role

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402787 )

Change subject: logstash: create compound role
..


logstash: create compound role

Change-Id: I42647a09dc824d6efd86537a105fafed03ffa230
---
R hieradata/role/common/logstash.yaml
D hieradata/role/common/logstash/apifeatureusage.yaml
R hieradata/role/eqiad/logstash.yaml
M manifests/site.pp
A modules/role/manifests/logstash.pp
5 files changed, 12 insertions(+), 6 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/logstash/collector.yaml 
b/hieradata/role/common/logstash.yaml
similarity index 94%
rename from hieradata/role/common/logstash/collector.yaml
rename to hieradata/role/common/logstash.yaml
index 0afd970..0f96405 100644
--- a/hieradata/role/common/logstash/collector.yaml
+++ b/hieradata/role/common/logstash.yaml
@@ -1,3 +1,6 @@
+role::logstash::apifeatureusage::elastic_hosts:
+  - search.svc.codfw.wmnet
+  - search.svc.eqiad.wmnet
 cluster: logstash
 admin::groups:
   - logstash-roots
@@ -52,4 +55,3 @@
   - cn=ops,ou=groups,dc=wikimedia,dc=org
   - cn=nda,ou=groups,dc=wikimedia,dc=org
   - cn=wmf,ou=groups,dc=wikimedia,dc=org
-
diff --git a/hieradata/role/common/logstash/apifeatureusage.yaml 
b/hieradata/role/common/logstash/apifeatureusage.yaml
deleted file mode 100644
index be0a322..000
--- a/hieradata/role/common/logstash/apifeatureusage.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-role::logstash::apifeatureusage::elastic_hosts:
-  - search.svc.codfw.wmnet
-  - search.svc.eqiad.wmnet
diff --git a/hieradata/role/eqiad/kibana.yaml 
b/hieradata/role/eqiad/logstash.yaml
similarity index 100%
rename from hieradata/role/eqiad/kibana.yaml
rename to hieradata/role/eqiad/logstash.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 3b4fa25..8c5f1a0 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1267,11 +1267,12 @@
 
 # logstash collectors (Ganeti VM)
 node 'logstash1007.eqiad.wmnet' {
-role(logstash::collector, kibana, logstash::apifeatureusage, 
logstash::eventlogging)
+role(logstash)
+include ::role::logstash::eventlogging
 include ::lvs::realserver
 }
 node /^logstash100[8-9]\.eqiad\.wmnet$/ {
-role(logstash::collector, kibana, logstash::apifeatureusage)
+role(logstash)
 include ::lvs::realserver
 }
 
diff --git a/modules/role/manifests/logstash.pp 
b/modules/role/manifests/logstash.pp
new file mode 100644
index 000..3e17e07
--- /dev/null
+++ b/modules/role/manifests/logstash.pp
@@ -0,0 +1,6 @@
+class role::logstash {
+system::role { 'logstash': }
+include ::role::logstash::collector
+include ::role::kibana
+include ::role::logstash::apifeatureusage
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402787
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I42647a09dc824d6efd86537a105fafed03ffa230
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: reorganize labs host to use one role() call per node

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402786 )

Change subject: site.pp: reorganize labs host to use one role() call per node
..


site.pp: reorganize labs host to use one role() call per node

Change-Id: I11cb24767dbd7fa1d4d5e69e702e0c376859abcd
---
R hieradata/role/common/labs/monitoring.yaml
M manifests/site.pp
A modules/role/manifests/labs/monitoring.pp
3 files changed, 15 insertions(+), 10 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/labs/prometheus.yaml 
b/hieradata/role/common/labs/monitoring.yaml
similarity index 100%
rename from hieradata/role/common/labs/prometheus.yaml
rename to hieradata/role/common/labs/monitoring.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 49e8038..3b4fa25 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1140,8 +1140,8 @@
 #  - silver (wikitech.wikimedia.org), and
 #  - californium (horizon.wikimedia.org)
 node 'labtestweb2001.wikimedia.org' {
-role(wmcs::openstack::labtest::web,
-  mariadb::wikitech)
+role(wmcs::openstack::labtest::web)
+include ::role::mariadb::wikitech
 include ::base::firewall
 include ::ldap::role::client::labs
 
@@ -1150,11 +1150,7 @@
 
 # Labs Graphite and StatsD host
 node 'labmon1001.eqiad.wmnet' {
-role(labs::graphite,
-  labs::prometheus,
-  grafana::labs)
-include ::standard
-include ::base::firewall
+role(labs::monitoring)
 }
 
 # role spare until pushed into service via T165784
@@ -1192,7 +1188,8 @@
 }
 
 node 'labsdb1004.eqiad.wmnet' {
-role(postgres::master, labs::db::slave)
+role(postgres::master)
+include ::role::labs::db::slave
 }
 
 node 'labsdb1005.eqiad.wmnet' {
@@ -1992,8 +1989,8 @@
 
 # Silver is the new home of the wikitech web server.
 node 'silver.wikimedia.org' {
-role(wmcs::openstack::main::wikitech,
-  mariadb::wikitech)
+role(wmcs::openstack::main::wikitech)
+include ::role::mariadb::wikitech
 include ::base::firewall
 include ::standard
 
diff --git a/modules/role/manifests/labs/monitoring.pp 
b/modules/role/manifests/labs/monitoring.pp
new file mode 100644
index 000..30ef498
--- /dev/null
+++ b/modules/role/manifests/labs/monitoring.pp
@@ -0,0 +1,8 @@
+class role::labs::monitoring {
+system::role { 'labs::monitoring': }
+include ::role::labs::graphite
+include ::role::labs::prometheus
+include ::role::grafana::labs
+include standard
+include ::base::firewall
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402786
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I11cb24767dbd7fa1d4d5e69e702e0c376859abcd
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kripton: one role() call

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402785 )

Change subject: kripton: one role() call
..


kripton: one role() call

Change-Id: I3d3f75227382943fbccd31ce5c20c92ad361b5f2
---
D hieradata/role/common/kafka/analytics/burrow.yaml
M hieradata/role/common/webserver_misc_apps.yaml
M manifests/site.pp
3 files changed, 9 insertions(+), 8 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/kafka/analytics/burrow.yaml 
b/hieradata/role/common/kafka/analytics/burrow.yaml
deleted file mode 100644
index 82a7289..000
--- a/hieradata/role/common/kafka/analytics/burrow.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-burrow::lagcheck_intervals: 100
-burrow::httpserver_port: 8000
-burrow::consumer_groups:
-  - eventlogging_processor_client_side_00
-  - eventlogging_consumer_mysql_00
-  - eventlogging_consumer_mysql_eventbus_00
-  - eventlogging_consumer_files_00
diff --git a/hieradata/role/common/webserver_misc_apps.yaml 
b/hieradata/role/common/webserver_misc_apps.yaml
index 7ad0377..7818b94 100644
--- a/hieradata/role/common/webserver_misc_apps.yaml
+++ b/hieradata/role/common/webserver_misc_apps.yaml
@@ -1 +1,8 @@
 standard::has_default_mail_relay: false
+burrow::lagcheck_intervals: 100
+burrow::httpserver_port: 8000
+burrow::consumer_groups:
+  - eventlogging_processor_client_side_00
+  - eventlogging_consumer_mysql_00
+  - eventlogging_consumer_mysql_eventbus_00
+  - eventlogging_consumer_files_00
diff --git a/manifests/site.pp b/manifests/site.pp
index 15e9db2..49e8038 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1093,7 +1093,8 @@
 # profile::racktables - https://racktables.wikimedia.org
 # kafka::analytics::burrow is a Kafka consumer lag monitor
 node 'krypton.eqiad.wmnet' {
-role(webserver_misc_apps,kafka::analytics::burrow)
+role(webserver_misc_apps)
+include ::role::kafka::analytics::burrow
 }
 
 node /kubernetes[12]00[1-4]\.(codfw|eqiad)\.wmnet/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/402785
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3d3f75227382943fbccd31ce5c20c92ad361b5f2
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kafka: create compound roles, one role() call per node defin...

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402784 )

Change subject: kafka: create compound roles, one role() call per node 
definition
..


kafka: create compound roles, one role() call per node definition

Change-Id: If38f8b18967c883fa7982b5b16751f68e1d69d03
---
A hieradata/role/common/kafka/analytics.yaml
D hieradata/role/common/kafka/analytics/broker.yaml
A hieradata/role/common/kafka/main.yaml
M manifests/site.pp
A modules/role/manifests/kafka/analytics.pp
A modules/role/manifests/kafka/main.pp
6 files changed, 51 insertions(+), 30 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/kafka/analytics.yaml 
b/hieradata/role/common/kafka/analytics.yaml
new file mode 100644
index 000..d714f2f
--- /dev/null
+++ b/hieradata/role/common/kafka/analytics.yaml
@@ -0,0 +1,11 @@
+admin::groups:
+  - analytics-admins
+cluster: analytics_kafka
+nagios_group: analytics_eqiad
+burrow::lagcheck_intervals: 100
+burrow::httpserver_port: 8000
+burrow::consumer_groups:
+  - eventlogging_processor_client_side_00
+  - eventlogging_consumer_mysql_00
+  - eventlogging_consumer_mysql_eventbus_00
+  - eventlogging_consumer_files_00
diff --git a/hieradata/role/common/kafka/analytics/broker.yaml 
b/hieradata/role/common/kafka/analytics/broker.yaml
deleted file mode 100644
index 79be700..000
--- a/hieradata/role/common/kafka/analytics/broker.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-admin::groups:
-  - analytics-admins
-cluster: analytics_kafka
-nagios_group: analytics_eqiad
diff --git a/hieradata/role/common/kafka/main.yaml 
b/hieradata/role/common/kafka/main.yaml
new file mode 100644
index 000..201622d
--- /dev/null
+++ b/hieradata/role/common/kafka/main.yaml
@@ -0,0 +1,5 @@
+cluster: eventbus
+admin::groups:
+  - eventbus-admins
+role::lvs::realserver::pools:
+  eventbus: {}
diff --git a/manifests/site.pp b/manifests/site.pp
index edf431e..15e9db2 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1066,38 +1066,14 @@
 
 # Analytics Kafka Brokers
 node /kafka10(12|13|14|20|22|23)\.eqiad\.wmnet/ {
-# Kafka brokers are routed via IPv6 so that
-# other DCs can address without public IPv4
-# addresses.
-interface::add_ip6_mapped { 'main': }
-
-role(kafka::analytics::broker,
-# Mirror all other Kafka cluster data into the analytics Kafka cluster.
-kafka::analytics::mirror,
-# Mirror main Kafka cluster data to Jumbo Kafka cluster.
-# NOTE:  this is only running on the analytics Kafka brokers because
-# of a 0.11 client compatibility issue.  Ideally this role would
-# be included on the jumbo brokers instead.  But, since we need to 
consume
-# from a 0.9 cluster (main), we need to use a non 0.11 MirrorMaker 
version,
-# which is not available on the Kafka jumbo brokers, since they are 
0.11.
-kafka::jumbo::mirror,
-ipsec)
-
-include ::standard
-include ::base::firewall
+role(kafka::analytics)
 }
 
 # Kafka Brokers - main-eqiad and main-codfw Kafka clusters.
 # For now, eventlogging-service-eventbus is also colocated
 # on these brokers.
 node /kafka[12]00[123]\.(eqiad|codfw)\.wmnet/ {
-role(kafka::main::broker,
-# Mirror eqiad.* topics from main-eqiad into main-codfw,
-# or mirror codfw.* topics from main-codfw into main-eqiad.
-kafka::main::mirror,
-eventbus::eventbus)
-
-include ::standard
+role(kafka::main)
 }
 
 # kafka-jumbo is a large general purpose Kafka cluster.
diff --git a/modules/role/manifests/kafka/analytics.pp 
b/modules/role/manifests/kafka/analytics.pp
new file mode 100644
index 000..d9c5fa9
--- /dev/null
+++ b/modules/role/manifests/kafka/analytics.pp
@@ -0,0 +1,23 @@
+# Compound role for analytics kafka
+class role::kafka::analytics {
+system::role { 'kafka_analytics': }
+# Kafka brokers are routed via IPv6 so that
+# other DCs can address without public IPv4
+# addresses.
+interface::add_ip6_mapped { 'main': }
+
+include ::role::kafka::analytics::broker
+# Mirror all other Kafka cluster data into the analytics Kafka cluster.
+include ::role::kafka::analytics::mirror
+# Mirror main Kafka cluster data to Jumbo Kafka cluster.
+# NOTE:  this is only running on the analytics Kafka brokers because
+# of a 0.11 client compatibility issue.  Ideally this role would
+# be included on the jumbo brokers instead.  But, since we need to consume
+# from a 0.9 cluster (main), we need to use a non 0.11 MirrorMaker version,
+# which is not available on the Kafka jumbo brokers, since they are 0.11.
+include ::role::kafka::jumbo::mirror
+include ::role::ipsec
+
+include ::standard
+include ::base::firewall
+}
diff --git a/modules/role/manifests/kafka/main.pp 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role() call for iron.wikimedia.org

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402390 )

Change subject: site.pp: one role() call for iron.wikimedia.org
..


site.pp: one role() call for iron.wikimedia.org

Change-Id: I6e26758d4828e3aed55aae3aa7d122474e377b5e
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index e009d1b..edf431e 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1055,7 +1055,8 @@
 description => 'Experimental Yubico two factor authentication bastion',
 }
 interface::add_ip6_mapped { 'main': }
-role(bastionhost::twofa, access_new_install)
+role(bastionhost::twofa)
+include ::role::access_new_install
 }
 
 # The host is going to be decom as part of T181518

-- 
To view, visit https://gerrit.wikimedia.org/r/402390
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6e26758d4828e3aed55aae3aa7d122474e377b5e
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::installserver: create meta-role for installserver

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402389 )

Change subject: role::installserver: create meta-role for installserver
..


role::installserver: create meta-role for installserver

Change-Id: I9c53980fa51ea062a967feaf21b71ff2636e0160
---
D hieradata/role/common/installserver/dhcp.yaml
M manifests/site.pp
A modules/role/manifests/installserver.pp
3 files changed, 12 insertions(+), 10 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/installserver/dhcp.yaml 
b/hieradata/role/common/installserver/dhcp.yaml
deleted file mode 100644
index 8bb5fd9..000
--- a/hieradata/role/common/installserver/dhcp.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-admin::groups:
-  - datacenter-ops
diff --git a/manifests/site.pp b/manifests/site.pp
index fd5863f..e009d1b 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1041,14 +1041,7 @@
 
 # replaced carbon and install1001/install2001 (T132757, T84380, T156440)
 node /^install[12]002\.wikimedia\.org$/ {
-role(installserver::tftp,
-installserver::dhcp,
-installserver::http,
-installserver::proxy,
-installserver::preseed,
-aptrepo::wikimedia)
-
-interface::add_ip6_mapped { 'main': }
+role(installserver)
 }
 
 # Phabricator
diff --git a/modules/role/manifests/installserver.pp 
b/modules/role/manifests/installserver.pp
new file mode 100644
index 000..4d9e90a
--- /dev/null
+++ b/modules/role/manifests/installserver.pp
@@ -0,0 +1,11 @@
+class role::installserver {
+system::role { 'installserver': }
+include ::role::installserver::tftp
+include ::role::installserver::dhcp
+include ::role::installserver::http
+include ::role::installserver::proxy
+include ::role::installserver::preseed
+include ::role::aptrepo::wikimedia
+
+interface::add_ip6_mapped { 'main': }
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402389
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9c53980fa51ea062a967feaf21b71ff2636e0160
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: graphite: reorganize roles, one role() call per node

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402388 )

Change subject: graphite: reorganize roles, one role() call per node
..


graphite: reorganize roles, one role() call per node

Also, move role::statsd to a profile

Change-Id: I4b4126d9a25a4b70275eac2e86c48b11a245d4e5
---
A hieradata/role/codfw/graphite/primary.yaml
A hieradata/role/common/graphite/primary.yaml
A hieradata/role/eqiad/graphite/primary.yaml
M manifests/site.pp
R modules/profile/manifests/statsd.pp
A modules/role/manifests/graphite/primary.pp
M modules/role/manifests/graphite/production.pp
7 files changed, 24 insertions(+), 7 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/codfw/graphite/primary.yaml 
b/hieradata/role/codfw/graphite/primary.yaml
new file mode 100644
index 000..509c8bc
--- /dev/null
+++ b/hieradata/role/codfw/graphite/primary.yaml
@@ -0,0 +1,3 @@
+graphite::web::cluster_servers:
+  - graphite2001.codfw.wmnet
+  - graphite2002.codfw.wmnet
diff --git a/hieradata/role/common/graphite/primary.yaml 
b/hieradata/role/common/graphite/primary.yaml
new file mode 100644
index 000..9acf843
--- /dev/null
+++ b/hieradata/role/common/graphite/primary.yaml
@@ -0,0 +1,3 @@
+admin::groups:
+  - perf-roots
+graphite::web::uwsgi_processes: 16
diff --git a/hieradata/role/eqiad/graphite/primary.yaml 
b/hieradata/role/eqiad/graphite/primary.yaml
new file mode 100644
index 000..5ce29cc
--- /dev/null
+++ b/hieradata/role/eqiad/graphite/primary.yaml
@@ -0,0 +1,3 @@
+graphite::web::cluster_servers:
+  - graphite1001.eqiad.wmnet
+  - graphite1003.eqiad.wmnet
diff --git a/manifests/site.pp b/manifests/site.pp
index 1e71e7c..fd5863f 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1011,8 +1011,12 @@
 
 # Primary graphite machines
 node 'graphite1001.eqiad.wmnet' {
-role(graphite::production, statsd, performance::site, graphite::alerts,
-restbase::alerts, graphite::alerts::reqstats, elasticsearch::alerts)
+role(graphite::primary)
+# TODO: move the roles below to ::role::alerting::host
+include ::role::graphite::alerts
+include ::role::restbase::alerts
+include ::role::graphite::alerts::reqstats
+include ::role::elasticsearch::alerts
 }
 
 # graphite test machine, currently with SSD caching + spinning disks
@@ -1022,17 +1026,17 @@
 
 # graphite additional machine, for additional space
 node 'graphite1003.eqiad.wmnet' {
-role(graphite::production, statsd)
+role(graphite::production)
 }
 
 # Primary graphite machines
 node 'graphite2001.codfw.wmnet' {
-role(graphite::production, statsd, performance::site)
+role(graphite::primary)
 }
 
 # graphite additional machine, for additional space
 node 'graphite2002.codfw.wmnet' {
-role(graphite::production, statsd)
+role(graphite::production)
 }
 
 # replaced carbon and install1001/install2001 (T132757, T84380, T156440)
diff --git a/modules/role/manifests/statsd.pp 
b/modules/profile/manifests/statsd.pp
similarity index 98%
rename from modules/role/manifests/statsd.pp
rename to modules/profile/manifests/statsd.pp
index e5e576f..8a01ca6 100644
--- a/modules/role/manifests/statsd.pp
+++ b/modules/profile/manifests/statsd.pp
@@ -5,7 +5,7 @@
 # as well as the set of statsite backends that listen on these ports.
 #
 # filtertags: labs-project-graphite
-class role::statsd {
+class profile::statsd {
 
 class { '::statsd_proxy':
 server_port   => 8125,
diff --git a/modules/role/manifests/graphite/primary.pp 
b/modules/role/manifests/graphite/primary.pp
new file mode 100644
index 000..dbc9f47
--- /dev/null
+++ b/modules/role/manifests/graphite/primary.pp
@@ -0,0 +1,4 @@
+class role::graphite::primary {
+include ::role::graphite::production
+include ::role::performance::site
+}
diff --git a/modules/role/manifests/graphite/production.pp 
b/modules/role/manifests/graphite/production.pp
index b0b33b0..e46cac1 100644
--- a/modules/role/manifests/graphite/production.pp
+++ b/modules/role/manifests/graphite/production.pp
@@ -9,6 +9,7 @@
 
 include ::standard
 include ::base::firewall
+include ::profile::statsd # all graphite hosts also include statsd
 
 class { 'role::graphite::base':
 storage_dir  => $storage_dir,
@@ -117,4 +118,3 @@
 srange => '$PRODUCTION_NETWORKS',
 }
 }
-

-- 
To view, visit https://gerrit.wikimedia.org/r/402388
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4b4126d9a25a4b70275eac2e86c48b11a245d4e5
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kafka: create compound roles, one role() call per node defin...

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402784 )

Change subject: kafka: create compound roles, one role() call per node 
definition
..

kafka: create compound roles, one role() call per node definition

Change-Id: If38f8b18967c883fa7982b5b16751f68e1d69d03
---
A hieradata/role/common/kafka/analytics.yaml
D hieradata/role/common/kafka/analytics/broker.yaml
M manifests/site.pp
A modules/role/manifests/kafka/analytics.pp
A modules/role/manifests/kafka/main.pp
5 files changed, 46 insertions(+), 30 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/84/402784/1

diff --git a/hieradata/role/common/kafka/analytics.yaml 
b/hieradata/role/common/kafka/analytics.yaml
new file mode 100644
index 000..d714f2f
--- /dev/null
+++ b/hieradata/role/common/kafka/analytics.yaml
@@ -0,0 +1,11 @@
+admin::groups:
+  - analytics-admins
+cluster: analytics_kafka
+nagios_group: analytics_eqiad
+burrow::lagcheck_intervals: 100
+burrow::httpserver_port: 8000
+burrow::consumer_groups:
+  - eventlogging_processor_client_side_00
+  - eventlogging_consumer_mysql_00
+  - eventlogging_consumer_mysql_eventbus_00
+  - eventlogging_consumer_files_00
diff --git a/hieradata/role/common/kafka/analytics/broker.yaml 
b/hieradata/role/common/kafka/analytics/broker.yaml
deleted file mode 100644
index 79be700..000
--- a/hieradata/role/common/kafka/analytics/broker.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-admin::groups:
-  - analytics-admins
-cluster: analytics_kafka
-nagios_group: analytics_eqiad
diff --git a/manifests/site.pp b/manifests/site.pp
index edf431e..15e9db2 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1066,38 +1066,14 @@
 
 # Analytics Kafka Brokers
 node /kafka10(12|13|14|20|22|23)\.eqiad\.wmnet/ {
-# Kafka brokers are routed via IPv6 so that
-# other DCs can address without public IPv4
-# addresses.
-interface::add_ip6_mapped { 'main': }
-
-role(kafka::analytics::broker,
-# Mirror all other Kafka cluster data into the analytics Kafka cluster.
-kafka::analytics::mirror,
-# Mirror main Kafka cluster data to Jumbo Kafka cluster.
-# NOTE:  this is only running on the analytics Kafka brokers because
-# of a 0.11 client compatibility issue.  Ideally this role would
-# be included on the jumbo brokers instead.  But, since we need to 
consume
-# from a 0.9 cluster (main), we need to use a non 0.11 MirrorMaker 
version,
-# which is not available on the Kafka jumbo brokers, since they are 
0.11.
-kafka::jumbo::mirror,
-ipsec)
-
-include ::standard
-include ::base::firewall
+role(kafka::analytics)
 }
 
 # Kafka Brokers - main-eqiad and main-codfw Kafka clusters.
 # For now, eventlogging-service-eventbus is also colocated
 # on these brokers.
 node /kafka[12]00[123]\.(eqiad|codfw)\.wmnet/ {
-role(kafka::main::broker,
-# Mirror eqiad.* topics from main-eqiad into main-codfw,
-# or mirror codfw.* topics from main-codfw into main-eqiad.
-kafka::main::mirror,
-eventbus::eventbus)
-
-include ::standard
+role(kafka::main)
 }
 
 # kafka-jumbo is a large general purpose Kafka cluster.
diff --git a/modules/role/manifests/kafka/analytics.pp 
b/modules/role/manifests/kafka/analytics.pp
new file mode 100644
index 000..d9c5fa9
--- /dev/null
+++ b/modules/role/manifests/kafka/analytics.pp
@@ -0,0 +1,23 @@
+# Compound role for analytics kafka
+class role::kafka::analytics {
+system::role { 'kafka_analytics': }
+# Kafka brokers are routed via IPv6 so that
+# other DCs can address without public IPv4
+# addresses.
+interface::add_ip6_mapped { 'main': }
+
+include ::role::kafka::analytics::broker
+# Mirror all other Kafka cluster data into the analytics Kafka cluster.
+include ::role::kafka::analytics::mirror
+# Mirror main Kafka cluster data to Jumbo Kafka cluster.
+# NOTE:  this is only running on the analytics Kafka brokers because
+# of a 0.11 client compatibility issue.  Ideally this role would
+# be included on the jumbo brokers instead.  But, since we need to consume
+# from a 0.9 cluster (main), we need to use a non 0.11 MirrorMaker version,
+# which is not available on the Kafka jumbo brokers, since they are 0.11.
+include ::role::kafka::jumbo::mirror
+include ::role::ipsec
+
+include ::standard
+include ::base::firewall
+}
diff --git a/modules/role/manifests/kafka/main.pp 
b/modules/role/manifests/kafka/main.pp
new file mode 100644
index 000..8ca7176
--- /dev/null
+++ b/modules/role/manifests/kafka/main.pp
@@ -0,0 +1,10 @@
+# Compound role for the Kafka "main" cluster
+class role::kafka::main {
+include ::role::kafka::main::broker
+# Mirror eqiad.* topics from main-eqiad into main-codfw,
+# or mirror codfw.* topics from main-codfw into 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: reorganize labs host to use one role() call per node

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402786 )

Change subject: site.pp: reorganize labs host to use one role() call per node
..

site.pp: reorganize labs host to use one role() call per node

Change-Id: I11cb24767dbd7fa1d4d5e69e702e0c376859abcd
---
R hieradata/role/common/labs/monitoring.yaml
M manifests/site.pp
A modules/role/manifests/labs/monitoring.pp
3 files changed, 15 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/86/402786/1

diff --git a/hieradata/role/common/labs/prometheus.yaml 
b/hieradata/role/common/labs/monitoring.yaml
similarity index 100%
rename from hieradata/role/common/labs/prometheus.yaml
rename to hieradata/role/common/labs/monitoring.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 49e8038..3b4fa25 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1140,8 +1140,8 @@
 #  - silver (wikitech.wikimedia.org), and
 #  - californium (horizon.wikimedia.org)
 node 'labtestweb2001.wikimedia.org' {
-role(wmcs::openstack::labtest::web,
-  mariadb::wikitech)
+role(wmcs::openstack::labtest::web)
+include ::role::mariadb::wikitech
 include ::base::firewall
 include ::ldap::role::client::labs
 
@@ -1150,11 +1150,7 @@
 
 # Labs Graphite and StatsD host
 node 'labmon1001.eqiad.wmnet' {
-role(labs::graphite,
-  labs::prometheus,
-  grafana::labs)
-include ::standard
-include ::base::firewall
+role(labs::monitoring)
 }
 
 # role spare until pushed into service via T165784
@@ -1192,7 +1188,8 @@
 }
 
 node 'labsdb1004.eqiad.wmnet' {
-role(postgres::master, labs::db::slave)
+role(postgres::master)
+include ::role::labs::db::slave
 }
 
 node 'labsdb1005.eqiad.wmnet' {
@@ -1992,8 +1989,8 @@
 
 # Silver is the new home of the wikitech web server.
 node 'silver.wikimedia.org' {
-role(wmcs::openstack::main::wikitech,
-  mariadb::wikitech)
+role(wmcs::openstack::main::wikitech)
+include ::role::mariadb::wikitech
 include ::base::firewall
 include ::standard
 
diff --git a/modules/role/manifests/labs/monitoring.pp 
b/modules/role/manifests/labs/monitoring.pp
new file mode 100644
index 000..30ef498
--- /dev/null
+++ b/modules/role/manifests/labs/monitoring.pp
@@ -0,0 +1,8 @@
+class role::labs::monitoring {
+system::role { 'labs::monitoring': }
+include ::role::labs::graphite
+include ::role::labs::prometheus
+include ::role::grafana::labs
+include standard
+include ::base::firewall
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402786
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I11cb24767dbd7fa1d4d5e69e702e0c376859abcd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: kripton: one role() call

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402785 )

Change subject: kripton: one role() call
..

kripton: one role() call

Change-Id: I3d3f75227382943fbccd31ce5c20c92ad361b5f2
---
D hieradata/role/common/kafka/analytics/burrow.yaml
M hieradata/role/common/webserver_misc_apps.yaml
M manifests/site.pp
3 files changed, 9 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/85/402785/1

diff --git a/hieradata/role/common/kafka/analytics/burrow.yaml 
b/hieradata/role/common/kafka/analytics/burrow.yaml
deleted file mode 100644
index 82a7289..000
--- a/hieradata/role/common/kafka/analytics/burrow.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-burrow::lagcheck_intervals: 100
-burrow::httpserver_port: 8000
-burrow::consumer_groups:
-  - eventlogging_processor_client_side_00
-  - eventlogging_consumer_mysql_00
-  - eventlogging_consumer_mysql_eventbus_00
-  - eventlogging_consumer_files_00
diff --git a/hieradata/role/common/webserver_misc_apps.yaml 
b/hieradata/role/common/webserver_misc_apps.yaml
index 7ad0377..7818b94 100644
--- a/hieradata/role/common/webserver_misc_apps.yaml
+++ b/hieradata/role/common/webserver_misc_apps.yaml
@@ -1 +1,8 @@
 standard::has_default_mail_relay: false
+burrow::lagcheck_intervals: 100
+burrow::httpserver_port: 8000
+burrow::consumer_groups:
+  - eventlogging_processor_client_side_00
+  - eventlogging_consumer_mysql_00
+  - eventlogging_consumer_mysql_eventbus_00
+  - eventlogging_consumer_files_00
diff --git a/manifests/site.pp b/manifests/site.pp
index 15e9db2..49e8038 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1093,7 +1093,8 @@
 # profile::racktables - https://racktables.wikimedia.org
 # kafka::analytics::burrow is a Kafka consumer lag monitor
 node 'krypton.eqiad.wmnet' {
-role(webserver_misc_apps,kafka::analytics::burrow)
+role(webserver_misc_apps)
+include ::role::kafka::analytics::burrow
 }
 
 node /kubernetes[12]00[1-4]\.(codfw|eqiad)\.wmnet/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/402785
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3d3f75227382943fbccd31ce5c20c92ad361b5f2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: rationalize prometheus, puppetmaster frontends

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402789 )

Change subject: site.pp: rationalize prometheus, puppetmaster frontends
..

site.pp: rationalize prometheus, puppetmaster frontends

* Created a compound role for prometheus
* Incorporated the additional roles inside role::puppetmaster::frontend

Change-Id: Icd70ef861dcadeeae7df0415a5c2779679c5e144
---
R hieradata/role/codfw/prometheus.yaml
R hieradata/role/eqiad/prometheus.yaml
M manifests/site.pp
R modules/profile/manifests/access_new_install.pp
A modules/profile/manifests/ipmi/mgmt.pp
D modules/role/manifests/ipmi/mgmt.pp
A modules/role/manifests/prometheus.pp
M modules/role/manifests/puppetmaster/frontend.pp
8 files changed, 29 insertions(+), 30 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/89/402789/1

diff --git a/hieradata/role/common/prometheus/ops.yaml 
b/hieradata/role/codfw/prometheus.yaml
similarity index 100%
rename from hieradata/role/common/prometheus/ops.yaml
rename to hieradata/role/codfw/prometheus.yaml
diff --git a/hieradata/role/eqiad/prometheus/ops.yaml 
b/hieradata/role/eqiad/prometheus.yaml
similarity index 100%
rename from hieradata/role/eqiad/prometheus/ops.yaml
rename to hieradata/role/eqiad/prometheus.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 812fe74..f5fb926 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1825,14 +1825,8 @@
 }
 
 node /^prometheus200[34]\.codfw\.wmnet$/ {
-role(prometheus::ops, prometheus::global, prometheus::services, 
prometheus::analytics,
-  prometheus::k8s)
-
+role(prometheus)
 include ::base::firewall
-include ::standard
-include ::lvs::realserver
-
-interface::add_ip6_mapped { 'main': }
 }
 
 node 'puppetcompiler1001.eqiad.wmnet' {
@@ -1841,21 +1835,11 @@
 }
 
 node /^prometheus100[34]\.eqiad\.wmnet$/ {
-role(prometheus::ops, prometheus::global, prometheus::services, 
prometheus::analytics,
-  prometheus::k8s)
-
-include ::lvs::realserver
-
-interface::add_ip6_mapped { 'main': }
+role(prometheus)
 }
 
 node /^puppetmaster[12]001\.(codfw|eqiad)\.wmnet$/ {
-role(
-ipmi::mgmt, access_new_install,
-puppetmaster::frontend,
-)
-include ::standard
-interface::add_ip6_mapped { 'main': }
+role(puppetmaster::frontend)
 }
 
 node /^puppetmaster[12]002\.(codfw|eqiad)\.wmnet$/ {
diff --git a/modules/role/manifests/access_new_install.pp 
b/modules/profile/manifests/access_new_install.pp
similarity index 96%
rename from modules/role/manifests/access_new_install.pp
rename to modules/profile/manifests/access_new_install.pp
index 4a80193..af6339f 100644
--- a/modules/role/manifests/access_new_install.pp
+++ b/modules/profile/manifests/access_new_install.pp
@@ -2,7 +2,7 @@
 #  to set up the initial puppet run.
 # This key is dangerous, do not deploy widely!
 # Also install a convenience script to ssh in using this key
-class role::access_new_install {
+class profile::access_new_install {
 file { '/root/.ssh/new_install':
 owner => 'root',
 group => 'root',
diff --git a/modules/profile/manifests/ipmi/mgmt.pp 
b/modules/profile/manifests/ipmi/mgmt.pp
new file mode 100644
index 000..0c9b1a3
--- /dev/null
+++ b/modules/profile/manifests/ipmi/mgmt.pp
@@ -0,0 +1,4 @@
+# IPMItool mgmt hosts
+class profile::ipmi::mgmt {
+class { '::ipmi::mgmt': }
+}
diff --git a/modules/role/manifests/ipmi/mgmt.pp 
b/modules/role/manifests/ipmi/mgmt.pp
deleted file mode 100644
index 176ab45..000
--- a/modules/role/manifests/ipmi/mgmt.pp
+++ /dev/null
@@ -1,10 +0,0 @@
-# IPMItool mgmt hosts
-class role::ipmi::mgmt {
-
-system::role { 'ipmi::mgmt':
-description => 'IPMI Management'
-}
-
-include ::ipmi::mgmt
-
-}
diff --git a/modules/role/manifests/prometheus.pp 
b/modules/role/manifests/prometheus.pp
new file mode 100644
index 000..f475c21
--- /dev/null
+++ b/modules/role/manifests/prometheus.pp
@@ -0,0 +1,14 @@
+class role::prometheus {
+system::role { 'prometheus::server':  }
+include ::role::prometheus::ops
+include ::role::prometheus::global
+include ::role::prometheus::services
+include ::role::prometheus::analytics
+include ::role::prometheus::k8s
+
+include ::standard
+# TODO: use role::lvs::realserver instead
+include ::lvs::realserver
+
+interface::add_ip6_mapped { 'main': }
+}
diff --git a/modules/role/manifests/puppetmaster/frontend.pp 
b/modules/role/manifests/puppetmaster/frontend.pp
index f6ee73d..33b6be6 100644
--- a/modules/role/manifests/puppetmaster/frontend.pp
+++ b/modules/role/manifests/puppetmaster/frontend.pp
@@ -5,6 +5,7 @@
 description => 'Puppetmaster frontend'
 }
 
+include ::standard
 include ::base::firewall
 
 include ::profile::backup::host
@@ -17,4 +18,10 @@
 # config-master.wikimedia.org
 include 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: fix more cases of multiple roles being declared

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402788 )

Change subject: site.pp: fix more cases of multiple roles being declared
..

site.pp: fix more cases of multiple roles being declared

Change-Id: Ifdc95d6c1f20e013b997b443e55f24e4aedcfa39
---
R hieradata/role/common/paws_internal.yaml
M manifests/site.pp
A modules/role/manifests/paws_internal.pp
3 files changed, 18 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/88/402788/1

diff --git a/hieradata/role/common/paws_internal/jupyterhub.yaml 
b/hieradata/role/common/paws_internal.yaml
similarity index 100%
rename from hieradata/role/common/paws_internal/jupyterhub.yaml
rename to hieradata/role/common/paws_internal.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 8c5f1a0..812fe74 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1429,7 +1429,8 @@
 }
 
 node /^ms-fe1005\.eqiad\.wmnet$/ {
-role(swift::proxy, swift::stats_reporter)
+role(swift::proxy)
+include ::role::swift::stats_reporter
 include ::lvs::realserver
 }
 
@@ -1456,7 +1457,8 @@
 }
 
 node /^ms-fe2005\.codfw\.wmnet$/ {
-role(swift::proxy, swift::stats_reporter)
+role(swift::proxy)
+include ::role::swift::stats_reporter
 include ::lvs::realserver
 }
 
@@ -1673,12 +1675,14 @@
 
 # mw logging host codfw
 node 'mwlog2001.codfw.wmnet' {
-role(xenon, logging::mediawiki::udp2log)
+role(logging::mediawiki::udp2log)
+include ::role::xenon
 }
 
 # mw logging host eqiad
 node 'mwlog1001.eqiad.wmnet' {
-role(xenon, logging::mediawiki::udp2log)
+role(logging::mediawiki::udp2log)
+include ::role::xenon
 }
 
 node 'mx1001.wikimedia.org' {
@@ -1705,14 +1709,13 @@
 
 # Experimental Jupyter notebook servers
 node 'notebook1001.eqiad.wmnet' {
-role(paws_internal::jupyterhub, analytics_cluster::client, 
paws_internal::mysql_access)
-include ::standard
+role(paws_internal)
 }
 
 # cluster management (cumin master) + other management tools
 node 'neodymium.eqiad.wmnet' {
-role(mgmt::drac_ilo, cluster::management)
-
+role(cluster::management)
+include ::role::mgmt::drac_ilo
 interface::add_ip6_mapped { 'main': }
 }
 
diff --git a/modules/role/manifests/paws_internal.pp 
b/modules/role/manifests/paws_internal.pp
new file mode 100644
index 000..b5741d4
--- /dev/null
+++ b/modules/role/manifests/paws_internal.pp
@@ -0,0 +1,7 @@
+class role::paws_internal {
+system::role { 'PAWS internal': }
+include ::standard
+include ::role::paws_internal::jupyterhub
+include ::role::analytics_cluster::client
+include ::role::paws_internal::mysql_access
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402788
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifdc95d6c1f20e013b997b443e55f24e4aedcfa39
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: create compound role

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402787 )

Change subject: logstash: create compound role
..

logstash: create compound role

Change-Id: I42647a09dc824d6efd86537a105fafed03ffa230
---
R hieradata/role/common/logstash.yaml
D hieradata/role/common/logstash/apifeatureusage.yaml
D hieradata/role/common/logstash/elasticsearch.yaml
R hieradata/role/eqiad/logstash.yaml
M manifests/site.pp
A modules/role/manifests/logstash.pp
6 files changed, 12 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/87/402787/1

diff --git a/hieradata/role/common/logstash/collector.yaml 
b/hieradata/role/common/logstash.yaml
similarity index 94%
rename from hieradata/role/common/logstash/collector.yaml
rename to hieradata/role/common/logstash.yaml
index 0afd970..0f96405 100644
--- a/hieradata/role/common/logstash/collector.yaml
+++ b/hieradata/role/common/logstash.yaml
@@ -1,3 +1,6 @@
+role::logstash::apifeatureusage::elastic_hosts:
+  - search.svc.codfw.wmnet
+  - search.svc.eqiad.wmnet
 cluster: logstash
 admin::groups:
   - logstash-roots
@@ -52,4 +55,3 @@
   - cn=ops,ou=groups,dc=wikimedia,dc=org
   - cn=nda,ou=groups,dc=wikimedia,dc=org
   - cn=wmf,ou=groups,dc=wikimedia,dc=org
-
diff --git a/hieradata/role/common/logstash/apifeatureusage.yaml 
b/hieradata/role/common/logstash/apifeatureusage.yaml
deleted file mode 100644
index be0a322..000
--- a/hieradata/role/common/logstash/apifeatureusage.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-role::logstash::apifeatureusage::elastic_hosts:
-  - search.svc.codfw.wmnet
-  - search.svc.eqiad.wmnet
diff --git a/hieradata/role/common/logstash/elasticsearch.yaml 
b/hieradata/role/common/logstash/elasticsearch.yaml
deleted file mode 100644
index 932f0ff..000
--- a/hieradata/role/common/logstash/elasticsearch.yaml
+++ /dev/null
@@ -1,28 +0,0 @@
-# General configs
-cluster: logstash
-admin::groups:
-  - logstash-roots
-  - elasticsearch-roots
-
-# ES-specific
-# NOTE: cluster_name must be kept in sync with the backend
-# node config in hieradata/role/common/collector.yaml
-elasticsearch::auto_create_index: true
-elasticsearch::cluster_name: production-logstash-eqiad
-elasticsearch::expected_nodes: 3
-elasticsearch::heap_memory: '30G'
-elasticsearch::minimum_master_nodes: 2
-elasticsearch::recover_after_nodes: 2
-elasticsearch::recover_after_time: '1m'
-elasticsearch::unicast_hosts:
-  - logstash1004.eqiad.wmnet
-  - logstash1005.eqiad.wmnet
-  - logstash1006.eqiad.wmnet
-logstash::cluster_hosts:
-  - logstash1004.eqiad.wmnet
-  - logstash1005.eqiad.wmnet
-  - logstash1006.eqiad.wmnet
-  - logstash1007.eqiad.wmnet
-  - logstash1008.eqiad.wmnet
-  - logstash1009.eqiad.wmnet
-elasticsearch::data_dir: '/var/lib/elasticsearch'
diff --git a/hieradata/role/eqiad/kibana.yaml 
b/hieradata/role/eqiad/logstash.yaml
similarity index 100%
rename from hieradata/role/eqiad/kibana.yaml
rename to hieradata/role/eqiad/logstash.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 3b4fa25..8c5f1a0 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1267,11 +1267,12 @@
 
 # logstash collectors (Ganeti VM)
 node 'logstash1007.eqiad.wmnet' {
-role(logstash::collector, kibana, logstash::apifeatureusage, 
logstash::eventlogging)
+role(logstash)
+include ::role::logstash::eventlogging
 include ::lvs::realserver
 }
 node /^logstash100[8-9]\.eqiad\.wmnet$/ {
-role(logstash::collector, kibana, logstash::apifeatureusage)
+role(logstash)
 include ::lvs::realserver
 }
 
diff --git a/modules/role/manifests/logstash.pp 
b/modules/role/manifests/logstash.pp
new file mode 100644
index 000..3e17e07
--- /dev/null
+++ b/modules/role/manifests/logstash.pp
@@ -0,0 +1,6 @@
+class role::logstash {
+system::role { 'logstash': }
+include ::role::logstash::collector
+include ::role::kibana
+include ::role::logstash::apifeatureusage
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402787
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I42647a09dc824d6efd86537a105fafed03ffa230
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role() call for iron.wikimedia.org

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402390 )

Change subject: site.pp: one role() call for iron.wikimedia.org
..

site.pp: one role() call for iron.wikimedia.org

Change-Id: I6e26758d4828e3aed55aae3aa7d122474e377b5e
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/90/402390/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 393e398..b28587b 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1047,7 +1047,8 @@
 description => 'Experimental Yubico two factor authentication bastion',
 }
 interface::add_ip6_mapped { 'main': }
-role(bastionhost::twofa, access_new_install)
+role(bastionhost::twofa)
+include ::role::access_new_install
 }
 
 # The host is going to be decom as part of T181518

-- 
To view, visit https://gerrit.wikimedia.org/r/402390
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6e26758d4828e3aed55aae3aa7d122474e377b5e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: graphite: reorganize roles, one role() call per node

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402388 )

Change subject: graphite: reorganize roles, one role() call per node
..

graphite: reorganize roles, one role() call per node

Also, move role::statsd to a profile

Change-Id: I4b4126d9a25a4b70275eac2e86c48b11a245d4e5
---
A hieradata/role/codfw/graphite/primary.yaml
A hieradata/role/common/graphite/primary.yaml
A hieradata/role/eqiad/graphite/primary.yaml
M manifests/site.pp
R modules/profile/manifests/statsd.pp
A modules/role/manifests/graphite/primary.pp
M modules/role/manifests/graphite/production.pp
7 files changed, 24 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/88/402388/1

diff --git a/hieradata/role/codfw/graphite/primary.yaml 
b/hieradata/role/codfw/graphite/primary.yaml
new file mode 100644
index 000..509c8bc
--- /dev/null
+++ b/hieradata/role/codfw/graphite/primary.yaml
@@ -0,0 +1,3 @@
+graphite::web::cluster_servers:
+  - graphite2001.codfw.wmnet
+  - graphite2002.codfw.wmnet
diff --git a/hieradata/role/common/graphite/primary.yaml 
b/hieradata/role/common/graphite/primary.yaml
new file mode 100644
index 000..9acf843
--- /dev/null
+++ b/hieradata/role/common/graphite/primary.yaml
@@ -0,0 +1,3 @@
+admin::groups:
+  - perf-roots
+graphite::web::uwsgi_processes: 16
diff --git a/hieradata/role/eqiad/graphite/primary.yaml 
b/hieradata/role/eqiad/graphite/primary.yaml
new file mode 100644
index 000..5ce29cc
--- /dev/null
+++ b/hieradata/role/eqiad/graphite/primary.yaml
@@ -0,0 +1,3 @@
+graphite::web::cluster_servers:
+  - graphite1001.eqiad.wmnet
+  - graphite1003.eqiad.wmnet
diff --git a/manifests/site.pp b/manifests/site.pp
index a9a879f..f2ef6a4 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1003,8 +1003,12 @@
 
 # Primary graphite machines
 node 'graphite1001.eqiad.wmnet' {
-role(graphite::production, statsd, performance::site, graphite::alerts,
-restbase::alerts, graphite::alerts::reqstats, elasticsearch::alerts)
+role(graphite::primary)
+# TODO: move the roles below to ::role::alerting::host
+include ::role::graphite::alerts
+include ::role::restbase::alerts
+include ::role::graphite::alerts::reqstats
+include ::role::elasticsearch::alerts
 }
 
 # graphite test machine, currently with SSD caching + spinning disks
@@ -1014,17 +1018,17 @@
 
 # graphite additional machine, for additional space
 node 'graphite1003.eqiad.wmnet' {
-role(graphite::production, statsd)
+role(graphite::production)
 }
 
 # Primary graphite machines
 node 'graphite2001.codfw.wmnet' {
-role(graphite::production, statsd, performance::site)
+role(graphite::primary)
 }
 
 # graphite additional machine, for additional space
 node 'graphite2002.codfw.wmnet' {
-role(graphite::production, statsd)
+role(graphite::production)
 }
 
 # replaced carbon and install1001/install2001 (T132757, T84380, T156440)
diff --git a/modules/role/manifests/statsd.pp 
b/modules/profile/manifests/statsd.pp
similarity index 98%
rename from modules/role/manifests/statsd.pp
rename to modules/profile/manifests/statsd.pp
index e5e576f..8a01ca6 100644
--- a/modules/role/manifests/statsd.pp
+++ b/modules/profile/manifests/statsd.pp
@@ -5,7 +5,7 @@
 # as well as the set of statsite backends that listen on these ports.
 #
 # filtertags: labs-project-graphite
-class role::statsd {
+class profile::statsd {
 
 class { '::statsd_proxy':
 server_port   => 8125,
diff --git a/modules/role/manifests/graphite/primary.pp 
b/modules/role/manifests/graphite/primary.pp
new file mode 100644
index 000..dbc9f47
--- /dev/null
+++ b/modules/role/manifests/graphite/primary.pp
@@ -0,0 +1,4 @@
+class role::graphite::primary {
+include ::role::graphite::production
+include ::role::performance::site
+}
diff --git a/modules/role/manifests/graphite/production.pp 
b/modules/role/manifests/graphite/production.pp
index b0b33b0..e46cac1 100644
--- a/modules/role/manifests/graphite/production.pp
+++ b/modules/role/manifests/graphite/production.pp
@@ -9,6 +9,7 @@
 
 include ::standard
 include ::base::firewall
+include ::profile::statsd # all graphite hosts also include statsd
 
 class { 'role::graphite::base':
 storage_dir  => $storage_dir,
@@ -117,4 +118,3 @@
 srange => '$PRODUCTION_NETWORKS',
 }
 }
-

-- 
To view, visit https://gerrit.wikimedia.org/r/402388
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4b4126d9a25a4b70275eac2e86c48b11a245d4e5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::installserver: create meta-role for installserver

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402389 )

Change subject: role::installserver: create meta-role for installserver
..

role::installserver: create meta-role for installserver

Change-Id: I9c53980fa51ea062a967feaf21b71ff2636e0160
---
D hieradata/role/common/installserver/dhcp.yaml
M manifests/site.pp
A modules/role/manifests/installserver.pp
3 files changed, 12 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/89/402389/1

diff --git a/hieradata/role/common/installserver/dhcp.yaml 
b/hieradata/role/common/installserver/dhcp.yaml
deleted file mode 100644
index 8bb5fd9..000
--- a/hieradata/role/common/installserver/dhcp.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-admin::groups:
-  - datacenter-ops
diff --git a/manifests/site.pp b/manifests/site.pp
index f2ef6a4..393e398 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1033,14 +1033,7 @@
 
 # replaced carbon and install1001/install2001 (T132757, T84380, T156440)
 node /^install[12]002\.wikimedia\.org$/ {
-role(installserver::tftp,
-installserver::dhcp,
-installserver::http,
-installserver::proxy,
-installserver::preseed,
-aptrepo::wikimedia)
-
-interface::add_ip6_mapped { 'main': }
+role(installserver)
 }
 
 # Phabricator
diff --git a/modules/role/manifests/installserver.pp 
b/modules/role/manifests/installserver.pp
new file mode 100644
index 000..4d9e90a
--- /dev/null
+++ b/modules/role/manifests/installserver.pp
@@ -0,0 +1,11 @@
+class role::installserver {
+system::role { 'installserver': }
+include ::role::installserver::tftp
+include ::role::installserver::dhcp
+include ::role::installserver::http
+include ::role::installserver::proxy
+include ::role::installserver::preseed
+include ::role::aptrepo::wikimedia
+
+interface::add_ip6_mapped { 'main': }
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/402389
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9c53980fa51ea062a967feaf21b71ff2636e0160
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: labtestservices2001: one role() call

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401555 )

Change subject: labtestservices2001: one role() call
..


labtestservices2001: one role() call

Change-Id: I8298dfaf9128bafe08da972a34574a83ca2ecf0f
---
M manifests/site.pp
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index 9ddd340..effdbbf 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -982,8 +982,8 @@
 }
 
 node 'labtestservices2001.wikimedia.org' {
-role(wmcs::openstack::labtest::services,
-  openldap::labtest)
+role(wmcs::openstack::labtest::services)
+include ::role::openldap::labtest
 interface::add_ip6_mapped { 'main': }
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/401555
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8298dfaf9128bafe08da972a34574a83ca2ecf0f
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: wmflib: simplify the role() function, convert to the new API

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402345 )

Change subject: wmflib: simplify the role() function, convert to the new API
..

wmflib: simplify the role() function, convert to the new API

Converting to the new API makes some things possible, namely:
- Clean access to the top scope
- Cleanly require the function is called exactly once
- Use call_function for 'include'

but on the other hand makes it impossible to verify if we're calling the
function from the node scope. Please note that this happens for a reason
- messing with the scope in function can cause undefined behaviour and
indeed I found edge cases where this can happen, for instance when
trying to access the variables before they're defined can make them null.

Change-Id: I32631c84a7f6340a0d7de2ae57dfbef87015c46b
---
A modules/wmflib/lib/puppet/functions/role.rb
D modules/wmflib/lib/puppet/parser/functions/role.rb
M modules/wmflib/spec/functions/role_spec.rb
3 files changed, 73 insertions(+), 101 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/45/402345/1

diff --git a/modules/wmflib/lib/puppet/functions/role.rb 
b/modules/wmflib/lib/puppet/functions/role.rb
new file mode 100644
index 000..877624e
--- /dev/null
+++ b/modules/wmflib/lib/puppet/functions/role.rb
@@ -0,0 +1,54 @@
+# == Function: role ( string $role_name [, string $... ] )
+#
+# Declare the _roles variable (or add keys to it), and if the role class
+# role::$role_name is in the scope, include it. This is roughly a
+# shortcut for
+#
+# $::_roles[foo] = true
+# include role::foo
+#
+# and has the notable advantage over that the syntax is shorter. Also,
+# this function  will refuse to run anywhere but at the node scope,
+# thus making any additional role added to a node explicit.
+#
+# If you have more than one role to declare, you MUST do that in one
+# single role stanza, or you would encounter unexpected behaviour. If
+# you do, an exception will be raised.
+#
+# This function is very useful with our "role" hiera backend if you
+# have global configs that are role-based
+#
+# === Example
+#
+# node /^www\d+/ {
+# role mediawiki::appserver  # this will load the 
role::mediawiki::appserver class
+# include ::standard  #this class will use hiera lookups defined for the 
role.
+# }
+#
+# node monitoring.local {
+# role icinga, ganglia::collector #GOOD
+# }
+#
+# node monitoring2.local {
+# role icinga
+# role ganglia::collector #BAD, issues a warning
+# }
+Puppet::Functions.create_function(:role) do
+  dispatch :main do
+param 'String', :main_role
+  end
+  def main(main_role)
+scope = closure_scope
+# Check if the function has already been called
+if scope.include? '_role'
+  raise Puppet::ParseError, "role() can only be called once per node"
+end
+role = main_role.gsub(/^::/, '')
+# Backwards compat
+scope['_roles'] = { role => true }
+# This is where we're going in the future
+scope['_role'] = role
+rolename = 'role::' + role
+call_function('include', rolename)
+  end
+end
diff --git a/modules/wmflib/lib/puppet/parser/functions/role.rb 
b/modules/wmflib/lib/puppet/parser/functions/role.rb
deleted file mode 100644
index 1fae3c2..000
--- a/modules/wmflib/lib/puppet/parser/functions/role.rb
+++ /dev/null
@@ -1,85 +0,0 @@
-# == Function: role ( string $role_name [, string $... ] )
-#
-# Declare the _roles variable (or add keys to it), and if the role class
-# role::$role_name is in the scope, include it. This is roughly a
-# shortcut for
-#
-# $::_roles[foo] = true
-# include role::foo
-#
-# and has the notable advantage over that the syntax is shorter. Also,
-# this function  will refuse to run anywhere but at the node scope,
-# thus making any additional role added to a node explicit.
-#
-# If you have more than one role to declare, you MUST do that in one
-# single role stanza, or you would encounter unexpected behaviour. If
-# you do, an exception will be raised.
-#
-# This function is very useful with our "role" hiera backend if you
-# have global configs that are role-based
-#
-# === Example
-#
-# node /^www\d+/ {
-# role mediawiki::appserver  # this will load the 
role::mediawiki::appserver class
-# include ::standard  #this class will use hiera lookups defined for the 
role.
-# }
-#
-# node monitoring.local {
-# role icinga, ganglia::collector #GOOD
-# }
-#
-# node monitoring2.local {
-# role icinga
-# role ganglia::collector #BAD, issues a warning
-# }
-
-module Puppet::Parser::Functions
-  newfunction(:role, :arity => -2) do |args|
-# This will add to the catalog, and to the node specifically:
-# - A global 'role' hash with the 'role::#{arg}' key set to true;
-# if the variable is present, append to it
-# - Include class role::#{arg} if present
-
-# Prevent use outside of node definitions
-

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hiera: first step of simplification

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402347 )

Change subject: hiera: first step of simplification
..

hiera: first step of simplification

In an effort to simplify our hiera backends before migrating to hiera 5,
do as follows:
- Remove the role backend from the list of backends
- Add the role/{common,$site} paths to the hierarchy for nuyaml
- While we're at it, simplify the directory structure (which was
somewhat convoluted and caused confusion) for the role hiera directories.

Change-Id: If10176ac3dd0f32408a0ccd3b539a29843bdcf8f
---
R hieradata/role/codfw/cache::misc.yaml
R hieradata/role/codfw/cache::text.yaml
R hieradata/role/codfw/cache::upload.yaml
R hieradata/role/codfw/elasticsearch::cirrus.yaml
R hieradata/role/codfw/graphite::production.yaml
R hieradata/role/codfw/kubernetes::master.yaml
R hieradata/role/codfw/kubernetes::worker.yaml
R hieradata/role/codfw/lvs::balancer.yaml
R hieradata/role/codfw/maps::master.yaml
R hieradata/role/codfw/maps::slave.yaml
R hieradata/role/codfw/maps::test::master.yaml
R hieradata/role/codfw/maps::test::slave.yaml
R hieradata/role/codfw/maps::test::vectortiles_master.yaml
R hieradata/role/codfw/mediawiki::canary_appserver.yaml
R hieradata/role/codfw/ores::redis.yaml
R hieradata/role/codfw/parsoid::production.yaml
R hieradata/role/codfw/prometheus::ops.yaml
R hieradata/role/codfw/puppetmaster::backend.yaml
R hieradata/role/codfw/puppetmaster::frontend.yaml
R hieradata/role/codfw/restbase::production.yaml
R hieradata/role/codfw/restbase::test_cluster.yaml
R hieradata/role/codfw/swift::proxy.yaml
R hieradata/role/codfw/swift::storage.yaml
R hieradata/role/codfw/thumbor::mediawiki.yaml
R hieradata/role/codfw/wmcs::openstack::labtest::control.yaml
R hieradata/role/codfw/wmcs::openstack::labtest::net.yaml
R hieradata/role/codfw/wmcs::openstack::labtest::puppetmaster::frontend.yaml
R hieradata/role/codfw/wmcs::openstack::labtest::services.yaml
R hieradata/role/codfw/wmcs::openstack::labtest::virt.yaml
R hieradata/role/codfw/wmcs::openstack::labtest::web.yaml
R hieradata/role/common/analytics_cluster::coordinator.yaml
R hieradata/role/common/analytics_cluster::hadoop::master.yaml
R hieradata/role/common/analytics_cluster::hadoop::standby.yaml
R hieradata/role/common/analytics_cluster::hadoop::worker.yaml
R hieradata/role/common/analytics_cluster::hue.yaml
R hieradata/role/common/analytics_cluster::webserver.yaml
R hieradata/role/common/bastionhost::general.yaml
R hieradata/role/common/bastionhost::pop.yaml
R hieradata/role/common/cache::canary.yaml
R hieradata/role/common/cache::misc.yaml
R hieradata/role/common/cache::text.yaml
R hieradata/role/common/cache::upload.yaml
R hieradata/role/common/ci::master.yaml
R hieradata/role/common/cluster::management.yaml
R hieradata/role/common/docker::registry.yaml
R hieradata/role/common/druid::analytics::worker.yaml
R hieradata/role/common/druid::public::worker.yaml
D hieradata/role/common/dumps/generation/worker/dumper.yaml
C hieradata/role/common/dumps::generation::worker::dumper.yaml
R hieradata/role/common/dumps::generation::worker::dumper_misc.yaml
R hieradata/role/common/dumps::generation::worker::testbed.yaml
R hieradata/role/common/dumps::web::htmldumps.yaml
R hieradata/role/common/elasticsearch::cirrus.yaml
R hieradata/role/common/elasticsearch::relforge.yaml
R hieradata/role/common/etcd::kubernetes.yaml
R hieradata/role/common/etcd::networking.yaml
R hieradata/role/common/eventbus::eventbus.yaml
R hieradata/role/common/eventlogging::analytics.yaml
R hieradata/role/common/graphite::production.yaml
R hieradata/role/common/installserver::dhcp.yaml
R hieradata/role/common/jobqueue_redis::master.yaml
R hieradata/role/common/jobqueue_redis::slave.yaml
R hieradata/role/common/kafka::analytics::broker.yaml
R hieradata/role/common/kafka::analytics::burrow.yaml
R hieradata/role/common/kafka::jumbo::broker.yaml
R hieradata/role/common/kubernetes::master.yaml
R hieradata/role/common/kubernetes::staging::etcd.yaml
R hieradata/role/common/kubernetes::staging::master.yaml
R hieradata/role/common/kubernetes::staging::worker.yaml
R hieradata/role/common/kubernetes::worker.yaml
R hieradata/role/common/labs::db::replica.yaml
R hieradata/role/common/labs::nfs::backup.yaml
R hieradata/role/common/labs::nfs::misc.yaml
R hieradata/role/common/labs::nfs::primary.yaml
R hieradata/role/common/labs::nfs::secondary.yaml
R hieradata/role/common/labs::openstack::nova::api.yaml
R hieradata/role/common/labs::openstack::nova::compute.yaml
R hieradata/role/common/labs::openstack::nova::controller.yaml
R hieradata/role/common/labs::prometheus.yaml
R hieradata/role/common/labs::puppetmaster.yaml
R hieradata/role/common/logging::mediawiki::udp2log.yaml
R hieradata/role/common/logstash::apifeatureusage.yaml
R hieradata/role/common/logstash::collector.yaml
R hieradata/role/common/logstash::elasticsearch.yaml
R 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hiera: port nuyaml to hiera 3

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402346 )

Change subject: hiera: port nuyaml to hiera 3
..

hiera: port nuyaml to hiera 3

This allows, among other things, full support for the lookup() function
found in recent puppet versions

Change-Id: I9866b2ae002fdb340e7cf99f33d65af4b72027a2
---
M modules/wmflib/lib/hiera/backend/nuyaml_backend.rb
1 file changed, 22 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/46/402346/1

diff --git a/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb 
b/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb
index 9dd48d2..b1b2dc7 100644
--- a/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb
+++ b/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb
@@ -75,7 +75,7 @@
 @expand_path = config[:expand_path] || []
   end
 
-  def get_path(key, scope, source)
+  def get_path(key, scope, source, context)
 config_section = :nuyaml
 # Special case: regex
 if %r{^regex/}.match(source)
@@ -107,7 +107,7 @@
 # with fairly small yaml files as opposed to a very large one.
 # Example:
 # $apache::mpm::worker will be in common/apache/mpm.yaml
-paths = @expand_path.map{ |x| Backend.parse_string(x, scope) }
+paths = @expand_path.map{ |x| Backend.parse_string(x, scope, {}, 
context) }
 if paths.include? source
   namespaces = key.gsub(/^::/, '').split('::')
   namespaces.pop
@@ -120,19 +120,19 @@
 Backend.datafile(config_section, scope, source, "yaml")
   end
 
-  def plain_lookup(key, data, scope)
+  def plain_lookup(key, data, scope, context)
   return nil unless data.include?(key)
-  Backend.parse_answer(data[key], scope)
+  Backend.parse_answer(data[key], scope, {}, context)
   end
 
-  def regex_lookup(key, matchon, data, scope)
+  def regex_lookup(key, matchon, data, scope, context)
 data.each do |label, datahash|
   r = datahash["__regex"]
   Hiera.debug("Scanning label #{label} for matches to '#{r}' in 
'#{matchon}' ")
   next unless r.match(matchon)
   Hiera.debug("Label #{label} matches; searching within it")
   next unless datahash.include?(key)
-  return Backend.parse_answer(datahash[key], scope)
+  return Backend.parse_answer(datahash[key], scope, {}, context)
 end
 return nil
   rescue => detail
@@ -140,7 +140,7 @@
 return nil
   end
 
-  def lookup(key, scope, order_override, resolution_type)
+  def lookup(key, scope, order_override, resolution_type, context)
 answer = nil
 
 Hiera.debug("Looking up #{key}")
@@ -148,7 +148,7 @@
 Backend.datasources(scope, order_override) do |source|
   Hiera.debug("Loading info from #{source} for #{key}")
 
-  yamlfile = get_path(key, scope, source)
+  yamlfile = get_path(key, scope, source, context)
 
   Hiera.debug("Searching for #{key} in #{yamlfile}")
 
@@ -166,9 +166,9 @@
 
   if %r{regex/(.*)$} =~ source
 matchto = Regexp.last_match(1)
-new_answer = regex_lookup(key, matchto, data, scope)
+new_answer = regex_lookup(key, matchto, data, scope, context)
   else
-new_answer = plain_lookup(key, data, scope)
+new_answer = plain_lookup(key, data, scope, context)
   end
 
   next if new_answer.nil?
@@ -179,7 +179,7 @@
   Hiera.debug("Found #{key} in #{source}")
 
   # for array resolution we just append to the array whatever
-  # we find, we then goes onto the next file and keep adding to
+  # we find, we then go onto the next file and keep adding to
   # the array
   #
   # for priority searches we break after the first found data
@@ -190,16 +190,24 @@
 raise Exception, "Hiera type mismatch: expected Array and got 
#{new_answer.class}" unless new_answer.kind_of?(Array) || 
new_answer.kind_of?(String)
 answer ||= []
 answer << new_answer
-  when :hash
+  when :hash, Hash
 raise Exception, "Hiera type mismatch: expected Hash and got 
#{new_answer.class}" unless new_answer.kind_of? Hash
 answer ||= {}
-answer = Backend.merge_answer(new_answer, answer)
+if resolution_type == :hash
+  rt = nil
+else
+  rt = resolution_type
+end
+
+answer = Backend.merge_answer(new_answer, answer, rt)
   else
 answer = new_answer
 break
   end
 end
-
+if answer.nil?
+  throw(:no_such_key)
+end
 answer
   end
 end

-- 
To view, visit https://gerrit.wikimedia.org/r/402346
To unsubscribe, 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging: create compound role, consolidate hiera

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401554 )

Change subject: eventlogging: create compound role, consolidate hiera
..


eventlogging: create compound role, consolidate hiera

Change-Id: I6d6802fbb29daa714f3cadeab55a98350419dc65
---
D hieradata/hosts/eventlog1001.yaml
R hieradata/role/common/eventlogging/analytics.yaml
M manifests/site.pp
A modules/role/manifests/eventlogging/analytics.pp
4 files changed, 14 insertions(+), 12 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/eventlog1001.yaml 
b/hieradata/hosts/eventlog1001.yaml
deleted file mode 100644
index 86f9540..000
--- a/hieradata/hosts/eventlog1001.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-admin::groups:
-  - eventlogging-admins
-  - eventlogging-roots
diff --git a/hieradata/role/common/eventlogging.yaml 
b/hieradata/role/common/eventlogging/analytics.yaml
similarity index 100%
rename from hieradata/role/common/eventlogging.yaml
rename to hieradata/role/common/eventlogging/analytics.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index d4c0c31..acbcaf4 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -843,15 +843,7 @@
 # Receives log data from Kafka processes it, and broadcasts
 # to Kafka Schema based topics.
 node 'eventlog1001.eqiad.wmnet' {
-role(eventlogging::analytics::zeromq,
-eventlogging::analytics::processor,
-eventlogging::analytics::mysql,
-eventlogging::analytics::files,
-logging::mediawiki::errors)
-
-include ::standard
-include ::base::firewall
-interface::add_ip6_mapped { 'main': }
+role(eventlogging::analytics)
 }
 
 # virtual machine for mailman list server
diff --git a/modules/role/manifests/eventlogging/analytics.pp 
b/modules/role/manifests/eventlogging/analytics.pp
new file mode 100644
index 000..f794721
--- /dev/null
+++ b/modules/role/manifests/eventlogging/analytics.pp
@@ -0,0 +1,13 @@
+class role::eventlogging::analytics {
+system::role { 'eventlogging_host':
+description => 'eventlogging host'
+}
+include ::standard
+include ::base::firewall
+interface::add_ip6_mapped { 'main': }
+include ::role::eventlogging::analytics::zeromq
+include ::role::eventlogging::analytics::processor
+include ::role::eventlogging::analytics::mysql
+include ::role::eventlogging::analytics::files
+include ::role::logging::mediawiki::errors
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/401554
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6d6802fbb29daa714f3cadeab55a98350419dc65
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: monitoring: create role::alerting_host

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401553 )

Change subject: monitoring: create role::alerting_host
..


monitoring: create role::alerting_host

Change-Id: Ia461dc99b07784db78fd62acb96c16910a92b48d
---
M manifests/site.pp
A modules/role/manifests/alerting_host.pp
2 files changed, 17 insertions(+), 2 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  Alexandros Kosiaris: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index 82e71a5..d4c0c31 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -701,8 +701,7 @@
 
 # icinga based monitoring hosts (einsteinium = eqiad, tegmen = codfw)
 node 'einsteinium.wikimedia.org', 'tegmen.wikimedia.org' {
-role(icinga, tcpircbot, certspotter)
-interface::add_ip6_mapped { 'main': }
+role(alerting_host)
 }
 
 node /^elastic101[7-9]\.eqiad\.wmnet/ {
diff --git a/modules/role/manifests/alerting_host.pp 
b/modules/role/manifests/alerting_host.pp
new file mode 100644
index 000..42be71e
--- /dev/null
+++ b/modules/role/manifests/alerting_host.pp
@@ -0,0 +1,16 @@
+# = Class: role::alerting_host
+#
+# Sets up a full production alerting host, including
+# an icinga instance, tcpircbot, and certspotter
+#
+# = Parameters
+#
+class role::alerting_host {
+system::role{ 'alerting_host':
+description => 'central host for health checking and alerting'
+}
+include ::role::icinga
+include ::role::tcpircbot
+include ::role::certspotter
+interface::add_ip6_mapped { 'main': }
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/401553
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia461dc99b07784db78fd62acb96c16910a92b48d
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role for dbstore2001.codfw.wmnet

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401552 )

Change subject: site.pp: one role for dbstore2001.codfw.wmnet
..


site.pp: one role for dbstore2001.codfw.wmnet

Move the inclusion of the backup_mydumper role out of the role()
function call.

Change-Id: I9a2797ae8361a428ec5cfbd077533318f1ce6616
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index eba85dd..82e71a5 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -625,7 +625,8 @@
 }
 
 node 'dbstore2001.codfw.wmnet' {
-role(mariadb::dbstore_multiinstance, mariadb::backup_mydumper)
+role(mariadb::dbstore_multiinstance)
+include ::role::mariadb::backup_mydumper
 }
 
 node 'dbstore2002.codfw.wmnet' {

-- 
To view, visit https://gerrit.wikimedia.org/r/401552
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9a2797ae8361a428ec5cfbd077533318f1ce6616
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Jcrespo 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: simplify role() keyword call for cache::canary

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401551 )

Change subject: site.pp: simplify role() keyword call for cache::canary
..


site.pp: simplify role() keyword call for cache::canary

Change-Id: I84313de76b2e4d0fcef7c23c7457e8c34c746f17
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index 3843ecd..eba85dd 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -185,7 +185,8 @@
 
 # cp1008: prod-like SSL test host
 node 'cp1008.wikimedia.org' {
-role(cache::canary, authdns::testns)
+role(cache::canary)
+include ::role::authdns::testns
 interface::add_ip6_mapped { 'main': }
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/401551
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I84313de76b2e4d0fcef7c23c7457e8c34c746f17
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: add ipsec to basic roles

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401550 )

Change subject: cache: add ipsec to basic roles
..


cache: add ipsec to basic roles

Change-Id: I96430566dd486630d393457ef33460b0581078ed
---
M manifests/site.pp
M modules/role/manifests/cache/misc.pp
M modules/role/manifests/cache/text.pp
M modules/role/manifests/cache/upload.pp
4 files changed, 22 insertions(+), 11 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  Ema: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index e0edbe8..3843ecd 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -191,7 +191,7 @@
 
 node /^cp10(45|5[18]|61)\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc)
 }
 
 node 'cp1046.eqiad.wmnet', 'cp1047.eqiad.wmnet', 'cp1059.eqiad.wmnet', 
'cp1060.eqiad.wmnet' {
@@ -201,22 +201,22 @@
 
 node /^cp10(4[89]|50|6[234]|7[1-4]|99)\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 node /^cp10(5[2-5]|6[5-8])\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node /^cp20(0[147]|1[0369]|23)\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node /^cp20(0[258]|1[147]|2[0246])\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 node /^cp20(0[39]|15|21)\.codfw\.wmnet$/ {
@@ -226,7 +226,7 @@
 
 node /^cp20(06|1[28]|25)\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc)
 }
 
 node /^cp300[3-6]\.esams\.wmnet$/ {
@@ -236,7 +236,7 @@
 
 node /^cp30(0[789]|10)\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc)
 }
 
 node 'cp3022.esams.wmnet' {
@@ -245,12 +245,12 @@
 
 node /^cp30[34][0123]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node /^cp30[34][4-9]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 #
@@ -264,12 +264,12 @@
 
 node /^cp402[1-6]\.ulsfo\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 node /^cp40(2[789]|3[012])\.ulsfo\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node 'darmstadtium.eqiad.wmnet' {
diff --git a/modules/role/manifests/cache/misc.pp 
b/modules/role/manifests/cache/misc.pp
index fa8944a..8f4e20b 100644
--- a/modules/role/manifests/cache/misc.pp
+++ b/modules/role/manifests/cache/misc.pp
@@ -6,4 +6,5 @@
 include ::profile::cache::base
 include ::profile::cache::ssl::unified
 include ::profile::cache::misc
+include ::role::ipsec
 }
diff --git a/modules/role/manifests/cache/text.pp 
b/modules/role/manifests/cache/text.pp
index 1cb4d10..8d362c7 100644
--- a/modules/role/manifests/cache/text.pp
+++ b/modules/role/manifests/cache/text.pp
@@ -13,4 +13,9 @@
 # and sends them to the 'statsv' topic in Kafka. A kafka consumer
 # (called 'statsv') then consumes these and emits metrics.
 include ::profile::cache::kafka::statsv
+
+# TODO: refactor all this so that we have separate roles for production 
and labs
+if $::realm == 'production' {
+include ::role::ipsec
+}
 }
diff --git a/modules/role/manifests/cache/upload.pp 
b/modules/role/manifests/cache/upload.pp
index 2e85df3..25602f3 100644
--- a/modules/role/manifests/cache/upload.pp
+++ b/modules/role/manifests/cache/upload.pp
@@ -9,4 +9,9 @@
 include ::profile::cache::base
 include ::profile::cache::ssl::unified
 include ::profile::cache::upload
+
+# TODO: refactor all this so that we have separate roles for production 
and labs
+if $::realm == 'production' {
+include ::role::ipsec
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/401550
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I96430566dd486630d393457ef33460b0581078ed
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: use role keyword for striker::web only on californium

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401549 )

Change subject: site.pp: use role keyword for striker::web only on californium
..


site.pp: use role keyword for striker::web only on californium

No hiera lookup is performed on the other role, so we can safely move it
out of the role() call.

Change-Id: I89f4090ba48e4bfbfc982be9662ec36c389e460c
---
M manifests/site.pp
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index 7cd34b6..e0edbe8 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -125,8 +125,8 @@
 # and Toolforge admin console AKA Striker
 #  It's proxied by the misc-web varnishes
 node 'californium.wikimedia.org' {
-role(wmcs::openstack::main::horizon,
-  striker::web)
+role(striker::web)
+include ::role::wmcs::openstack::main::horizon
 include ::standard
 include ::base::firewall
 include ::ldap::role::client::labs

-- 
To view, visit https://gerrit.wikimedia.org/r/401549
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I89f4090ba48e4bfbfc982be9662ec36c389e460c
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: bastionhost: add role for caching PoPs

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401548 )

Change subject: bastionhost: add role for caching PoPs
..


bastionhost: add role for caching PoPs

This is a role for bastionhosts in our caching PoPs, where they also
host the installserver tftp, the prometheus collector and the ipmi
management tools.

This patch adds role::ipmi::mgmt to the esams bastion for uniformity.

Change-Id: I8504878d64cb7612abcdf01b6f92539f3693ae3f
---
A hieradata/role/common/bastionhost/pop.yaml
M manifests/site.pp
A modules/role/manifests/bastionhost/pop.pp
3 files changed, 23 insertions(+), 11 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/bastionhost/pop.yaml 
b/hieradata/role/common/bastionhost/pop.yaml
new file mode 100644
index 000..3f8f20e
--- /dev/null
+++ b/hieradata/role/common/bastionhost/pop.yaml
@@ -0,0 +1,10 @@
+cluster: misc
+profile::base::domain_search:
+  - wikimedia.org
+  - eqiad.wmnet
+  - codfw.wmnet
+  - esams.wmnet
+  - ulsfo.wmnet
+  - eqsin.wmnet
+admin::groups:
+  - all-users
diff --git a/manifests/site.pp b/manifests/site.pp
index 1215fdb..7cd34b6 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -94,28 +94,20 @@
 
 # Bastion in the Netherlands (replaced bast3001)
 node 'bast3002.wikimedia.org' {
-role(bastionhost::general,
-installserver::tftp,
-prometheus::ops)
+role(bastionhost::pop)
 
 interface::add_ip6_mapped { 'main': }
 }
 
 # Bastion in California
 node 'bast4001.wikimedia.org' {
-role(bastionhost::general,
-ipmi::mgmt,
-installserver::tftp,
-prometheus::ops)
+role(bastionhost::pop)
 
 interface::add_ip6_mapped { 'main': }
 }
 
 node 'bast4002.wikimedia.org' {
-role(bastionhost::general,
-ipmi::mgmt,
-installserver::tftp,
-prometheus::ops)
+role(bastionhost::pop)
 
 interface::add_ip6_mapped { 'main': }
 }
diff --git a/modules/role/manifests/bastionhost/pop.pp 
b/modules/role/manifests/bastionhost/pop.pp
new file mode 100644
index 000..ade59e5
--- /dev/null
+++ b/modules/role/manifests/bastionhost/pop.pp
@@ -0,0 +1,10 @@
+# bastion host for all users in a caching PoP
+class role::bastionhost::pop {
+system::role { 'bastionhost::pop':
+description => 'Bastion host for all shell users in a caching Pop',
+}
+require ::role::bastionhost::general
+require ::role::installserver::tftp
+require ::role::prometheus::ops
+require ::role::ipmi::mgmt
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/401548
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8504878d64cb7612abcdf01b6f92539f3693ae3f
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: convert dns recursors to single role

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401547 )

Change subject: site.pp: convert dns recursors to single role
..


site.pp: convert dns recursors to single role

Change-Id: Id448dd97ad201f0a4bf978909c30aa20c80bfd97
---
R hieradata/role/codfw/recursor.yaml
R hieradata/role/common/recursor.yaml
R hieradata/role/eqiad/recursor.yaml
R hieradata/role/esams/recursor.yaml
M manifests/site.pp
A modules/role/manifests/recursor.pp
6 files changed, 14 insertions(+), 6 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/codfw/dnsrecursor.yaml 
b/hieradata/role/codfw/recursor.yaml
similarity index 100%
rename from hieradata/role/codfw/dnsrecursor.yaml
rename to hieradata/role/codfw/recursor.yaml
diff --git a/hieradata/role/common/dnsrecursor.yaml 
b/hieradata/role/common/recursor.yaml
similarity index 100%
rename from hieradata/role/common/dnsrecursor.yaml
rename to hieradata/role/common/recursor.yaml
diff --git a/hieradata/role/eqiad/dnsrecursor.yaml 
b/hieradata/role/eqiad/recursor.yaml
similarity index 100%
rename from hieradata/role/eqiad/dnsrecursor.yaml
rename to hieradata/role/eqiad/recursor.yaml
diff --git a/hieradata/role/esams/dnsrecursor.yaml 
b/hieradata/role/esams/recursor.yaml
similarity index 100%
rename from hieradata/role/esams/dnsrecursor.yaml
rename to hieradata/role/esams/recursor.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index ab0787b..8949779 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -10,7 +10,7 @@
 # Node definitions (alphabetic order)
 
 node 'acamar.wikimedia.org' {
-role(dnsrecursor, ntp)
+role(recursor)
 
 # use achernar (directly) + eqiad LVS (avoid self-dep)
 $nameservers_override = [ '208.80.153.42', '208.80.154.254' ]
@@ -19,7 +19,7 @@
 }
 
 node 'achernar.wikimedia.org' {
-role(dnsrecursor, ntp)
+role(recursor)
 
 # use acamar (directly) + eqiad LVS (avoid self-dep)
 $nameservers_override = [ '208.80.153.12', '208.80.154.254' ]
@@ -149,7 +149,7 @@
 
 # DNS recursor
 node 'chromium.wikimedia.org' {
-role(dnsrecursor, ntp)
+role(recursor)
 
 # use hydrogen (directly) + codfw LVS (avoid self-dep)
 $nameservers_override = [ '208.80.154.50', '208.80.153.254' ]
@@ -890,7 +890,7 @@
 
 # DNS recursor
 node 'hydrogen.wikimedia.org' {
-role(dnsrecursor, ntp)
+role(recursor)
 
 # use chromium (directly) + codfw LVS (avoid self-dep)
 $nameservers_override = [ '208.80.154.157', '208.80.153.254' ]
@@ -1391,7 +1391,7 @@
 }
 
 node 'maerlant.wikimedia.org' {
-role(dnsrecursor, ntp)
+role(recursor)
 
 interface::add_ip6_mapped { 'main': }
 }
@@ -1747,7 +1747,7 @@
 }
 
 node 'nescio.wikimedia.org' {
-role(dnsrecursor, ntp)
+role(recursor)
 
 interface::add_ip6_mapped { 'main': }
 }
diff --git a/modules/role/manifests/recursor.pp 
b/modules/role/manifests/recursor.pp
new file mode 100644
index 000..9062f51
--- /dev/null
+++ b/modules/role/manifests/recursor.pp
@@ -0,0 +1,8 @@
+# == class role::recursor
+#
+# Class for configuring production dns recursors that use
+# both class role::dnsrecursor and role::ntp
+class role::recursor {
+require role::dnsrecursor
+require role::ntp
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/401547
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id448dd97ad201f0a4bf978909c30aa20c80bfd97
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: api: move mw1189-1200 to role::spare::system

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401482 )

Change subject: api: move mw1189-1200 to role::spare::system
..


api: move mw1189-1200 to role::spare::system

Also remove them from conftool

Bug: T183895
Change-Id: Ia25effaafda0d0f2656a94efc0584b03154fbde6
---
M conftool-data/node/eqiad.yaml
M manifests/site.pp
2 files changed, 4 insertions(+), 15 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index 8f54087..48d4950 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -25,17 +25,6 @@
 mw1336.eqiad.wmnet: [apache2,nginx]
 mw1337.eqiad.wmnet: [apache2,nginx]
   api_appserver:
-mw1189.eqiad.wmnet: [apache2,nginx]
-mw1190.eqiad.wmnet: [apache2,nginx]
-mw1191.eqiad.wmnet: [apache2,nginx]
-mw1192.eqiad.wmnet: [apache2,nginx]
-mw1193.eqiad.wmnet: [apache2,nginx]
-mw1194.eqiad.wmnet: [apache2,nginx]
-mw1195.eqiad.wmnet: [apache2,nginx]
-mw1197.eqiad.wmnet: [apache2,nginx]
-mw1198.eqiad.wmnet: [apache2,nginx]
-mw1199.eqiad.wmnet: [apache2,nginx]
-mw1200.eqiad.wmnet: [apache2,nginx]
 mw1201.eqiad.wmnet: [apache2,nginx]
 mw1202.eqiad.wmnet: [apache2,nginx]
 mw1203.eqiad.wmnet: [apache2,nginx]
diff --git a/manifests/site.pp b/manifests/site.pp
index ddfb6df..6604232 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1510,13 +1510,13 @@
 role(mediawiki::canary_appserver)
 }
 
-# mw1180-1188 are apaches
-node /^mw118[0-8]\.eqiad\.wmnet$/ {
+# mw1180-1200 are spares
+node /^mw1(1[8-9][0-9]|200)\.eqiad\.wmnet$/ {
 role(spare::system)
 }
 
-# mw1189-1208 are api apaches
-node /^mw1(189|19[0-9]|20[0-8])\.eqiad\.wmnet$/ {
+# mw1201-1208 are api apaches
+node /^mw120[1-8]\.eqiad\.wmnet$/ {
 role(mediawiki::appserver::api)
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/401482
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia25effaafda0d0f2656a94efc0584b03154fbde6
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: hieradata: remove old leftovers

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401480 )

Change subject: hieradata: remove old leftovers
..


hieradata: remove old leftovers

Change-Id: I6d4bbcc0c94ba7c1436622b5bc5255c58f130b5e
---
D hieradata/hosts/mw1168.yaml
D hieradata/hosts/mw1169.yaml
2 files changed, 0 insertions(+), 19 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/mw1168.yaml b/hieradata/hosts/mw1168.yaml
deleted file mode 100644
index 5875d84..000
--- a/hieradata/hosts/mw1168.yaml
+++ /dev/null
@@ -1,10 +0,0 @@
-profile::mediawiki::jobrunner::runners:
-  transcode: 8
-  transcode_prioritized: 6
-
-hhvm::extra::fcgi:
-  hhvm:
-server:
-  connection_timeout_seconds: 86400
-  thread_count: 20
-  max_execution_time: 86400
diff --git a/hieradata/hosts/mw1169.yaml b/hieradata/hosts/mw1169.yaml
deleted file mode 100644
index ccf184a..000
--- a/hieradata/hosts/mw1169.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-profile::mediawiki::jobrunner::runners:
-  transcode: 8
-  transcode_prioritized: 6
-hhvm::extra::fcgi:
-  hhvm:
-server:
-  connection_timeout_seconds: 86400
-  thread_count: 20
-  max_execution_time: 86400

-- 
To view, visit https://gerrit.wikimedia.org/r/401480
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6d4bbcc0c94ba7c1436622b5bc5255c58f130b5e
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: appservers: move mw1180-1188 to role::spare::system

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401479 )

Change subject: appservers: move mw1180-1188 to role::spare::system
..


appservers: move mw1180-1188 to role::spare::system

Also, remove them from conftool-data

Bug: T183895
Change-Id: Ibb45f449767088ff429723ba103579133dd42ccc
---
M conftool-data/node/eqiad.yaml
M manifests/site.pp
2 files changed, 1 insertion(+), 10 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  Muehlenhoff: Looks good to me, but someone else must approve
  Elukey: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/conftool-data/node/eqiad.yaml b/conftool-data/node/eqiad.yaml
index 9357ded..8f54087 100644
--- a/conftool-data/node/eqiad.yaml
+++ b/conftool-data/node/eqiad.yaml
@@ -81,15 +81,6 @@
 mw1316.eqiad.wmnet: [apache2,nginx]
 mw1317.eqiad.wmnet: [apache2,nginx]
   appserver:
-mw1180.eqiad.wmnet: [apache2,nginx]
-mw1181.eqiad.wmnet: [apache2,nginx]
-mw1182.eqiad.wmnet: [apache2,nginx]
-mw1183.eqiad.wmnet: [apache2,nginx]
-mw1184.eqiad.wmnet: [apache2,nginx]
-mw1185.eqiad.wmnet: [apache2,nginx]
-mw1186.eqiad.wmnet: [apache2,nginx]
-mw1187.eqiad.wmnet: [apache2,nginx]
-mw1188.eqiad.wmnet: [apache2,nginx]
 mw1209.eqiad.wmnet: [apache2,nginx]
 mw1210.eqiad.wmnet: [apache2,nginx]
 mw1211.eqiad.wmnet: [apache2,nginx]
diff --git a/manifests/site.pp b/manifests/site.pp
index 89fa933..ddfb6df 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1512,7 +1512,7 @@
 
 # mw1180-1188 are apaches
 node /^mw118[0-8]\.eqiad\.wmnet$/ {
-role(mediawiki::appserver)
+role(spare::system)
 }
 
 # mw1189-1208 are api apaches

-- 
To view, visit https://gerrit.wikimedia.org/r/401479
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibb45f449767088ff429723ba103579133dd42ccc
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mediawiki::appserver::api: add load monitoring

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401714 )

Change subject: mediawiki::appserver::api: add load monitoring
..


mediawiki::appserver::api: add load monitoring

We've had quite a few cases of HHVM API appservers with high cpu usage
causing noitceable latencies to users; the easiest way to detect such
deadlocks is quite simply checking the machine CPU usage/load - at least
that's what I do manually.

This change won't solve the issue per-se, but it will make ops aware of
what is going on proactively.

Bug: T182568
Bug: T184048
Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
---
A modules/profile/manifests/mediawiki/api.pp
M modules/role/manifests/mediawiki/appserver/api.pp
2 files changed, 23 insertions(+), 8 deletions(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/profile/manifests/mediawiki/api.pp 
b/modules/profile/manifests/mediawiki/api.pp
new file mode 100644
index 000..0f0e9d6
--- /dev/null
+++ b/modules/profile/manifests/mediawiki/api.pp
@@ -0,0 +1,22 @@
+# == Class profile::mediawiki::api
+#
+# Specific settings for the mediawiki API servers
+class profile::mediawiki::api {
+# Using fastcgi we need more local ports
+sysctl::parameters { 'raise_port_range':
+values   => { 'net.ipv4.local_port_range' => '22500 65535', },
+priority => 90,
+}
+
+# Check the load to detect clearly hosts hanging (see T184048, T182568)
+$nproc = $facts['processorcount']
+$warning = join([ $nproc * 0.95, $nproc * 0.8, $nproc * 0.75], ',')
+$critical = join([ $nproc * 1.5, $nproc * 1.1, $nproc * 1], ',')
+# Since we're checking the load, that is already a moving average, we can
+# alarm at the first occurrence
+nrpe::monitor_service { 'cpu_load':
+description  => 'High CPU load on API appserver',
+nrpe_command => "/usr/lib/nagios/plugins/check_load -w ${warning} -c 
${critical}",
+retries  => 1,
+}
+}
diff --git a/modules/role/manifests/mediawiki/appserver/api.pp 
b/modules/role/manifests/mediawiki/appserver/api.pp
index 83494bb..c36d4b7 100644
--- a/modules/role/manifests/mediawiki/appserver/api.pp
+++ b/modules/role/manifests/mediawiki/appserver/api.pp
@@ -5,12 +5,5 @@
 include ::profile::base::firewall
 include ::profile::prometheus::apache_exporter
 include ::profile::prometheus::hhvm_exporter
-
-# Using fastcgi we need more local ports
-sysctl::parameters { 'raise_port_range':
-values   => {
-'net.ipv4.local_port_range' => '22500 65535',
-},
-priority => 90,
-}
+include ::profile::mediawiki::api
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/401714
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mediawiki::appserver::api: add load monitoring

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401714 )

Change subject: mediawiki::appserver::api: add load monitoring
..

mediawiki::appserver::api: add load monitoring

We've had quite a few cases of HHVM API appservers with high cpu usage
causing noitceable latencies to users; the easiest way to detect such
deadlocks is quite simply checking the machine CPU usage/load - at least
that's what I do manually.

This change won't solve the issue per-se, but it will make ops aware of
what is going on proactively.

Bug: T182568, T184048
Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
---
M modules/role/manifests/mediawiki/appserver/api.pp
1 file changed, 11 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/14/401714/1

diff --git a/modules/role/manifests/mediawiki/appserver/api.pp 
b/modules/role/manifests/mediawiki/appserver/api.pp
index 83494bb..39d2ffb 100644
--- a/modules/role/manifests/mediawiki/appserver/api.pp
+++ b/modules/role/manifests/mediawiki/appserver/api.pp
@@ -13,4 +13,15 @@
 },
 priority => 90,
 }
+
+# Check the load to detect clearly hosts hanging (see T184048, T182568)
+$nproc = $facts['processorcount']
+$warning = join([ $nproc * 0.95, $nproc * 0.8, $nproc * 0.75], ',')
+$critical = join([ $nproc * 1.5, $nproc * 1.1, $nproc * 1], ',')
+# Since we're checking the load, that is already a moving average, we can
+# alarm at the first occurrence
+nrpe::monitor_service { 'cpu_load':
+command => "check_load -w ${warning} -c ${critical}",
+retries => 1,
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/401714
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eventlogging: create compound role, consolidate hiera

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401554 )

Change subject: eventlogging: create compound role, consolidate hiera
..

eventlogging: create compound role, consolidate hiera

Change-Id: I6d6802fbb29daa714f3cadeab55a98350419dc65
---
D hieradata/hosts/eventlog1001.yaml
R hieradata/role/common/eventlogging/analytics/host.yaml
M manifests/site.pp
A modules/role/manifests/eventlogging/analytics/host.pp
4 files changed, 14 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/54/401554/1

diff --git a/hieradata/hosts/eventlog1001.yaml 
b/hieradata/hosts/eventlog1001.yaml
deleted file mode 100644
index 86f9540..000
--- a/hieradata/hosts/eventlog1001.yaml
+++ /dev/null
@@ -1,3 +0,0 @@
-admin::groups:
-  - eventlogging-admins
-  - eventlogging-roots
diff --git a/hieradata/role/common/eventlogging.yaml 
b/hieradata/role/common/eventlogging/analytics/host.yaml
similarity index 100%
rename from hieradata/role/common/eventlogging.yaml
rename to hieradata/role/common/eventlogging/analytics/host.yaml
diff --git a/manifests/site.pp b/manifests/site.pp
index 39e0ef6..30f6a83 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -824,15 +824,7 @@
 # Receives log data from Kafka processes it, and broadcasts
 # to Kafka Schema based topics.
 node 'eventlog1001.eqiad.wmnet' {
-role(eventlogging::analytics::zeromq,
-eventlogging::analytics::processor,
-eventlogging::analytics::mysql,
-eventlogging::analytics::files,
-logging::mediawiki::errors)
-
-include ::standard
-include ::base::firewall
-interface::add_ip6_mapped { 'main': }
+role(eventlogging::analytics::host)
 }
 
 # virtual machine for mailman list server
diff --git a/modules/role/manifests/eventlogging/analytics/host.pp 
b/modules/role/manifests/eventlogging/analytics/host.pp
new file mode 100644
index 000..35d9549
--- /dev/null
+++ b/modules/role/manifests/eventlogging/analytics/host.pp
@@ -0,0 +1,13 @@
+class role::eventlogging::analytics::host {
+system::role { 'eventlogging_host':
+description => 'eventlogging host'
+}
+include ::standard
+include ::base::firewall
+interface::add_ip6_mapped { 'main': }
+include ::role::eventlogging::analytics::zeromq
+include ::role::eventlogging::analytics::processor
+include ::role::eventlogging::analytics::mysql
+include ::role::eventlogging::analytics::files
+include ::role::logging::mediawiki::errors
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/401554
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6d6802fbb29daa714f3cadeab55a98350419dc65
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: simplify role() keyword call for cache::canary

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401551 )

Change subject: site.pp: simplify role() keyword call for cache::canary
..

site.pp: simplify role() keyword call for cache::canary

Change-Id: I84313de76b2e4d0fcef7c23c7457e8c34c746f17
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/51/401551/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 826e734..68665a4 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -185,7 +185,8 @@
 
 # cp1008: prod-like SSL test host
 node 'cp1008.wikimedia.org' {
-role(cache::canary, authdns::testns)
+role(cache::canary)
+include ::role::authdns::testns
 interface::add_ip6_mapped { 'main': }
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/401551
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I84313de76b2e4d0fcef7c23c7457e8c34c746f17
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: one role for dbstore2001.codfw.wmnet

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401552 )

Change subject: site.pp: one role for dbstore2001.codfw.wmnet
..

site.pp: one role for dbstore2001.codfw.wmnet

Move the inclusion of the backup_mydumper role out of the role()
function call.

Change-Id: I9a2797ae8361a428ec5cfbd077533318f1ce6616
---
M manifests/site.pp
1 file changed, 2 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/52/401552/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 68665a4..579eee4 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -606,7 +606,8 @@
 }
 
 node 'dbstore2001.codfw.wmnet' {
-role(mariadb::dbstore_multiinstance, mariadb::backup_mydumper)
+role(mariadb::dbstore_multiinstance)
+include ::role::mariadb::backup_mydumper
 }
 
 node 'dbstore2002.codfw.wmnet' {

-- 
To view, visit https://gerrit.wikimedia.org/r/401552
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9a2797ae8361a428ec5cfbd077533318f1ce6616
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: monitoring: create role::alerting_host

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401553 )

Change subject: monitoring: create role::alerting_host
..

monitoring: create role::alerting_host

Change-Id: Ia461dc99b07784db78fd62acb96c16910a92b48d
---
M manifests/site.pp
A modules/role/manifests/alerting_host.pp
2 files changed, 17 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/53/401553/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 579eee4..39e0ef6 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -682,8 +682,7 @@
 
 # icinga based monitoring hosts (einsteinium = eqiad, tegmen = codfw)
 node 'einsteinium.wikimedia.org', 'tegmen.wikimedia.org' {
-role(icinga, tcpircbot, certspotter)
-interface::add_ip6_mapped { 'main': }
+role(alerting_host)
 }
 
 node /^elastic101[7-9]\.eqiad\.wmnet/ {
diff --git a/modules/role/manifests/alerting_host.pp 
b/modules/role/manifests/alerting_host.pp
new file mode 100644
index 000..42be71e
--- /dev/null
+++ b/modules/role/manifests/alerting_host.pp
@@ -0,0 +1,16 @@
+# = Class: role::alerting_host
+#
+# Sets up a full production alerting host, including
+# an icinga instance, tcpircbot, and certspotter
+#
+# = Parameters
+#
+class role::alerting_host {
+system::role{ 'alerting_host':
+description => 'central host for health checking and alerting'
+}
+include ::role::icinga
+include ::role::tcpircbot
+include ::role::certspotter
+interface::add_ip6_mapped { 'main': }
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/401553
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia461dc99b07784db78fd62acb96c16910a92b48d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: add ipsec to basic roles

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401550 )

Change subject: cache: add ipsec to basic roles
..

cache: add ipsec to basic roles

Change-Id: I96430566dd486630d393457ef33460b0581078ed
---
M manifests/site.pp
M modules/role/manifests/cache/misc.pp
M modules/role/manifests/cache/text.pp
M modules/role/manifests/cache/upload.pp
4 files changed, 14 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/50/401550/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 1c8a0db..826e734 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -191,7 +191,7 @@
 
 node /^cp10(45|5[18]|61)\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc)
 }
 
 node 'cp1046.eqiad.wmnet', 'cp1047.eqiad.wmnet', 'cp1059.eqiad.wmnet', 
'cp1060.eqiad.wmnet' {
@@ -201,22 +201,22 @@
 
 node /^cp10(4[89]|50|6[234]|7[1-4]|99)\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 node /^cp10(5[2-5]|6[5-8])\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node /^cp20(0[147]|1[0369]|23)\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node /^cp20(0[258]|1[147]|2[0246])\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 node /^cp20(0[39]|15|21)\.codfw\.wmnet$/ {
@@ -226,7 +226,7 @@
 
 node /^cp20(06|1[28]|25)\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc)
 }
 
 node /^cp300[3-6]\.esams\.wmnet$/ {
@@ -236,7 +236,7 @@
 
 node /^cp30(0[789]|10)\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc)
 }
 
 node 'cp3022.esams.wmnet' {
@@ -245,12 +245,12 @@
 
 node /^cp30[34][0123]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node /^cp30[34][4-9]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 #
@@ -264,12 +264,12 @@
 
 node /^cp402[1-6]\.ulsfo\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload)
 }
 
 node /^cp40(2[789]|3[012])\.ulsfo\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text)
 }
 
 node 'darmstadtium.eqiad.wmnet' {
diff --git a/modules/role/manifests/cache/misc.pp 
b/modules/role/manifests/cache/misc.pp
index fa8944a..8f4e20b 100644
--- a/modules/role/manifests/cache/misc.pp
+++ b/modules/role/manifests/cache/misc.pp
@@ -6,4 +6,5 @@
 include ::profile::cache::base
 include ::profile::cache::ssl::unified
 include ::profile::cache::misc
+include ::role::ipsec
 }
diff --git a/modules/role/manifests/cache/text.pp 
b/modules/role/manifests/cache/text.pp
index 16a2d40..2463e16 100644
--- a/modules/role/manifests/cache/text.pp
+++ b/modules/role/manifests/cache/text.pp
@@ -8,4 +8,5 @@
 include ::profile::cache::base
 include ::profile::cache::ssl::unified
 include ::profile::cache::text
+include ::role::ipsec
 }
diff --git a/modules/role/manifests/cache/upload.pp 
b/modules/role/manifests/cache/upload.pp
index 2e85df3..f8f68a9 100644
--- a/modules/role/manifests/cache/upload.pp
+++ b/modules/role/manifests/cache/upload.pp
@@ -9,4 +9,5 @@
 include ::profile::cache::base
 include ::profile::cache::ssl::unified
 include ::profile::cache::upload
+include ::role::ipsec
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/401550
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I96430566dd486630d393457ef33460b0581078ed
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site.pp: use role keyword for striker::web only on californium

[Giuseppe Lavagetto (Code Review)]

Giuseppe Lavagetto has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401549 )

Change subject: site.pp: use role keyword for striker::web only on californium
..

site.pp: use role keyword for striker::web only on californium

No hiera lookup is performed on the other role, so we can safely move it
out of the role() call.

Change-Id: I89f4090ba48e4bfbfc982be9662ec36c389e460c
---
M manifests/site.pp
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/49/401549/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 0caeec2..1c8a0db 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -125,8 +125,8 @@
 # and Toolforge admin console AKA Striker
 #  It's proxied by the misc-web varnishes
 node 'californium.wikimedia.org' {
-role(wmcs::openstack::main::horizon,
-  striker::web)
+role(striker::web)
+include ::wmcs::openstack::main::horizon
 include ::standard
 include ::base::firewall
 include ::ldap::role::client::labs

-- 
To view, visit https://gerrit.wikimedia.org/r/401549
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I89f4090ba48e4bfbfc982be9662ec36c389e460c
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits