[MediaWiki-commits] [Gerrit] operations...pybal[master]: Fix Server testInitialize test case

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405308 )

Change subject: Fix Server testInitialize test case
..

Fix Server testInitialize test case

pybal.server.Server.initialize was being overridden with a MagicMock
in test_coordinator.py, which interfered with unit tests in
test_server.py

server.py now has 100% unit testing coverage.

Change-Id: I2e7d185557e2bc887817d220edcfc64645fe4b95
---
M pybal/test/test_coordinator.py
M pybal/test/test_server.py
2 files changed, 10 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/08/405308/1

diff --git a/pybal/test/test_coordinator.py b/pybal/test/test_coordinator.py
index b6ab1a9..c0dab8d 100644
--- a/pybal/test/test_coordinator.py
+++ b/pybal/test/test_coordinator.py
@@ -31,8 +31,6 @@
 self.coordinator.lvsservice.getDepoolThreshold = mock.MagicMock(
 return_value=0.5)
 
-pybal.server.Server.initialize = mock.MagicMock()
-
 def tearDown(self):
 self.coordinator.configObserver.reloadTask.stop()
 
@@ -47,7 +45,8 @@
 server.up = True
 server.enabled = True
 
-def test2serversCanDepool(self):
+@mock.patch('pybal.server.Server.initialize')
+def test2serversCanDepool(self, mock_initialize):
 servers = {
 'cp1045.eqiad.wmnet': {},
 'cp1046.eqiad.wmnet': {},
@@ -71,7 +70,8 @@
 # cannot depool.
 self.assertFalse(self.coordinator.canDepool())
 
-def test4serversCanDepool(self):
+@mock.patch('pybal.server.Server.initialize')
+def test4serversCanDepool(self, mock_initialize):
 servers = {
 'cp1045.eqiad.wmnet': {},
 'cp1046.eqiad.wmnet': {},
diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index 017b3fa..de33e12 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -102,14 +102,13 @@
 self.assertEqual(len(self.server.monitors), 0)
 
 def testInitialize(self):
-def callback(result):
-self.assertTrue(isinstance(result, bool))
-self.assertEquals(self.server.ready, result)
+def cb(result):
+ self.assertIsInstance(result, bool)
+ self.assertEquals(self.server.ready, result)
 
-self.server.createMonitoringInstances = mock.MagicMock()
-deferred = self.server.initialize(self.mockCoordinator)
-deferred.addCallback(callback)
-return deferred
+d = self.server.initialize(self.mockCoordinator)
+d.addCallback(cb)
+return d
 
 @mock.patch('pybal.server.Server.createMonitoringInstances')
 def testReady(self, mock_createMonitoringInstances):

-- 
To view, visit https://gerrit.wikimedia.org/r/405308
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2e7d185557e2bc887817d220edcfc64645fe4b95
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Stop pybal on all failures to properly parse 'monitors'

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405298 )

Change subject: Stop pybal on all failures to properly parse 'monitors'
..

Stop pybal on all failures to properly parse 'monitors'

When a monitor module failed to import, pybal was made to exit in order
to prevent it from silently doing nothing. However, a failure parsing
the 'monitors' configuration option would only log an error and still
not have the desired effect. Stop pybal in more cases when it fails to
interpret 'monitors'.

Also complete unit testing coverage of createMonitoringInstances

Change-Id: I804af7081f645f4d2e90499a2394c40712a52241
---
M pybal/server.py
M pybal/test/test_server.py
2 files changed, 46 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/98/405298/1

diff --git a/pybal/server.py b/pybal/server.py
index 38a9224..3622322 100644
--- a/pybal/server.py
+++ b/pybal/server.py
@@ -181,37 +181,41 @@
 lvsservice = self.lvsservice
 
 try:
-monitorlist = eval(lvsservice.configuration['monitors'])
-except KeyError:
-log.critical(
-"LVS service {} does not have a 'monitors' configuration 
option set.".format(
-lvsservice.name)
-)
-raise
+try:
+monitorlist = eval(lvsservice.configuration['monitors'])
+except KeyError:
+log.critical(
+"LVS service {} does not have a 'monitors' configuration 
option set.".format(
+lvsservice.name)
+)
+raise
 
-if type(monitorlist) != list:
-msg = "option 'monitors' in LVS service section {} is not a python 
list"
-log.err(msg.format(lvsservice.name))
-else:
-for monitorname in monitorlist:
-try:
-monitormodule = importlib.import_module(
-"pybal.monitors.{}".format(monitorname.lower()))
-except ImportError:
-log.err("Monitor {} does not exist".format(monitorname))
-except Exception:
-log.critical("Cannot import 
pybal.monitors.{}".format(monitorname))
-# An exception was raised importing the given monitor
-# module. Instead of just logging the problem, stop PyBal
-# as the admin might think everything is fine and all
-# checks are green, while in fact no check is being
-# performed.
-reactor.stop()
-else:
-monitorclass = getattr(monitormodule, monitorname + 
'MonitoringProtocol')
-monitor = monitorclass(coordinator, self, 
lvsservice.configuration)
-self.addMonitor(monitor)
-monitor.run()
+if not isinstance(monitorlist, list):
+msg = "option 'monitors' in LVS service section {} is not a 
python list"
+log.critical(msg.format(lvsservice.name))
+raise Exception(msg.format(lvsservice.name))
+else:
+for monitorname in monitorlist:
+try:
+monitormodule = importlib.import_module(
+"pybal.monitors.{}".format(monitorname.lower()))
+except ImportError:
+log.err("Monitor {} does not 
exist".format(monitorname))
+except Exception:
+log.critical("Cannot import 
pybal.monitors.{}".format(monitorname))
+raise
+else:
+monitorclass = getattr(monitormodule, monitorname + 
'MonitoringProtocol')
+monitor = monitorclass(coordinator, self, 
lvsservice.configuration)
+self.addMonitor(monitor)
+monitor.run()
+except Exception:
+# An exception was raised importing a monitor
+# module. Instead of just logging the problem, stop PyBal
+# as the admin might think everything is fine and all
+# checks are green, while in fact no check is being
+# performed.
+reactor.stop()
 
 def calcStatus(self):
 """AND quantification of monitor.up over all monitoring instances of a 
single Server"""
diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index 1df0121..017b3fa 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -124,10 +124,6 @@
 self.assertFalse(self.server.ready)
 
 def testCreateMonitoringInstances(self):
-assert 'monitors' not in self.config
-self.assertRaises(KeyError,
-

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Fix RuntimeError in Server.merge

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405295 )

Change subject: Fix RuntimeError in Server.merge
..

Fix RuntimeError in Server.merge

If a disallowed config key/type was present in configuration,
it would be deleted from the dict while looping over it.

Change-Id: I5f1027c7695e4ade48333abcc65a14d0cecc4a45
---
M pybal/server.py
M pybal/test/test_server.py
2 files changed, 9 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/95/405295/1

diff --git a/pybal/server.py b/pybal/server.py
index f1ba983..38a9224 100644
--- a/pybal/server.py
+++ b/pybal/server.py
@@ -28,7 +28,7 @@
 DEF_WEIGHT = 10
 
 # Set of attributes allowed to be overridden in a server list
-allowedConfigKeys = [ ('host', str), ('weight', int), ('enabled', bool) ]
+allowedConfigKeys = { ('host', str), ('weight', int), ('enabled', bool) }
 
 def __init__(self, host, lvsservice, addressFamily=None):
 """Constructor"""
@@ -243,12 +243,12 @@
 def merge(self, configuration):
 """Merges in configuration from a dictionary of (allowed) attributes"""
 
-for key, value in configuration.iteritems():
-if (key, type(value)) not in self.allowedConfigKeys:
-del configuration[key]
-
+filteredConfig = {k:v
+for (k,v)
+in configuration.iteritems()
+if (k, type(v)) in self.allowedConfigKeys}
 # Overwrite configuration
-self.__dict__.update(configuration)
+self.__dict__.update(filteredConfig)
 self.maintainState()
 self.modified = True# Indicate that this instance previously 
existed
 
diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index 7ff0a31..ec6cedd 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -33,8 +33,7 @@
 'host': "example1.example.com",
 'weight': 66,
 'enabled': True,
-# FIXME: bug in Server.merge
-#'rogue': "this attribute should not be merged"
+'rogue': "this attribute should not be merged"
 }
 
 def tearDown(self):
@@ -170,6 +169,8 @@
 self.assertEquals(self.server.host, self.exampleConfigDict['host'])
 self.assertEquals(self.server.weight, self.exampleConfigDict['weight'])
 self.assertEquals(self.server.enabled, 
self.exampleConfigDict['enabled'])
+self.assertNotIn(self.exampleConfigDict['rogue'], self.server.__dict__)
+del self.exampleConfigDict['rogue']
 self.assertDictContainsSubset(self.exampleConfigDict, 
self.server.__dict__)
 
 def testDumpState(self):

-- 
To view, visit https://gerrit.wikimedia.org/r/405295
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5f1027c7695e4ade48333abcc65a14d0cecc4a45
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add unit testing of monitor creation/loading

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405297 )

Change subject: Add unit testing of monitor creation/loading
..

Add unit testing of monitor creation/loading

Add a mock monitor to aid unit testing

Change-Id: I8e0c21605f7c130426d61ebef72afc1f1a4bc208
---
M pybal/monitors/__init__.py
M pybal/test/test_server.py
2 files changed, 16 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/97/405297/1

diff --git a/pybal/monitors/__init__.py b/pybal/monitors/__init__.py
index d0743a7..d64e73e 100644
--- a/pybal/monitors/__init__.py
+++ b/pybal/monitors/__init__.py
@@ -5,4 +5,4 @@
 The monitors package contains all (complete) monitoring implementations of 
PyBal
 """
 
-__all__ = [ 'proxyfetch', 'idleconnection', 'runcommand', 'dnsquery' ]
+__all__ = [ 'proxyfetch', 'idleconnection', 'runcommand', 'dnsquery', 'mock' ]
diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index 3d62f83..1df0121 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -13,7 +13,7 @@
 import pybal.server
 
 from twisted.python import failure
-from twisted.internet.reactor import getDelayedCalls
+from twisted.internet import reactor
 
 from .fixtures import PyBalTestCase, StubLVSService
 
@@ -38,7 +38,7 @@
 }
 
 def tearDown(self):
-for call in getDelayedCalls():
+for call in reactor.getDelayedCalls():
 if call.func.func_name == 'maybeParseConfig':
 call.cancel()
 
@@ -131,7 +131,19 @@
 self.config['monitors'] = "[ \"NonexistentMonitor\" ]"
 self.server.createMonitoringInstances(self.mockCoordinator)
 
-# TODO: test creation of a (mock) monitor
+def testCreateMockMonitoringInstance(self):
+self.config['monitors'] = "[ \"Mock\" ]"
+self.server.createMonitoringInstances(self.mockCoordinator)
+self.assertTrue(self.server.monitors)
+self.assertTrue(all({m.active for m in self.server.monitors}))
+
+@mock.patch('twisted.internet.reactor.stop')
+@mock.patch('importlib.import_module')
+def testCreateFailingMockMonitoringInstance(self, mocked_import_module, 
mock_reactor):
+mocked_import_module.side_effect = RuntimeError("Similating runtime 
error to aid unit testing")
+self.config['monitors'] = "[ \"Mock\" ]"
+self.server.createMonitoringInstances(self.mockCoordinator)
+self.assertTrue(mock_reactor.assert_called)
 
 def testCalcStatus(self):
 self.mockMonitor.up = True

-- 
To view, visit https://gerrit.wikimedia.org/r/405297
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I8e0c21605f7c130426d61ebef72afc1f1a4bc208
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Test all branches of the Server constructor

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405296 )

Change subject: Test all branches of the Server constructor
..

Test all branches of the Server constructor

Change-Id: Iabb44b4cbe8457ccc0cc456137de0e15166cc696
---
M pybal/test/test_server.py
1 file changed, 8 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/96/405296/1

diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index ec6cedd..3d62f83 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -8,6 +8,7 @@
 """
 
 import mock
+import socket
 
 import pybal.server
 
@@ -41,6 +42,13 @@
 if call.func.func_name == 'maybeParseConfig':
 call.cancel()
 
+def testInit(self):
+self.assertEquals(self.server.addressFamily, socket.AF_INET)
+
+server = pybal.server.Server(
+'example.com', self.lvsservice, addressFamily=socket.AF_INET6)
+self.assertEquals(server.addressFamily, socket.AF_INET6)
+
 def testEq(self):
 self.assertEquals(self.server, self.server)
 

-- 
To view, visit https://gerrit.wikimedia.org/r/405296
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iabb44b4cbe8457ccc0cc456137de0e15166cc696
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Improve and clarify error handling of DNS lookups

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405284 )

Change subject: Improve and clarify error handling of DNS lookups
..

Improve and clarify error handling of DNS lookups

Add a test for a nonexistent DNS name.

Rename _hostnameResolved to _allLookupsCompleted to better express its
function.

Consume all errors in the DeferredList because they're not useful and
error handling is done by _allLookupsCompleted.

Change-Id: I125951079cd07ef71964c5e576c3168d55c17d37
---
M pybal/server.py
M pybal/test/test_server.py
2 files changed, 20 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/84/405284/1

diff --git a/pybal/server.py b/pybal/server.py
index 48b35fc..f1ba983 100644
--- a/pybal/server.py
+++ b/pybal/server.py
@@ -11,6 +11,7 @@
 
 from twisted.internet import defer, reactor
 from twisted.names import client, dns
+from twisted.names.error import AuthoritativeDomainError
 from twisted.python import failure
 
 from pybal import util
@@ -89,7 +90,8 @@
 lookups.append(client.lookupIPV6Address(self.host, timeout
 ).addCallback(self._lookupFinished, socket.AF_INET6, query))
 
-return defer.DeferredList(lookups).addBoth(self._hostnameResolved)
+return defer.DeferredList(lookups, consumeErrors=True
+).addCallback(self._allLookupsCompleted)
 
 def _lookupFinished(self, (answers, authority, additional), addressFamily, 
query):
 ips = set([socket.inet_ntop(addressFamily, r.payload.address)
@@ -108,7 +110,7 @@
 
 return ips
 
-def _hostnameResolved(self, result):
+def _allLookupsCompleted(self, results):
 # Pick *1* main ip address to use. Prefer any existing one
 # if still available.
 
@@ -127,11 +129,12 @@
 try:
 if not self.ip or self.ip not in ip_addresses:
 self.ip = random.choice(list(ip_addresses))
-# TODO: (re)pool
 except IndexError:
-return failure.Failure() # TODO: be more specific?
+errmsg = "Could not resolve {} to IP addresses for AF {})".format(
+self.host, self.addressFamily)
+raise AuthoritativeDomainError(errmsg)
 else:
-return True
+return self.ip
 
 def destroy(self):
 self.enabled = False
diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index e6d6bc1..7ff0a31 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -72,12 +72,23 @@
 
 def testResolveHostname(self):
 def callback(result):
-self.assertTrue((result == True or isinstance(result, 
failure.Failure)))
+self.assertIsNotNone(result)
 
 deferred = self.server.resolveHostname()
 deferred.addCallback(callback)
 return deferred
 
+def testResolveNonexistentHostname(self):
+def errback(err):
+self.assertTrue(isinstance(err, failure.Failure))
+return True
+
+nonexistentServer = pybal.server.Server(
+'nonexistent.example.com', self.lvsservice)
+deferred = nonexistentServer.resolveHostname()
+deferred.addErrback(errback)
+return deferred
+
 def testDestroy(self):
 self.server.destroy()
 self.assertFalse(self.server.enabled)

-- 
To view, visit https://gerrit.wikimedia.org/r/405284
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I125951079cd07ef71964c5e576c3168d55c17d37
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Expand test coverage of server.py

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404762 )

Change subject: Expand test coverage of server.py
..

Expand test coverage of server.py

Change-Id: I56c521a1a4f24c73839f76bcf52afb11efa739e1
---
M pybal/test/test_server.py
1 file changed, 52 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/62/404762/1

diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index 01da0d9..e6d6bc1 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -9,12 +9,12 @@
 
 import mock
 
-import pybal.coordinator
+import pybal.server
 
 from twisted.python import failure
 from twisted.internet.reactor import getDelayedCalls
 
-from .fixtures import PyBalTestCase
+from .fixtures import PyBalTestCase, StubLVSService
 
 class ServerTestCase(PyBalTestCase):
 """Test case for `pybal.server.Server`."""
@@ -23,9 +23,10 @@
 super(ServerTestCase, self).setUp()
 
 self.server = pybal.server.Server(
-'example.com', mock.MagicMock())
+'example.com', self.lvsservice)
 
 self.mockMonitor = mock.MagicMock()
+self.mockCoordinator = mock.MagicMock()
 self.server.addMonitor(self.mockMonitor)
 
 self.exampleConfigDict = {
@@ -40,6 +41,26 @@
 for call in getDelayedCalls():
 if call.func.func_name == 'maybeParseConfig':
 call.cancel()
+
+def testEq(self):
+self.assertEquals(self.server, self.server)
+
+# Create a Server instance with different hostname
+otherServer = pybal.server.Server('other.example.com', self.lvsservice)
+self.assertNotEqual(self.server, otherServer)
+
+# Create a Server instance with equal hostname but different LVSService
+otherLVSService = StubLVSService(
+'otherservice',
+(self.protocol, self.ip, self.port, self.scheduler),
+self.config)
+otherServer = pybal.server.Server('example.com', otherLVSService)
+self.assertNotEqual(self.server, otherServer)
+
+def testHash(self):
+# Create a Server instance with different hostname
+otherServer = pybal.server.Server('other.example.com', self.lvsservice)
+self.assertNotEqual(hash(self.server), hash(otherServer))
 
 def testAddMonitor(self):
 self.assertIn(self.mockMonitor, self.server.monitors)
@@ -68,9 +89,31 @@
 self.assertEquals(self.server.ready, result)
 
 self.server.createMonitoringInstances = mock.MagicMock()
-deferred = self.server.initialize(coordinator=mock.MagicMock())
+deferred = self.server.initialize(self.mockCoordinator)
 deferred.addCallback(callback)
 return deferred
+
+@mock.patch('pybal.server.Server.createMonitoringInstances')
+def testReady(self, mock_createMonitoringInstances):
+r = self.server._ready(True, self.mockCoordinator)
+self.assertTrue(r)
+self.assertTrue(self.server.ready)
+mock_createMonitoringInstances.assert_called()
+
+def testInitFailed(self):
+r = self.server._initFailed(failure.Failure(Exception("Fake failure")))
+self.assertFalse(r)
+self.assertFalse(self.server.ready)
+
+def testCreateMonitoringInstances(self):
+assert 'monitors' not in self.config
+self.assertRaises(KeyError,
+self.server.createMonitoringInstances, self.mockCoordinator)
+
+self.config['monitors'] = "[ \"NonexistentMonitor\" ]"
+self.server.createMonitoringInstances(self.mockCoordinator)
+
+# TODO: test creation of a (mock) monitor
 
 def testCalcStatus(self):
 self.mockMonitor.up = True
@@ -95,6 +138,11 @@
 self.assertFalse(self.server.calcStatus())
 self.assertTrue(self.server.calcPartialStatus())
 
+def testTextStatus(self):
+textStatus = self.server.textStatus()
+self.assertTrue(isinstance(textStatus, str))
+self.assertEquals(len(textStatus.split('/')), 3)
+
 def testMaintainState(self):
 self.server.pooled = True
 self.server.enabled = False

-- 
To view, visit https://gerrit.wikimedia.org/r/404762
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I56c521a1a4f24c73839f76bcf52afb11efa739e1
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Separate out coordinator.Server into its own module

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404713 )

Change subject: Separate out coordinator.Server into its own module
..

Separate out coordinator.Server into its own module

Change-Id: I925f0abe553cd82603014cd8a0760be8bc8ad880
---
M pybal/coordinator.py
A pybal/server.py
M pybal/test/test_coordinator.py
M pybal/test/test_server.py
4 files changed, 277 insertions(+), 264 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/13/404713/1

diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index a4b4e16..b7b3591 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -6,270 +6,14 @@
 
 LVS Squid balancer/monitor for managing the Wikimedia Squid servers using LVS
 """
-import importlib
-import random
-import socket
 
-from twisted.internet import defer, reactor
-from twisted.names import client, dns
-from twisted.python import failure
+from twisted.internet import defer
 
 from pybal import config, util
 from pybal.metrics import Counter, Gauge
+import pybal.server
 
 log = util.log
-
-
-class Server:
-"""
-Class that maintains configuration and state of a single (real)server
-"""
-
-# Defaults
-DEF_STATE = True
-DEF_WEIGHT = 10
-
-# Set of attributes allowed to be overridden in a server list
-allowedConfigKeys = [ ('host', str), ('weight', int), ('enabled', bool) ]
-
-def __init__(self, host, lvsservice, addressFamily=None):
-"""Constructor"""
-
-self.host = host
-self.lvsservice = lvsservice
-if addressFamily:
-self.addressFamily = addressFamily
-else:
-self.addressFamily = (':' in self.lvsservice.ip) and 
socket.AF_INET6 or socket.AF_INET
-self.ip = None
-self.port = 80
-self.ip4_addresses = set()
-self.ip6_addresses = set()
-self.monitors = set()
-
-# A few invariants that SHOULD be maintained (but currently may not 
be):
-# P0: pooled => enabled /\ ready
-# P1: up => pooled \/ !enabled \/ !ready
-# P2: pooled => up \/ !canDepool
-
-self.weight = self.DEF_WEIGHT
-self.up = False
-self.pooled = False
-self.enabled = True
-self.ready = False
-self.modified = None
-
-def __eq__(self, other):
-return isinstance(other, Server) and self.host == other.host and 
self.lvsservice == other.lvsservice
-
-def __hash__(self):
-return hash(self.host)
-
-def addMonitor(self, monitor):
-"""Adds a monitor instance to the set"""
-
-self.monitors.add(monitor)
-
-def removeMonitors(self):
-"""Removes all monitors"""
-
-for monitor in self.monitors:
-monitor.stop()
-
-self.monitors.clear()
-
-def resolveHostname(self):
-"""Attempts to resolve the server's hostname to an IP address for 
better reliability."""
-
-timeout = [1, 2, 5]
-lookups = []
-
-query = dns.Query(self.host, dns.A)
-lookups.append(client.lookupAddress(self.host, timeout
-).addCallback(self._lookupFinished, socket.AF_INET, query))
-
-query = dns.Query(self.host, dns.)
-lookups.append(client.lookupIPV6Address(self.host, timeout
-).addCallback(self._lookupFinished, socket.AF_INET6, query))
-
-return defer.DeferredList(lookups).addBoth(self._hostnameResolved)
-
-def _lookupFinished(self, (answers, authority, additional), addressFamily, 
query):
-ips = set([socket.inet_ntop(addressFamily, r.payload.address)
-   for r in answers
-   if r.name == query.name and r.type == query.type])
-
-if query.type == dns.A:
-self.ip4_addresses = ips
-elif query.type == dns.:
-self.ip6_addresses = ips
-
-# TODO: expire TTL
-#if self.ip:
-#minTTL = min([r.ttl for r in answers
-#  if r.name == query.name and r.type == query.type])
-
-return ips
-
-def _hostnameResolved(self, result):
-# Pick *1* main ip address to use. Prefer any existing one
-# if still available.
-
-addr = " ".join(
-list(self.ip4_addresses) + list(self.ip6_addresses))
-msg = "Resolved {} to addresses {}".format(self.host, addr)
-log.debug(msg)
-
-ip_addresses = {
-socket.AF_INET:
-self.ip4_addresses,
-socket.AF_INET6:
-self.ip6_addresses
-}[self.addressFamily]
-
-try:
-if not self.ip or self.ip not in ip_addresses:
-self.ip = random.choice(list(ip_addresses))
-# TODO: (re)pool
-except IndexError:
-return failure.Failure() # TODO: be more specific?
-else:
-return True
-
-def destroy(self):
-  

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add unit test cases for Server

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404704 )

Change subject: Add unit test cases for Server
..

Add unit test cases for Server

Basic test cases for most of the Server class, with the notable
exception of method createMonitoringInstances.

Change-Id: Icdc4f9753e95859623b512634c314649fcfda56d
---
A pybal/test/test_server.py
1 file changed, 128 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/04/404704/1

diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
new file mode 100644
index 000..3a39d33
--- /dev/null
+++ b/pybal/test/test_server.py
@@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+"""
+  PyBal unit tests
+  
+
+  This module contains tests for `pybal.monitor`.
+
+"""
+
+import mock
+
+import pybal.coordinator
+
+from twisted.python import failure
+from twisted.internet.reactor import getDelayedCalls
+
+from .fixtures import PyBalTestCase
+
+class ServerTestCase(PyBalTestCase):
+"""Test case for `pybal.coordinator.Server`."""
+
+def setUp(self):
+super(ServerTestCase, self).setUp()
+
+self.server = pybal.coordinator.Server(
+'example.com', mock.MagicMock())
+
+self.mockMonitor = mock.MagicMock()
+self.server.addMonitor(self.mockMonitor)
+
+self.exampleConfigDict = {
+'host': "example1.example.com",
+'weight': 66,
+'enabled': True,
+# FIXME: bug in Server.merge
+#'rogue': "this attribute should not be merged"
+}
+
+def tearDown(self):
+for call in getDelayedCalls():
+if call.func.func_name == 'maybeParseConfig':
+call.cancel()
+
+def testAddMonitor(self):
+self.assertIn(self.mockMonitor, self.server.monitors)
+
+def testRemoveMonitors(self):
+self.server.removeMonitors()
+self.assertEqual(len(self.server.monitors), 0)
+self.mockMonitor.stop.assert_called()
+
+def testResolveHostname(self):
+def callback(result):
+self.assertTrue((result == True or isinstance(result, 
failure.Failure)))
+
+deferred = self.server.resolveHostname()
+deferred.addCallback(callback)
+return deferred
+
+def testDestroy(self):
+self.server.destroy()
+self.assertFalse(self.server.enabled)
+self.assertEqual(len(self.server.monitors), 0)
+
+def testInitialize(self):
+def callback(result):
+self.assertTrue(isinstance(result, bool))
+self.assertEquals(self.server.ready, result)
+
+self.server.createMonitoringInstances = mock.MagicMock()
+deferred = self.server.initialize(coordinator=mock.MagicMock())
+deferred.addCallback(callback)
+return deferred
+
+def testCalcStatus(self):
+self.mockMonitor.up = True
+self.assertTrue(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+m = mock.MagicMock()
+m.up = True
+self.server.addMonitor(m)
+self.assertTrue(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+m.up = False
+self.assertFalse(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+self.mockMonitor.up = False
+self.assertFalse(self.server.calcPartialStatus())
+
+# Currently, no monitors implies False Status
+self.server.removeMonitors()
+self.assertFalse(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+def testMaintainState(self):
+self.server.pooled = True
+self.server.enabled = False
+self.server.maintainState()
+self.assertFalse(self.server.pooled)
+
+self.server.pooled = False
+self.server.enabled = True
+self.server.maintainState()
+self.assertFalse(self.server.up)
+
+def testMerge(self):
+self.server.merge(self.exampleConfigDict)
+self.assertEquals(self.server.host, self.exampleConfigDict['host'])
+self.assertEquals(self.server.weight, self.exampleConfigDict['weight'])
+self.assertEquals(self.server.enabled, 
self.exampleConfigDict['enabled'])
+self.assertDictContainsSubset(self.exampleConfigDict, 
self.server.__dict__)
+
+def testDumpState(self):
+state = self.server.dumpState()
+self.assertLessEqual(
+{'pooled', 'weight', 'up', 'enabled'},
+set(state.keys()))
+
+def testBuildServer(self):
+server = self.server.buildServer(
+hostName=self.exampleConfigDict['host'],
+configuration=self.exampleConfigDict,
+lvsservice=mock.MagicMock())
+self.assertTrue(isinstance(server, pybal.coordinator.Server))
+

[MediaWiki-commits] [Gerrit] operations...pybal[master]: [WiP] Support per-service-IP BGP MED values

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/393097 )

Change subject: [WiP] Support per-service-IP BGP MED values
..

[WiP] Support per-service-IP BGP MED values

Add a per-service 'bgp-med' configuation option that allows overriding
the global MED value per service.

Note that as multiple services may share a service IP (e.g. port 80 and
443), the MED values are required to be equal.

These changes also add scaffolding for later support for announcing and
retracting service IPs based on the service health.

Bug: T165764
Change-Id: I08687e8072dd8f2ae88833a1660d154ae28c7add
---
M pybal/bgpfailover.py
M pybal/coordinator.py
M pybal/ipvs.py
3 files changed, 59 insertions(+), 27 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/97/393097/1

diff --git a/pybal/bgpfailover.py b/pybal/bgpfailover.py
index 1b6865f..70811ed 100755
--- a/pybal/bgpfailover.py
+++ b/pybal/bgpfailover.py
@@ -22,6 +22,7 @@
 
 prefixes = {}
 peerings = {}
+ipServices = {}
 
 def __init__(self, globalConfig):
 if not globalConfig.getboolean('bgp', False):
@@ -35,29 +36,22 @@
 myASN = self.globalConfig.getint('bgp-local-asn')
 asPath = self.globalConfig.get('bgp-as-path', str(myASN))
 asPath = [int(asn) for asn in asPath.split()]
-med = self.globalConfig.getint('bgp-med', 0)
-baseAttrs = [bgp.OriginAttribute(), bgp.ASPathAttribute(asPath)]
-if med: baseAttrs.append(bgp.MEDAttribute(med))
 
-attributes = {}
+defaultMED = self.globalConfig.getint('bgp-med', 0)
+
 try:
-attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
-
bgp.NextHopAttribute(self.globalConfig['bgp-nexthop-ipv4'])])
+nexthopIPv4 = self.globalConfig['bgp-nexthop-ipv4']
 except KeyError:
 if (bgp.AFI_INET, bgp.SAFI_UNICAST) in BGPFailover.prefixes:
 raise ValueError("IPv4 BGP NextHop (global configuration 
variable 'bgp-nexthop-ipv4') not set")
 
 try:
-attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
-bgp.MPReachNLRIAttribute((bgp.AFI_INET6, bgp.SAFI_UNICAST,
- 
bgp.IPv6IP(self.globalConfig['bgp-nexthop-ipv6']), []))])
+nexthopIPv6 = self.globalConfig['bgp-nexthop-ipv6']
 except KeyError:
 if (bgp.AFI_INET6, bgp.SAFI_UNICAST) in BGPFailover.prefixes:
 raise ValueError("IPv6 BGP NextHop (global configuration 
variable 'bgp-nexthop-ipv6') not set")
 
-advertisements = set([bgp.Advertisement(prefix, attributes[af], af)
-  for af in attributes.keys()
-  for prefix in BGPFailover.prefixes.get(af, 
set())])
+advertisements = self.buildAdvertisements(myASN, asPath, 
nexthopIPv4, nexthopIPv6, defaultMED)
 
 bgpPeerAddress = self.globalConfig.get('bgp-peer-address', 
'').strip()
 if bgpPeerAddress[0] != '[': bgpPeerAddress = "[ \"{}\" 
]".format(bgpPeerAddress)
@@ -66,7 +60,7 @@
 
 for peerAddr in peerAddresses:
 peering = bgp.NaiveBGPPeering(myASN, peerAddr)
-peering.setEnabledAddressFamilies(set(attributes.keys()))
+peering.setEnabledAddressFamilies(set(self.prefixes.keys()))
 peering.setAdvertisements(advertisements)
 
 log.info("Starting BGP session with peer {}".format(peerAddr))
@@ -102,13 +96,49 @@
 peering.setAdvertisements(set())
 return peering.manualStop()
 
+def buildAdvertisements(self, myASN, asPath, nexthopIPv4, nexthopIPv6, 
defaultMED):
+baseAttrs = [bgp.OriginAttribute(), bgp.ASPathAttribute(asPath)]
+
+advertisements = set()
+for af in self.prefixes:
+for prefix in self.prefixes[af]:
+if af[0] == (bgp.AFI_INET):
+attrList = baseAttrs + [bgp.NextHopAttribute(nexthopIPv4)]
+elif af[0] == (bgp.AFI_INET6):
+attrList = (baseAttrs
++ [bgp.MPReachNLRIAttribute((af, 
bgp.IPv6IP(nexthopIPv6), []))])
+else:
+raise ValueError("Unsupported address family 
{}".format(af))
+
+# This service IP may use a non-default MED
+med = self.ipServices[prefix][0]['med'] # Guaranteed to exist, 
may be None
+attrList.append(bgp.MEDAttribute(med or defaultMED))
+
+attributes = bgp.FrozenAttributeDict(attrList)
+advertisements.add(bgp.Advertisement(prefix, attributes, af))
+
+return advertisements
+
 @classmethod
-   

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Support multiple BGP peerings

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/393066 )

Change subject: Support multiple BGP peerings
..

Support multiple BGP peerings

Allow Pybal to setup BGP peerings with multiple peers.
Global configuration option bgp-peer-address can now be a list and
contain multiple addresses.

Bug: T180069
Change-Id: Ie008f62c3667c20e947ee8d193632a46513188e9
---
M pybal/bgpfailover.py
1 file changed, 29 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/66/393066/1

diff --git a/pybal/bgpfailover.py b/pybal/bgpfailover.py
index b967837..8055957 100755
--- a/pybal/bgpfailover.py
+++ b/pybal/bgpfailover.py
@@ -18,23 +18,26 @@
 
 
 class BGPFailover:
-"""Class for maintaining a BGP session to a router for IP address 
failover"""
+"""Class for maintaining BGP sessions to routers for IP address failover"""
 
 prefixes = {}
-peerings = []
+peerings = {}
 
 def __init__(self, globalConfig):
-self.globalConfig = globalConfig
+if not globalConfig.getboolean('bgp', False):
+return
 
-if self.globalConfig.getboolean('bgp', False):
-self.setup()
+self.globalConfig = globalConfig
+self.setup()
 
 def setup(self):
 try:
-self.bgpPeering = 
bgp.NaiveBGPPeering(myASN=self.globalConfig.getint('bgp-local-asn'),
-  
peerAddr=self.globalConfig.get('bgp-peer-address'))
-
-asPath = [int(asn) for asn in self.globalConfig.get('bgp-as-path', 
str(self.bgpPeering.myASN)).split()]
+myASN = self.globalConfig.getint('bgp-local-asn')
+asPath = [int(asn)
+for asn
+in self.globalConfig.get('bgp-as-path',
+str(myASN)
+).split()]
 med = self.globalConfig.getint('bgp-med', 0)
 baseAttrs = [bgp.OriginAttribute(), bgp.ASPathAttribute(asPath)]
 if med: baseAttrs.append(bgp.MEDAttribute(med))
@@ -59,15 +62,25 @@
   for af in attributes.keys()
   for prefix in BGPFailover.prefixes.get(af, 
set())])
 
-self.bgpPeering.setEnabledAddressFamilies(set(attributes.keys()))
-self.bgpPeering.setAdvertisements(advertisements)
-self.bgpPeering.automaticStart()
+bgpPeerAddress = self.globalConfig.get('bgp-peer-address', 
'').strip()
+if bgpPeerAddress[0] != '[': bgpPeerAddress = "[ \"{}\" 
]".format(bgpPeerAddress)
+peerAddresses = eval(bgpPeerAddress)
+assert isinstance(peerAddresses, list)
+
+for peerAddr in peerAddresses:
+peering = bgp.NaiveBGPPeering(myASN, peerAddr)
+peering.setEnabledAddressFamilies(set(attributes.keys()))
+peering.setAdvertisements(advertisements)
+
+log.info("Starting BGP session with peer {}".format(peerAddr))
+peering.automaticStart()
+self.peerings[peerAddr] = peering
+reactor.addSystemEventTrigger('before', 'shutdown', 
self.closeSession, peering)
+
 except Exception:
-log.critical("Could not set up BGP peering instance.")
+log.critical("Could not set up BGP peering instances.")
 raise
 else:
-BGPFailover.peerings.append(self.bgpPeering)
-reactor.addSystemEventTrigger('before', 'shutdown', 
self.closeSession, self.bgpPeering)
 
 # Bind on the IPs listed in 'bgp_local_ips'. Default to
 # localhost v4 and v6 if no IPs have been specified in the
@@ -79,7 +92,7 @@
 try:
 reactor.listenTCP(
 bgp_local_port,
-bgp.BGPServerFactory({self.bgpPeering.peerAddr: 
self.bgpPeering}),
+bgp.BGPServerFactory(self.peerings),
 interface=ip)
 except CannotListenError as e:
 log.critical(

-- 
To view, visit https://gerrit.wikimedia.org/r/393066
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie008f62c3667c20e947ee8d193632a46513188e9
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Instrument the Twisted reactor with Prometheus metrics

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371636 )

Change subject: Instrument the Twisted reactor with Prometheus metrics
..

Instrument the Twisted reactor with Prometheus metrics

instrumented_reactor adds InstrumentedReactor which derives from either
the EPollReactor or SelectReactor (depending on availability on the
platform), and decorates the most important methods with Prometheus
timing metrics.

Change-Id: Ie7cccd14687808b1c4d4a224d82b8b4ed8d0cbce
---
M pybal/__init__.py
A pybal/instrumented_reactor.py
M pybal/metrics.py
3 files changed, 94 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/36/371636/1

diff --git a/pybal/__init__.py b/pybal/__init__.py
index ade8bc3..a6418e7 100644
--- a/pybal/__init__.py
+++ b/pybal/__init__.py
@@ -4,6 +4,11 @@
 
 The pybal package contains all PyBal modules
 """
+
+# Use our own instrumented epoll reactor
+from pybal import instrumented_reactor
+instrumented_reactor.install()
+
 import test
 
 from .version import *
diff --git a/pybal/instrumented_reactor.py b/pybal/instrumented_reactor.py
new file mode 100644
index 000..7cc0ac2
--- /dev/null
+++ b/pybal/instrumented_reactor.py
@@ -0,0 +1,64 @@
+"""
+reactor.py
+
+An instrumented reactor (epoll or select).
+To install the reactor (and you should do this before any connections,
+listeners or connectors are added)::
+
+from pybal import instrumented_reactor
+instrumented_reactor.install()
+"""
+
+from pybal.metrics import Summary
+
+try:
+from twisted.internet.epollreactor import EPollReactor
+ancestor = EPollReactor
+except ImportError:
+from twisted.internet.selectreactor import SelectReactor
+ancestor = SelectReactor
+
+class InstrumentedReactor(ancestor):
+metric_keywords = {
+'namespace': 'pybal',
+'subsystem': 'reactor'
+}
+metrics = {
+'iteration_duration':
+Summary(
+'iteration_duration',
+'Reactor iteration latency',
+**metric_keywords),
+'do_read_or_write_duration':
+Summary(
+'do_read_or_write_duration',
+'doReadOrWrite latency',
+labelnames=('selectable', 'method'),
+**metric_keywords),
+'run_until_current_duration':
+Summary(
+'run_until_current_duration',
+'runUntilCurrent latency',
+**metric_keywords)
+}
+
+if ancestor.__name__ == 'EPollReactor':
+doPoll = 
metrics['iteration_duration'].time()(EPollReactor.doPoll.__func__)
+doIteration = doPoll
+elif ancestor.__name__ == 'SelectReactor':
+doSelect = 
metrics['iteration_duration'].time()(SelectReactor.doSelect.__func__)
+doIteration = doSelect
+
+runUntilCurrent = 
metrics['run_until_current_duration'].time()(ancestor.runUntilCurrent.__func__)
+
+def _doReadOrWrite(self, selectable, method):
+with self.metrics['do_read_or_write_duration'].labels(
+type(selectable).__name__, method).time():
+super(InstrumentedReactor, self)._doReadOrWrite(selectable, 
method)
+
+def install():
+"""
+Install the instrumented epoll() reactor.
+"""
+from twisted.internet.main import installReactor
+installReactor(InstrumentedReactor())
diff --git a/pybal/metrics.py b/pybal/metrics.py
index 4ccb064..74de644 100644
--- a/pybal/metrics.py
+++ b/pybal/metrics.py
@@ -10,24 +10,44 @@
 except ImportError:
 metrics_implementation = 'dummy'
 
-class DummyMetric(object):
-def __init__(self, **kwargs):
+class DummyTimer(object):
+def __call__(self, func):
+def wrap(*args, **kwargs):
+func(*args, **kwargs)
+return wrap
+
+def __enter__(self, *args, **kwargs):
+pass
+def __exit__(self, *args, **kwargs):
 pass
 
-def labels(self, **kwargs):
+class DummyMetric(object):
+def __init__(self, *args, **kwargs):
+pass
+
+def labels(self, *args, **kwargs):
 return self
 
 class DummyCounter(DummyMetric):
-def inc(self, **kwargs):
+def inc(self, *args, **kwargs):
 pass
 
 class DummyGauge(DummyMetric):
-def set(**kwargs):
+def set(self, *args, **kwargs):
 pass
+
+class DummySummary(DummyMetric):
+def observe(self, *args, **kwargs):
+pass
+
+def time(self):
+return DummyTimer()
 
 if metrics_implementation == 'prometheus':
 Counter = prometheus_client.Counter
 Gauge = prometheus_client.Gauge
+Summary = prometheus_client.Summary
 else:
 Counter = DummyCounter
 Gauge = DummyGauge
+Summary = DummySummary

-- 
To view, visit https://gerrit.wikimedia.org/r/371636
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: 

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add metric pybal_service_depool_threshold

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371185 )

Change subject: Add metric pybal_service_depool_threshold
..

Add metric pybal_service_depool_threshold

Reports the depool_threshold from pybal.conf, useful for plotting
against pooled & total servers.

Bug: T171710
Change-Id: Ia3d712002d11cc8e9096ee49814bd53b4a9547df
---
M pybal/coordinator.py
1 file changed, 8 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/85/371185/1

diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index 22ce835..a526b5b 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -316,6 +316,10 @@
 'could_not_depool_total',
 'Pybal could not depool a server because too many down',
 **metric_keywords),
+'depool_threshold': Gauge(
+'depool_threshold',
+"Threshold of up servers vs total servers below which pybal can't 
depool any more",
+**metric_keywords),
 }
 
 def __init__(self, lvsservice, configUrl):
@@ -333,6 +337,10 @@
 self.configObserver = config.ConfigurationObserver.fromUrl(self, 
configUrl)
 self.configObserver.startObserving()
 
+self.metrics['depool_threshold'].labels(
+**self.metric_labels
+).set(self.lvsservice.getDepoolThreshold())
+
 def __str__(self):
 return "[%s]" % self.lvsservice.name
 

-- 
To view, visit https://gerrit.wikimedia.org/r/371185
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia3d712002d11cc8e9096ee49814bd53b4a9547df
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add pybal service (coordinator) metrics

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371126 )

Change subject: Add pybal service (coordinator) metrics
..

Add pybal service (coordinator) metrics

Change-Id: I4a4e067a0bb56a77cb3123b14942e5e470702998
---
M pybal/coordinator.py
1 file changed, 75 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/26/371126/1

diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index c828ca3..401c9a5 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -15,6 +15,7 @@
 from twisted.python import failure
 
 from pybal import config, util
+from pybal.metrics import Counter, Gauge
 
 log = util.log
 
@@ -280,6 +281,43 @@
 
 intvLoadServers = 60
 
+metric_keywords = {
+'labelnames': ('service', ),
+'namespace': 'pybal',
+'subsystem': 'service'
+}
+
+metrics = {
+'servers': Gauge(
+'servers',
+'Amount of servers',
+**metric_keywords),
+'servers_enabled': Gauge(
+'servers_enabled',
+'Amount of enabled servers',
+**metric_keywords),
+'servers_up': Gauge(
+'servers_up',
+'Amount of up servers',
+**metric_keywords),
+'servers_pooled': Gauge(
+'servers_pooled',
+'Amount of pooled servers',
+**metric_keywords),
+'can_depool': Gauge(
+'can_depool',
+'Can depool more servers',
+**metric_keywords),
+'pooled_down_servers': Counter(
+'pooled_down_servers',
+'Amount of down servers pooled because too many down',
+**metric_keywords),
+'could_not_depool_total': Counter(
+'could_not_depool_total',
+'Pybal could not depool a server because too many down',
+**metric_keywords),
+}
+
 def __init__(self, lvsservice, configUrl):
 """Constructor"""
 
@@ -291,6 +329,10 @@
 self.serverInitDeferredList = defer.Deferred()
 self.configObserver = config.ConfigurationObserver.fromUrl(self, 
configUrl)
 self.configObserver.startObserving()
+
+self.metric_labels = {
+'service': self.lvsservice.name
+}
 
 def __str__(self):
 return "[%s]" % self.lvsservice.name
@@ -358,11 +400,14 @@
 if self.canDepool():
 self.lvsservice.removeServer(server)
 self.pooledDownServers.discard(server)
+self.metrics['servers_pooled'].labels(**self.metric_labels).dec()
 else:
 self.pooledDownServers.add(server)
 msg = "Could not depool server " \
   "{} because of too many down!".format(server.host)
 log.error(msg, system=self.lvsservice.name)
+
self.metrics['could_not_depool_total'].labels(**self.metric_labels).inc()
+self._updatePooledDownMetrics()
 
 def repool(self, server):
 """
@@ -374,16 +419,19 @@
 
 if not server.pooled:
 self.lvsservice.addServer(server)
+self.metrics['servers_pooled'].labels(**self.metric_labels).inc()
 else:
 msg = "Leaving previously pooled but down server {} pooled"
 log.info(msg.format(server.host), system=self.lvsservice.name)
 
 # If it had been pooled in down state before, remove it from the list
 self.pooledDownServers.discard(server)
+self._updatePooledDownMetrics()
 
 # See if we can depool any servers that could not be depooled before
 while len(self.pooledDownServers) > 0 and self.canDepool():
 self.depool(self.pooledDownServers.pop())
+self.metrics['servers_pooled'].labels(**self.metric_labels).dec()
 
 def canDepool(self):
 """Returns a boolean denoting whether another server can be depooled"""
@@ -438,6 +486,9 @@
 # Wait for all new servers to finish initializing
 self.serverInitDeferredList = 
defer.DeferredList(initList).addCallback(self._serverInitDone)
 
+# Update metrics
+self._updateServerMetrics()
+
 def _serverInitDone(self, result):
 """Called when all (new) servers have finished initializing"""
 
@@ -445,3 +496,27 @@
 
 # Assign the updated list of enabled servers to the LVSService instance
 self.assignServers()
+
+def _updateServerMetrics(self):
+"""Update gauge metrics for servers on config change"""
+self.metrics['servers'].labels(
+**self.metric_labels
+).set(
+len(self.servers))
+self.metrics['servers_enabled'].labels(
+**self.metric_labels
+).set(
+len([s for s in self.servers.itervalues() if server.enabled]))
+self.metrics['servers_up'].labels(
+

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add monitoring specific metric to IdleConnection

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371117 )

Change subject: Add monitoring specific metric to IdleConnection
..

Add monitoring specific metric to IdleConnection

pybal_monitor_idleconnection_connections_failed_total counts connections
failing to establish.
pybal_monitor_idleconnection_connections_lost_total counts connections
lost for unclean reasons.

Bug: T171710
Change-Id: I9acbcd2c859cc9213e40b7f7019a85be49ca72f8
---
M pybal/monitors/idleconnection.py
1 file changed, 31 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/17/371117/1

diff --git a/pybal/monitors/idleconnection.py b/pybal/monitors/idleconnection.py
index 16bde75..04f6dc3 100644
--- a/pybal/monitors/idleconnection.py
+++ b/pybal/monitors/idleconnection.py
@@ -6,6 +6,7 @@
 """
 
 from pybal import monitor, util
+from pybal.metrics import Counter
 
 from twisted.internet import reactor, protocol
 import logging
@@ -31,6 +32,26 @@
 KEEPALIVE_INTERVAL = 30
 
 __name__ = 'IdleConnection'
+
+metric_labelnames = ('service', 'host', 'monitor')
+metric_keywords = {
+'namespace': 'pybal',
+'subsystem': 'monitor_' + __name__.lower()
+}
+
+idleconnection_metrics = {
+'connections_failed_total': Counter(
+'connections_failed_total',
+'Connections failed to establish',
+labelnames=metric_labelnames + ('reason',),
+**metric_keywords),
+'connections_lost_total': Counter(
+'connections_lost_total',
+'Connections lost uncleanly',
+labelnames=metric_labelnames + ('reason',),
+**metric_keywords)
+}
+
 
 def __init__(self, coordinator, server, configuration):
 """Constructor"""
@@ -76,6 +97,11 @@
 
 self.report("%s failed." % self._report_prefix(), level=logging.WARN)
 
+self.idleconnection_metrics['connections_failed_total'].labels(
+reason=reason.type.__name__,
+**self.metric_labels
+).inc()
+
 # Slowly reconnect
 self.retry(connector)
 
@@ -95,6 +121,11 @@
 
 self.report("%s lost." % self._report_prefix(), level=logging.INFO)
 
+self.idleconnection_metrics['connections_lost_total'].labels(
+reason=reason.type.__name__,
+**self.metric_labels
+).inc()
+
 # Slowly reconnect
 self.retry(connector)
 

-- 
To view, visit https://gerrit.wikimedia.org/r/371117
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9acbcd2c859cc9213e40b7f7019a85be49ca72f8
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add monitoring specific metric to IdleConnection

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371110 )

Change subject: Add monitoring specific metric to IdleConnection
..

Add monitoring specific metric to IdleConnection

Counter connection_failed_total counts connections ending for
unclean reasons.

Bug: T171710
Change-Id: I07809ba636b02a634e91ec86e12215bc9b8b1ad6
---
M pybal/monitors/idleconnection.py
1 file changed, 24 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/10/371110/1

diff --git a/pybal/monitors/idleconnection.py b/pybal/monitors/idleconnection.py
index fd105dd..5c40751 100644
--- a/pybal/monitors/idleconnection.py
+++ b/pybal/monitors/idleconnection.py
@@ -6,6 +6,7 @@
 """
 
 from pybal import monitor
+from pybal.metrics import Counter
 
 from twisted.internet import reactor, protocol
 import logging
@@ -30,6 +31,21 @@
 KEEPALIVE_INTERVAL = 30
 
 __name__ = 'IdleConnection'
+
+metric_labelnames = ('service', 'host', 'monitor')
+metric_keywords = {
+'namespace': 'pybal',
+'subsystem': 'monitor_' + __name__.lower()
+}
+
+idleconnection_metrics = {
+'connection_failed_total': Counter(
+'connection_failed_total',
+'HTTP(S) request duration',
+labelnames=metric_labelnames,
+**metric_keywords)
+}
+
 
 def __init__(self, coordinator, server, configuration):
 """Constructor"""
@@ -75,6 +91,10 @@
 
 self.report("%s failed." % self._report_prefix(), level=logging.WARN)
 
+self.idleconnection_metrics['connection_failed_total'].labels(
+**self.metric_labels
+).inc()
+
 # Slowly reconnect
 self.retry(connector)
 
@@ -94,6 +114,10 @@
 
 self.report("%s lost." % self._report_prefix(), level=logging.INFO)
 
+self.idleconnection_metrics['connection_failed_total'].labels(
+**self.metric_labels
+).inc()
+
 # Slowly reconnect
 self.retry(connector)
 

-- 
To view, visit https://gerrit.wikimedia.org/r/371110
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I07809ba636b02a634e91ec86e12215bc9b8b1ad6
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add monitoring specific metric to RunCommand

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371105 )

Change subject: Add monitoring specific metric to RunCommand
..

Add monitoring specific metric to RunCommand

pybal_monitor_runcommand_run_duration_seconds is a gauge that represents
the run latency, differentiated by the 'result' label
(successful/failed).

Bug: T171710
Change-Id: I2aeea030955c1b2b1abe872e6ed8f5b334c09069
---
M pybal/monitors/runcommand.py
1 file changed, 30 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/05/371105/1

diff --git a/pybal/monitors/runcommand.py b/pybal/monitors/runcommand.py
index a0ca0ec..b10afce 100644
--- a/pybal/monitors/runcommand.py
+++ b/pybal/monitors/runcommand.py
@@ -7,11 +7,13 @@
 
 from pybal import monitor
 from pybal.util import log
+from pybal.metrics import Gauge
 
 import os, sys, signal, errno
 import logging
 
 from twisted.internet import reactor, process, error
+from twisted.python.runtime import seconds
 
 class ProcessGroupProcess(process.Process, object):
 """
@@ -89,6 +91,20 @@
 
 TIMEOUT_RUN = 20
 
+metric_labelnames = ('service', 'host', 'monitor')
+metric_keywords = {
+'namespace': 'pybal',
+'subsystem': 'monitor_' + __name__.lower()
+}
+
+runcommand_metrics = {
+'run_duration_seconds': Gauge(
+'run_duration_seconds',
+'Command duration',
+labelnames=metric_labelnames + ('result', 'exitcode'),
+**metric_keywords)
+}
+
 def __init__(self, coordinator, server, configuration={}):
 """Constructor"""
 
@@ -131,6 +147,7 @@
 def runCommand(self):
 """Periodically called method that does a single uptime check."""
 
+self.checkStartTime = seconds()
 self.runningProcess = self._spawnProcess(self, self.command, 
[self.command] + self.arguments,
  sessionLeader=True, 
timeout=(self.timeout or None))
 
@@ -157,10 +174,23 @@
 Called when the process has ended
 """
 
+duration = seconds() - self.checkStartTime
 if reason.check(error.ProcessDone):
 self._resultUp()
+result = 'successful'
+exitcode = 0
 elif reason.check(error.ProcessTerminated):
 self._resultDown(reason.getErrorMessage())
+result = 'failed'
+exitcode = reason.value.exitCode
+else:
+result = None
+exitcode = None
+
+self.runcommand_metrics['run_duration_seconds'].labels(
+result=result, exitcode=exitcode,
+**self.metric_labels
+).set(duration)
 
 # Schedule the next check
 if self.active:

-- 
To view, visit https://gerrit.wikimedia.org/r/371105
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2aeea030955c1b2b1abe872e6ed8f5b334c09069
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add monitoring specific metric to DNSQuery

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371104 )

Change subject: Add monitoring specific metric to DNSQuery
..

Add monitoring specific metric to DNSQuery

pybal_monitor_dnsquery_request_duration_seconds is a gauge that
represents the request latency, differentiated by the 'result' label
(successful/failed).

Bug: T171710
Change-Id: Ia4814427f11fea266532e13c771e81becfceda52
---
M pybal/monitors/dnsquery.py
1 file changed, 28 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/04/371104/1

diff --git a/pybal/monitors/dnsquery.py b/pybal/monitors/dnsquery.py
index 32c7a2a..0b0c2fa 100644
--- a/pybal/monitors/dnsquery.py
+++ b/pybal/monitors/dnsquery.py
@@ -6,6 +6,7 @@
 """
 
 from pybal import monitor
+from pybal.metrics import Gauge
 
 from twisted.internet import reactor, defer
 from twisted.names import client, dns, error
@@ -29,6 +30,19 @@
  error.DNSQueryRefusedError, error.DNSQueryTimeoutError,
  error.DNSServerError, error.DNSUnknownError)
 
+metric_labelnames = ('service', 'host', 'monitor')
+metric_keywords = {
+'namespace': 'pybal',
+'subsystem': 'monitor_' + __name__.lower()
+}
+
+dnsquery_metrics = {
+'request_duration_seconds': Gauge(
+'request_duration_seconds',
+'DNS query duration',
+labelnames=metric_labelnames + ('result',),
+**metric_keywords)
+}
 
 def __init__(self, coordinator, server, configuration):
 """Constructor"""
@@ -98,9 +112,15 @@
 else:
 resultStr = None
 
-self.report('DNS query successful, %.3f s' % (runtime.seconds() - 
self.checkStartTime)
+duration = runtime.seconds() - self.checkStartTime
+self.report('DNS query successful, %.3f s' % (duration)
 + (resultStr and (': ' + resultStr) or ""))
 self._resultUp()
+
+self.dnsquery_metrics['request_duration_seconds'].labels(
+result='successful',
+**self.metric_labels
+).set(duration)
 
 return answers, authority, additional
 
@@ -127,13 +147,19 @@
 else:
 errorStr = str(failure)
 
+duration = runtime.seconds() - self.checkStartTime
 self.report(
-'DNS query failed, %.3f s' % (runtime.seconds() - 
self.checkStartTime),
+'DNS query failed, %.3f s' % (duration),
 level=logging.ERROR
 )
 
 self._resultDown(errorStr)
 
+self.dnsquery_metrics['request_duration_seconds'].labels(
+result='failed',
+**self.metric_labels
+).set(duration)
+
 failure.trap(*self.catchList)
 
 def _checkFinished(self, result):

-- 
To view, visit https://gerrit.wikimedia.org/r/371104
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia4814427f11fea266532e13c771e81becfceda52
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Handle failing TCP socket options in IdleConnection

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371102 )

Change subject: Handle failing TCP socket options in IdleConnection
..

Handle failing TCP socket options in IdleConnection

Linux socket options TCP_KEEPIDLE are not portable, fail gracefully
if they don't exist (e.g. on other platforms).

Change-Id: I79041b4baabeeb537ed9d4cb4a249f97f75a35aa
---
M pybal/monitors/idleconnection.py
1 file changed, 8 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/02/371102/1

diff --git a/pybal/monitors/idleconnection.py b/pybal/monitors/idleconnection.py
index fd105dd..16bde75 100644
--- a/pybal/monitors/idleconnection.py
+++ b/pybal/monitors/idleconnection.py
@@ -5,13 +5,14 @@
 Monitor class implementations for PyBal
 """
 
-from pybal import monitor
+from pybal import monitor, util
 
 from twisted.internet import reactor, protocol
 import logging
 
 import socket
 
+log = util.log
 
 class IdleConnectionMonitoringProtocol(monitor.MonitoringProtocol, 
protocol.ReconnectingClientFactory):
 """
@@ -106,9 +107,12 @@
 if self.transport is not None and self.keepAlive:
 sock = self.transport.getHandle()
 sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPIDLE, 
self.keepAliveIdle)
-sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPCNT, 
self.keepAliveRetries)
-sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPINTVL, 
self.keepAliveInterval)
+try:
+sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPIDLE, 
self.keepAliveIdle)
+sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPCNT, 
self.keepAliveRetries)
+sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPINTVL, 
self.keepAliveInterval)
+except AttributeError:
+log.warn("Could not set TCP_KEEPIDLE, TCP_KEEPCNT, 
TCP_KEEPINTVL socket options (not Linux?)")
 
 # Set status to up
 self._resultUp()

-- 
To view, visit https://gerrit.wikimedia.org/r/371102
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I79041b4baabeeb537ed9d4cb4a249f97f75a35aa
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add monitoring specific metric to ProxyFetch

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371061 )

Change subject: Add monitoring specific metric to ProxyFetch
..

Add monitoring specific metric to ProxyFetch

pybal_monitor_proxyfetch_request_duration_seconds is a gauge that
represents the request latency, differentiated by the 'result' label
(up/down).

Bug: T171710
Change-Id: I18fc7bdc679d5c339487d2d87c515d1254d37d4b
---
M pybal/monitors/proxyfetch.py
1 file changed, 27 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/61/371061/1

diff --git a/pybal/monitors/proxyfetch.py b/pybal/monitors/proxyfetch.py
index 3468e73..7334684 100644
--- a/pybal/monitors/proxyfetch.py
+++ b/pybal/monitors/proxyfetch.py
@@ -6,6 +6,7 @@
 """
 
 from pybal import monitor, util
+from pybal.metrics import Gauge
 
 from twisted.internet import reactor, defer
 from twisted.web import client
@@ -49,6 +50,18 @@
 from twisted.internet import error
 from twisted.web import error as weberror
 catchList = ( defer.TimeoutError, weberror.Error, error.ConnectError, 
error.DNSLookupError )
+
+metric_labelnames = ('service', 'host', 'monitor')
+metric_keywords = {
+'namespace': 'pybal',
+'subsystem': 'monitor_' + __name__.lower()
+}
+
+monitor.MonitoringProtocol.metrics['request_duration_seconds'] = Gauge(
+'request_duration_seconds',
+'HTTP(S) request duration',
+labelnames=metric_labelnames + ('result',), # TODO: statuscode
+**metric_keywords)
 
 def __init__(self, coordinator, server, configuration={}):
 """Constructor"""
@@ -115,8 +128,14 @@
 def _fetchSuccessful(self, result):
 """Called when getProxyPage is finished successfully."""
 
-self.report('Fetch successful, %.3f s' % (seconds() - 
self.checkStartTime))
+duration = seconds() - self.checkStartTime
+self.report('Fetch successful, %.3f s' % (duration))
 self._resultUp()
+
+self.metrics['request_duration_seconds'].labels(
+result='up',
+**self.metric_labels
+).set(duration)
 
 return result
 
@@ -127,11 +146,17 @@
 if failure.check(defer.CancelledError):
 return None
 
-self.report('Fetch failed, %.3f s' % (seconds() - self.checkStartTime),
+duration = seconds() - self.checkStartTime
+self.report('Fetch failed, %.3f s' % (duration),
 level=logging.WARN)
 
 self._resultDown(failure.getErrorMessage())
 
+self.metrics['request_duration_seconds'].labels(
+result='down',
+**self.metric_labels
+).set(duration)
+
 failure.trap(*self.catchList)
 
 def _checkFinished(self, result):

-- 
To view, visit https://gerrit.wikimedia.org/r/371061
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I18fc7bdc679d5c339487d2d87c515d1254d37d4b
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add generic monitor metrics

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/371060 )

Change subject: Add generic monitor metrics
..

Add generic monitor metrics

pybal_monitor_{up,down}_result_total count the number of up and down
check run results, not transitions
pybal_monitor_status is a gauge that represents whether the monitor
is up or down.

Bug: T171710
Change-Id: I2c28d40019338930ad093c407b85a9c235427723
---
M pybal/metrics.py
M pybal/monitor.py
2 files changed, 14 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/60/371060/1

diff --git a/pybal/metrics.py b/pybal/metrics.py
index f44acf5..1178970 100644
--- a/pybal/metrics.py
+++ b/pybal/metrics.py
@@ -21,7 +21,13 @@
 def inc(**kwargs):
 pass
 
+class DummyGauge(DummyMetric):
+def set(**kwargs):
+pass
+
 if metrics_implementation == 'prometheus':
 Counter = prometheus_client.Counter
+Gauge = prometheus_client.Gauge
 else:
 Counter = DummyCounter
+Gauge = DummyGauge
diff --git a/pybal/monitor.py b/pybal/monitor.py
index d8d187f..96d1ca0 100644
--- a/pybal/monitor.py
+++ b/pybal/monitor.py
@@ -7,7 +7,7 @@
 from twisted.internet import reactor
 from . import util
 import logging
-from pybal.metrics import Counter
+from pybal.metrics import Counter, Gauge
 
 _log = util._log
 
@@ -30,6 +30,9 @@
 metrics = {
 'up_transitions_total': Counter('up_transitions_total', 'Monitor up 
transition count', **metric_keywords),
 'down_transitions_total': Counter('down_transitions_total', 'Monitor 
down transition count', **metric_keywords),
+'up_result_total': Counter('up_result_total', 'Monitor up result 
count', **metric_keywords),
+'down_result_total': Counter('down_result_total', 'Monitor down result 
count', **metric_keywords),
+'status': Gauge('status', 'Monitor up status', **metric_keywords)
 }
 
 def __init__(self, coordinator, server, configuration={}, reactor=reactor):
@@ -70,6 +73,7 @@
 """Sets own monitoring state to Up and notifies the coordinator
 if this implies a state change.
 """
+self.metrics['up_result_total'].labels(**self.metric_labels).inc()
 if self.active and self.up is False or self.firstCheck:
 self.up = True
 self.firstCheck = False
@@ -77,11 +81,12 @@
 self.coordinator.resultUp(self)
 
 
self.metrics['up_transitions_total'].labels(**self.metric_labels).inc()
-
+self.metrics['status'].labels(**self.metric_labels).set(1)
 
 def _resultDown(self, reason=None):
 """Sets own monitoring state to Down and notifies the
 coordinator if this implies a state change."""
+self.metrics['down_result_total'].labels(**self.metric_labels).inc()
 if self.active and self.up is True or self.firstCheck:
 self.up = False
 self.firstCheck = False
@@ -89,6 +94,7 @@
 self.coordinator.resultDown(self, reason)
 
 
self.metrics['down_transitions_total'].labels(**self.metric_labels).inc()
+self.metrics['status'].labels(**self.metric_labels).set(0)
 
 def report(self, text, level=logging.DEBUG):
 """Common method for reporting/logging check results."""

-- 
To view, visit https://gerrit.wikimedia.org/r/371060
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2c28d40019338930ad093c407b85a9c235427723
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Allow BGP socket to listen on specific IPs and port

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/370894 )

Change subject: Allow BGP socket to listen on specific IPs and port
..


Allow BGP socket to listen on specific IPs and port

Add global config option bgp-local-ips (default empty: all IPs) which
takes a set of IPs for BGP to bind to.
Add global config option bgp-local-port to override the
default TCP port 179.

Error handling has been altered to log a critical message and raise
an exception if pybal fails to bind to the port on on one of the IP
addresses, instead of ignoring it.

Bug: T103882
Change-Id: I0e4a1cae515602eea8ab07bffb61b9009d5a995a
---
M pybal/bgpfailover.py
1 file changed, 18 insertions(+), 5 deletions(-)



diff --git a/pybal/bgpfailover.py b/pybal/bgpfailover.py
index 5392bcb..b967837 100755
--- a/pybal/bgpfailover.py
+++ b/pybal/bgpfailover.py
@@ -8,6 +8,7 @@
 """
 
 from twisted.internet import reactor
+from twisted.internet.error import CannotListenError
 
 from pybal.util import log
 try:
@@ -67,11 +68,23 @@
 else:
 BGPFailover.peerings.append(self.bgpPeering)
 reactor.addSystemEventTrigger('before', 'shutdown', 
self.closeSession, self.bgpPeering)
-try:
-# Try to listen on the BGP port, not fatal if fails
-reactor.listenTCP(bgp.PORT, 
bgp.BGPServerFactory({self.bgpPeering.peerAddr: self.bgpPeering}))
-except Exception:
-pass
+
+# Bind on the IPs listed in 'bgp_local_ips'. Default to
+# localhost v4 and v6 if no IPs have been specified in the
+# configuration.
+bgp_local_ips = eval(self.globalConfig.get('bgp-local-ips', 
'[""]'))
+bgp_local_port = self.globalConfig.getint('bgp-local-port', 
bgp.PORT)
+# Try to listen on the BGP port, not fatal if fails
+for ip in bgp_local_ips:
+try:
+reactor.listenTCP(
+bgp_local_port,
+bgp.BGPServerFactory({self.bgpPeering.peerAddr: 
self.bgpPeering}),
+interface=ip)
+except CannotListenError as e:
+log.critical(
+"Could not listen for BGP connections: " + str(e))
+raise
 
 def closeSession(self, peering):
 log.info("Clearing session to {}".format(peering.peerAddr))

-- 
To view, visit https://gerrit.wikimedia.org/r/370894
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0e4a1cae515602eea8ab07bffb61b9009d5a995a
Gerrit-PatchSet: 3
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Allow BGP socket to listen on specific IPs only

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/370894 )

Change subject: Allow BGP socket to listen on specific IPs only
..

Allow BGP socket to listen on specific IPs only

Adds a global config option bgp_local_ips (default empty: all IPv4 and
all IPv6 IPs) which takes a set of IPs for BGP to bind to.

Error handling has been altered to log a critical message and raise
an exception if pybal fails to bind to one of the IP addresses,
instead of ignoring it.

Bug: T103882
Change-Id: I0e4a1cae515602eea8ab07bffb61b9009d5a995a
---
M pybal/bgpfailover.py
1 file changed, 15 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/94/370894/1

diff --git a/pybal/bgpfailover.py b/pybal/bgpfailover.py
index 5392bcb..f63b274 100755
--- a/pybal/bgpfailover.py
+++ b/pybal/bgpfailover.py
@@ -67,11 +67,21 @@
 else:
 BGPFailover.peerings.append(self.bgpPeering)
 reactor.addSystemEventTrigger('before', 'shutdown', 
self.closeSession, self.bgpPeering)
-try:
-# Try to listen on the BGP port, not fatal if fails
-reactor.listenTCP(bgp.PORT, 
bgp.BGPServerFactory({self.bgpPeering.peerAddr: self.bgpPeering}))
-except Exception:
-pass
+
+# Bind on the IPs listed in 'bgp_local_ips'. Default to
+# localhost v4 and v6 if no IPs have been specified in the
+# configuration.
+bgp_local_ips = eval(configdict.get('bgp_local_ips', '["", ""]'))
+# Try to listen on the BGP port, not fatal if fails
+for ip in bgp_local_ips:
+try:
+reactor.listenTCP(
+bgp.PORT,
+bgp.BGPServerFactory({self.bgpPeering.peerAddr: 
self.bgpPeering}),
+interface=ip)
+except CannotListenError:
+log.critical("Could not listen for BGP connections on ip " 
+ ip)
+raise
 
 def closeSession(self, peering):
 log.info("Clearing session to {}".format(peering.peerAddr))

-- 
To view, visit https://gerrit.wikimedia.org/r/370894
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0e4a1cae515602eea8ab07bffb61b9009d5a995a
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add BGP.parse{KeepAlive, Notification} test cases

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/356620 )

Change subject: Add BGP.parse{KeepAlive,Notification} test cases
..

Add BGP.parse{KeepAlive,Notification} test cases

Change-Id: I57c71ddd12fe0c9f271debd4dcbda99181025484
---
M pybal/bgp/test/test_bgp.py
1 file changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/20/356620/1

diff --git a/pybal/bgp/test/test_bgp.py b/pybal/bgp/test/test_bgp.py
index 309b2a1..34eba77 100644
--- a/pybal/bgp/test/test_bgp.py
+++ b/pybal/bgp/test/test_bgp.py
@@ -230,3 +230,13 @@
 msgdata = bytearray(bytes(update)[bgp.HDR_LEN:])
 msgdata[0] += 66
 self.assertRaises(Exception, self.proto.parseUpdate, msgdata)
+
+def testParseKeepAlive(self):
+self.assertRaises(bgp.BadMessageLength, self.proto.parseKeepAlive, b' 
')
+
+def testParseNotification(self):
+msg = (struct.pack('!BB', bgp.ERR_MSG_UPDATE,
+bgp.ERR_MSG_UPDATE_MALFORMED_ASPATH) + b"Unit test")
+self.assertEquals(self.proto.parseNotification(msg),
+(bgp.ERR_MSG_UPDATE, bgp.ERR_MSG_UPDATE_MALFORMED_ASPATH, b"Unit 
test"))
+self.assertRaises(bgp.BadMessageLength, self.proto.parseNotification, 
b' ')

-- 
To view, visit https://gerrit.wikimedia.org/r/356620
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I57c71ddd12fe0c9f271debd4dcbda99181025484
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add basic BGP.parseUpdate test case

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/356612 )

Change subject: Add basic BGP.parseUpdate test case
..

Add basic BGP.parseUpdate test case

Change-Id: Ib90d61298f40b6bb0c8ae94c5edd6c56112325bc
---
M pybal/bgp/test/test_bgp.py
1 file changed, 32 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/12/356612/1

diff --git a/pybal/bgp/test/test_bgp.py b/pybal/bgp/test/test_bgp.py
index 4b77ba4..309b2a1 100644
--- a/pybal/bgp/test/test_bgp.py
+++ b/pybal/bgp/test/test_bgp.py
@@ -198,3 +198,35 @@
 mock_method.reset_mock()
 self.proto.parseOpen(self.MSG_DATA_OPEN[bgp.HDR_LEN:])
 mock_method.assert_called_with(bgp.ERR_MSG_OPEN_BAD_BGP_ID)
+
+def testParseUpdate(self):
+# Test empty UPDATE
+update = bgp.BGPUpdateMessage()
+self.assertEquals(self.proto.parseUpdate(bytes(update)[bgp.HDR_LEN:]), 
([], [], []))
+
+# Add withdrawals
+withdrawals = [ip.IPPrefix("192.168.{0}.0/24".format(i)) for i in 
range(100)]
+update.addSomeWithdrawals(set(withdrawals))
+r = self.proto.parseUpdate(bytes(update)[bgp.HDR_LEN:])
+self.assertListEqual(sorted(r[0]), withdrawals)
+self.assertEquals(r[1:], ([], []))
+
+# Add some attributes
+update.addAttributes(BGPUpdateMessageTestCase.attrs)
+r = self.proto.parseUpdate(bytes(update)[bgp.HDR_LEN:])
+self.assertListEqual(sorted(r[0]), withdrawals)
+self.assertEquals(len(r[1]), len(BGPUpdateMessageTestCase.attrs))
+self.assertEquals(r[2:], ([], ))
+
+# ...and some NLRI
+nlri = [ip.IPPrefix("10.{0}.3.0/24".format(i)) for i in range(100)]
+update.addSomeNLRI(set(nlri))
+r = self.proto.parseUpdate(bytes(update)[bgp.HDR_LEN:])
+self.assertListEqual(sorted(r[0]), withdrawals)
+self.assertEquals(len(r[1]), len(BGPUpdateMessageTestCase.attrs))
+self.assertListEqual(sorted(r[2]), nlri)
+
+# Test a malformed message
+msgdata = bytearray(bytes(update)[bgp.HDR_LEN:])
+msgdata[0] += 66
+self.assertRaises(Exception, self.proto.parseUpdate, msgdata)

-- 
To view, visit https://gerrit.wikimedia.org/r/356612
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib90d61298f40b6bb0c8ae94c5edd6c56112325bc
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Fix IPPrefix value comparisons with different packed paddings

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/356611 )

Change subject: Fix IPPrefix value comparisons with different packed paddings
..

Fix IPPrefix value comparisons with different packed paddings

self.prefix is optionally padded up to the maximum length, so
this needs to be taken into account when comparing.

Change-Id: I7c7a7f052b1ab370ec797850e8b1a0090b93ece4
---
M pybal/bgp/ip.py
1 file changed, 6 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/11/356611/1

diff --git a/pybal/bgp/ip.py b/pybal/bgp/ip.py
index a9f207e..0d78475 100644
--- a/pybal/bgp/ip.py
+++ b/pybal/bgp/ip.py
@@ -90,7 +90,9 @@
 
 def __eq__(self, other):
 # FIXME: masked ips
-return isinstance(other, IPPrefix) and self.prefixlen == 
other.prefixlen and self.prefix == other.prefix
+return (isinstance(other, IPPrefix)
+and self.prefixlen == other.prefixlen
+and self.packed(pad=True) == other.packed(pad=True))
 
 def __ne__(self, other):
 return not self.__eq__(other)
@@ -118,6 +120,9 @@
 def _packedMaxLen(self):
 return (self.addressfamily == AFI_INET6 and 16 or 4)
 
+def _packedMinLen(self):
+return -(-self.prefixlen // 8)
+
 def ipToInt(self):
 return reduce(lambda x, y: x * 256 + y, map(ord, self.prefix))
 

-- 
To view, visit https://gerrit.wikimedia.org/r/356611
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7c7a7f052b1ab370ec797850e8b1a0090b93ece4
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add BGP.parseOpen unit test cases

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355795 )

Change subject: Add BGP.parseOpen unit test cases
..

Add BGP.parseOpen unit test cases

Change-Id: I3209fa9fe7b6ac7df18794e8dd8d8209c9419031
---
M pybal/bgp/test/test_bgp.py
1 file changed, 38 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/95/355795/1

diff --git a/pybal/bgp/test/test_bgp.py b/pybal/bgp/test/test_bgp.py
index f7e4b02..4b77ba4 100644
--- a/pybal/bgp/test/test_bgp.py
+++ b/pybal/bgp/test/test_bgp.py
@@ -9,7 +9,7 @@
 
 from .. import ip, bgp
 
-import unittest, mock
+import unittest, mock, struct
 
 from twisted.test import proto_helpers
 from twisted.python.failure import Failure
@@ -86,6 +86,10 @@
 self.assertEquals(self.msg.freeSpace(), bgp.MAX_LEN-len(self.msg))
 
 class BGPTestCase(unittest.TestCase):
+MSG_DATA_OPEN = 
(b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' +
+ 
b'\x00+\x01\x04\xfcX\x00\xb4\x7f\x7f\x7f\x7f\x0e\x02\x0c\x01\x04' +
+ b'\x00\x01\x00\x01\x01\x04\x00\x02\x00\x01')
+
 def setUp(self):
 self.factory = bgp.BGPPeering(myASN=64600, peerAddr='127.0.0.1')
 # FIXME: Should configure this in a better way in bgp.FSM
@@ -137,10 +141,7 @@
 
 def testSendOpen(self):
 self.proto.sendOpen()
-self.assertEqual(self.tr.value(),
-
b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' +
-b'\x00+\x01\x04\xfcX\x00\xb4\x7f\x7f\x7f\x7f\x0e\x02\x0c\x01\x04' +
-b'\x00\x01\x00\x01\x01\x04\x00\x02\x00\x01')
+self.assertEqual(self.tr.value(), self.MSG_DATA_OPEN)
 
 def testSendUpdate(self):
 withdrawals = [ip.IPPrefix('192.168.99.0/24')]
@@ -165,3 +166,35 @@
 self.assertEqual(self.tr.value(),
 
b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' +
 b'\x00#\x03\x03\x0bArbitrary data')
+
+def testParseOpen(self):
+# parseOpen rejects our own bgpId
+msgdata = bytearray(self.MSG_DATA_OPEN[bgp.HDR_LEN:])
+msgdata[5:9] = [1,2,3,4]
+bgpId = ip.IPv4IP('1.2.3.4').ipToInt()
+struct.pack_into('!I', msgdata, 5, bgpId)
+t = self.proto.parseOpen(str(msgdata))
+self.assertEquals(t, (bgp.VERSION, self.factory.myASN, 
self.proto.fsm.holdTime, bgpId))
+
+self.assertRaises(bgp.BadMessageLength, self.proto.parseOpen, 
str(msgdata[:3]))
+
+with mock.patch.object(self.proto.fsm, 'openMessageError') as 
mock_method:
+msgdata[0] = 66
+self.proto.parseOpen(str(msgdata))
+mock_method.assert_called_with(bgp.ERR_MSG_OPEN_UNSUP_VERSION, 
chr(bgp.VERSION))
+
+mock_method.reset_mock()
+msgdata[0] = bgp.VERSION
+msgdata[1:3] = [0, 0]
+self.proto.parseOpen(str(msgdata))
+mock_method.assert_called_with(bgp.ERR_MSG_OPEN_BAD_PEER_AS)
+
+mock_method.reset_mock()
+msgdata[1:3] = [2, 3]
+msgdata[5:9] = [0]*4
+self.proto.parseOpen(str(msgdata))
+mock_method.assert_called_with(bgp.ERR_MSG_OPEN_BAD_BGP_ID)
+
+mock_method.reset_mock()
+self.proto.parseOpen(self.MSG_DATA_OPEN[bgp.HDR_LEN:])
+mock_method.assert_called_with(bgp.ERR_MSG_OPEN_BAD_BGP_ID)

-- 
To view, visit https://gerrit.wikimedia.org/r/355795
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3209fa9fe7b6ac7df18794e8dd8d8209c9419031
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add basic unit tests for protocol BGP send methods

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355445 )

Change subject: Add basic unit tests for protocol BGP send methods
..

Add basic unit tests for protocol BGP send methods

Change-Id: I24500ac658cd1befa3a62e6b35af216da894e46d
---
M pybal/bgp/test/test_bgp.py
1 file changed, 47 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/45/355445/1

diff --git a/pybal/bgp/test/test_bgp.py b/pybal/bgp/test/test_bgp.py
index 4cad2be..f7e4b02 100644
--- a/pybal/bgp/test/test_bgp.py
+++ b/pybal/bgp/test/test_bgp.py
@@ -29,6 +29,11 @@
 self.assertEquals(attr.value, [(2, [])])
 
 class BGPUpdateMessageTestCase(unittest.TestCase):
+attrs = bgp.FrozenAttributeDict(
+[bgp.OriginAttribute(),
+bgp.ASPathAttribute([64600, 64601]),
+bgp.NextHopAttribute("192.0.2.1"),
+bgp.MEDAttribute(100)])
 
 def setUp(self):
 self.msg = bgp.BGPUpdateMessage()
@@ -36,12 +41,6 @@
 self.assertEquals(len(self.msg.msg), 4)
 self.assertEquals(len(self.msg), 23)
 self.assertIn("UPDATE", repr(self.msg))
-
-self.attrs = bgp.FrozenAttributeDict(
-[bgp.OriginAttribute(),
-bgp.ASPathAttribute([64600, 64601]),
-bgp.NextHopAttribute("192.0.2.1"),
-bgp.MEDAttribute(100)])
 
 def testAddSomeWithdrawals(self):
 self.assertEquals(self.msg.addSomeWithdrawals(set()), 0)
@@ -91,6 +90,13 @@
 self.factory = bgp.BGPPeering(myASN=64600, peerAddr='127.0.0.1')
 # FIXME: Should configure this in a better way in bgp.FSM
 self.factory.fsm.allowAutomaticStart = False
+
+# FIXME: Make Factory param
+af = [(bgp.AFI_INET, bgp.SAFI_UNICAST), (bgp.AFI_INET6, 
bgp.SAFI_UNICAST)]
+self.factory.setEnabledAddressFamilies(af)
+# FIXME: make Factory param
+self.factory.bgpId = ip.IPv4IP('127.127.127.127').ipToInt()
+
 self.proto = self.factory.buildProtocol(IPv4Address('TCP', 
'127.0.0.1', 0))
 self.proto.fsm.allowAutomaticStart = False
 self.tr = proto_helpers.StringTransportWithDisconnection()
@@ -99,8 +105,10 @@
 
 def tearDown(self):
 # The BGPPeering factory keeps its own separate FSM
-self.factory.fsm.idleHoldTimer.cancel()
-self.proto.fsm.idleHoldTimer.cancel()
+for fsm in [self.factory.fsm, self.proto.fsm]:
+for timer in (fsm.connectRetryTimer, fsm.holdTimer, 
fsm.keepAliveTimer,
+  fsm.delayOpenTimer, fsm.idleHoldTimer):
+timer.cancel()
 
 def testConnectionLost(self):
 failure = Failure(ConnectionLost("Unit test"))
@@ -126,3 +134,34 @@
 self.assertTrue(self.proto.disconnected)
 self.assertTrue(self.tr.disconnecting or not self.tr.connected)
 mock_method.assert_called()
+
+def testSendOpen(self):
+self.proto.sendOpen()
+self.assertEqual(self.tr.value(),
+
b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' +
+b'\x00+\x01\x04\xfcX\x00\xb4\x7f\x7f\x7f\x7f\x0e\x02\x0c\x01\x04' +
+b'\x00\x01\x00\x01\x01\x04\x00\x02\x00\x01')
+
+def testSendUpdate(self):
+withdrawals = [ip.IPPrefix('192.168.99.0/24')]
+nlri = [ip.IPPrefix('172.24.0.0/17')]
+
+self.proto.sendUpdate(withdrawals, BGPUpdateMessageTestCase.attrs, 
nlri)
+self.assertEqual(self.tr.value()[:19],
+
b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' +
+b'\x00:\x02')
+self.assertEqual(len(self.tr.value()), 58)
+
+def testSendKeepAlive(self):
+self.proto.sendKeepAlive()
+self.assertEqual(self.tr.value(),
+
b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x13\x04')
+
+def testSendNotification(self):
+self.proto.sendNotification(
+bgp.ERR_MSG_UPDATE,
+bgp.ERR_MSG_UPDATE_MALFORMED_ASPATH,
+"Arbitrary data")
+self.assertEqual(self.tr.value(),
+
b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' +
+b'\x00#\x03\x03\x0bArbitrary data')

-- 
To view, visit https://gerrit.wikimedia.org/r/355445
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I24500ac658cd1befa3a62e6b35af216da894e46d
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add bgp.ip unit test cases

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355425 )

Change subject: Add bgp.ip unit test cases
..

Add bgp.ip unit test cases

Completes 100% statement coverage

Change-Id: I0048d5cdfa81401223cbc3782f01dd8f816c1616
---
M pybal/bgp/test/test_ip.py
1 file changed, 23 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/25/355425/1

diff --git a/pybal/bgp/test/test_ip.py b/pybal/bgp/test/test_ip.py
index fc46197..f5c44df 100644
--- a/pybal/bgp/test/test_ip.py
+++ b/pybal/bgp/test/test_ip.py
@@ -74,9 +74,13 @@
 self.assertEquals(ip.IPv6IP('2620:0:862:ed1a::1').packed(pad=True),
 b'\x26\x20\0\0\x08\x62\xed\x1a\0\0\0\0\0\0\0\x01')
 
+def testPackedInit(self):
+self.assertEquals(str(ip.IPv6IP(packed=b'\xFF'*16)), 
':'.join(['']*8))
+
 def testValueError(self):
 with self.assertRaises(ValueError):
 ip.IPv6IP()
+self.assertRaises(ValueError, ip.IPv6IP, '1:2:3:4:5:6:7:8:9')
 
 class IPPrefixTestCase(TestCase):
 
@@ -88,6 +92,7 @@
 self.assertEquals(len(prefix), 32)
 self.assertEquals(prefix, ip.IPv4IP('91.198.174.192'))
 self.assertEquals(str(prefix), '91.198.174.192/32')
+self.assertEquals(repr(prefix), "'91.198.174.192/32'")
 
 prefix = ip.IPPrefix('192.168.1.2/24')
 self.assertEquals(len(prefix), 24)
@@ -109,6 +114,24 @@
 self.assertEquals(len(prefix), 32)
 prefix.mask(len(prefix))
 
+def testIPPrefix(self):
+p1 = ip.IPv4IP('192.0.2.66')
+p2 = ip.IPPrefix(p1)
+self.assertIsNot(p1, p2)
+self.assertEquals(str(p2), '192.0.2.66/32')
+
+def testIPTuple(self):
+ipbstr = b'\x01\x02\x03\x04'
+p = ip.IPPrefix((ipbstr, 32), addressfamily=ip.AFI_INET)
+self.assertEquals(str(p), "1.2.3.4/32")
+self.assertRaises(ValueError, ip.IPPrefix, (ipbstr, 32))
+
+self.assertRaises(ValueError, ip.IPPrefix, (, 128), ip.AFI_INET6)
+
+def testValueError(self):
+self.assertRaises(ValueError, ip.IPPrefix, '1:2:3:4:5:6:7:8:9/128')
+self.assertRaises(ValueError, ip.IPPrefix, None)
+
 def testComparisons(self):
 p1, p2 = ip.IPPrefix('1.2.3.4/8'), ip.IPPrefix('1.2.3.4/16')
 self.assertNotEqual(p1, p2)

-- 
To view, visit https://gerrit.wikimedia.org/r/355425
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0048d5cdfa81401223cbc3782f01dd8f816c1616
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Move BGP tests into a sub package and add to pybal test suite

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355416 )

Change subject: Move BGP tests into a sub package and add to pybal test suite
..

Move BGP tests into a sub package and add to pybal test suite

While the BGP package is shipped with pybal, ensure its (separate)
tests run as part of the pybal test suite.

Change-Id: If6e0b9aa433bfbb61e47404e04347584842d921c
---
A pybal/bgp/test/__init__.py
R pybal/bgp/test/test_bgp.py
R pybal/bgp/test/test_ip.py
M pybal/test/__init__.py
4 files changed, 15 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/16/355416/1

diff --git a/pybal/bgp/test/__init__.py b/pybal/bgp/test/__init__.py
new file mode 100644
index 000..720c357
--- /dev/null
+++ b/pybal/bgp/test/__init__.py
@@ -0,0 +1,12 @@
+# -*- coding: utf-8 -*-
+"""
+  PyBal unit tests
+  
+
+  This module is the main entry point for the unit tests.
+
+"""
+# flake8: noqa
+
+from .test_ip import *
+from .test_bgp import *
diff --git a/pybal/bgp/test_bgp.py b/pybal/bgp/test/test_bgp.py
similarity index 99%
rename from pybal/bgp/test_bgp.py
rename to pybal/bgp/test/test_bgp.py
index 42a734b..9bb2f76 100644
--- a/pybal/bgp/test_bgp.py
+++ b/pybal/bgp/test/test_bgp.py
@@ -7,8 +7,7 @@
 
 """
 
-import ip
-import bgp
+from bgp import ip, bgp
 
 import unittest, mock
 
diff --git a/pybal/bgp/test_ip.py b/pybal/bgp/test/test_ip.py
similarity index 99%
rename from pybal/bgp/test_ip.py
rename to pybal/bgp/test/test_ip.py
index c062f46..ab7b7ba 100644
--- a/pybal/bgp/test_ip.py
+++ b/pybal/bgp/test/test_ip.py
@@ -6,7 +6,7 @@
   This module contains tests for `bgp.ip`.
 
 """
-import ip
+from bgp import ip
 
 from unittest import TestCase
 
diff --git a/pybal/test/__init__.py b/pybal/test/__init__.py
index 5806cf3..b99e444 100644
--- a/pybal/test/__init__.py
+++ b/pybal/test/__init__.py
@@ -13,3 +13,4 @@
 from .test_monitor import *
 from .test_util import *
 from .test_instrumentation import *
+from ..bgp.test import *

-- 
To view, visit https://gerrit.wikimedia.org/r/355416
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If6e0b9aa433bfbb61e47404e04347584842d921c
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add some protocol BGP class test cases

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355415 )

Change subject: Add some protocol BGP class test cases
..

Add some protocol BGP class test cases

Change-Id: I9b5fc5b5cf760cea54c677cf06dfbd82c0dd5e22
---
M pybal/bgp/test_bgp.py
1 file changed, 46 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/15/355415/1

diff --git a/pybal/bgp/test_bgp.py b/pybal/bgp/test_bgp.py
index 3615341..42a734b 100644
--- a/pybal/bgp/test_bgp.py
+++ b/pybal/bgp/test_bgp.py
@@ -10,8 +10,12 @@
 import ip
 import bgp
 
-import unittest
+import unittest, mock
 
+from twisted.test import proto_helpers
+from twisted.python.failure import Failure
+from twisted.internet.error import ConnectionLost
+from twisted.internet.address import IPv4Address, IPv6Address
 
 class AttributeTestCase(unittest.TestCase):
 
@@ -82,3 +86,44 @@
 
 def testFreeSpace(self):
 self.assertEquals(self.msg.freeSpace(), bgp.MAX_LEN-len(self.msg))
+
+class BGPTestCase(unittest.TestCase):
+def setUp(self):
+self.factory = bgp.BGPPeering(myASN=64600, peerAddr='127.0.0.1')
+# FIXME: Should configure this in a better way in bgp.FSM
+self.factory.fsm.allowAutomaticStart = False
+self.proto = self.factory.buildProtocol(IPv4Address('TCP', 
'127.0.0.1', 0))
+self.proto.fsm.allowAutomaticStart = False
+self.tr = proto_helpers.StringTransportWithDisconnection()
+self.tr.protocol = self.proto
+self.assertRaises(AttributeError, self.proto.makeConnection, self.tr)
+
+def tearDown(self):
+# The BGPPeering factory keeps its own separate FSM
+self.factory.fsm.idleHoldTimer.cancel()
+self.proto.fsm.idleHoldTimer.cancel()
+
+def testConnectionLost(self):
+failure = Failure(ConnectionLost("Unit test"))
+
+with mock.patch.object(self.proto.fsm, 'connectionFailed') as 
mock_method:
+self.proto.connectionLost(failure)
+mock_method.assert_called()
+
+mock_method.reset_mock()
+self.proto.disconnected = True
+self.proto.connectionLost(failure)
+mock_method.assert_not_called()
+
+def testDataReceived(self):
+d = b"Unit testing data"
+self.proto.dataReceived(d)
+self.assertIn(d, self.proto.receiveBuffer)
+
+def testCloseConnection(self):
+with mock.patch.object(self.proto.fsm, 'connectionFailed') as 
mock_method:
+self.tr.connected = True
+self.proto.closeConnection()
+self.assertTrue(self.proto.disconnected)
+self.assertTrue(self.tr.disconnecting or not self.tr.connected)
+mock_method.assert_called()

-- 
To view, visit https://gerrit.wikimedia.org/r/355415
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9b5fc5b5cf760cea54c677cf06dfbd82c0dd5e22
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add IPv4IP, IPv6IP and IPPrefix test cases

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355214 )

Change subject: Add IPv4IP, IPv6IP and IPPrefix test cases
..

Add IPv4IP, IPv6IP and IPPrefix test cases

Change-Id: I48dd95d6e4d5194a4d6e90fe5599086a5befaf60
---
M pybal/bgp/test_ip.py
1 file changed, 64 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/14/355214/1

diff --git a/pybal/bgp/test_ip.py b/pybal/bgp/test_ip.py
index ed70930..bbbc429 100644
--- a/pybal/bgp/test_ip.py
+++ b/pybal/bgp/test_ip.py
@@ -18,16 +18,40 @@
 
 self.assertEquals(prefix.addressfamily, ip.AFI_INET)
 self.assertEquals(prefix.prefixlen, 32)
+self.assertEquals(len(prefix), 32)
 
 def testPrefixInt(self):
 prefix = ip.IPv4IP(2130706433)
 self.assertEquals(prefix.prefixlen, 32)
+self.assertEquals(len(prefix), 32)
 self.assertEquals(str(prefix), '127.0.0.1')
 
 def testPrefixOctets(self):
 prefix = ip.IPv4IP((0x7f, 0x0, 0x0, 0x1))
 self.assertEquals(prefix.prefixlen, 32)
+self.assertEquals(len(prefix), 32)
 self.assertEquals(str(prefix), '127.0.0.1')
+
+def testComparisons(self):
+p1, p2 = ip.IPv4IP('127.0.0.1'), ip.IPv4IP('127.0.0.2')
+self.assertNotEqual(p1, p2)
+self.assertLess(p1, p2)
+self.assertEquals(p1, ip.IPv4IP(2130706433))
+self.assertLessEqual(p1, p2)
+self.assertLessEqual(p1, ip.IPv4IP(2130706433))
+self.assertNotEqual(hash(p1), hash(p2))
+
+def testIPToInt(self):
+prefix = ip.IPv4IP('127.0.0.1')
+self.assertEquals(prefix.ipToInt(), 2130706433)
+
+def testNetmask(self):
+prefix = ip.IPv4IP('127.0.0.1')
+self.assertEquals(prefix.netmask(), -1)
+
+def testPacked(self):
+prefix = ip.IPv4IP('127.0.0.1')
+self.assertEquals(prefix.packed(pad=True), b'\x7F\0\0\x01')
 
 class IPv6IPTestCase(TestCase):
 
@@ -36,7 +60,47 @@
 
 self.assertEquals(prefix.addressfamily, ip.AFI_INET6)
 self.assertEquals(prefix.prefixlen, 128)
+self.assertEquals(len(prefix), 128)
+self.assertEquals(str(prefix), '2620:0:862:ed1a:0:0:0:1')
+
+def testComparisons(self):
+p1, p2 = ip.IPv6IP('fe80::1'), ip.IPv6IP('2620:0:862:ed1a::1')
+self.assertNotEqual(p1, p2)
+self.assertGreater(p1, p2)
+self.assertGreaterEqual(p1, p2)
+self.assertNotEqual(hash(p1), hash(p2))
+
+def testPacked(self):
+self.assertEquals(ip.IPv6IP('2620:0:862:ed1a::1').packed(pad=True),
+b'\x26\x20\0\0\x08\x62\xed\x1a\0\0\0\0\0\0\0\x01')
 
 def testValueError(self):
 with self.assertRaises(ValueError):
 ip.IPv6IP()
+
+class IPPrefixTestCase(TestCase):
+
+def testPrefixStr(self):
+prefix = ip.IPPrefix('91.198.174.192/32')
+
+self.assertEquals(prefix.addressfamily, ip.AFI_INET)
+self.assertEquals(prefix.prefixlen, 32)
+self.assertEquals(len(prefix), 32)
+self.assertEquals(prefix, ip.IPv4IP('91.198.174.192'))
+self.assertEquals(str(prefix), '91.198.174.192/32')
+
+prefix = ip.IPPrefix('192.168.1.2/24')
+self.assertEquals(len(prefix), 24)
+prefix.mask(len(prefix))
+self.assertEquals(prefix, ip.IPPrefix('192.168.1.0/24'))
+
+prefix = ip.IPPrefix('2620:0:863::/46')
+self.assertEquals(len(prefix), 46)
+self.assertEquals(str(prefix), '2620:0:863:0:0:0:0:0/46')
+
+def testComparisons(self):
+p1, p2 = ip.IPPrefix('1.2.3.4/8'), ip.IPPrefix('1.2.3.4/16')
+self.assertNotEqual(p1, p2)
+self.assertLess(p1, p2)
+self.assertLessEqual(p1, p2)
+self.assertNotEqual(hash(p1), hash(p2))

-- 
To view, visit https://gerrit.wikimedia.org/r/355214
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I48dd95d6e4d5194a4d6e90fe5599086a5befaf60
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Fix IPPrefix IPv6 string padding bug

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355213 )

Change subject: Fix IPPrefix IPv6 string padding bug
..

Fix IPPrefix IPv6 string padding bug

When an IPv6 prefix starting or ending with '::' would be passed
to the IPPrefix constructor, self.prefix was padded incorrectly.

Change-Id: I3173d52cf67f2220b2d63fc78f1fa8761a184a4c
---
M pybal/bgp/ip.py
1 file changed, 8 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/13/355213/1

diff --git a/pybal/bgp/ip.py b/pybal/bgp/ip.py
index cf01579..444da9c 100644
--- a/pybal/bgp/ip.py
+++ b/pybal/bgp/ip.py
@@ -58,6 +58,12 @@
 self.prefix = b"".join([chr(int(o)) for o in 
prefix.split('.')])
 elif self.addressfamily == AFI_INET6:
 self.prefix = bytearray()
+# If prefix starts or ends with :: then .split() returns two
+# empty strings
+if prefix.startswith('::'):
+prefix = '0' + prefix
+elif prefix.endswith('::'):
+prefix = prefix + '0'
 hexlist = prefix.split(":")
 if len(hexlist) > 8:
 raise ValueError()
@@ -66,8 +72,7 @@
 if hexstr is not "":
 self.prefix += struct.pack('!H', int(hexstr, 16))
 else:
-zeroCount = 8 - len(hexlist) + 1
-self.prefix += struct.pack('!%dH' % zeroCount, *((0,) 
* zeroCount))
+self.prefix.extend('\0\0' * (8 - len(hexlist) + 1))
 self.prefix = bytes(self.prefix)
 
 self.prefixlen = int(prefixlen)
@@ -130,7 +135,7 @@
 
 def packed(self, pad=False):
 if pad:
-return self.prefix + '\0' * (self._packedMaxLen() - 
len(self.prefix))
+return self.prefix.ljust(self._packedMaxLen(), '\0')
 else:
 return self.prefix
 

-- 
To view, visit https://gerrit.wikimedia.org/r/355213
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3173d52cf67f2220b2d63fc78f1fa8761a184a4c
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add GPLv2 header to bgp/ip.py

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355152 )

Change subject: Add GPLv2 header to bgp/ip.py
..

Add GPLv2 header to bgp/ip.py

Change-Id: Ibe8ff8e7e81f8d0abe19afd85c09262bca9e9143
---
M pybal/bgp/ip.py
1 file changed, 13 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/52/355152/1

diff --git a/pybal/bgp/ip.py b/pybal/bgp/ip.py
index e63c813..cf01579 100644
--- a/pybal/bgp/ip.py
+++ b/pybal/bgp/ip.py
@@ -1,6 +1,19 @@
 # ip.py
 # Copyright (c) 2007 by Mark Bergsma 
 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+
 import struct
 
 # Constants

-- 
To view, visit https://gerrit.wikimedia.org/r/355152
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibe8ff8e7e81f8d0abe19afd85c09262bca9e9143
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add BGPUpdateMessage attribute method test cases

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/355123 )

Change subject: Add BGPUpdateMessage attribute method test cases
..

Add BGPUpdateMessage attribute method test cases

Change-Id: Id0f790b04e25ffec7ad4acc263598af3d0f07d6d
---
M pybal/bgp/test_bgp.py
1 file changed, 18 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/23/355123/1

diff --git a/pybal/bgp/test_bgp.py b/pybal/bgp/test_bgp.py
index 9db5ff6..a5aba1c 100644
--- a/pybal/bgp/test_bgp.py
+++ b/pybal/bgp/test_bgp.py
@@ -31,6 +31,13 @@
 self.msg = bgp.BGPUpdateMessage()
 self.assertEquals(self.msg.msgLenOffset, 16)
 self.assertEquals(len(self.msg.msg), 4)
+self.assertEquals(len(self.msg), 23)
+
+self.attrs = bgp.FrozenAttributeDict(
+[bgp.OriginAttribute(),
+bgp.ASPathAttribute([64600, 64601]),
+bgp.NextHopAttribute("192.0.2.1"),
+bgp.MEDAttribute(100)])
 
 def testAddSomeWithdrawals(self):
 self.assertEquals(self.msg.addSomeWithdrawals(set()), 0)
@@ -45,3 +52,14 @@
 # Not all prefixes will fit within maxLen
 self.assertEquals(self.msg.addSomeWithdrawals(prefixset), 813)
 self.assertEquals(len(prefixset), 211)
+
+def testAddAttributes(self):
+self.msg.addAttributes(bgp.FrozenAttributeDict({}))
+self.assertEqual(len(self.msg), 23)
+self.msg.addAttributes(self.attrs)
+self.assertEqual(len(self.msg), 50)
+
+def testClearAttributes(self):
+self.msg.addAttributes(self.attrs)
+self.msg.clearAttributes()
+self.assertEquals(len(self.msg), 23)

-- 
To view, visit https://gerrit.wikimedia.org/r/355123
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id0f790b04e25ffec7ad4acc263598af3d0f07d6d
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add GPLv2 license header to bgp.py

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354955 )

Change subject: Add GPLv2 license header to bgp.py
..

Add GPLv2 license header to bgp.py

Change-Id: I21ae050bc40dfc360e1ff9a73d9586ffccdb905c
---
M pybal/bgp.py
1 file changed, 14 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/55/354955/1

diff --git a/pybal/bgp.py b/pybal/bgp.py
index bf04d03..caf2926 100644
--- a/pybal/bgp.py
+++ b/pybal/bgp.py
@@ -1,5 +1,18 @@
 # bgp.py
-# Copyright (c) 2007 by Mark Bergsma 
+# Copyright (c) 2007-2017 by Mark Bergsma 
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
 
 """
 A (partial) implementation of the BGP 4 protocol (RFC4271).

-- 
To view, visit https://gerrit.wikimedia.org/r/354955
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I21ae050bc40dfc360e1ff9a73d9586ffccdb905c
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Allow for withdrawals and NLRI to be sent in the same UPDATE

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354723 )

Change subject: Allow for withdrawals and NLRI to be sent in the same UPDATE
..

Allow for withdrawals and NLRI to be sent in the same UPDATE

For inet-unicast UPDATEs, allow withdrawn routes and new NLRI
to be shared within the same packet.

Change-Id: I7d67f908f53f1c973e9a8751ded0bf9ca5702132
---
M pybal/bgp.py
1 file changed, 28 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/23/354723/1

diff --git a/pybal/bgp.py b/pybal/bgp.py
index 6bd179e..de8c4d8 100644
--- a/pybal/bgp.py
+++ b/pybal/bgp.py
@@ -1520,6 +1520,15 @@
 self.attrCount += added
 return added
 
+def clearAttributes(self):
+"""
+Removes all previously added attributes from the packet.
+"""
+
+del self.msg[2][2:]
+self.msg[0:2] = 0, 0
+self._updateMsgLen()
+
 def addSomeNLRI(self, nlriSet):
 """
 Incrementally adds as many nlri to the UPDATE message as will
@@ -2500,16 +2509,32 @@
 withdrawalPrefixSet = set([w.prefix for w in withdrawals])
 adPrefixSet = set([ad.prefix for ad in advertisements])
 
+bgpupdate = BGPUpdateMessage()
 # Start with withdrawals, if there are any
 while len(withdrawals) > 0:
-bgpupdate = BGPUpdateMessage()
 prefixesAdded = 
bgpupdate.addSomeWithdrawals(withdrawalPrefixSet)
 if prefixesAdded == 0:
 raise ValueError("Could not add any withdrawals")
+if len(withdrawals) > 0:
+# We overflowed the packet
+self.estabProtocol.sendMessage(bgpupdate)
+bgpupdate = BGPUpdateMessage()
+
+# Attempt to add all attributes and (some) NLRI to the existing
+# packet, to optimize for the common case of small updates
+try:
+bgpupdate.addAttributes(attributes)
+except ValueError:
+# Alas, didn't fit. Just send out.
+self.estabProtocol.sendMessage(bgpupdate)
+else:
+prefixesAdded = bgpupdate.addSomeNLRI(adPrefixSet)
+if prefixesAdded == 0:
+# Nevermind
+bgpupdate.clearAttributes()
 self.estabProtocol.sendMessage(bgpupdate)
 
-# Start with a clean slate, RFC4760 forbids sending the same
-# prefix in withdrawals and any NLRI or MPReachNLRI/MPUnreachNLRI
+# Start with a clean slate
 while len(adPrefixSet) > 0:
 bgpupdate = BGPUpdateMessage()
 # For inet-unicast, we need to add the complete set of

-- 
To view, visit https://gerrit.wikimedia.org/r/354723
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7d67f908f53f1c973e9a8751ded0bf9ca5702132
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Use a bytearray to build IPPrefix

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354711 )

Change subject: Use a bytearray to build IPPrefix
..

Use a bytearray to build IPPrefix

bytearray() should offer better performance, and bytes/bytearrays
should aid conversion to Python 3.

Change-Id: I48b7a71f384402e80224a0db34250805944d158a
---
M pybal/bgp.py
1 file changed, 4 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/11/354711/1

diff --git a/pybal/bgp.py b/pybal/bgp.py
index 33650b9..c7179da 100644
--- a/pybal/bgp.py
+++ b/pybal/bgp.py
@@ -202,16 +202,16 @@
 raise ValueError()
 else:
 # Assume prefix is a sequence of octets
-self.prefix = "".join(map(chr, prefix))
+self.prefix = b"".join(map(chr, prefix))
 elif type(ipprefix) is str:
 # textual form
 prefix, prefixlen = ipprefix.split('/')
 self.addressfamily = addressfamily or (':' in prefix and AFI_INET6 
or AFI_INET)
 
 if self.addressfamily == AFI_INET:
-self.prefix = "".join([chr(int(o)) for o in prefix.split('.')])
+self.prefix = b"".join([chr(int(o)) for o in 
prefix.split('.')])
 elif self.addressfamily == AFI_INET6:
-self.prefix = ""
+self.prefix = bytearray()
 hexlist = prefix.split(":")
 if len(hexlist) > 8:
 raise ValueError()
@@ -222,6 +222,7 @@
 else:
 zeroCount = 8 - len(hexlist) + 1
 self.prefix += struct.pack('!%dH' % zeroCount, *((0,) 
* zeroCount))
+self.prefix = bytes(self.prefix)
 
 self.prefixlen = int(prefixlen)
 else:

-- 
To view, visit https://gerrit.wikimedia.org/r/354711
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I48b7a71f384402e80224a0db34250805944d158a
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Adapt NaiveBGPPeering to support UPDATE message overflow

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354686 )

Change subject: Adapt NaiveBGPPeering to support UPDATE message overflow
..

Adapt NaiveBGPPeering to support UPDATE message overflow

Add {MPReachNLRI,MPUnreachNLRI}.addSomePrefixes to fit
as many prefixes into these attributes as will fit within
maxLen.

Adapt BGPUpdateMessage add methods to addSomeWithdrawals,
addSomeAttributes, addSomeNLRI to fit as many prefixes or
attributes as will fit the (remainder) of the BGP UPDATE
packet, BGPMessage.freeSpace().
BGPUpdateMessage.addAttributes is retained; it adds all
attributes or throws an exception if it doesn't fit.

BGP.sendMessage() is added to send a previously crafted
BGPMessage instance.

BGP.encodeSomePrefixes() is added to fit as many prefixes
into a bytearray as will fit within maxLen.

NaiveBGPPeering._sendUpdates() and support methods have
been rewritten to support UPDATE message overflows.

Change-Id: I29177f9f5ff806fee77e1f2c6dc361900d4b1064
---
M pybal/bgp.py
1 file changed, 252 insertions(+), 57 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/86/354686/1

diff --git a/pybal/bgp.py b/pybal/bgp.py
index e430cdd..33650b9 100644
--- a/pybal/bgp.py
+++ b/pybal/bgp.py
@@ -761,6 +761,24 @@
 """
 self.value = self.value[0:3] + (list(prefixes), )
 
+def addSomePrefixes(self, prefixSet, maxLen):
+"""
+Add as many prefixes from prefixSet as will fit within maxLen,
+removing added prefixes from prefixSet.
+Returns the number of prefixes added.
+"""
+# FIXME: Optimize to prevent double encoding
+
+bArray = bytearray()
+origPrefixSet = frozenset(prefixSet)
+count = BGP.encodeSomePrefixes(
+prefixSet=prefixSet,
+bArray=bArray,
+offset=0,
+maxLen=maxLen)
+self.value = self.value[0:3] + (list(origPrefixSet - prefixSet), )
+return count
+
 class MPUnreachNLRIAttribute(BaseMPAttribute):
 name = 'MP Unreach NLRI'
 typeCode = ATTR_TYPE_MP_UNREACH_NLRI
@@ -785,7 +803,7 @@
 
 def encode(self):
 afi, safi, nlri = self.value
-encodedNLRI = BGP.encoddePrefixes(nlri)
+encodedNLRI = BGP.encodedPrefixes(nlri)
 length = 3 + len(encodedNLRI)
 
 return struct.pack('!BBHHB', self.flags(), self.typeCode, length, afi, 
safi) + encodedNLRI
@@ -795,6 +813,24 @@
 Adds a (copied) list of prefixes to this attribute's NLRI
 """
 self.value = self.value[0:2] + (list(prefixes), )
+
+def addSomePrefixes(self, prefixSet, maxLen):
+"""
+Add as many prefixes from prefixSet as will fit within maxLen,
+removing added prefixes from prefixSet.
+Returns the number of prefixes added.
+"""
+# FIXME: Optimize to prevent double encoding
+
+bArray = bytearray()
+origPrefixSet = frozenset(prefixSet)
+count = BGP.encodeSomePrefixes(
+prefixSet=prefixSet,
+bArray=bArray,
+offset=0,
+maxLen=maxLen)
+self.value = self.value[0:2] + (list(origPrefixSet - prefixSet), )
+return count
 
 def __hash__(self):
 return hash(self.value[0:2]) ^ hash(frozenset(self.value[2]))
@@ -1391,7 +1427,28 @@
 len(self),
 self.msgtype)
 
-def _append(self, buf, data, lenOffset=None):
+def freeSpace(self):
+"""
+Returns the available free space in the packet
+"""
+return MAX_LEN - len(self)
+
+def _updateRecordLen(self, buf, offset=None, length=None):
+"""
+Updates the length of the variable length field at the given
+offset in the provided buffer
+"""
+
+struct.pack_into('!H', buf, offset, length)
+
+def _updateMsgLen(self):
+"""
+Updates the length of the message in the message header
+"""
+
+struct.pack_into('!H', self.msg[0], self.msgLenOffset, len(self))
+
+def _appendAll(self, buf, data, lenOffset=None):
 """
 Appends variable records (e.g. NLRI, attributes) and updates
 the variable length and total message size.
@@ -1400,9 +1457,9 @@
 newSize = len(self) + len(data)
 if newSize <= MAX_LEN:
 buf.extend(data)
-if lenOffset:
-struct.pack_into('!H', buf, lenOffset, len(buf) - lenOffset - 
2)
-struct.pack_into('!H', self.msg[0], self.msgLenOffset, newSize)
+if lenOffset is not None:
+self._updateRecordLen(buf, lenOffset, len(buf) - lenOffset - 2)
+self._updateMsgLen()
 else:
 raise ValueError("New message size %s would exceed MAX_LEN %d" %
 (newSize, MAX_LEN))
@@ -1422,29 +1479,61 @@
 + ", [%d:%d] NLRI>" % 

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Use a bytearray to encode prefixes in BGP.encodePrefixes

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354685 )

Change subject: Use a bytearray to encode prefixes in BGP.encodePrefixes
..

Use a bytearray to encode prefixes in BGP.encodePrefixes

Converts back to str before returning, as some depending code
(including Twisted) can't handle bytearrays yet.

Change-Id: I0ccc231cce89ebcaf27f6c340ffea59dec35ed67
---
M pybal/bgp.py
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/85/354685/1

diff --git a/pybal/bgp.py b/pybal/bgp.py
index 5f186f0..e430cdd 100644
--- a/pybal/bgp.py
+++ b/pybal/bgp.py
@@ -1886,16 +1886,16 @@
 def encodePrefixes(prefixes):
 """Encodes a list of IPPrefix"""
 
-prefixData = ""
+prefixData = bytearray()
 for prefix in prefixes:
 octetLen, remainder = len(prefix) / 8, len(prefix) % 8
 if remainder > 0:
 # prefix length doesn't fall on octet boundary
 octetLen += 1
 
-prefixData += struct.pack('!B', len(prefix)) + 
prefix.packed()[:octetLen]
+prefixData.extend(struct.pack('!B', len(prefix)) + 
prefix.packed()[:octetLen])
 
-return prefixData
+return str(prefixData)
 
 
 class BGPFactory(protocol.Factory):

-- 
To view, visit https://gerrit.wikimedia.org/r/354685
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0ccc231cce89ebcaf27f6c340ffea59dec35ed67
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Create new BGP message classes for incremental construction

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354684 )

Change subject: Create new BGP message classes for incremental construction
..

Create new BGP message classes for incremental construction

The existing construct* methods in the BGP class are simple
and work well, but are not really flexible or optimal for large
messages such as UPDATEs.

Introduce a new ancestor class BGPMessage and derived BGPUpdateMessage
for efficient and incremental package construction.

Also move BGP.constructHeader to BGPMessage.prependHeader as a static
method.

Change-Id: I542ca34c09de45005b083c3f5f7797653f664989
---
M pybal/bgp.py
1 file changed, 101 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/84/354684/1

diff --git a/pybal/bgp.py b/pybal/bgp.py
index 10787bc..5f186f0 100644
--- a/pybal/bgp.py
+++ b/pybal/bgp.py
@@ -1348,7 +1348,103 @@
 self.protocol.closeConnection()
 # Remove from connections list
 if self.bgpPeering: self.bgpPeering.connectionClosed(self.protocol)
-
+
+class BGPMessage(object):
+msgtype = None
+msgLenOffset = 16
+
+def __init__(self):
+self.msg = (bytearray(HDR_LEN), )
+self.constructHeader()
+
+def __repr__(self):
+msgType = {
+MSG_OPEN: "OPEN",
+MSG_UPDATE: "UPDATE",
+MSG_NOTIFICATION: "NOTIFICATION",
+MSG_KEEPALIVE: "KEEPALIVE",
+
+}.get(self.msgtype, "Invalid")
+return "" % (msgType, len(self))
+
+def __str__(self):
+return "".join([str(part) for part in self.msg])
+
+def __len__(self):
+return sum([len(part) for part in self.msg])
+
+def __getitem__(self, i):
+return buffer(self.msg[i])
+
+@staticmethod
+def prependHeader(message, type):
+"""Prepends the mandatory header to a constructed BGP message"""
+
+return struct.pack('!16sHB',
+   chr(255)*16,
+   len(message)+HDR_LEN,
+   type) + message
+
+def constructHeader(self, buffer=None):
+struct.pack_into('!16sHB', (buffer or self.msg[0]), 0,
+chr(255)*16,
+len(self),
+self.msgtype)
+
+def _append(self, buf, data, lenOffset=None):
+"""
+Appends variable records (e.g. NLRI, attributes) and updates
+the variable length and total message size.
+"""
+
+newSize = len(self) + len(data)
+if newSize <= MAX_LEN:
+buf.extend(data)
+if lenOffset:
+struct.pack_into('!H', buf, lenOffset, len(buf) - lenOffset - 
2)
+struct.pack_into('!H', self.msg[0], self.msgLenOffset, newSize)
+else:
+raise ValueError("New message size %s would exceed MAX_LEN %d" %
+(newSize, MAX_LEN))
+
+class BGPUpdateMessage(BGPMessage):
+msgtype = MSG_UPDATE
+
+def __init__(self):
+super(BGPUpdateMessage, self).__init__()
+self.msg = (self.msg[0], bytearray(2), bytearray(2), bytearray())
+self.withdrCount, self.attrCount, self.nlriCount = 0, 0, 0
+
+def __repr__(self):
+return (super(BGPUpdateMessage, self).__repr__()[:-1]
++ ", [%d:%d] withdrawals" % (self.withdrCount, len(self.msg[1]))
++ ", [%d:%d] attributes" % (self.attrCount, len(self.msg[2]))
++ ", [%d:%d] NLRI>" % (self.nlriCount, len(self.msg[3])))
+
+
+def addWithdrawals(self, withdrawals):
+"""
+Incrementally adds withdrawals to the UPDATE message.
+Does not attempt to remove duplicates.
+"""
+self._append(self.msg[1], BGP.encodePrefixes(withdrawals))
+self.withdrCount += len(withdrawals)
+
+def addAttributes(self, attributes):
+"""
+Incrementally adds NLRI attributes to the UPDATE message.
+"""
+
+self._append(self.msg[2], BGP.encodeAttributes(attributes))
+self.attrCount += len(attributes)
+
+def addNLRI(self, nlri):
+"""
+Incrementally adds NLRI to the UPDATE message.
+"""
+
+self._append(self.msg[3], BGP.encodePrefixes(nlri))
+self.nlriCount += len(nlri)
 
 class BGP(protocol.Protocol):
 """Protocol class for BGP 4"""
@@ -1450,14 +1546,6 @@
 
 self.transport.write(self.constructNotification(error, suberror, data))
 
-def constructHeader(self, message, type):
-"""Prepends the mandatory header to a constructed BGP message"""
-
-return struct.pack('!16sHB',
-   chr(255)*16,
-   len(message)+19,
-   type) + message
-
 def constructOpen(self):
 """Constructs a BGP Open message"""
 
@@ -1472,7 +1560,7 @@
   self.fsm.holdTime,
   

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Cleanup whitespace

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/354683 )

Change subject: Cleanup whitespace
..

Cleanup whitespace

Change-Id: Ib5cfef66d06ec7c05d5eaa89cd30a0c83b8f
---
M pybal/bgp.py
1 file changed, 422 insertions(+), 422 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/83/354683/1

diff --git a/pybal/bgp.py b/pybal/bgp.py
index 47d21c4..10787bc 100644
--- a/pybal/bgp.py
+++ b/pybal/bgp.py
@@ -122,11 +122,11 @@
 class NotificationSent(BGPException):
 def __init__(self, protocol, error, suberror, data=''):
 BGPException.__init__(self, protocol)
-
+
 self.error = error
 self.suberror = suberror
 self.data = data
-
+
 def __str__(self):
 return repr((self.error, self.suberror, self.data))
 
@@ -136,7 +136,7 @@
 class AttributeException(BGPException):
 def __init__(self, suberror, data=''):
 BGPException.__init__(self)
-
+
 self.error = ERR_MSG_UPDATE
 self.suberror = suberror
 self.data = data
@@ -147,27 +147,27 @@
 """
 Interface for notifications from the BGP protocol / FSM
 """
-
+
 def notificationSent(self, protocol, error, suberror, data):
 """
 Called when a BGP Notification message was sent.
 """
-
+
 def connectionClosed(self, protocol):
 """
 Called when the BGP connection has been closed (in error or not).
 """
-
+
 def completeInit(self, protocol):
 """
 Called when BGP resources should be initialized.
 """
-
+
 def sessionEstablished(self, protocol):
 """
 Called when the BGP session has reached the Established state
 """
-
+
 def connectRetryEvent(self, protocol):
 """
 Called when the connect-retry timer expires. A new connection should
@@ -177,17 +177,17 @@
 # TODO: Replace by some better third party classes or rewrite
 class IPPrefix(object):
 """Class that represents an IP prefix"""
-
+
 def __init__(self, ipprefix, addressfamily=None):
 self.prefix = None # packed ip string
-
+
 if isinstance(ipprefix, IPPrefix):
 self.prefix, self.prefixlen, self.addressfamily = ipprefix.prefix, 
ipprefix.prefixlen, ipprefix.addressfamily
 elif type(ipprefix) is tuple:
 # address family must be specified
 if not addressfamily:
 raise ValueError()
-
+
 self.addressfamily = addressfamily
 
 prefix, self.prefixlen = ipprefix
@@ -221,51 +221,51 @@
 self.prefix += struct.pack('!H', int(hexstr, 16))
 else:
 zeroCount = 8 - len(hexlist) + 1
-self.prefix += struct.pack('!%dH' % zeroCount, *((0,) 
* zeroCount))  
-
+self.prefix += struct.pack('!%dH' % zeroCount, *((0,) 
* zeroCount))
+
 self.prefixlen = int(prefixlen)
 else:
 raise ValueError()
-
+
 def __repr__(self):
 return repr(str(self))
-
+
 def __str__(self):
 if self.addressfamily == AFI_INET:
 return '.'.join([str(ord(o)) for o in self.packed(pad=True)]) + 
'/%d' % self.prefixlen
 elif self.addressfamily == AFI_INET6:
-return ':'.join([hex(o)[2:] for o in struct.unpack('!8H', 
self.packed(pad=True))]) + '/%d' % self.prefixlen   
-
+return ':'.join([hex(o)[2:] for o in struct.unpack('!8H', 
self.packed(pad=True))]) + '/%d' % self.prefixlen
+
 def __eq__(self, other):
 # FIXME: masked ips
 return isinstance(other, IPPrefix) and self.prefixlen == 
other.prefixlen and self.prefix == other.prefix
-
+
 def __ne__(self, other):
 return not self.__eq__(other)
-
+
 def __lt__(self, other):
 return self.prefix < other.prefix or \
 (self.prefix == other.prefix and self.prefixlen < other.prefixlen)
-
+
 def __le__(self, other):
 return self.__lt__(other) or self.__eq__(other)
-
+
 def __gt__(self, other):
 return self.prefix > other.prefix or \
 (self.prefix == other.prefix and self.prefixlen > other.prefixlen)
-
+
 def __ge__(self, other):
 return self.__gt__(other) or self.__eq__(other)
-
+
 def __hash__(self):
 return hash(self.prefix) ^ hash(self.prefixlen)
-
+
 def __len__(self):
 return self.prefixlen
 
 def _packedMaxLen(self):
 return (self.addressfamily == AFI_INET6 and 16 or 4)
-
+
 def ipToInt(self):
 return reduce(lambda x, y: x * 256 + y, map(ord, self.prefix))
 
@@ -275,12 +275,12 @@
 def mask(self, prefixlen, shorten=False):
 # DEBUG
 assert len(self.prefix) 

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add pyenv and pydev config files to .gitignore

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/353988 )

Change subject: Add pyenv and pydev config files to .gitignore
..

Add pyenv and pydev config files to .gitignore

Change-Id: I2abb6d70651f833562d5bccd607aa1c0b4569684
---
M .gitignore
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/88/353988/1

diff --git a/.gitignore b/.gitignore
index f322c84..01a68e7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,3 +5,7 @@
 _trial_temp
 build
 dist
+.project
+.pydevproject
+.settings
+.python-version

-- 
To view, visit https://gerrit.wikimedia.org/r/353988
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2abb6d70651f833562d5bccd607aa1c0b4569684
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set hhvm.server.request_timeout_seconds to 60s

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/326144 )

Change subject: Set hhvm.server.request_timeout_seconds to 60s
..

Set hhvm.server.request_timeout_seconds to 60s

We've seen various indications that PHP's max_execution_timeout
does not work in FCGI context. This is supported by
https://github.com/facebook/hhvm/issues/4209

Setting hhvm.server.request_timeout_seconds in the fcgi server
config should make the limit of 60s effective again (however
the request queue as well, before execution starts). It doesn't
solve the issue of setting per-request timeouts in PHP.

Bug: T97192
Bug: T149421
Change-Id: Ic41979c3624baa464080e40633a22897e39514e6
---
M modules/hhvm/manifests/init.pp
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/44/326144/1

diff --git a/modules/hhvm/manifests/init.pp b/modules/hhvm/manifests/init.pp
index 133ac3b..1fbbbfc 100644
--- a/modules/hhvm/manifests/init.pp
+++ b/modules/hhvm/manifests/init.pp
@@ -173,6 +173,7 @@
 enable => true,
 ttl=> to_seconds('5 minutes'),
 },
+request_timeout_seconds => 60,
 },
 },
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/326144
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic41979c3624baa464080e40633a22897e39514e6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dns[master]: Reflect new FPC3 ports after cr1-/cr2-eqiad FPC5 decommissio...

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Reflect new FPC3 ports after cr1-/cr2-eqiad FPC5 decommissioning
..


Reflect new FPC3 ports after cr1-/cr2-eqiad FPC5 decommissioning

Bug: T149196
Change-Id: I30606173b9d0ac3fa393a10fec9666fe51060704
---
M templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
M templates/153.80.208.in-addr.arpa
M templates/154.80.208.in-addr.arpa
3 files changed, 4 insertions(+), 4 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa 
b/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
index 3a006f9..b8740f4 100644
--- a/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
+++ b/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
@@ -256,7 +256,7 @@
 
 $ORIGIN 7.0.e.f.{{ zonename }}.
 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   xe-5-0-1.cr2-codfw.wikimedia.org.
-2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   xe-5-2-3.cr2-eqiad.wikimedia.org.
+2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   xe-3-2-3.cr2-eqiad.wikimedia.org.
 
 ; Loopback IPs
 
diff --git a/templates/153.80.208.in-addr.arpa 
b/templates/153.80.208.in-addr.arpa
index fc79040..1f468a5 100644
--- a/templates/153.80.208.in-addr.arpa
+++ b/templates/153.80.208.in-addr.arpa
@@ -91,7 +91,7 @@
 
 ; 208.80.153.204/31 (cr2-codfw <--> cr2-eqiad)
 204 1H  IN PTR  xe-5-0-1.cr2-codfw.wikimedia.org.
-205 1H  IN PTR  xe-5-2-3.cr2-eqiad.wikimedia.org.
+205 1H  IN PTR  xe-3-2-3.cr2-eqiad.wikimedia.org.
 
 ; 208.80.153.206/31 (cr1-codfw <--> mr1-codfw)
 206 1H  IN PTR  xe-5-3-3.cr1-codfw.wikimedia.org.
diff --git a/templates/154.80.208.in-addr.arpa 
b/templates/154.80.208.in-addr.arpa
index 28181bb..694a74e 100644
--- a/templates/154.80.208.in-addr.arpa
+++ b/templates/154.80.208.in-addr.arpa
@@ -141,14 +141,14 @@
 209 1H  IN PTR  xe-1-1-0.cr1-eqord.wikimedia.org.
 
 ; 208.80.154.216/31 (cr1-eqiad <--> pfw-eqiad)
-216 1H  IN PTR  xe-5-0-3.cr1-eqiad.wikimedia.org.
+216 1H  IN PTR  xe-3-3-2.cr1-eqiad.wikimedia.org.
 217 1H  IN PTR  xe-6-0-0.pfw-eqiad.wikimedia.org.
 
 ; 208.80.154.218 - 219 (loopbacks)
 218 1H  IN PTR  pfw-eqiad.wikimedia.org.
 
 ; 208.80.154.220/31 (cr2-eqiad <--> pfw-eqiad)
-220 1H  IN PTR  xe-5-0-3.cr2-eqiad.wikimedia.org.
+220 1H  IN PTR  xe-3-3-2.cr2-eqiad.wikimedia.org.
 221 1H  IN PTR  xe-15-0-0.pfw-eqiad.wikimedia.org.
 
 ; 208.80.154.222/31 (pfw-eqiad <--> pfw-codfw)

-- 
To view, visit https://gerrit.wikimedia.org/r/319617
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I30606173b9d0ac3fa393a10fec9666fe51060704
Gerrit-PatchSet: 4
Gerrit-Project: operations/dns
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/dns[master]: Reflect new FPC3 ports after cr1-/cr2-eqiad FPC5 decommissio...

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319617

Change subject: Reflect new FPC3 ports after cr1-/cr2-eqiad FPC5 decommissioning
..

Reflect new FPC3 ports after cr1-/cr2-eqiad FPC5 decommissioning

Change-Id: I30606173b9d0ac3fa393a10fec9666fe51060704
---
M templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
M templates/153.80.208.in-addr.arpa
M templates/154.80.208.in-addr.arpa
3 files changed, 4 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dns 
refs/changes/17/319617/1

diff --git a/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa 
b/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
index 3a006f9..b8740f4 100644
--- a/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
+++ b/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
@@ -256,7 +256,7 @@
 
 $ORIGIN 7.0.e.f.{{ zonename }}.
 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   xe-5-0-1.cr2-codfw.wikimedia.org.
-2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   xe-5-2-3.cr2-eqiad.wikimedia.org.
+2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   xe-3-2-3.cr2-eqiad.wikimedia.org.
 
 ; Loopback IPs
 
diff --git a/templates/153.80.208.in-addr.arpa 
b/templates/153.80.208.in-addr.arpa
index fc79040..1f468a5 100644
--- a/templates/153.80.208.in-addr.arpa
+++ b/templates/153.80.208.in-addr.arpa
@@ -91,7 +91,7 @@
 
 ; 208.80.153.204/31 (cr2-codfw <--> cr2-eqiad)
 204 1H  IN PTR  xe-5-0-1.cr2-codfw.wikimedia.org.
-205 1H  IN PTR  xe-5-2-3.cr2-eqiad.wikimedia.org.
+205 1H  IN PTR  xe-3-2-3.cr2-eqiad.wikimedia.org.
 
 ; 208.80.153.206/31 (cr1-codfw <--> mr1-codfw)
 206 1H  IN PTR  xe-5-3-3.cr1-codfw.wikimedia.org.
diff --git a/templates/154.80.208.in-addr.arpa 
b/templates/154.80.208.in-addr.arpa
index 28181bb..550c380 100644
--- a/templates/154.80.208.in-addr.arpa
+++ b/templates/154.80.208.in-addr.arpa
@@ -141,14 +141,14 @@
 209 1H  IN PTR  xe-1-1-0.cr1-eqord.wikimedia.org.
 
 ; 208.80.154.216/31 (cr1-eqiad <--> pfw-eqiad)
-216 1H  IN PTR  xe-5-0-3.cr1-eqiad.wikimedia.org.
+216 1H  IN PTR  xe-3-0-3.cr1-eqiad.wikimedia.org.
 217 1H  IN PTR  xe-6-0-0.pfw-eqiad.wikimedia.org.
 
 ; 208.80.154.218 - 219 (loopbacks)
 218 1H  IN PTR  pfw-eqiad.wikimedia.org.
 
 ; 208.80.154.220/31 (cr2-eqiad <--> pfw-eqiad)
-220 1H  IN PTR  xe-5-0-3.cr2-eqiad.wikimedia.org.
+220 1H  IN PTR  xe-3-0-3.cr2-eqiad.wikimedia.org.
 221 1H  IN PTR  xe-15-0-0.pfw-eqiad.wikimedia.org.
 
 ; 208.80.154.222/31 (pfw-eqiad <--> pfw-codfw)

-- 
To view, visit https://gerrit.wikimedia.org/r/319617
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I30606173b9d0ac3fa393a10fec9666fe51060704
Gerrit-PatchSet: 1
Gerrit-Project: operations/dns
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update README.md for BACKPORTS=yes option

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Update README.md for BACKPORTS=yes option
..


Update README.md for BACKPORTS=yes option

Change-Id: Ib07e30b1710c7b22265b9bf9d6ac1ad6481707f4
---
M modules/package_builder/README.md
1 file changed, 9 insertions(+), 0 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/package_builder/README.md 
b/modules/package_builder/README.md
index 794f2c4..445731d 100644
--- a/modules/package_builder/README.md
+++ b/modules/package_builder/README.md
@@ -76,6 +76,15 @@
 The commands above are equivalent and will both build a package for the
 jessie distribution using the Wikimedia apt repository.
 
+Backports repos
+===
+
+Packages from the Debian backports repositories can be used to satisfy
+dependencies as well. To use the backports repository for the distribution
+selected (e.g. jessie-backports), use:
+
+DIST=jessie BACKPORTS=yes pdebuild
+
 git-buildpackage
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/302679
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib07e30b1710c7b22265b9bf9d6ac1ad6481707f4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update README.md for BACKPORTS=yes option

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302679

Change subject: Update README.md for BACKPORTS=yes option
..

Update README.md for BACKPORTS=yes option

Change-Id: Ib07e30b1710c7b22265b9bf9d6ac1ad6481707f4
---
M modules/package_builder/README.md
1 file changed, 9 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/79/302679/1

diff --git a/modules/package_builder/README.md 
b/modules/package_builder/README.md
index 794f2c4..445731d 100644
--- a/modules/package_builder/README.md
+++ b/modules/package_builder/README.md
@@ -76,6 +76,15 @@
 The commands above are equivalent and will both build a package for the
 jessie distribution using the Wikimedia apt repository.
 
+Backports repos
+===
+
+Packages from the Debian backports repositories can be used to satisfy
+dependencies as well. To use the backports repository for the distribution
+selected (e.g. jessie-backports), use:
+
+DIST=jessie BACKPORTS=yes pdebuild
+
 git-buildpackage
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/302679
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib07e30b1710c7b22265b9bf9d6ac1ad6481707f4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Convert back to tabs, rename .prefs file

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Convert back to tabs, rename .prefs file
..


Convert back to tabs, rename .prefs file

The bash <<- operator only works with tabs, not spaces, which caused
the apt preferences file to be indented and fail to work.

Change-Id: I63fae273ac8e12d41e642c7a639e883e03c2aad3
---
M modules/package_builder/templates/D02backports.erb
1 file changed, 8 insertions(+), 8 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 3052229..eb2451d 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -2,14 +2,14 @@
 
 # Avoid running hooks if backports is not specified
 if [ "${BACKPORTS}" = "yes" ]; then
-cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list <<-'EOF'
-deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
-deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
+   cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list 
<<-'EOF'
+   deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
+   deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
 EOF
-cat > /etc/apt/preferences.d/<%= @distribution %>.pref <<-'EOF'
-Package: *
-Pin: release a=<%= @distribution %>-backports
-Pin-Priority: 500
+   cat > /etc/apt/preferences.d/<%= @distribution %>-backports.pref 
<<-'EOF'
+   Package: *
+   Pin: release a=<%= @distribution %>-backports
+   Pin-Priority: 500
 EOF
-apt-get update
+   apt-get update
 fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302670
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I63fae273ac8e12d41e642c7a639e883e03c2aad3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Convert back to tabs, rename .prefs file

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302670

Change subject: Convert back to tabs, rename .prefs file
..

Convert back to tabs, rename .prefs file

The bash <<- operator only works with tabs, not spaces, which caused
the apt preferences file to be indented and fail to work.

Change-Id: I63fae273ac8e12d41e642c7a639e883e03c2aad3
---
M modules/package_builder/templates/D02backports.erb
1 file changed, 8 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/70/302670/1

diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 3052229..eb2451d 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -2,14 +2,14 @@
 
 # Avoid running hooks if backports is not specified
 if [ "${BACKPORTS}" = "yes" ]; then
-cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list <<-'EOF'
-deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
-deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
+   cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list 
<<-'EOF'
+   deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
+   deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
 EOF
-cat > /etc/apt/preferences.d/<%= @distribution %>.pref <<-'EOF'
-Package: *
-Pin: release a=<%= @distribution %>-backports
-Pin-Priority: 500
+   cat > /etc/apt/preferences.d/<%= @distribution %>-backports.pref 
<<-'EOF'
+   Package: *
+   Pin: release a=<%= @distribution %>-backports
+   Pin-Priority: 500
 EOF
-apt-get update
+   apt-get update
 fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302670
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I63fae273ac8e12d41e642c7a639e883e03c2aad3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: C/p error

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: C/p error
..


C/p error

Change-Id: I2c7f07f503561ebbb27235f6a480447942429efd
---
M modules/package_builder/templates/pbuilderrc.erb
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Mark Bergsma: Verified; Looks good to me, approved



diff --git a/modules/package_builder/templates/pbuilderrc.erb 
b/modules/package_builder/templates/pbuilderrc.erb
index 4695e38..5a5fe5d 100644
--- a/modules/package_builder/templates/pbuilderrc.erb
+++ b/modules/package_builder/templates/pbuilderrc.erb
@@ -7,7 +7,7 @@
 # Default to not use wikimedia repos
 WIKIMEDIA=${WIKIMEDIA:-"no"}
 # Default to not use distribution backports
-BACKPORTS=${WIKIMEDIA:-"no"}
+BACKPORTS=${BACKPORTS:-"no"}
 
 # If DIST has been declared like dist-wikimedia, handle it like as if the user
 # wanted to specify WIKIMEDIA="yes" which will satisfy build dependencies from

-- 
To view, visit https://gerrit.wikimedia.org/r/302475
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2c7f07f503561ebbb27235f6a480447942429efd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: C/p error

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302475

Change subject: C/p error
..

C/p error

Change-Id: I2c7f07f503561ebbb27235f6a480447942429efd
---
M modules/package_builder/templates/pbuilderrc.erb
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/75/302475/1

diff --git a/modules/package_builder/templates/pbuilderrc.erb 
b/modules/package_builder/templates/pbuilderrc.erb
index 4695e38..5a5fe5d 100644
--- a/modules/package_builder/templates/pbuilderrc.erb
+++ b/modules/package_builder/templates/pbuilderrc.erb
@@ -7,7 +7,7 @@
 # Default to not use wikimedia repos
 WIKIMEDIA=${WIKIMEDIA:-"no"}
 # Default to not use distribution backports
-BACKPORTS=${WIKIMEDIA:-"no"}
+BACKPORTS=${BACKPORTS:-"no"}
 
 # If DIST has been declared like dist-wikimedia, handle it like as if the user
 # wanted to specify WIKIMEDIA="yes" which will satisfy build dependencies from

-- 
To view, visit https://gerrit.wikimedia.org/r/302475
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2c7f07f503561ebbb27235f6a480447942429efd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Pin $dist-backports at priority 500

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Pin $dist-backports at priority 500
..


Pin $dist-backports at priority 500

Because -backports has priority 100 by default, those packages are
not considered when there's any other version in any other repo.
Set to the same value as the default repo, so highest version nr wins.

Change-Id: I43174931c1086f1760b21bc55384a647d025c570
---
M modules/package_builder/templates/D02backports.erb
1 file changed, 5 insertions(+), 0 deletions(-)

Approvals:
  Mark Bergsma: Verified; Looks good to me, approved



diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 94d7f5a..3052229 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -6,5 +6,10 @@
 deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
 deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
 EOF
+cat > /etc/apt/preferences.d/<%= @distribution %>.pref <<-'EOF'
+Package: *
+Pin: release a=<%= @distribution %>-backports
+Pin-Priority: 500
+EOF
 apt-get update
 fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302472
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I43174931c1086f1760b21bc55384a647d025c570
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Pin $dist-backports at priority 500

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302472

Change subject: Pin $dist-backports at priority 500
..

Pin $dist-backports at priority 500

Because -backports has priority 100 by default, those packages are
not considered when there's any other version in any other repo.
Set to the same value as the default repo, so highest version nr wins.

Change-Id: I43174931c1086f1760b21bc55384a647d025c570
---
M modules/package_builder/templates/D02backports.erb
1 file changed, 5 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/72/302472/1

diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 94d7f5a..3052229 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -6,5 +6,10 @@
 deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
 deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
 EOF
+cat > /etc/apt/preferences.d/<%= @distribution %>.pref <<-'EOF'
+Package: *
+Pin: release a=<%= @distribution %>-backports
+Pin-Priority: 500
+EOF
 apt-get update
 fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302472
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I43174931c1086f1760b21bc55384a647d025c570
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Hardcode $dist-backports components

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302466

Change subject: Hardcode $dist-backports components
..

Hardcode $dist-backports components

Change-Id: I57432bbfb54ecba8db8d5ec19d3e38e066a5c863
---
M modules/package_builder/templates/D02backports.erb
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/66/302466/1

diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 7033d16..94d7f5a 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -3,8 +3,8 @@
 # Avoid running hooks if backports is not specified
 if [ "${BACKPORTS}" = "yes" ]; then
 cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list <<-'EOF'
-deb <%= @upstream_mirror %> <%= @distribution %>-backports <%= @components 
%>
-deb-src <%= @upstream_mirror %> <%= @distribution %>-backports <%= 
@components %>
+deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
+deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
 EOF
 apt-get update
 fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302466
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I57432bbfb54ecba8db8d5ec19d3e38e066a5c863
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Hardcode $dist-backports components

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Hardcode $dist-backports components
..


Hardcode $dist-backports components

Change-Id: I57432bbfb54ecba8db8d5ec19d3e38e066a5c863
---
M modules/package_builder/templates/D02backports.erb
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Mark Bergsma: Verified; Looks good to me, approved



diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 7033d16..94d7f5a 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -3,8 +3,8 @@
 # Avoid running hooks if backports is not specified
 if [ "${BACKPORTS}" = "yes" ]; then
 cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list <<-'EOF'
-deb <%= @upstream_mirror %> <%= @distribution %>-backports <%= @components 
%>
-deb-src <%= @upstream_mirror %> <%= @distribution %>-backports <%= 
@components %>
+deb <%= @upstream_mirror %> <%= @distribution %>-backports main contrib 
non-free
+deb-src <%= @upstream_mirror %> <%= @distribution %>-backports main 
contrib non-free
 EOF
 apt-get update
 fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302466
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I57432bbfb54ecba8db8d5ec19d3e38e066a5c863
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Introduce $upstream_mirror parameter

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Introduce $upstream_mirror parameter
..


Introduce $upstream_mirror parameter

We've always separated mirrors of upstream repos (Debian/Ubuntu) from
our own local repositories. Most hooks need our own repos, but in
some cases we need to refer to upstream.

Change-Id: I0013ce52664e7f6723855b45157bdd2e5b23494d
---
M modules/package_builder/manifests/pbuilder_hook.pp
M modules/package_builder/templates/D02backports.erb
M modules/package_builder/templates/pbuilderrc.erb
3 files changed, 5 insertions(+), 2 deletions(-)

Approvals:
  Mark Bergsma: Verified; Looks good to me, approved



diff --git a/modules/package_builder/manifests/pbuilder_hook.pp 
b/modules/package_builder/manifests/pbuilder_hook.pp
index 5cf9533..2cd78ec 100644
--- a/modules/package_builder/manifests/pbuilder_hook.pp
+++ b/modules/package_builder/manifests/pbuilder_hook.pp
@@ -3,6 +3,7 @@
 $distribution='jessie',
 $components='main',
 $mirror='http://apt.wikimedia.org/wikimedia',
+$upstream_mirror='http://mirrors.wikimedia.org/debian',
 $basepath='/var/cache/pbuilder',
 ) {
 file { "${basepath}/hooks/${distribution}":
diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 2a9f576..7033d16 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -3,8 +3,8 @@
 # Avoid running hooks if backports is not specified
 if [ "${BACKPORTS}" = "yes" ]; then
 cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list <<-'EOF'
-deb <%= @mirror %> <%= @distribution %>-backports <%= @components %>
-deb-src <%= @mirror %> <%= @distribution %>-backports <%= @components %>
+deb <%= @upstream_mirror %> <%= @distribution %>-backports <%= @components 
%>
+deb-src <%= @upstream_mirror %> <%= @distribution %>-backports <%= 
@components %>
 EOF
 apt-get update
 fi
diff --git a/modules/package_builder/templates/pbuilderrc.erb 
b/modules/package_builder/templates/pbuilderrc.erb
index 0177b03..4695e38 100644
--- a/modules/package_builder/templates/pbuilderrc.erb
+++ b/modules/package_builder/templates/pbuilderrc.erb
@@ -6,6 +6,8 @@
 DIST=${DIST:-"$(lsb_release --short --codename)"}
 # Default to not use wikimedia repos
 WIKIMEDIA=${WIKIMEDIA:-"no"}
+# Default to not use distribution backports
+BACKPORTS=${WIKIMEDIA:-"no"}
 
 # If DIST has been declared like dist-wikimedia, handle it like as if the user
 # wanted to specify WIKIMEDIA="yes" which will satisfy build dependencies from

-- 
To view, visit https://gerrit.wikimedia.org/r/302464
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0013ce52664e7f6723855b45157bdd2e5b23494d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Introduce $upstream_mirror parameter

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302464

Change subject: Introduce $upstream_mirror parameter
..

Introduce $upstream_mirror parameter

We've always separated mirrors of upstream repos (Debian/Ubuntu) from
our own local repositories. Most hooks need our own repos, but in
some cases we need to refer to upstream.

Change-Id: I0013ce52664e7f6723855b45157bdd2e5b23494d
---
M modules/package_builder/manifests/pbuilder_hook.pp
M modules/package_builder/templates/D02backports.erb
M modules/package_builder/templates/pbuilderrc.erb
3 files changed, 5 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/64/302464/1

diff --git a/modules/package_builder/manifests/pbuilder_hook.pp 
b/modules/package_builder/manifests/pbuilder_hook.pp
index 5cf9533..2cd78ec 100644
--- a/modules/package_builder/manifests/pbuilder_hook.pp
+++ b/modules/package_builder/manifests/pbuilder_hook.pp
@@ -3,6 +3,7 @@
 $distribution='jessie',
 $components='main',
 $mirror='http://apt.wikimedia.org/wikimedia',
+$upstream_mirror='http://mirrors.wikimedia.org/debian',
 $basepath='/var/cache/pbuilder',
 ) {
 file { "${basepath}/hooks/${distribution}":
diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
index 2a9f576..7033d16 100755
--- a/modules/package_builder/templates/D02backports.erb
+++ b/modules/package_builder/templates/D02backports.erb
@@ -3,8 +3,8 @@
 # Avoid running hooks if backports is not specified
 if [ "${BACKPORTS}" = "yes" ]; then
 cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list <<-'EOF'
-deb <%= @mirror %> <%= @distribution %>-backports <%= @components %>
-deb-src <%= @mirror %> <%= @distribution %>-backports <%= @components %>
+deb <%= @upstream_mirror %> <%= @distribution %>-backports <%= @components 
%>
+deb-src <%= @upstream_mirror %> <%= @distribution %>-backports <%= 
@components %>
 EOF
 apt-get update
 fi
diff --git a/modules/package_builder/templates/pbuilderrc.erb 
b/modules/package_builder/templates/pbuilderrc.erb
index 0177b03..4695e38 100644
--- a/modules/package_builder/templates/pbuilderrc.erb
+++ b/modules/package_builder/templates/pbuilderrc.erb
@@ -6,6 +6,8 @@
 DIST=${DIST:-"$(lsb_release --short --codename)"}
 # Default to not use wikimedia repos
 WIKIMEDIA=${WIKIMEDIA:-"no"}
+# Default to not use distribution backports
+BACKPORTS=${WIKIMEDIA:-"no"}
 
 # If DIST has been declared like dist-wikimedia, handle it like as if the user
 # wanted to specify WIKIMEDIA="yes" which will satisfy build dependencies from

-- 
To view, visit https://gerrit.wikimedia.org/r/302464
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0013ce52664e7f6723855b45157bdd2e5b23494d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add BACKPORTS=yes hook to include $DIST-backports apt source

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Add BACKPORTS=yes hook to include $DIST-backports apt source
..


Add BACKPORTS=yes hook to include $DIST-backports apt source

Change-Id: I3bae4d811678036930cecc0330e581081ac0cb03
---
M modules/package_builder/manifests/pbuilder_hook.pp
A modules/package_builder/templates/D02backports.erb
2 files changed, 19 insertions(+), 0 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/package_builder/manifests/pbuilder_hook.pp 
b/modules/package_builder/manifests/pbuilder_hook.pp
index ea79b9a..5cf9533 100644
--- a/modules/package_builder/manifests/pbuilder_hook.pp
+++ b/modules/package_builder/manifests/pbuilder_hook.pp
@@ -20,6 +20,14 @@
 content => template('package_builder/D01apt.wikimedia.org.erb'),
 }
 
+file { "${basepath}/hooks/${distribution}/D02backports":
+ensure  => present,
+owner   => 'root',
+group   => 'root',
+mode=> '0555',
+content => template('package_builder/D02backports.erb'),
+}
+
 file { "${basepath}/hooks/${distribution}/D05localsources":
 ensure  => present,
 owner   => 'root',
@@ -31,5 +39,6 @@
 
 # Dependency info
 File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D01apt.wikimedia.org"]
+File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D02backports"]
 File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D05localsources"]
 }
diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
new file mode 100755
index 000..2a9f576
--- /dev/null
+++ b/modules/package_builder/templates/D02backports.erb
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+# Avoid running hooks if backports is not specified
+if [ "${BACKPORTS}" = "yes" ]; then
+cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list <<-'EOF'
+deb <%= @mirror %> <%= @distribution %>-backports <%= @components %>
+deb-src <%= @mirror %> <%= @distribution %>-backports <%= @components %>
+EOF
+apt-get update
+fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302440
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3bae4d811678036930cecc0330e581081ac0cb03
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add BACKPORTS=yes hook to include $DIST-backports apt source

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302440

Change subject: Add BACKPORTS=yes hook to include $DIST-backports apt source
..

Add BACKPORTS=yes hook to include $DIST-backports apt source

Change-Id: I3bae4d811678036930cecc0330e581081ac0cb03
---
M modules/package_builder/manifests/pbuilder_hook.pp
A modules/package_builder/templates/D02backports.erb
2 files changed, 19 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/40/302440/1

diff --git a/modules/package_builder/manifests/pbuilder_hook.pp 
b/modules/package_builder/manifests/pbuilder_hook.pp
index ea79b9a..5cf9533 100644
--- a/modules/package_builder/manifests/pbuilder_hook.pp
+++ b/modules/package_builder/manifests/pbuilder_hook.pp
@@ -20,6 +20,14 @@
 content => template('package_builder/D01apt.wikimedia.org.erb'),
 }
 
+file { "${basepath}/hooks/${distribution}/D02backports":
+ensure  => present,
+owner   => 'root',
+group   => 'root',
+mode=> '0555',
+content => template('package_builder/D02backports.erb'),
+}
+
 file { "${basepath}/hooks/${distribution}/D05localsources":
 ensure  => present,
 owner   => 'root',
@@ -31,5 +39,6 @@
 
 # Dependency info
 File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D01apt.wikimedia.org"]
+File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D02backports"]
 File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D05localsources"]
 }
diff --git a/modules/package_builder/templates/D02backports.erb 
b/modules/package_builder/templates/D02backports.erb
new file mode 100644
index 000..050a9aa
--- /dev/null
+++ b/modules/package_builder/templates/D02backports.erb
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+# Avoid running hooks if wikimedia is not specified
+if [ "${BACKPORTS}" = "yes" ]; then
+   cat > /etc/apt/sources.list.d/<%= @distribution %>-backports.list 
<<-'EOF'
+   deb <%= @mirror %> <%= @distribution %>-backports <%= @components %>
+   deb-src <%= @mirror %> <%= @distribution %>-backports <%= @components %>
+EOF
+   apt-get update
+fi

-- 
To view, visit https://gerrit.wikimedia.org/r/302440
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3bae4d811678036930cecc0330e581081ac0cb03
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Move cp3022 to standard puppet role after reinstall

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Move cp3022 to standard puppet role after reinstall
..


Move cp3022 to standard puppet role after reinstall

Change-Id: I7ee60a1f26947f78fbae9bde5ecf167244484f57
---
M manifests/site.pp
1 file changed, 5 insertions(+), 1 deletion(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/manifests/site.pp b/manifests/site.pp
index 97546c2..175ede9 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -366,11 +366,15 @@
 role spare::system # to be decommed (T130883)
 }
 
-node /^cp30(19|2[0-2])\.esams\.wmnet$/ {
+node /^cp30(19|2[0-1])\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
 role spare::system # to be decommed (T130883)
 }
 
+node 'cp3022.esams.wmnet' {
+include standard
+}
+
 node /^cp30[34][0123]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
 role cache::text, ipsec

-- 
To view, visit https://gerrit.wikimedia.org/r/302419
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7ee60a1f26947f78fbae9bde5ecf167244484f57
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Move cp3022 to standard puppet role after reinstall

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/302419

Change subject: Move cp3022 to standard puppet role after reinstall
..

Move cp3022 to standard puppet role after reinstall

Change-Id: I7ee60a1f26947f78fbae9bde5ecf167244484f57
---
M manifests/site.pp
1 file changed, 5 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/19/302419/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 97546c2..7c88137 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -366,11 +366,15 @@
 role spare::system # to be decommed (T130883)
 }
 
-node /^cp30(19|2[0-2])\.esams\.wmnet$/ {
+node /^cp30(19|2[0-1])\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
 role spare::system # to be decommed (T130883)
 }
 
+node cp3022.esams.wmnet {
+   include standard
+}
+
 node /^cp30[34][0123]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
 role cache::text, ipsec

-- 
To view, visit https://gerrit.wikimedia.org/r/302419
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7ee60a1f26947f78fbae9bde5ecf167244484f57
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] [Beta] Change-prop: Adjust the ORES URI in BetaCluster - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: [Beta] Change-prop: Adjust the ORES URI in BetaCluster
..


[Beta] Change-prop: Adjust the ORES URI in BetaCluster

Change-Id: If67925417967a00ba278ce37e5527382ec61bab2
---
M hieradata/labs/deployment-prep/common.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/labs/deployment-prep/common.yaml 
b/hieradata/labs/deployment-prep/common.yaml
index 826591d..d839122 100644
--- a/hieradata/labs/deployment-prep/common.yaml
+++ b/hieradata/labs/deployment-prep/common.yaml
@@ -18,6 +18,7 @@
 citoid::zotero_port: 1969
 changeprop::purge_host: deployment-cache-text04.deployment-prep.eqiad.wmflabs
 changeprop::concurrency: 20
+changeprop::ores_uri: 
http://deployment-sca03.deployment-prep.eqiad.wmflabs:8081
 graphoid::allowed_domains:
   http:
 - wmflabs.org

-- 
To view, visit https://gerrit.wikimedia.org/r/299169
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If67925417967a00ba278ce37e5527382ec61bab2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mobrovac 
Gerrit-Reviewer: Ladsgroup 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: Mobrovac 
Gerrit-Reviewer: Ppchelko 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add codfw power to the total aggregates - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Add codfw power to the total aggregates
..


Add codfw power to the total aggregates

Change-Id: I51282bcb7b63286ecfeab60655c4f4e21fd92d51
---
M modules/torrus/files/xmlconfig/facilities_aggregates.xml
1 file changed, 3 insertions(+), 3 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/torrus/files/xmlconfig/facilities_aggregates.xml 
b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
index 84c6177..d11c937 100644
--- a/modules/torrus/files/xmlconfig/facilities_aggregates.xml
+++ b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
@@ -9,10 +9,10 @@
 

 
-   
+   
 

-   
+   



@@ -86,7 +86,7 @@
 


-   
+   


 

-- 
To view, visit https://gerrit.wikimedia.org/r/286455
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I51282bcb7b63286ecfeab60655c4f4e21fd92d51
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add codfw power to the total aggregates - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/286455

Change subject: Add codfw power to the total aggregates
..

Add codfw power to the total aggregates

Change-Id: I51282bcb7b63286ecfeab60655c4f4e21fd92d51
---
M modules/torrus/files/xmlconfig/facilities_aggregates.xml
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/55/286455/1

diff --git a/modules/torrus/files/xmlconfig/facilities_aggregates.xml 
b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
index 84c6177..d11c937 100644
--- a/modules/torrus/files/xmlconfig/facilities_aggregates.xml
+++ b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
@@ -9,10 +9,10 @@
 

 
-   
+   
 

-   
+   



@@ -86,7 +86,7 @@
 


-   
+   


 

-- 
To view, visit https://gerrit.wikimedia.org/r/286455
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I51282bcb7b63286ecfeab60655c4f4e21fd92d51
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Provide Torrus with the (specific) SNMP community used for c... - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Provide Torrus with the (specific) SNMP community used for 
codfw PDUs
..


Provide Torrus with the (specific) SNMP community used for codfw PDUs

Change-Id: Ia566f325d2990409e531597c0ae54a7fe11f6980
---
M modules/torrus/files/xmlconfig/power.xml
M modules/torrus/templates/site-global.xml.erb
2 files changed, 6 insertions(+), 0 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/torrus/files/xmlconfig/power.xml 
b/modules/torrus/files/xmlconfig/power.xml
index 23b9ca0..77a447c 100644
--- a/modules/torrus/files/xmlconfig/power.xml
+++ b/modules/torrus/files/xmlconfig/power.xml
@@ -2795,6 +2795,8 @@

 

+   
+



diff --git a/modules/torrus/templates/site-global.xml.erb 
b/modules/torrus/templates/site-global.xml.erb
index a9ab9f0..2ceb60f 100644
--- a/modules/torrus/templates/site-global.xml.erb
+++ b/modules/torrus/templates/site-global.xml.erb
@@ -30,6 +30,10 @@
last24h-small,last24h,lastweek,lastmonth,lastyear

 
+   
+   "/>
+   
+
 
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/286447
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia566f325d2990409e531597c0ae54a7fe11f6980
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Provide Torrus with the (specific) SNMP community used for c... - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/286447

Change subject: Provide Torrus with the (specific) SNMP community used for 
codfw PDUs
..

Provide Torrus with the (specific) SNMP community used for codfw PDUs

Change-Id: Ia566f325d2990409e531597c0ae54a7fe11f6980
---
M modules/torrus/files/xmlconfig/power.xml
M modules/torrus/templates/site-global.xml.erb
2 files changed, 6 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/47/286447/1

diff --git a/modules/torrus/files/xmlconfig/power.xml 
b/modules/torrus/files/xmlconfig/power.xml
index 23b9ca0..77a447c 100644
--- a/modules/torrus/files/xmlconfig/power.xml
+++ b/modules/torrus/files/xmlconfig/power.xml
@@ -2795,6 +2795,8 @@

 

+   
+



diff --git a/modules/torrus/templates/site-global.xml.erb 
b/modules/torrus/templates/site-global.xml.erb
index a9ab9f0..2ceb60f 100644
--- a/modules/torrus/templates/site-global.xml.erb
+++ b/modules/torrus/templates/site-global.xml.erb
@@ -30,6 +30,10 @@
last24h-small,last24h,lastweek,lastmonth,lastyear

 
+   
+   "/>
+   
+
 
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/286447
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia566f325d2990409e531597c0ae54a7fe11f6980
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add missing tokensets for codfw PDUs - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Add missing tokensets for codfw PDUs
..


Add missing tokensets for codfw PDUs

Change-Id: I85677a8b2d4a3d4d315c1411a039bca7005893e4
---
M modules/torrus/files/xmlconfig/power.xml
1 file changed, 229 insertions(+), 3 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/torrus/files/xmlconfig/power.xml 
b/modules/torrus/files/xmlconfig/power.xml
index 16e8c42..23b9ca0 100644
--- a/modules/torrus/files/xmlconfig/power.xml
+++ b/modules/torrus/files/xmlconfig/power.xml
@@ -5820,14 +5820,15 @@



-   
-   
+
+   


+



-   
+



@@ -5877,6 +5878,231 @@


 
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+

 


-- 
To view, visit 

[MediaWiki-commits] [Gerrit] Add missing tokensets for codfw PDUs - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/286420

Change subject: Add missing tokensets for codfw PDUs
..

Add missing tokensets for codfw PDUs

Change-Id: I85677a8b2d4a3d4d315c1411a039bca7005893e4
---
M modules/torrus/files/xmlconfig/power.xml
1 file changed, 229 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/20/286420/1

diff --git a/modules/torrus/files/xmlconfig/power.xml 
b/modules/torrus/files/xmlconfig/power.xml
index 16e8c42..23b9ca0 100644
--- a/modules/torrus/files/xmlconfig/power.xml
+++ b/modules/torrus/files/xmlconfig/power.xml
@@ -5820,14 +5820,15 @@



-   
-   
+
+   


+



-   
+



@@ -5877,6 +5878,231 @@


 
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+
+   
+   
+   
+   
+   
+   
+

 

[MediaWiki-commits] [Gerrit] Fix Facilities power aggregates - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Fix Facilities power aggregates
..


Fix Facilities power aggregates

Fix stack underflow in expr-Power_Measured
Add eqiad row D to expr-Power_Measured_eqiad
Fix stack underflow in expr-Power_Estimated
Update estimated values for site power usage

Change-Id: Iad55ceb55624da01d87687c152e285ebca30186b
---
M modules/torrus/files/xmlconfig/facilities_aggregates.xml
1 file changed, 9 insertions(+), 5 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/torrus/files/xmlconfig/facilities_aggregates.xml 
b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
index e8c8ea8..5f5b7fd 100644
--- a/modules/torrus/files/xmlconfig/facilities_aggregates.xml
+++ b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
@@ -5,15 +5,19 @@



-   
+   
 
-   
+   
 

-   
-   
+   
+   
+   
+   
+   
+   
 
-   
+   
 



-- 
To view, visit https://gerrit.wikimedia.org/r/286416
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iad55ceb55624da01d87687c152e285ebca30186b
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Fix Facilities power aggregates - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/286416

Change subject: Fix Facilities power aggregates
..

Fix Facilities power aggregates

Fix stack underflow in expr-Power_Measured
Add eqiad row D to expr-Power_Measured_eqiad
Fix stack underflow in expr-Power_Estimated
Update estimated values for site power usage

Change-Id: Iad55ceb55624da01d87687c152e285ebca30186b
---
M modules/torrus/files/xmlconfig/facilities_aggregates.xml
1 file changed, 9 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/16/286416/1

diff --git a/modules/torrus/files/xmlconfig/facilities_aggregates.xml 
b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
index e8c8ea8..0058167 100644
--- a/modules/torrus/files/xmlconfig/facilities_aggregates.xml
+++ b/modules/torrus/files/xmlconfig/facilities_aggregates.xml
@@ -5,15 +5,19 @@



-   
+   
 
-   
+   
 

-   
-   
+   
+   
+   
+   
+   
+   
 
-   
+   
 



-- 
To view, visit https://gerrit.wikimedia.org/r/286416
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iad55ceb55624da01d87687c152e285ebca30186b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add BGP MED support - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/267251

Change subject: Add BGP MED support
..

Add BGP MED support

Adds a global pybal.conf configuration option 'bgp-med' which sets a
BGP MED attribute on all NLRI if non-zero. This can be used e.g. for
BGP failover between multiple PyBal instances where the router selects
the routes for the the PyBal instance with the lowest MED value as sent
by PyBal.

Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
---
M pybal/pybal.py
1 file changed, 6 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/51/267251/1

diff --git a/pybal/pybal.py b/pybal/pybal.py
index cbffd3a..dd439e1 100755
--- a/pybal/pybal.py
+++ b/pybal/pybal.py
@@ -467,21 +467,20 @@
   
peerAddr=self.globalConfig.get('bgp-peer-address'))
 
 asPath = [int(asn) for asn in self.globalConfig.get('bgp-as-path', 
str(self.bgpPeering.myASN)).split()]
-attributes = {}
+med = self.globalConfig.getint('bgp-med', 0)
+baseAttrs = [bgp.OriginAttribute(), bgp.ASPathAttribute(asPath)]
+if med: baseAttrs.append(bgp.MEDAttribute(med))
 
+attributes = {}
 try:
-attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 
bgp.NextHopAttribute(self.globalConfig['bgp-nexthop-ipv4'])])
 except KeyError:
 if (bgp.AFI_INET, bgp.SAFI_UNICAST) in BGPFailover.prefixes:
 raise ValueError("IPv4 BGP NextHop (global configuration 
variable 'bgp-nexthop-ipv4') not set")
 
 try:
-attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 bgp.MPReachNLRIAttribute((bgp.AFI_INET6, bgp.SAFI_UNICAST,
  
bgp.IPv6IP(self.globalConfig['bgp-nexthop-ipv6']), []))])
 except KeyError:

-- 
To view, visit https://gerrit.wikimedia.org/r/267251
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add BGP MED support - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Add BGP MED support
..


Add BGP MED support

Adds a global pybal.conf configuration option 'bgp-med' which sets a
BGP MED attribute on all NLRI if non-zero. This can be used e.g. for
BGP failover between multiple PyBal instances where the router selects
the routes for the the PyBal instance with the lowest MED value as sent
by PyBal.

Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
---
M pybal/pybal.py
1 file changed, 6 insertions(+), 7 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/pybal.py b/pybal/pybal.py
index cbffd3a..dd439e1 100755
--- a/pybal/pybal.py
+++ b/pybal/pybal.py
@@ -467,21 +467,20 @@
   
peerAddr=self.globalConfig.get('bgp-peer-address'))
 
 asPath = [int(asn) for asn in self.globalConfig.get('bgp-as-path', 
str(self.bgpPeering.myASN)).split()]
-attributes = {}
+med = self.globalConfig.getint('bgp-med', 0)
+baseAttrs = [bgp.OriginAttribute(), bgp.ASPathAttribute(asPath)]
+if med: baseAttrs.append(bgp.MEDAttribute(med))
 
+attributes = {}
 try:
-attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 
bgp.NextHopAttribute(self.globalConfig['bgp-nexthop-ipv4'])])
 except KeyError:
 if (bgp.AFI_INET, bgp.SAFI_UNICAST) in BGPFailover.prefixes:
 raise ValueError("IPv4 BGP NextHop (global configuration 
variable 'bgp-nexthop-ipv4') not set")
 
 try:
-attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 bgp.MPReachNLRIAttribute((bgp.AFI_INET6, bgp.SAFI_UNICAST,
  
bgp.IPv6IP(self.globalConfig['bgp-nexthop-ipv6']), []))])
 except KeyError:

-- 
To view, visit https://gerrit.wikimedia.org/r/267251
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add IPv6 support to all monitors - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/267008

Change subject: Add IPv6 support to all monitors
..

Add IPv6 support to all monitors

Previously, monitors would only monitor over IPv4 by picking
a random IP from the IPv4 address set only on every connection.

With this change they will *only* monitor with the primary IP
(IPv4 or IPv6, depending on the address family of the service)
that was selected to be pooled by IPVS.

Change-Id: I21c1575e180cbda792a5eacc004fafd4a624e03b
---
M pybal/monitors/dnsquery.py
M pybal/monitors/idleconnection.py
M pybal/monitors/proxyfetch.py
3 files changed, 6 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/08/267008/1

diff --git a/pybal/monitors/dnsquery.py b/pybal/monitors/dnsquery.py
index 33c3938..1341ca8 100644
--- a/pybal/monitors/dnsquery.py
+++ b/pybal/monitors/dnsquery.py
@@ -52,7 +52,7 @@
 super(DNSQueryMonitoringProtocol, self).run()
 
 # Create a resolver
-self.resolver = client.createResolver([(ip, 53) for ip in 
self.server.ip4_addresses])
+self.resolver = client.createResolver([(self.server.ip, 53)])
 
 if not self.checkCall or not self.checkCall.active():
 self.checkCall = reactor.callLater(self.intvCheck, self.check)
diff --git a/pybal/monitors/idleconnection.py b/pybal/monitors/idleconnection.py
index 1c34634..42c556d 100644
--- a/pybal/monitors/idleconnection.py
+++ b/pybal/monitors/idleconnection.py
@@ -10,7 +10,6 @@
 from twisted.internet import reactor, protocol
 import logging
 
-import random
 import socket
 
 
@@ -130,9 +129,4 @@
 def _connect(self, *args, **kwargs):
 """Starts a TCP connection attempt"""
 
-try:
-host = random.choice(self.server.ip4_addresses)
-except (TypeError, IndexError):
-host = self.server.host
-
-self.reactor.connectTCP(host, self.server.port, self, *args, **kwargs)
+self.reactor.connectTCP(self.server.ip, self.server.port, self, *args, 
**kwargs)
diff --git a/pybal/monitors/proxyfetch.py b/pybal/monitors/proxyfetch.py
index 201c2ef..3468e73 100644
--- a/pybal/monitors/proxyfetch.py
+++ b/pybal/monitors/proxyfetch.py
@@ -10,7 +10,7 @@
 from twisted.internet import reactor, defer
 from twisted.web import client
 from twisted.python.runtime import seconds
-import logging
+import logging, random
 
 log = util.log
 
@@ -96,16 +96,13 @@
 # FIXME: Use GET as a workaround for a Twisted bug with 
HEAD/Content-length
 # where it expects a body and throws a PartialDownload failure
 
-import random
 url = random.choice(self.URL)
-try:
-host = random.choice(self.server.ip4_addresses)
-except (TypeError, IndexError):
-host = self.server.host
 
 self.checkStartTime = seconds()
 self.getPageDeferred = self.getProxyPage(
-url, method='GET', host=host,
+url,
+method='GET',
+host=self.server.ip,
 port=self.server.port,
 status=self.expectedStatus,
 timeout=self.toGET,

-- 
To view, visit https://gerrit.wikimedia.org/r/267008
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I21c1575e180cbda792a5eacc004fafd4a624e03b
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] admin: introduce restbase-admins group - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: admin: introduce restbase-admins group
..


admin: introduce restbase-admins group

also deploy where needed

Change-Id: Iedebf878900c513aad43cac9617f0d1e62eaa951
---
M hieradata/hosts/rutherfordium.yaml
M hieradata/role/common/restbase.yaml
M modules/admin/data/data.yaml
3 files changed, 11 insertions(+), 0 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/rutherfordium.yaml 
b/hieradata/hosts/rutherfordium.yaml
index c3ce322..b90b0dd 100644
--- a/hieradata/hosts/rutherfordium.yaml
+++ b/hieradata/hosts/rutherfordium.yaml
@@ -51,6 +51,7 @@
   - releasers-mobile
   - researchers
   - restbase-roots
+  - restbase-admins
   - restricted
   - snapshot-admins
   - statistics-admins
diff --git a/hieradata/role/common/restbase.yaml 
b/hieradata/role/common/restbase.yaml
index 1fb5478..b0c8d5f 100644
--- a/hieradata/role/common/restbase.yaml
+++ b/hieradata/role/common/restbase.yaml
@@ -21,3 +21,4 @@
 
 admin::groups:
   - restbase-roots
+  - restbase-admins
diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index 034867e..28caa2d 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -440,6 +440,15 @@
 description: Group of users running penetration tests
 gid: 768
 members: []
+  restbase-admins:
+gid: 769
+description: group of restbase admins
+members: [gwicke]
+privileges: ['ALL = NOPASSWD: /usr/sbin/service cassandra *',
+ 'ALL = (cassandra) NOPASSWD: ALL',
+ 'ALL = NOPASSWD: /usr/sbin/service restbase *',
+ 'ALL = (restbase) NOPASSWD: ALL',
+ 'ALL = NOPASSWD: /bin/journalctl *']
 users:
   rush:
 ensure: present

-- 
To view, visit https://gerrit.wikimedia.org/r/263349
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iedebf878900c513aad43cac9617f0d1e62eaa951
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] admin: remove gwicke from {parsoid, restbase}-roots - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: admin: remove gwicke from {parsoid,restbase}-roots
..


admin: remove gwicke from {parsoid,restbase}-roots

Change-Id: I8c51562cee0acd6771d58a18f876fa673093b81c
---
M modules/admin/data/data.yaml
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index 269b913..034867e 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -19,7 +19,7 @@
   parsoid-roots:
 gid: 701
 description: RT 5934
-members: [gwicke, catrope]
+members: [catrope]
 privileges: ['ALL = (ALL) NOPASSWD: ALL']
   parsoid-admin:
 gid: 702
@@ -292,7 +292,7 @@
   restbase-roots:
 gid: 744
 description: people who have full root on restbase nodes
-members: [eevans, gwicke, mobrovac]
+members: [eevans, mobrovac]
 privileges: ['ALL = (ALL) NOPASSWD: ALL']
   citoid-users:
 gid: 745

-- 
To view, visit https://gerrit.wikimedia.org/r/263346
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8c51562cee0acd6771d58a18f876fa673093b81c
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Enable RPS on eth0 on labstores - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/261598

Change subject: Enable RPS on eth0 on labstores
..

Enable RPS on eth0 on labstores

During recent Labs NFS overload situations, we've seen timeouts on
CPU #0 and ksoftirqd/0 consuming high amount of CPU. A quick look
in /proc/interrupts reveals why: all network interrupts (and
megasas as well) are being handled by CPU #0. No irqbalance running
either.

This naive patch enables Receive Packet Steering (RPS), which will
hopefully improve the situation, but may need further tweaking.

Change-Id: I28780014759fb0cbf4d0123cae1ae0c233a515b2
---
M modules/role/manifests/labs/nfs/fileserver.pp
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/98/261598/1

diff --git a/modules/role/manifests/labs/nfs/fileserver.pp 
b/modules/role/manifests/labs/nfs/fileserver.pp
index ee01956..29b2bdd 100644
--- a/modules/role/manifests/labs/nfs/fileserver.pp
+++ b/modules/role/manifests/labs/nfs/fileserver.pp
@@ -7,6 +7,9 @@
 include standard
 include ::labstore::fileserver
 
+# Enable RPS/RSS to balance IRQs over CPUs
+interface::rps { 'eth0': }
+
 class { '::labstore::monitoring':
 monitor_iface => $monitor,
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/261598
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I28780014759fb0cbf4d0123cae1ae0c233a515b2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add BGP MED support - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Add BGP MED support
..


Add BGP MED support

Adds a global pybal.conf configuration option 'bgp-med' which sets a
BGP MED attribute on all NLRI if non-zero. This can be used e.g. for
BGP failover between multiple PyBal instances where the router selects
the routes for the the PyBal instance with the lowest MED value as sent
by PyBal.

Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
---
M pybal/pybal.py
1 file changed, 6 insertions(+), 7 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  Faidon Liambotis: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/pybal/pybal.py b/pybal/pybal.py
index 4f7863b..31d6ee4 100755
--- a/pybal/pybal.py
+++ b/pybal/pybal.py
@@ -472,21 +472,20 @@
   
peerAddr=self.globalConfig.get('bgp-peer-address'))
 
 asPath = [int(asn) for asn in self.globalConfig.get('bgp-as-path', 
str(self.bgpPeering.myASN)).split()]
-attributes = {}
+med = self.globalConfig.getint('bgp-med', 0)
+baseAttrs = [bgp.OriginAttribute(), bgp.ASPathAttribute(asPath)]
+if med: baseAttrs.append(bgp.MEDAttribute(med))
 
+attributes = {}
 try:
-attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 
bgp.NextHopAttribute(self.globalConfig['bgp-nexthop-ipv4'])])
 except KeyError:
 if (bgp.AFI_INET, bgp.SAFI_UNICAST) in BGPFailover.prefixes:
 raise ValueError("IPv4 BGP NextHop (global configuration 
variable 'bgp-nexthop-ipv4') not set")
 
 try:
-attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 bgp.MPReachNLRIAttribute((bgp.AFI_INET6, bgp.SAFI_UNICAST,
  
bgp.IPv6IP(self.globalConfig['bgp-nexthop-ipv6']), []))])
 except KeyError:

-- 
To view, visit https://gerrit.wikimedia.org/r/255544
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
Gerrit-PatchSet: 2
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: bgp-med
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: Ori.livneh 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Make all IdleConnection TCP KeepAlive parameters configurable - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/256942

Change subject: Make all IdleConnection TCP KeepAlive parameters configurable
..

Make all IdleConnection TCP KeepAlive parameters configurable

This commit creates several configuration options for TCP KeepAlive
in IdleConnection, instead of using the hardcoded values. The default
values have also been changed to be slightly less aggressive.

Change-Id: I1f5e0993f1901def199e0fb4d3790836dc7db2f2
---
M pybal/monitors/idleconnection.py
1 file changed, 12 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/42/256942/1

diff --git a/pybal/monitors/idleconnection.py b/pybal/monitors/idleconnection.py
index 5ac7ea0..1c34634 100644
--- a/pybal/monitors/idleconnection.py
+++ b/pybal/monitors/idleconnection.py
@@ -25,7 +25,10 @@
 
 TIMEOUT_CLEAN_RECONNECT = 3
 MAX_DELAY = 300
-KEEPALIVE_RETRIES = 10
+KEEPALIVE = True
+KEEPALIVE_RETRIES = 3
+KEEPALIVE_IDLE = 10
+KEEPALIVE_INTERVAL = 30
 
 __name__ = 'IdleConnection'
 
@@ -37,8 +40,10 @@
 
 self.toCleanReconnect = self._getConfigInt('timeout-clean-reconnect', 
self.TIMEOUT_CLEAN_RECONNECT)
 self.maxDelay = self._getConfigInt('max-delay', self.MAX_DELAY)
-self.keepAliveRetries = self._getConfigInt('keepalive-retries',
-   self.KEEPALIVE_RETRIES)
+self.keepAlive = self._getConfigBool('keepalive', self.KEEPALIVE)
+self.keepAliveRetries = self._getConfigInt('keepalive-retries', 
self.KEEPALIVE_RETRIES)
+self.keepAliveIdle = self._getConfigInt('keepalive-idle', 
self.KEEPALIVE_IDLE)
+self.keepAliveInterval = self._getConfigInt('keepalive-interval', 
self.KEEPALIVE_INTERVAL)
 
 def run(self):
 """Start the monitoring"""
@@ -96,13 +101,12 @@
 if not self.active:
 return
 
-if self.transport is not None:
+if self.transport is not None and self.keepAlive:
 sock = self.transport.getHandle()
 sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPIDLE, 1)
-sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPCNT,
-self.keepAliveRetries)
-sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPINTVL, 1)
+sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPIDLE, 
self.keepAliveIdle)
+sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPCNT, 
self.keepAliveRetries)
+sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPINTVL, 
self.keepAliveInterval)
 
 # Set status to up
 self._resultUp()

-- 
To view, visit https://gerrit.wikimedia.org/r/256942
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1f5e0993f1901def199e0fb4d3790836dc7db2f2
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add Mark's Yubikey based SSH pubkey - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/256419

Change subject: Add Mark's Yubikey based SSH pubkey
..

Add Mark's Yubikey based SSH pubkey

Change-Id: Ic1b63c4156882cf6e8e95c9420d002717d3b085f
---
M modules/admin/data/data.yaml
1 file changed, 3 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/19/256419/1

diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index 237b2b0..8d43b20 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -498,8 +498,9 @@
 gid: 500
 name: mark
 realname: Mark Bergsma
-ssh_keys: [ssh-rsa 
B3NzaC1yc2EBIwAAAIEAorTmQ0qlrxB3RL+GULLzex3k1Pg/c6tgLbKsl1A7Qo0B5XI4eNgfWwaAXUrKyQW3/9gwDH3YJ2eoOue0/BGhKX6voOTnNPeGE9ZbrufpPLT6DXDEbvpmXQd/qw8s0GxdftleHYl28av0nTZgKY+1/Oc+ZHNUN5YxmdGehWBvTXs=
-Mark\s main public key]
+ssh_keys:
+  - ssh-rsa 
B3NzaC1yc2EBIwAAAIEAorTmQ0qlrxB3RL+GULLzex3k1Pg/c6tgLbKsl1A7Qo0B5XI4eNgfWwaAXUrKyQW3/9gwDH3YJ2eoOue0/BGhKX6voOTnNPeGE9ZbrufpPLT6DXDEbvpmXQd/qw8s0GxdftleHYl28av0nTZgKY+1/Oc+ZHNUN5YxmdGehWBvTXs=
 Mark\s main public key
+  - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCGYwDexjSlyf1dMsVMr9VrEwnsEW90p3Ywgz/1Zl1EmYIrwLztKZ6V2bR6FbMx072YT6eEkg3m6dHxck5Q2Tx8QvUuUMoHzg18e/ZzsVZkG/cqtwrEXTRpg7O35xcdg0rYAbyyB0qo2z5gZWakZPd+h9hwfj2CUQ29aGRv83fsu2Ua5rbavU8OwAfndxJaBHDUqHsS/Mc+M3h6LF6XjhT9ELh0k7ZyPNhYTJ0N1VI4ROoxWI4/WRsclsSZtnWi7tYWP6WjhptezRGdbo2H/EMNffmISq5/n/1lkcAWGTIY3bppmBGe+f7CDcg8VN0i56Xe9+U7smlnqBSc17HZ8035
 mark@neo
 uid: 531
   jgreen:
 ensure: present

-- 
To view, visit https://gerrit.wikimedia.org/r/256419
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic1b63c4156882cf6e8e95c9420d002717d3b085f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add Mark's Yubikey based SSH pubkey - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Add Mark's Yubikey based SSH pubkey
..


Add Mark's Yubikey based SSH pubkey

Change-Id: Ic1b63c4156882cf6e8e95c9420d002717d3b085f
---
M modules/admin/data/data.yaml
1 file changed, 3 insertions(+), 2 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index 237b2b0..8d43b20 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -498,8 +498,9 @@
 gid: 500
 name: mark
 realname: Mark Bergsma
-ssh_keys: [ssh-rsa 
B3NzaC1yc2EBIwAAAIEAorTmQ0qlrxB3RL+GULLzex3k1Pg/c6tgLbKsl1A7Qo0B5XI4eNgfWwaAXUrKyQW3/9gwDH3YJ2eoOue0/BGhKX6voOTnNPeGE9ZbrufpPLT6DXDEbvpmXQd/qw8s0GxdftleHYl28av0nTZgKY+1/Oc+ZHNUN5YxmdGehWBvTXs=
-Mark\s main public key]
+ssh_keys:
+  - ssh-rsa 
B3NzaC1yc2EBIwAAAIEAorTmQ0qlrxB3RL+GULLzex3k1Pg/c6tgLbKsl1A7Qo0B5XI4eNgfWwaAXUrKyQW3/9gwDH3YJ2eoOue0/BGhKX6voOTnNPeGE9ZbrufpPLT6DXDEbvpmXQd/qw8s0GxdftleHYl28av0nTZgKY+1/Oc+ZHNUN5YxmdGehWBvTXs=
 Mark\s main public key
+  - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCGYwDexjSlyf1dMsVMr9VrEwnsEW90p3Ywgz/1Zl1EmYIrwLztKZ6V2bR6FbMx072YT6eEkg3m6dHxck5Q2Tx8QvUuUMoHzg18e/ZzsVZkG/cqtwrEXTRpg7O35xcdg0rYAbyyB0qo2z5gZWakZPd+h9hwfj2CUQ29aGRv83fsu2Ua5rbavU8OwAfndxJaBHDUqHsS/Mc+M3h6LF6XjhT9ELh0k7ZyPNhYTJ0N1VI4ROoxWI4/WRsclsSZtnWi7tYWP6WjhptezRGdbo2H/EMNffmISq5/n/1lkcAWGTIY3bppmBGe+f7CDcg8VN0i56Xe9+U7smlnqBSc17HZ8035
 mark@neo
 uid: 531
   jgreen:
 ensure: present

-- 
To view, visit https://gerrit.wikimedia.org/r/256419
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic1b63c4156882cf6e8e95c9420d002717d3b085f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Mark Bergsma 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add BGP MED support - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/255544

Change subject: Add BGP MED support
..

Add BGP MED support

Adds a global pybal.conf configuration option 'bgp-med' which sets a
BGP MED attribute on all NLRI if non-zero. This can be used e.g. for
BGP failover between multiple PyBal instances where the router selects
the routes for the the PyBal instance with the lowest MED value as sent
by PyBal.

Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
---
M pybal/pybal.py
1 file changed, 6 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/44/255544/1

diff --git a/pybal/pybal.py b/pybal/pybal.py
index e34bc66..b4ac395 100755
--- a/pybal/pybal.py
+++ b/pybal/pybal.py
@@ -469,21 +469,20 @@
   
peerAddr=self.globalConfig.get('bgp-peer-address'))
 
 asPath = [int(asn) for asn in self.globalConfig.get('bgp-as-path', 
str(self.bgpPeering.myASN)).split()]
-attributes = {}
+med = self.globalConfig.getint('bgp-med', 0)
+baseAttrs = [bgp.OriginAttribute(), bgp.ASPathAttribute(asPath)]
+if med: baseAttrs.append(bgp.MEDAttribute(med))
 
+attributes = {}
 try:
-attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 
bgp.NextHopAttribute(self.globalConfig['bgp-nexthop-ipv4'])])
 except KeyError:
 if (bgp.AFI_INET, bgp.SAFI_UNICAST) in BGPFailover.prefixes:
 raise ValueError("IPv4 BGP NextHop (global configuration 
variable 'bgp-nexthop-ipv4') not set")
 
 try:
-attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict([
-bgp.OriginAttribute(),
-bgp.ASPathAttribute(asPath),
+attributes[(bgp.AFI_INET6, bgp.SAFI_UNICAST)] = 
bgp.FrozenAttributeDict(baseAttrs + [
 bgp.MPReachNLRIAttribute((bgp.AFI_INET6, bgp.SAFI_UNICAST,
  
bgp.IPv6IP(self.globalConfig['bgp-nexthop-ipv6']), []))])
 except KeyError:

-- 
To view, visit https://gerrit.wikimedia.org/r/255544
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I589be24c275cc73a5c04312e68bdf509e8c2533f
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: bgp-med
Gerrit-Owner: Mark Bergsma 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Throttle RESTBase update jobs - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Throttle RESTBase update jobs
..


Throttle RESTBase update jobs

The Cassandra cluster backing RESTBase is currently under relatively high
load. These background jobs are responsible for most requests and basically
all writes. Throttling them slightly will decrease peak write load on
RESTBase, which should reduce the probability of OOM events while the cluster
is compacting away old revisions.

Note that we used to run on three job runners for a long time without issues 
[1],
when the Parsoid jobs were still enabled in parallel. The RESTBase jobs are
split in two queues, so that updates to an edited page itself are still
processed quickly (within seconds), while less urgent dependent updates are
somewhat throttled.

[1]: 2baa8353

Change-Id: Ic8cc415a4458db061d5d41067f6043617c1ab0af
---
M hieradata/role/common/mediawiki/jobrunner.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/mediawiki/jobrunner.yaml 
b/hieradata/role/common/mediawiki/jobrunner.yaml
index 3755a53..059b002 100644
--- a/hieradata/role/common/mediawiki/jobrunner.yaml
+++ b/hieradata/role/common/mediawiki/jobrunner.yaml
@@ -3,7 +3,7 @@
   - deployment
 mediawiki::jobrunner::runners_basic: 20
 mediawiki::jobrunner::runners_gwt: 1
-mediawiki::jobrunner::runners_restbase: 5
+mediawiki::jobrunner::runners_restbase: 3
 mediawiki::jobrunner::runners_translate: 1
 mediawiki::jobrunner::runners_upload: 7
 mediawiki::users::web: www-data

-- 
To view, visit https://gerrit.wikimedia.org/r/223588
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic8cc415a4458db061d5d41067f6043617c1ab0af
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: GWicke gwi...@wikimedia.org
Gerrit-Reviewer: Aaron Schulz asch...@wikimedia.org
Gerrit-Reviewer: BBlack bbl...@wikimedia.org
Gerrit-Reviewer: Dzahn dz...@wikimedia.org
Gerrit-Reviewer: Eevans eev...@wikimedia.org
Gerrit-Reviewer: Filippo Giunchedi fgiunch...@wikimedia.org
Gerrit-Reviewer: Giuseppe Lavagetto glavage...@wikimedia.org
Gerrit-Reviewer: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: Mobrovac mobro...@wikimedia.org
Gerrit-Reviewer: Ori.livneh o...@wikimedia.org
Gerrit-Reviewer: RobH r...@wikimedia.org
Gerrit-Reviewer: jenkins-bot 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Revert icinga: remove mark from SMS - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/197037

Change subject: Revert icinga: remove mark from SMS
..

Revert icinga: remove mark from SMS

This reverts commit 2e7d329dc5ee76a1abbe6d672728967591235da6.

Change-Id: I3d62a0f61fa5725ecf14a301d02464a554df49a1
---
M modules/icinga/files/contactgroups.cfg
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/37/197037/1

diff --git a/modules/icinga/files/contactgroups.cfg 
b/modules/icinga/files/contactgroups.cfg
index 38ee380..457dac7 100644
--- a/modules/icinga/files/contactgroups.cfg
+++ b/modules/icinga/files/contactgroups.cfg
@@ -6,7 +6,7 @@
 
 define contactgroup {
 contactgroup_name   sms
-members 
akosiaris,ariel,bblack,cmjohnson,dzahn,faidon,fgiunchedi,gage,jgreen,mpelletier,otto,robh,springle,tstarling,rush,glavagetto,yuvipanda
+members 
akosiaris,ariel,bblack,cmjohnson,dzahn,faidon,fgiunchedi,gage,jgreen,mark,mpelletier,otto,robh,springle,tstarling,rush,glavagetto,yuvipanda
 }
 
 define contactgroup {

-- 
To view, visit https://gerrit.wikimedia.org/r/197037
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3d62a0f61fa5725ecf14a301d02464a554df49a1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Revert icinga: remove mark from SMS - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Revert icinga: remove mark from SMS
..


Revert icinga: remove mark from SMS

This reverts commit 2e7d329dc5ee76a1abbe6d672728967591235da6.

Change-Id: I3d62a0f61fa5725ecf14a301d02464a554df49a1
---
M modules/icinga/files/contactgroups.cfg
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/icinga/files/contactgroups.cfg 
b/modules/icinga/files/contactgroups.cfg
index d0a0516..23fe5ce 100644
--- a/modules/icinga/files/contactgroups.cfg
+++ b/modules/icinga/files/contactgroups.cfg
@@ -6,7 +6,7 @@
 
 define contactgroup {
 contactgroup_name   sms
-members 
akosiaris,ariel,bblack,cmjohnson,dzahn,faidon,fgiunchedi,gage,jgreen,mpelletier,otto,robh,springle,tstarling,rush,glavagetto,yuvipanda
+members 
akosiaris,ariel,bblack,cmjohnson,dzahn,faidon,fgiunchedi,gage,jgreen,mark,mpelletier,otto,robh,springle,tstarling,rush,glavagetto,yuvipanda
 }
 
 define contactgroup {

-- 
To view, visit https://gerrit.wikimedia.org/r/197037
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3d62a0f61fa5725ecf14a301d02464a554df49a1
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: jenkins-bot 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Disable LWP SSL hostname verification - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/188553

Change subject: Disable LWP SSL hostname verification
..

Disable LWP SSL hostname verification

The RT mail gate appears to have broken recently, likely when the
RT SSL cert was replaced. Quick debugging showed that LWP::UserAgent
was complaining about SSL hostname verification failing. Setting
this environment variable turns that off.

Change-Id: I6ae6bcf2f365a4b4aa63834bd703fd8f541d395f
---
M templates/exim/exim4.conf.SMTP_IMAP_MM.erb
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/53/188553/1

diff --git a/templates/exim/exim4.conf.SMTP_IMAP_MM.erb 
b/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
index 4e51886..3e37f04 100644
--- a/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
+++ b/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
@@ -640,7 +640,7 @@
command = /usr/bin/rt-mailgate --queue $local_part \
--action ${if 
eq{$local_part_suffix}{-comment}{comment}{correspond}} \
--extension ticket --url https://rt.wikimedia.org
-   environment = $address_data
+   environment = $address_data : PERL_LWP_SSL_VERIFY_HOSTNAME=0
user = mail
group = mail
return_fail_output

-- 
To view, visit https://gerrit.wikimedia.org/r/188553
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6ae6bcf2f365a4b4aa63834bd703fd8f541d395f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Gerrit: limit max object size to 50M - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Gerrit: limit max object size to 50M
..


Gerrit: limit max object size to 50M

Why are you uploading objects larger than that?
It's almost always a bad idea.

Change-Id: I5131ff6599f169332b363e477c6e33757c5877b4
---
M modules/gerrit/templates/gerrit.config.erb
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Rush: Looks good to me, but someone else must approve
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified
  Dzahn: Looks good to me, but someone else must approve



diff --git a/modules/gerrit/templates/gerrit.config.erb 
b/modules/gerrit/templates/gerrit.config.erb
index 0cc8d1d..5a83bfc 100644
--- a/modules/gerrit/templates/gerrit.config.erb
+++ b/modules/gerrit/templates/gerrit.config.erb
@@ -154,3 +154,5 @@
 [commitmessage]
 maxSubjectLength = 100
 maxLineLength = 120
+[receive]
+maxObjectSizeLimit = 50m

-- 
To view, visit https://gerrit.wikimedia.org/r/187691
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I5131ff6599f169332b363e477c6e33757c5877b4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Chad ch...@wikimedia.org
Gerrit-Reviewer: Chad ch...@wikimedia.org
Gerrit-Reviewer: Dzahn dz...@wikimedia.org
Gerrit-Reviewer: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: Rush r...@wikimedia.org
Gerrit-Reviewer: jenkins-bot 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add no-gravity configuration option - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/187346

Change subject: Add no-gravity configuration option
..

Add no-gravity configuration option

If no-gravity = True on an LVS service, instead of removing a server
from IPVS for depooling, its weight is set to 0 instead.

This adds a lot of complications, as previously held invariants
(e.g. (server.pooled == server listed in IPVS)) no longer hold.

* server.pooled has been renamed to server.listed to clarify the
distinction between being used and being configured in IPVS
* server.listWeight has been added to distinguish between desired server
weight when pooled vs weight configured in IPVS

The existing code is also inconsistent with (boolean) server state
variables that indicate _intent_ and _current state_. This is (at
least partially) resolved by creating new variables:

* server.list vs server.listed
* server.pool vs server.pooled

These are now maintained as separate concerns in the code. Tests still
need to be adapted.

Bug: T86650
Change-Id: I1f8aec57401481520eec80c7f9bfd0ca5271f7c2
---
M pybal/ipvs.py
M pybal/pybal.py
2 files changed, 98 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/46/187346/1

diff --git a/pybal/ipvs.py b/pybal/ipvs.py
index 6d4aa76..25cba80 100644
--- a/pybal/ipvs.py
+++ b/pybal/ipvs.py
@@ -141,8 +141,8 @@
 cmd =  .join(['-a', cls.subCommandService(service), 
cls.subCommandServer(server)])
 
 # Include weight if specified
-if server.weight:
-cmd += ' -w %d' % server.weight
+if server.listWeight:
+cmd += ' -w %d' % server.listWeight
 
 return cmd
 
@@ -159,8 +159,8 @@
 cmd =  .join(['-e', cls.subCommandService(service), 
cls.subCommandServer(server)])
 
 # Include weight if specified
-if server.weight:
-cmd += ' -w %d' % server.weight
+if server.listWeight:
+cmd += ' -w %d' % server.listWeight
 
 return cmd
 
@@ -192,6 +192,7 @@
 self.configuration = configuration
 
 self.ipvsManager.DryRun = configuration.getboolean('dryrun', False)
+self.noGravity = configuration.getboolean('no-gravity', False)
 
 if self.configuration.getboolean('bgp', False):
 from pybal import BGPFailover
@@ -228,8 +229,15 @@
 [self.ipvsManager.commandEditServer(self.service(), server) 
for server in newServers  self.servers] + \
 [self.ipvsManager.commandRemoveServer(self.service(), server) 
for server in self.servers - newServers]
 
-self.servers = newServers
 self.ipvsManager.modifyState(cmdList)
+
+# Update server.listed  server.pooled with new status
+for server in self.servers + newServers:
+server.listed = server in newServers
+server.pooled = (server.listed and server.listWeight  0)
+
+self.servers = newServers
+
 
 def addServer(self, server):
 Adds (pools) a single Server to the LVS state
@@ -243,7 +251,8 @@
 self.servers.add(server)
 
 self.ipvsManager.modifyState(cmdList)
-server.pooled = True
+server.listed = True
+server.pooled = (server.weight != 0)
 
 def removeServer(self, server):
 Removes (depools) a single Server from the LVS state
@@ -252,8 +261,25 @@
 
 self.servers.remove(server) # May raise KeyError
 
-server.pooled = False
 self.ipvsManager.modifyState(cmdList)
+server.listed = False
+server.pooled = False
+
+
+def editServer(self, server):
+
+Changes the IPVS attributes of an existing server.
+
+Handles weight 0 as special (depooled) case
+Does NOT modify server.weight
+
+
+assert server.listed and server in self.servers
+
+cmdList = [self.ipvsManager.commandEditServer(self.service(), server)]
+self.ipvsManager.modifyState(cmdList)
+server.pooled = (server.weight != 0)
+
 
 def initServer(self, server):
 Initializes a server instance with LVS service specific 
configuration.
diff --git a/pybal/pybal.py b/pybal/pybal.py
index 6dae0d3..4a4249e 100644
--- a/pybal/pybal.py
+++ b/pybal/pybal.py
@@ -49,15 +49,19 @@
 self.monitors = set()
 
 # A few invariants that SHOULD be maintained (but currently may not 
be):
-# P0: pooled = enabled /\ ready
-# P1: up = pooled \/ !enabled \/ !ready
+# P0: list = enabled /\ ready
+# P1: up = pool \/ !enabled \/ !ready
 # P2: pooled = up \/ !canDepool
 
-self.weight = self.DEF_WEIGHT
-self.up = False
-self.pooled = False
-self.enabled = True
-self.ready = False
+self.weight = 

[MediaWiki-commits] [Gerrit] Remove init script for default udp2log instance on fluorine - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/187082

Change subject: Remove init script for default udp2log instance on fluorine
..

Remove init script for default udp2log instance on fluorine

Adds a default_instance parameter to misc::udp2log which, if false,
removes the default init script so it won't conflict with any
misc::udp2log::instance instances.

Bug: T87726
Change-Id: Ib7b0a618f221fff982e4683ee378a8300cba8bff
---
M manifests/misc/udp2log.pp
M manifests/role/logging.pp
2 files changed, 21 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/82/187082/1

diff --git a/manifests/misc/udp2log.pp b/manifests/misc/udp2log.pp
index b210470..a9db69d 100644
--- a/manifests/misc/udp2log.pp
+++ b/manifests/misc/udp2log.pp
@@ -6,7 +6,11 @@
 #
 # Parameters:
 #$monitor  - If true, monitoring scripts will be installed.  Default: true
-class misc::udp2log($monitor = true) {
+#$default_instance  - If false, remove init script for the default 
instance.  Default: true
+class misc::udp2log(
+$monitor = true,
+$default_instance = true
+) {
 
 include contacts::udp2log
 include misc::udp2log::udp_filter
@@ -42,6 +46,17 @@
 package { 'udplog':
 ensure = present,
 }
+
+if !$default_instance {
+file { '/etc/init.d/udp2log':
+require = Package[udp2log],
+ensure = absent
+}
+exec { '/usr/sbin/update-rc.d -f udp2log remove':
+subscribe   = File['/etc/init.d/udp2log'],
+refreshonly = true
+}
+}
 }
 
 # Class: misc::udp2log::rsyncd
diff --git a/manifests/role/logging.pp b/manifests/role/logging.pp
index 0cbb786..30d02b0 100644
--- a/manifests/role/logging.pp
+++ b/manifests/role/logging.pp
@@ -35,7 +35,11 @@
 class role::logging::mediawiki($monitor = true, $log_directory = 
'/home/wikipedia/logs' ) {
 system::role { role::logging:mediawiki: description = MediaWiki log 
collector }
 
-class { misc::udp2log: monitor = $monitor }
+class { misc::udp2log:
+monitor = $monitor,
+default_instance= false
+}
+
 include misc::udp2log::utilities,
 misc::udp2log::firewall
 

-- 
To view, visit https://gerrit.wikimedia.org/r/187082
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib7b0a618f221fff982e4683ee378a8300cba8bff
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Create a special alias router for RT - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/180641

Change subject: Create a special alias router for RT
..

Create a special alias router for RT

Moves the existing 'mailalias' Puppet type aliases to a simple
static file. As in this case there's no existing file to manage,
it's simpler.

Change-Id: I99292b51dd8448c068f4adbee3773176e06d3063
---
A modules/requesttracker/files/rt.aliases
M modules/requesttracker/manifests/aliases.pp
M templates/exim/exim4.conf.SMTP_IMAP_MM.erb
3 files changed, 28 insertions(+), 39 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/41/180641/1

diff --git a/modules/requesttracker/files/rt.aliases 
b/modules/requesttracker/files/rt.aliases
new file mode 100644
index 000..8a6204a
--- /dev/null
+++ b/modules/requesttracker/files/rt.aliases
@@ -0,0 +1,13 @@
+RT aliases
+
+ops-request: ops-priv...@lists.wikimedia.org
+phabricator: r...@phabricator.wikimedia.org
+foo: t...@phabricator.wikimedia.org
+ulsfo: ops-priv...@lists.wikimedia.org
+pmtpa: ops-priv...@lists.wikimedia.org
+todo: ops-priv...@lists.wikimedia.org
+eqiad: ops-priv...@lists.wikimedia.org
+esams: ops-priv...@lists.wikimedia.org
+network: ops-priv...@lists.wikimedia.org
+codfw: ops-priv...@lists.wikimedia.org
+core-ops: ops-priv...@lists.wikimedia.org
\ No newline at end of file
diff --git a/modules/requesttracker/manifests/aliases.pp 
b/modules/requesttracker/manifests/aliases.pp
index 22b1815..f381b0c 100644
--- a/modules/requesttracker/manifests/aliases.pp
+++ b/modules/requesttracker/manifests/aliases.pp
@@ -1,42 +1,5 @@
 class requesttracker::aliases {
-
-mailalias { 'ops-request':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'codfw':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'core-ops':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'eqiad':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'esams':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'network':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'pmtpa':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'todo':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'ulsfo':
-recipient = 'ops-priv...@lists.wikimedia.org',
-}
-
-mailalias { 'phabricator':
-recipient = 'r...@phabricator.wikimedia.org',
+file { '/etc/exim4/aliases/rt':
+source = modules/requesttracker/rt.aliases
 }
 }
diff --git a/templates/exim/exim4.conf.SMTP_IMAP_MM.erb 
b/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
index 7a9e3f9..cb8f1c3 100644
--- a/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
+++ b/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
@@ -363,6 +363,19 @@
 % if @rt_relay == true then -%
 # Mail destined for RT
 
+# Special alias file for the RT domain
+rt_aliases:
+   driver = redirect
+   domains = +local_domains
+   require_files = CONFDIR/aliases/rt
+   data = ${lookup{$local_part}lsearch*{CONFDIR/aliases/rt}}
+   qualify_preserve_domain
+   allow_fail
+   allow_defer
+   forbid_file
+   forbid_pipe
+   include_directory = CONFDIR
+
 # This router checks whether the local part consists of solely digits,
 # and assumes this is the ticket number of an existing ticket if this is
 # the case. It rewrites the address to the general queue, and puts the

-- 
To view, visit https://gerrit.wikimedia.org/r/180641
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I99292b51dd8448c068f4adbee3773176e06d3063
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Fix aliases file URL - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/180643

Change subject: Fix aliases file URL
..

Fix aliases file URL

Change-Id: I67c2d1a316ff03ef0c2a66f7394486ddca809a9d
---
M modules/requesttracker/manifests/aliases.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/43/180643/1

diff --git a/modules/requesttracker/manifests/aliases.pp 
b/modules/requesttracker/manifests/aliases.pp
index dcada8b..7d79b86 100644
--- a/modules/requesttracker/manifests/aliases.pp
+++ b/modules/requesttracker/manifests/aliases.pp
@@ -3,6 +3,6 @@
 owner   = root,
 group   = root,
 mode= 0444,
-source  = 'modules/requesttracker/rt.aliases',
+source  = 'puppet:///modules/requesttracker/rt.aliases',
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/180643
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I67c2d1a316ff03ef0c2a66f7394486ddca809a9d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Fix aliases file URL - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Fix aliases file URL
..


Fix aliases file URL

Change-Id: I67c2d1a316ff03ef0c2a66f7394486ddca809a9d
---
M modules/requesttracker/manifests/aliases.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Mark Bergsma: Verified; Looks good to me, approved



diff --git a/modules/requesttracker/manifests/aliases.pp 
b/modules/requesttracker/manifests/aliases.pp
index dcada8b..7d79b86 100644
--- a/modules/requesttracker/manifests/aliases.pp
+++ b/modules/requesttracker/manifests/aliases.pp
@@ -3,6 +3,6 @@
 owner   = root,
 group   = root,
 mode= 0444,
-source  = 'modules/requesttracker/rt.aliases',
+source  = 'puppet:///modules/requesttracker/rt.aliases',
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/180643
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I67c2d1a316ff03ef0c2a66f7394486ddca809a9d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Revert Add 'xenon' module for aggregating ext_xenon-produce... - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/179954

Change subject: Revert Add 'xenon' module for aggregating ext_xenon-produced 
traces.
..

Revert Add 'xenon' module for aggregating ext_xenon-produced traces.

Let's review such changes first.

This reverts commit 793c30b11d4801abf3249cc2cbe2b09908a64b26.

Change-Id: I4b7a2c91b964a0f4fab27dba50da547661b6b44a
---
M manifests/role/performance.pp
D manifests/role/xenon.pp
D modules/xenon/files/flamegraph.pl
D modules/xenon/files/xenon-generate-svgs
D modules/xenon/files/xenon-log
D modules/xenon/files/xenon-log.conf
D modules/xenon/manifests/init.pp
M templates/apache/sites/performance.wikimedia.org.erb
D templates/apache/sites/xenon.erb
9 files changed, 0 insertions(+), 1,137 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/54/179954/1

diff --git a/manifests/role/performance.pp b/manifests/role/performance.pp
index 14e2da2..226a957 100644
--- a/manifests/role/performance.pp
+++ b/manifests/role/performance.pp
@@ -5,7 +5,6 @@
 #
 class role::performance {
 include ::apache
-include ::apache::mod::proxy
 
 package { 'reporter/reporter':
 provider = 'trebuchet',
diff --git a/manifests/role/xenon.pp b/manifests/role/xenon.pp
deleted file mode 100644
index 138d350..000
--- a/manifests/role/xenon.pp
+++ /dev/null
@@ -1,25 +0,0 @@
-# == Class: role::xenon
-#
-# Aggregates and graphs stack trace snapshots from MediaWiki
-# application servers, showing where time is spent.
-#
-class role::xenon {
-include ::xenon
-
-class { '::redis':
-maxmemory = '1Mb',
-persist   = undef,
-redis_replication = undef,
-}
-
-Service['redis'] ~ Service['xenon']
-
-apache::site { 'xenon':
-content = template('apache/sites/xenon.erb'),
-}
-
-ferm::service { 'xenon_http':
-proto = 'tcp',
-port  = '80',
-}
-}
diff --git a/modules/xenon/files/flamegraph.pl 
b/modules/xenon/files/flamegraph.pl
deleted file mode 100755
index def5ed0..000
--- a/modules/xenon/files/flamegraph.pl
+++ /dev/null
@@ -1,846 +0,0 @@
-#!/usr/bin/perl -w
-#
-# flamegraph.plflame stack grapher.
-#
-# This takes stack samples and renders a call graph, allowing hot functions
-# and codepaths to be quickly identified.  Stack samples can be generated using
-# tools such as DTrace, perf, SystemTap, and Instruments.
-#
-# USAGE: ./flamegraph.pl [options] input.txt  graph.svg
-#
-#grep funcA input.txt | ./flamegraph.pl [options]  graph.svg
-#
-# Options are listed in the usage message (--help).
-#
-# The input is stack frames and sample counts formatted as single lines.  Each
-# frame in the stack is semicolon separated, with a space and count at the end
-# of the line.  These can be generated using DTrace with stackcollapse.pl,
-# and other tools using the stackcollapse variants.
-#
-# An optional extra column of counts can be provided to generate a differential
-# flame graph of the counts, colored red for more, and blue for less.  This
-# can be useful when using flame graphs for non-regression testing.
-# See the header comment in the difffolded.pl program for instructions.
-#
-# The output graph shows relative presence of functions in stack samples.  The
-# ordering on the x-axis has no meaning; since the data is samples, time order
-# of events is not known.  The order used sorts function names alphabetically.
-#
-# While intended to process stack samples, this can also process stack traces.
-# For example, tracing stacks for memory allocation, or resource usage.  You
-# can use --title to set the title to reflect the content, and --countname
-# to change samples to bytes etc.
-#
-# There are a few different palettes, selectable using --color.  By default,
-# the colors are selected at random (except for differentials).  Functions
-# called - will be printed gray, which can be used for stack separators (eg,
-# between user and kernel stacks).
-#
-# HISTORY
-#
-# This was inspired by Neelakanth Nadgir's excellent function_call_graph.rb
-# program, which visualized function entry and return trace events.  As Neel
-# wrote: The output displayed is inspired by Roch's CallStackAnalyzer which
-# was in turn inspired by the work on vftrace by Jan Boerhout.  See:
-# https://blogs.oracle.com/realneel/entry/visualizing_callstacks_via_dtrace_and
-#
-# Copyright 2011 Joyent, Inc.  All rights reserved.
-# Copyright 2011 Brendan Gregg.  All rights reserved.
-#
-# CDDL HEADER START
-#
-# The contents of this file are subject to the terms of the
-# Common Development and Distribution License (the License).
-# You may not use this file except in compliance with the License.
-#
-# You can obtain a copy of the license at docs/cddl1.txt or
-# http://opensource.org/licenses/CDDL-1.0.
-# See the License for the specific language governing 

[MediaWiki-commits] [Gerrit] Revert Apply role::xenon on fluorine - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/179955

Change subject: Revert Apply role::xenon on fluorine
..

Revert Apply role::xenon on fluorine

This reverts commit 1fecdaddce7e60579e08066f1bf44092e8f7890d.

Change-Id: I38c43801827f1d6e217eb4dc35de0db0596d37dd
---
M manifests/site.pp
1 file changed, 0 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/55/179955/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 337ed0d..ed661cb 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1209,7 +1209,6 @@
 
 include standard
 include misc::deployment::fatalmonitor
-include ::role::xenon
 
 class { 'admin':
 groups = [

-- 
To view, visit https://gerrit.wikimedia.org/r/179955
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I38c43801827f1d6e217eb4dc35de0db0596d37dd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Allocate frack management subnets - change (operations/dns)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/175718

Change subject: Allocate frack management subnets
..

Allocate frack management subnets

Change-Id: Ic9f84b6f1cc2b1d442d3a75701db33414bc5d4ad
---
M templates/10.in-addr.arpa
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dns 
refs/changes/18/175718/1

diff --git a/templates/10.in-addr.arpa b/templates/10.in-addr.arpa
index 092c0a2..0d2ca67 100644
--- a/templates/10.in-addr.arpa
+++ b/templates/10.in-addr.arpa
@@ -897,6 +897,8 @@
 ; 10.64.40.160/27 - frack-listenerdmz1-c-eqiad
 162 1H IN PTR   thulium.frack.eqiad.wmnet.
 
+; 10.64.40.192/27 - frack management eqiad
+
 ; 10.64.48.0/22 - private1-d-eqiad
 $ORIGIN 48.64.{{ zonename }}.
 
@@ -2949,6 +2951,8 @@
 ; 10.195.0.80/29 - frack-listenerdmz1-c-codfw
 82 1H IN PTR  saiph.frack.codfw.wmnet.
 
+; 10.195.0.96/27 - frack management codfw
+
 ; LEGACY
 
 $ORIGIN 1.230.{{ zonename }}.

-- 
To view, visit https://gerrit.wikimedia.org/r/175718
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic9f84b6f1cc2b1d442d3a75701db33414bc5d4ad
Gerrit-PatchSet: 1
Gerrit-Project: operations/dns
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Allocate labstore200[12] mgmt IPs - change (operations/dns)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/173313

Change subject: Allocate labstore200[12] mgmt IPs
..

Allocate labstore200[12] mgmt IPs

Change-Id: I0e1851a5542acdee48a196dd1d34d27ac03a7a41
---
M templates/10.in-addr.arpa
M templates/wmnet
2 files changed, 8 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/dns 
refs/changes/13/173313/1

diff --git a/templates/10.in-addr.arpa b/templates/10.in-addr.arpa
index c551aeb..98d9436 100644
--- a/templates/10.in-addr.arpa
+++ b/templates/10.in-addr.arpa
@@ -2806,6 +2806,10 @@
 246 1H IN PTR  wmf6139.mgmt.codfw.wmnet.
 247 1H IN PTR  ms-be2015.mgmt.codfw.wmnet.
 247 1H IN PTR  wmf6140.mgmt.codfw.wmnet.
+248 1H IN PTR  labstore2001.mgmt.codfw.wmnet.
+248 1H IN PTR  wmf3649.mgmt.codfw.wmnet.
+249 1H IN PTR  labstore2002.mgmt.codfw.wmnet.
+249 1H IN PTR  wmf3651.mgmt.codfw.wmnet.
 
 ; 10.195.0.0/25 - frack.codfw subnets
 $ORIGIN 0.195.{{ zonename }}.
diff --git a/templates/wmnet b/templates/wmnet
index 5318f5a..73dfb06 100644
--- a/templates/wmnet
+++ b/templates/wmnet
@@ -2177,6 +2177,8 @@
 labsdb2001  1H  IN A10.193.1.26
 labsdb2002  1H  IN A10.193.1.27
 labsdb2003  1H  IN A10.193.1.28
+labstore20011H  IN A10.193.1.248
+labstore20021H  IN A10.193.1.249
 lvs2001 1H  IN A10.193.1.1
 lvs2002 1H  IN A10.193.1.2
 lvs2003 1H  IN A10.193.1.3
@@ -2375,6 +2377,8 @@
 wmf3640 1H  IN A10.193.1.34
 wmf3641 1H  IN A10.193.1.50
 wmf3647 1H  IN A10.193.1.62
+wmf3649 1H  IN A10.193.1.248
+wmf3651 1H  IN A10.193.1.249
 wmf3662 1H  IN A10.192.1.12
 wmf3663 1H  IN A10.193.1.35
 wmf3664 1H  IN A10.193.1.36

-- 
To view, visit https://gerrit.wikimedia.org/r/173313
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0e1851a5542acdee48a196dd1d34d27ac03a7a41
Gerrit-PatchSet: 1
Gerrit-Project: operations/dns
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Allocate labstore200[12] mgmt IPs - change (operations/dns)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Allocate labstore200[12] mgmt IPs
..


Allocate labstore200[12] mgmt IPs

Change-Id: I0e1851a5542acdee48a196dd1d34d27ac03a7a41
---
M templates/10.in-addr.arpa
M templates/wmnet
2 files changed, 8 insertions(+), 0 deletions(-)

Approvals:
  Mark Bergsma: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/templates/10.in-addr.arpa b/templates/10.in-addr.arpa
index c551aeb..98d9436 100644
--- a/templates/10.in-addr.arpa
+++ b/templates/10.in-addr.arpa
@@ -2806,6 +2806,10 @@
 246 1H IN PTR  wmf6139.mgmt.codfw.wmnet.
 247 1H IN PTR  ms-be2015.mgmt.codfw.wmnet.
 247 1H IN PTR  wmf6140.mgmt.codfw.wmnet.
+248 1H IN PTR  labstore2001.mgmt.codfw.wmnet.
+248 1H IN PTR  wmf3649.mgmt.codfw.wmnet.
+249 1H IN PTR  labstore2002.mgmt.codfw.wmnet.
+249 1H IN PTR  wmf3651.mgmt.codfw.wmnet.
 
 ; 10.195.0.0/25 - frack.codfw subnets
 $ORIGIN 0.195.{{ zonename }}.
diff --git a/templates/wmnet b/templates/wmnet
index 5318f5a..73dfb06 100644
--- a/templates/wmnet
+++ b/templates/wmnet
@@ -2177,6 +2177,8 @@
 labsdb2001  1H  IN A10.193.1.26
 labsdb2002  1H  IN A10.193.1.27
 labsdb2003  1H  IN A10.193.1.28
+labstore20011H  IN A10.193.1.248
+labstore20021H  IN A10.193.1.249
 lvs2001 1H  IN A10.193.1.1
 lvs2002 1H  IN A10.193.1.2
 lvs2003 1H  IN A10.193.1.3
@@ -2375,6 +2377,8 @@
 wmf3640 1H  IN A10.193.1.34
 wmf3641 1H  IN A10.193.1.50
 wmf3647 1H  IN A10.193.1.62
+wmf3649 1H  IN A10.193.1.248
+wmf3651 1H  IN A10.193.1.249
 wmf3662 1H  IN A10.192.1.12
 wmf3663 1H  IN A10.193.1.35
 wmf3664 1H  IN A10.193.1.36

-- 
To view, visit https://gerrit.wikimedia.org/r/173313
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0e1851a5542acdee48a196dd1d34d27ac03a7a41
Gerrit-PatchSet: 1
Gerrit-Project: operations/dns
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: jenkins-bot 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add error logging for ipvsadm invocations - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/172215

Change subject: Add error logging for ipvsadm invocations
..

Add error logging for ipvsadm invocations

Rewrites the existing (synchronous) invocation of ipvsadm using
os.popen using Twisted's (asynchronous) spawnProcess.

UNTESTED MOCKUP

Change-Id: I2f58010d2e19004b2524e0858254f5b846478f6b
---
M pybal/ipvs.py
1 file changed, 29 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/15/172215/1

diff --git a/pybal/ipvs.py b/pybal/ipvs.py
index 2387b38..1e7f6b0 100644
--- a/pybal/ipvs.py
+++ b/pybal/ipvs.py
@@ -5,7 +5,28 @@
 LVS state/configuration classes for PyBal
 
 
-import os
+from twisted.internet import reactor, defer, protocol, error
+
+class IPVSProcessProtocol(protocol.ProcessProtocol):
+def __init__(self, cmdList):
+super(IPVSProcessProtocol, self).__init__()
+
+self.stderr = 
+self.cmdList = cmdList
+
+def connectionMade(self):
+# Send the ipvsadm commands
+self.transport.writeSequence(self.cmdList)
+self.transport.closeStdin()
+
+def errReceived(self, data):
+self.stderr += data
+
+def processExited(self, reason):
+if reason.check(error.ProcessTerminated):
+print ipvsadm exited with status %d when executing cmdlist %s % 
(reason.value.exitCode, self.cmdList)
+print ipvsadm stderr output:
+print self.stderr
 
 class IPVSManager:
 
@@ -23,16 +44,13 @@
 
 
 print cmdList
-if cls.DryRun: return
-
-command = [cls.ipvsPath, '-R']
-stdin = os.popen( .join(command), 'w')
-for line in cmdList:
-stdin.write(line + '\n')
-stdin.close()
-
-# FIXME: Check return code and act on failure
-
+if cls.DryRun: return defer.succeed(0)
+
+ipvsProcessProtocol = IPVSProcessProtocol(cmdList)
+return reactor.spawnProcess(ipvsProcessProtocol, cls.ipvsPath, 
[cls.ipvsPath, '-R'])
+
+# FIXME: Do something with this deferred
+
 modifyState = classmethod(modifyState)
 
 def subCommandService(service):

-- 
To view, visit https://gerrit.wikimedia.org/r/172215
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2f58010d2e19004b2524e0858254f5b846478f6b
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Add error logging for ipvsadm invocations - change (operations...pybal)

[Mark Bergsma (Code Review)]

Mark Bergsma has submitted this change and it was merged.

Change subject: Add error logging for ipvsadm invocations
..


Add error logging for ipvsadm invocations

Rewrites the existing (synchronous) invocation of ipvsadm using
os.popen using Twisted's (asynchronous) spawnProcess.

UNTESTED MOCKUP

Change-Id: I2f58010d2e19004b2524e0858254f5b846478f6b
---
M pybal/ipvs.py
1 file changed, 29 insertions(+), 11 deletions(-)

Objections:
  jenkins-bot: Fails



diff --git a/pybal/ipvs.py b/pybal/ipvs.py
index 2387b38..1e7f6b0 100644
--- a/pybal/ipvs.py
+++ b/pybal/ipvs.py
@@ -5,7 +5,28 @@
 LVS state/configuration classes for PyBal
 
 
-import os
+from twisted.internet import reactor, defer, protocol, error
+
+class IPVSProcessProtocol(protocol.ProcessProtocol):
+def __init__(self, cmdList):
+super(IPVSProcessProtocol, self).__init__()
+
+self.stderr = 
+self.cmdList = cmdList
+
+def connectionMade(self):
+# Send the ipvsadm commands
+self.transport.writeSequence(self.cmdList)
+self.transport.closeStdin()
+
+def errReceived(self, data):
+self.stderr += data
+
+def processExited(self, reason):
+if reason.check(error.ProcessTerminated):
+print ipvsadm exited with status %d when executing cmdlist %s % 
(reason.value.exitCode, self.cmdList)
+print ipvsadm stderr output:
+print self.stderr
 
 class IPVSManager:
 
@@ -23,16 +44,13 @@
 
 
 print cmdList
-if cls.DryRun: return
-
-command = [cls.ipvsPath, '-R']
-stdin = os.popen( .join(command), 'w')
-for line in cmdList:
-stdin.write(line + '\n')
-stdin.close()
-
-# FIXME: Check return code and act on failure
-
+if cls.DryRun: return defer.succeed(0)
+
+ipvsProcessProtocol = IPVSProcessProtocol(cmdList)
+return reactor.spawnProcess(ipvsProcessProtocol, cls.ipvsPath, 
[cls.ipvsPath, '-R'])
+
+# FIXME: Do something with this deferred
+
 modifyState = classmethod(modifyState)
 
 def subCommandService(service):

-- 
To view, visit https://gerrit.wikimedia.org/r/172215
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2f58010d2e19004b2524e0858254f5b846478f6b
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma m...@wikimedia.org
Gerrit-Reviewer: BBlack bbl...@wikimedia.org
Gerrit-Reviewer: Faidon Liambotis fai...@wikimedia.org
Gerrit-Reviewer: Giuseppe Lavagetto glavage...@wikimedia.org
Gerrit-Reviewer: Hashar has...@free.fr
Gerrit-Reviewer: Ori.livneh o...@wikimedia.org
Gerrit-Reviewer: jenkins-bot 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] Stop sending mail to the IMAP server (sanger) - change (operations/puppet)

[Mark Bergsma (Code Review)]

Mark Bergsma has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/164916

Change subject: Stop sending mail to the IMAP server (sanger)
..

Stop sending mail to the IMAP server (sanger)

Change-Id: I1eb1ae65263aac343625129609f0a93e345ac677
---
M templates/exim/exim4.conf.SMTP_IMAP_MM.erb
1 file changed, 0 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/16/164916/1

diff --git a/templates/exim/exim4.conf.SMTP_IMAP_MM.erb 
b/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
index ec4e868..514c2f0 100644
--- a/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
+++ b/templates/exim/exim4.conf.SMTP_IMAP_MM.erb
@@ -486,7 +486,6 @@
local_part_suffix_optional
transport = remote_smtp
route_list = *  aspmx.l.google.com
-   unseen
 
 # LDAP accounts
 ldap_account:
@@ -520,16 +519,6 @@

{user=cn=eximagent,ou=other,dc=corp,dc=wikimedia,dc=org pass=LDAPPASSWORD \

ldap:///ou=people,dc=corp,dc=wikimedia,dc=org?mail?sub?((objectClass=inetOrgPerson)(initials=${quote_ldap:$local_part}@$domain)(x121Address=1))}
 \
{$value}fail}
-
-# Send mail for IMAP accounts to the IMAP server
-imap:
-   driver = manualroute
-   domains = wikimedia.org
-   local_parts = lsearch;CONFDIR/imap_accounts
-   local_part_suffix = +*
-   local_part_suffix_optional
-   transport = remote_smtp
-   route_list = *  sanger.wikimedia.org
 
 # send phabricator.wm.org emails to Phabricator
 phabricator:

-- 
To view, visit https://gerrit.wikimedia.org/r/164916
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1eb1ae65263aac343625129609f0a93e345ac677
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Mark Bergsma m...@wikimedia.org

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits