Ori.livneh has uploaded a new change for review.
https://gerrit.wikimedia.org/r/317831
Change subject: coal: die if no events in N seconds
......................................................................
coal: die if no events in N seconds
Sometimes coal will keep running even when the publisher has stopped
sending it events for whatever reason, and then we have to restart it.
Automate this by dying if three minutes pass with no events.
Change-Id: Ic7b7b601028605afd24144090f49c98e001e2921
---
M modules/coal/files/coal
1 file changed, 8 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/31/317831/1
diff --git a/modules/coal/files/coal b/modules/coal/files/coal
index 18ae7bf..9863f12 100755
--- a/modules/coal/files/coal
+++ b/modules/coal/files/coal
@@ -140,15 +140,23 @@
poll.register(timer_fd, select.EPOLLIN)
syslog.syslog('Entering main loop.')
+
+ intervals_since_last_event = 0
+
while 1:
try:
for fd, _ in poll.poll():
if fd == timer_fd:
os.read(timer_fd, 1)
self.flush_data()
+ intervals_since_last_event += 1
+ if intervals_since_last_event > 2:
+ raise RuntimeError('No events in %d seconds.' % (
+ UPDATE_INTERVAL * intervals_since_last_event))
if fd == socket.fd:
while socket.events & zmq.POLLIN:
meta = socket.recv_json()
+ intervals_since_last_event = 0
self.handle_event(meta)
except IOError as e:
if e.errno != errno.EINTR:
--
To view, visit https://gerrit.wikimedia.org/r/317831
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic7b7b601028605afd24144090f49c98e001e2921
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ori.livneh <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
