Yuvipanda has uploaded a new change for review.
https://gerrit.wikimedia.org/r/203682
Change subject: tools: time out webservice commands after 30s waiting for job
......................................................................
tools: time out webservice commands after 30s waiting for job
Because the jobs are checked only once every 1s, it is possible
for a job to start and quit in the middle of a checking interval,
thus causing webservice commands to hang forever. This can easily
be triggered by starting a nodejs, uwsgi or tomcat job for a tool
that is not set up to have those jobs.
The ideal solution is to do precondition checks for each server
type here (or even autodetect, akin to heroku buildpacks), but
that is not here yet. Timeout after 30s of waiting for each
job operations
Change-Id: I860c03a26de6a8e5c0922328c1a21b309bcf8cc7
---
M modules/toollabs/files/webservice2
1 file changed, 10 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/82/203682/1
diff --git a/modules/toollabs/files/webservice2
b/modules/toollabs/files/webservice2
index 9170595..c4ed2bf 100644
--- a/modules/toollabs/files/webservice2
+++ b/modules/toollabs/files/webservice2
@@ -98,7 +98,7 @@
:param job_name: Name of job to look for
"""
- while True:
+ for i in xrange(30):
job = get_job_xml(job_name)
state = job.findtext('.//state') if job is not None else None
if job is None or 'r' not in state:
@@ -107,6 +107,10 @@
time.sleep(1)
else:
break
+ else:
+ # The loop ended without job being started
+ sys.stderr.write('Timeout: could not start job in 30s')
+ sys.exit(-1)
def wait_for_job_stopped(job_name):
@@ -115,7 +119,7 @@
:param job_name: Name of job to look for
"""
- while True:
+ for i in xrange(30):
job = get_job_xml(job_name)
if job is not None:
sys.stdout.write('.')
@@ -123,6 +127,10 @@
time.sleep(1)
else:
break
+ else:
+ # The loop ended without job being stopped
+ sys.stderr.write('Timeout: could not stop job in 30s')
+ sys.exit(-1)
# Setup constants that we would need later on
PREFIX = read_file('/etc/wmflabs-project', 'tools').strip() # project name
--
To view, visit https://gerrit.wikimedia.org/r/203682
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I860c03a26de6a8e5c0922328c1a21b309bcf8cc7
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Yuvipanda <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
