Paladox has uploaded a new change for review. https://gerrit.wikimedia.org/r/300705
Change subject: 2.1.0-391-gbc58ea3-wmf1precise1 ...................................................................... 2.1.0-391-gbc58ea3-wmf1precise1 Change-Id: I08c9b30da7f0d55d5ba0b436a5742c3af0a48d21 --- A .pbuilderrc M debian/changelog R debian/patches/0001-Ensure-the-repository-configuration-lock-is-released.patch D debian/patches/0001-wmf-soften-requirements.patch R debian/patches/0002-Replace-python-shebang-with-python2.7.patch R debian/patches/0003-Cloner-Implement-cache-no-hardlinks-argument.patch R debian/patches/0004-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch A debian/patches/0005-WMF-soften-paramiko-requirement.patch D debian/patches/0006-Properly-shutdown-apscheduler-on-reconfigure.patch A debian/patches/0006-WMF-soften-WebOb-requirement.patch D debian/patches/0007-GerritWatcher-add-poll_timeout.patch A debian/patches/0007-WMF-drop-requirement-ordereddict.patch D debian/patches/0008-Do-not-import-paramiko-before-daemonization.patch A debian/patches/0008-Gerrit-trailing-delay-is-now-configurable.patch D debian/patches/0009-Register-connections-when-testing-configuration.patch A debian/patches/0009-WMF-soften-pbr-requirement.patch A debian/patches/0010-WMF-constraint-apscheduler-to-3.1.0.patch M debian/patches/series M debian/rules 19 files changed, 330 insertions(+), 271 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/integration/zuul refs/changes/05/300705/1 diff --git a/.pbuilderrc b/.pbuilderrc new file mode 100644 index 0000000..5c5dc40 --- /dev/null +++ b/.pbuilderrc @@ -0,0 +1,3 @@ +BASEPATH=/mnt/pbuilder/base-precise-amd64.cow/ +BUILDRESULT=/mnt/pbuilder/result/precise-amd64/ +BUILDPLACE="/mnt/pbuilder/build/" diff --git a/debian/changelog b/debian/changelog index 1399df7..7f1bb3d 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,22 @@ +zuul (2.1.0-391-gbc58ea3-wmf1precise1) precise-wikimedia; urgency=medium + + * New upstream release + * Rebase patches + * Remove patches that have been merged upstream: + - Properly-shutdown-apscheduler-on-reconfigure.patch + - GerritWatcher-add-poll_timeout.patch + - Do-not-import-paramiko-before-daemonization.patch + - Register-connections-when-testing-configuration.patch + * Split our requirement tweaking patch wmf-soften-requirements.patch in + new atomic patches. + * Let us change the Gerrit event delay: + [12d23bf53] Gerrit trailing delay is now configurable + * Prevent dh_virtualenv from running tests by setting + DEB_BUILD_OPTIONS=nocheck . Apparently tries to run them outside of the + virtualenv/without using system packages. + + -- Antoine Musso <has...@free.fr> Fri, 22 Jul 2016 14:46:02 +0200 + zuul (2.1.0-151-g30a433b-wmf4precise1) precise-wikimedia; urgency=medium * Zuul server layout validation would not recognize multiple Gerrit diff --git a/debian/patches/0002-Ensure-the-repository-configuration-lock-is-released.patch b/debian/patches/0001-Ensure-the-repository-configuration-lock-is-released.patch similarity index 98% rename from debian/patches/0002-Ensure-the-repository-configuration-lock-is-released.patch rename to debian/patches/0001-Ensure-the-repository-configuration-lock-is-released.patch index 52db537..ff09636 100644 --- a/debian/patches/0002-Ensure-the-repository-configuration-lock-is-released.patch +++ b/debian/patches/0001-Ensure-the-repository-configuration-lock-is-released.patch @@ -24,7 +24,7 @@ 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/zuul/merger/merger.py b/zuul/merger/merger.py -index c6ae35d..fed8394 100644 +index 94933de..d7867af 100644 --- a/zuul/merger/merger.py +++ b/zuul/merger/merger.py @@ -70,7 +70,18 @@ class Repo(object): diff --git a/debian/patches/0001-wmf-soften-requirements.patch b/debian/patches/0001-wmf-soften-requirements.patch deleted file mode 100644 index 71062c7..0000000 --- a/debian/patches/0001-wmf-soften-requirements.patch +++ /dev/null @@ -1,44 +0,0 @@ -From: Antoine Musso <has...@free.fr> -Date: Mon, 4 Nov 2013 12:17:58 +0100 -Subject: wmf: soften requirements - -WebOb and paramiko have a minimal version which is enforced by OpenStack -but is probably not really needed. - -ordereddict and argparse are included in python 2.7 and we do not care -about python 2.6. - -apscheduler 3.1.0 fail with old setuptools -https://github.com/agronholm/apscheduler/issues/123 - -Change-Id: Ie2bffccd0b4aa8b505a5c7de8174a42d4395d9d7 ---- - requirements.txt | 9 ++++----- - 1 file changed, 4 insertions(+), 5 deletions(-) - -diff --git a/requirements.txt b/requirements.txt -index 77ac0a5..0c85d7a 100644 ---- a/requirements.txt -+++ b/requirements.txt -@@ -1,17 +1,16 @@ --pbr>=1.1.0 -+pbr>=0.5.21 - - PyYAML>=3.1.0 - Paste --WebOb>=1.2.3 --paramiko>=1.8.0,<2.0.0 -+WebOb -+paramiko<2.0.0 - GitPython>=0.3.3 --ordereddict - python-daemon>=2.0.4,<2.1.0 - extras - statsd>=1.0.0,<3.0 - voluptuous>=0.7 - gear>=0.5.7,<1.0.0 --apscheduler>=3.0 -+apscheduler>=3.0,<3.1.0 - PrettyTable>=0.6,<0.8 - babel>=1.0 - six>=1.6.0 diff --git a/debian/patches/0003-Replace-python-shebang-with-python2.7.patch b/debian/patches/0002-Replace-python-shebang-with-python2.7.patch similarity index 93% rename from debian/patches/0003-Replace-python-shebang-with-python2.7.patch rename to debian/patches/0002-Replace-python-shebang-with-python2.7.patch index 014feea..afd209b 100644 --- a/debian/patches/0003-Replace-python-shebang-with-python2.7.patch +++ b/debian/patches/0002-Replace-python-shebang-with-python2.7.patch @@ -36,7 +36,7 @@ # # Licensed under the Apache License, Version 2.0 (the "License"); diff --git a/tests/base.py b/tests/base.py -index 405caa0..4f41be9 100755 +index 38d2817..6d4f017 100755 --- a/tests/base.py +++ b/tests/base.py @@ -1,4 +1,4 @@ @@ -46,7 +46,7 @@ # Copyright 2012 Hewlett-Packard Development Company, L.P. # diff --git a/tests/test_cloner.py b/tests/test_cloner.py -index 137c157..1ce4a0a 100644 +index e3576bd..064a61c 100644 --- a/tests/test_cloner.py +++ b/tests/test_cloner.py @@ -1,4 +1,4 @@ @@ -56,7 +56,7 @@ # Copyright 2012 Hewlett-Packard Development Company, L.P. # Copyright 2014 Wikimedia Foundation Inc. diff --git a/tests/test_layoutvalidator.py b/tests/test_layoutvalidator.py -index 3dc3234..99732a5 100644 +index 46a8c7c..101e0d0 100644 --- a/tests/test_layoutvalidator.py +++ b/tests/test_layoutvalidator.py @@ -1,4 +1,4 @@ @@ -86,7 +86,7 @@ # Copyright 2012-2014 Hewlett-Packard Development Company, L.P. # diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py -index fe7c7cc..cbf1495 100755 +index 628775d..ac70091 100755 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1,4 +1,4 @@ @@ -96,7 +96,7 @@ # Copyright 2012 Hewlett-Packard Development Company, L.P. # diff --git a/tests/test_webapp.py b/tests/test_webapp.py -index b127c51..8a88261 100644 +index 94f097a..93d88a8 100644 --- a/tests/test_webapp.py +++ b/tests/test_webapp.py @@ -1,4 +1,4 @@ @@ -116,7 +116,7 @@ # Copyright 2014 Hewlett-Packard Development Company, L.P. # diff --git a/tools/trigger-job.py b/tools/trigger-job.py -index dff4e3f..4651d7d 100755 +index 7123afc..0c307ca 100755 --- a/tools/trigger-job.py +++ b/tools/trigger-job.py @@ -1,4 +1,4 @@ @@ -126,7 +126,7 @@ # # Licensed under the Apache License, Version 2.0 (the "License"); you may diff --git a/tools/zuul-changes.py b/tools/zuul-changes.py -index 9dbf504..d825ef1 100755 +index 8b854c7..e156268 100755 --- a/tools/zuul-changes.py +++ b/tools/zuul-changes.py @@ -1,4 +1,4 @@ @@ -136,7 +136,7 @@ # Copyright 2015 Hewlett-Packard Development Company, L.P. # diff --git a/zuul/cmd/__init__.py b/zuul/cmd/__init__.py -index 2902c50..7ee7900 100644 +index 5ffd431..120c62c 100644 --- a/zuul/cmd/__init__.py +++ b/zuul/cmd/__init__.py @@ -1,4 +1,4 @@ @@ -146,7 +146,7 @@ # Copyright 2013 OpenStack Foundation # diff --git a/zuul/cmd/client.py b/zuul/cmd/client.py -index 59ac419..e31f467 100644 +index 1ce2828..61183da 100644 --- a/zuul/cmd/client.py +++ b/zuul/cmd/client.py @@ -1,4 +1,4 @@ @@ -156,7 +156,7 @@ # Copyright 2013 OpenStack Foundation # diff --git a/zuul/cmd/cloner.py b/zuul/cmd/cloner.py -index c616aa1..a922a34 100755 +index 4f8b9f4..dbbe396 100755 --- a/zuul/cmd/cloner.py +++ b/zuul/cmd/cloner.py @@ -1,4 +1,4 @@ @@ -166,7 +166,7 @@ # Copyright 2014 Antoine "hashar" Musso # Copyright 2014 Wikimedia Foundation Inc. diff --git a/zuul/cmd/merger.py b/zuul/cmd/merger.py -index df215fd..5f51ee6 100644 +index 797a990..2807f9e 100644 --- a/zuul/cmd/merger.py +++ b/zuul/cmd/merger.py @@ -1,4 +1,4 @@ @@ -176,7 +176,7 @@ # Copyright 2013-2014 OpenStack Foundation # diff --git a/zuul/cmd/server.py b/zuul/cmd/server.py -index b1cd050..8861972 100755 +index 0b7538d..59834ad 100755 --- a/zuul/cmd/server.py +++ b/zuul/cmd/server.py @@ -1,4 +1,4 @@ diff --git a/debian/patches/0004-Cloner-Implement-cache-no-hardlinks-argument.patch b/debian/patches/0003-Cloner-Implement-cache-no-hardlinks-argument.patch similarity index 83% rename from debian/patches/0004-Cloner-Implement-cache-no-hardlinks-argument.patch rename to debian/patches/0003-Cloner-Implement-cache-no-hardlinks-argument.patch index 905027c..e6555d7 100644 --- a/debian/patches/0004-Cloner-Implement-cache-no-hardlinks-argument.patch +++ b/debian/patches/0003-Cloner-Implement-cache-no-hardlinks-argument.patch @@ -15,8 +15,8 @@ --- doc/source/cloner.rst | 5 +++++ zuul/cmd/cloner.py | 5 +++++ - zuul/lib/cloner.py | 11 ++++++++--- - 3 files changed, 18 insertions(+), 3 deletions(-) + zuul/lib/cloner.py | 10 +++++++--- + 3 files changed, 17 insertions(+), 3 deletions(-) diff --git a/doc/source/cloner.rst b/doc/source/cloner.rst index 70577cc..c0ca990 100644 @@ -32,10 +32,10 @@ +the cache directory is on the same disk as the workspace, git-clone +will hardlink git objects to speed up the process and save space. diff --git a/zuul/cmd/cloner.py b/zuul/cmd/cloner.py -index a922a34..63825e4 100755 +index dbbe396..0d045be 100755 --- a/zuul/cmd/cloner.py +++ b/zuul/cmd/cloner.py -@@ -57,6 +57,10 @@ class Cloner(zuul.cmd.ZuulApp): +@@ -59,6 +59,10 @@ class Cloner(zuul.cmd.ZuulApp): 'Can also be set via ZUUL_CACHE_DIR ' 'environment variable.' )) @@ -46,29 +46,28 @@ parser.add_argument('git_base_url', help='reference repo to clone from') parser.add_argument('projects', nargs='+', -@@ -145,6 +149,7 @@ class Cloner(zuul.cmd.ZuulApp): - clone_map_file=self.args.clone_map_file, - project_branches=project_branches, +@@ -153,6 +157,7 @@ class Cloner(zuul.cmd.ZuulApp): cache_dir=self.args.cache_dir, + zuul_newrev=self.args.zuul_newrev, + zuul_project=self.args.zuul_project, + cache_no_hardlinks=self.args.cache_no_hardlinks, ) cloner.execute() diff --git a/zuul/lib/cloner.py b/zuul/lib/cloner.py -index f0235a6..3a12a40 100644 +index 197c426..249fc30 100644 --- a/zuul/lib/cloner.py +++ b/zuul/lib/cloner.py -@@ -29,7 +29,8 @@ class Cloner(object): - +@@ -33,7 +33,7 @@ class Cloner(object): def __init__(self, git_base_url, projects, workspace, zuul_branch, zuul_ref, zuul_url, branch=None, clone_map_file=None, -- project_branches=None, cache_dir=None): -+ project_branches=None, cache_dir=None, -+ cache_no_hardlinks=None): + project_branches=None, cache_dir=None, zuul_newrev=None, +- zuul_project=None): ++ zuul_project=None, cache_no_hardlinks=None): self.clone_map = [] self.dests = None -@@ -37,6 +38,7 @@ class Cloner(object): +@@ -41,6 +41,7 @@ class Cloner(object): self.branch = branch self.git_url = git_base_url self.cache_dir = cache_dir @@ -76,7 +75,7 @@ self.projects = projects self.workspace = workspace self.zuul_branch = zuul_branch or '' -@@ -74,8 +76,11 @@ class Cloner(object): +@@ -82,8 +83,11 @@ class Cloner(object): if (self.cache_dir and os.path.exists(git_cache) and not repo_is_cloned): diff --git a/debian/patches/0005-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch b/debian/patches/0004-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch similarity index 97% rename from debian/patches/0005-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch rename to debian/patches/0004-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch index 01df976..de6f434 100644 --- a/debian/patches/0005-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch +++ b/debian/patches/0004-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch @@ -26,7 +26,7 @@ 2 files changed, 47 insertions(+), 15 deletions(-) diff --git a/tests/test_cloner.py b/tests/test_cloner.py -index 1ce4a0a..2f5f8a7 100644 +index 064a61c..1c7cdc1 100644 --- a/tests/test_cloner.py +++ b/tests/test_cloner.py @@ -15,6 +15,7 @@ @@ -67,10 +67,10 @@ self.worker.hold_jobs_in_build = True diff --git a/zuul/lib/cloner.py b/zuul/lib/cloner.py -index 3a12a40..77f52fa 100644 +index 249fc30..fd4d0e8 100644 --- a/zuul/lib/cloner.py +++ b/zuul/lib/cloner.py -@@ -71,25 +71,34 @@ class Cloner(object): +@@ -78,25 +78,34 @@ class Cloner(object): def cloneUpstream(self, project, dest): # Check for a cached git repo first git_cache = '%s/%s' % (self.cache_dir, project) diff --git a/debian/patches/0005-WMF-soften-paramiko-requirement.patch b/debian/patches/0005-WMF-soften-paramiko-requirement.patch new file mode 100644 index 0000000..aa8dc80 --- /dev/null +++ b/debian/patches/0005-WMF-soften-paramiko-requirement.patch @@ -0,0 +1,24 @@ +From: Antoine Musso <has...@free.fr> +Date: Fri, 22 Jul 2016 15:15:54 +0200 +Subject: WMF: soften paramiko requirement + +1.7 on Precise is good enough for us. + +Change-Id: I561e5ed2ff459af02910ef4bb1d803976970a48b +--- + requirements.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/requirements.txt b/requirements.txt +index 77ac0a5..2fe384b 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -3,7 +3,7 @@ pbr>=1.1.0 + PyYAML>=3.1.0 + Paste + WebOb>=1.2.3 +-paramiko>=1.8.0,<2.0.0 ++paramiko + GitPython>=0.3.3 + ordereddict + python-daemon>=2.0.4,<2.1.0 diff --git a/debian/patches/0006-Properly-shutdown-apscheduler-on-reconfigure.patch b/debian/patches/0006-Properly-shutdown-apscheduler-on-reconfigure.patch deleted file mode 100644 index ea5488f..0000000 --- a/debian/patches/0006-Properly-shutdown-apscheduler-on-reconfigure.patch +++ /dev/null @@ -1,44 +0,0 @@ -From: "James E. Blair" <jebl...@redhat.com> -Date: Thu, 12 May 2016 11:30:05 -0700 -Subject: Properly shutdown apscheduler on reconfigure - -This addresses a recent increase in test_idle failures. The -existing apscheduler was not being shut down during reconfigurations -which caused us to end up with two apschedulers running during the -test. This could cause a trigger event to fire while the test -was expecting the system to be idle. - -The triggers were not previously shut down during reconfigurations. -Instead, the timer trigger relied on having its postConfig method -called on each reconfiguration, where it would begin by cleaning -up any existing jobs. However, since the connections changes, -triggers are stopped, discarded, and recreated during reconfiguration. -Because the stop method of the timer trigger was not actually being -called, a new trigger was created each time we reconfigured, and old -ones were never cleaned up. - -This likely had a production impact as well, however, it was not -likely to be visible unless a configuration change altered the -scheduled times for periodic queues (in that case, we would see jobs -run at both the old and new times). - -Change-Id: Ia7c61984a9c47a9b1554a4ccb99309674dffec11 ---- - zuul/trigger/timer.py | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -diff --git a/zuul/trigger/timer.py b/zuul/trigger/timer.py -index d42e3db..f81312e 100644 ---- a/zuul/trigger/timer.py -+++ b/zuul/trigger/timer.py -@@ -40,8 +40,8 @@ class TimerTrigger(BaseTrigger): - self.log.debug("Adding event %s" % event) - self.sched.addEvent(event) - -- def _shutdown(self): -- self.apsched.stop() -+ def stop(self): -+ self.apsched.shutdown() - - def getEventFilters(self, trigger_conf): - def toList(item): diff --git a/debian/patches/0006-WMF-soften-WebOb-requirement.patch b/debian/patches/0006-WMF-soften-WebOb-requirement.patch new file mode 100644 index 0000000..6f97948 --- /dev/null +++ b/debian/patches/0006-WMF-soften-WebOb-requirement.patch @@ -0,0 +1,24 @@ +From: Antoine Musso <has...@free.fr> +Date: Fri, 22 Jul 2016 15:16:35 +0200 +Subject: WMF: soften WebOb requirement + +Precise version is good enough for us. + +Change-Id: I8ec22828b4c44f6869f06834a4123a1801ce97c1 +--- + requirements.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/requirements.txt b/requirements.txt +index 2fe384b..e06feb5 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -2,7 +2,7 @@ pbr>=1.1.0 + + PyYAML>=3.1.0 + Paste +-WebOb>=1.2.3 ++WebOb + paramiko + GitPython>=0.3.3 + ordereddict diff --git a/debian/patches/0007-GerritWatcher-add-poll_timeout.patch b/debian/patches/0007-GerritWatcher-add-poll_timeout.patch deleted file mode 100644 index 6af423a..0000000 --- a/debian/patches/0007-GerritWatcher-add-poll_timeout.patch +++ /dev/null @@ -1,35 +0,0 @@ -From: Tristan Cacqueray <tdeca...@redhat.com> -Date: Sun, 31 Jan 2016 11:15:36 -0500 -Subject: GerritWatcher: add poll_timeout - -So that zuul reload process can happen when gerrit stream is silent. -Otherwise the main loop doesn't exit out of the poll call and doesn't -evaluate the "while not self._stopped" location. - -Change-Id: If3129a65da6119acf69ad00e2e78c7ec82a49941 -Closes-Bug: https://storyboard.openstack.org/#!/story/2000472 ---- - zuul/connection/gerrit.py | 3 ++- - 1 file changed, 2 insertions(+), 1 deletion(-) - -diff --git a/zuul/connection/gerrit.py b/zuul/connection/gerrit.py -index 4671ff9..a1854f4 100644 ---- a/zuul/connection/gerrit.py -+++ b/zuul/connection/gerrit.py -@@ -132,6 +132,7 @@ class GerritEventConnector(threading.Thread): - - class GerritWatcher(threading.Thread): - log = logging.getLogger("gerrit.GerritWatcher") -+ poll_timeout = 500 - - def __init__(self, gerrit_connection, username, hostname, port=29418, - keyfile=None): -@@ -154,7 +155,7 @@ class GerritWatcher(threading.Thread): - poll = select.poll() - poll.register(stdout.channel) - while not self._stopped: -- ret = poll.poll() -+ ret = poll.poll(self.poll_timeout) - for (fd, event) in ret: - if fd == stdout.channel.fileno(): - if event == select.POLLIN: diff --git a/debian/patches/0007-WMF-drop-requirement-ordereddict.patch b/debian/patches/0007-WMF-drop-requirement-ordereddict.patch new file mode 100644 index 0000000..93f7b93 --- /dev/null +++ b/debian/patches/0007-WMF-drop-requirement-ordereddict.patch @@ -0,0 +1,23 @@ +From: Antoine Musso <has...@free.fr> +Date: Fri, 22 Jul 2016 15:17:53 +0200 +Subject: WMF: drop requirement ordereddict + +Only used for Python 2.6 and we use 2.7. + +Change-Id: I16d0ccaec46a79aaa11659213b7232e54a5a8ddd +--- + requirements.txt | 1 - + 1 file changed, 1 deletion(-) + +diff --git a/requirements.txt b/requirements.txt +index e06feb5..69484c7 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -5,7 +5,6 @@ Paste + WebOb + paramiko + GitPython>=0.3.3 +-ordereddict + python-daemon>=2.0.4,<2.1.0 + extras + statsd>=1.0.0,<3.0 diff --git a/debian/patches/0008-Do-not-import-paramiko-before-daemonization.patch b/debian/patches/0008-Do-not-import-paramiko-before-daemonization.patch deleted file mode 100644 index ea33e88..0000000 --- a/debian/patches/0008-Do-not-import-paramiko-before-daemonization.patch +++ /dev/null @@ -1,57 +0,0 @@ -From: Antoine Musso <has...@free.fr> -Date: Fri, 15 Jul 2016 14:29:39 +0200 -Subject: Do not import paramiko before daemonization - -The zuul-server would hang when establishing a Gerrit ssh connection due -to Random.Crypto() failling to acquire random number from /dev/urandom. -It would block on read() and never process. - -The root cause is the Zuul refactoring of Gerrit connections. The root -class of all zuul CLI imports zuul.lib.connections globally which -imports paramiko / Crypto.Random. - -When the Server command line invokes the daemonization, python-daemon -close all file descriptors. Including /dev/urandom. Then the daemonized -establish the SSH connection and fail to get random number because -Random.Crypto() locks on read() on a closed file description. - -See upstream issue: -https://github.com/paramiko/paramiko/issues/59 - -Stackoverflow answers by EarlCrapstone: -http://stackoverflow.com/questions/20636678/paramiko-inside-python-daemon-causes-ioerror - -The fix is in Paramiko 1.11.6, which make it use os.urandom: -https://github.com/paramiko/paramiko/commit/6f211115f49edcea7d23b764d7cf3a84ff12f5f0 - -Move the zuul.lib.connections import to the method that relies on it: -ZuulApp.configure.connections(). - -Bug: T137525 -Change-Id: Iea3288abfde6ff2b7339fae775ad5561009dc129 ---- - zuul/cmd/__init__.py | 5 +++-- - 1 file changed, 3 insertions(+), 2 deletions(-) - -diff --git a/zuul/cmd/__init__.py b/zuul/cmd/__init__.py -index 7ee7900..966d1f7 100644 ---- a/zuul/cmd/__init__.py -+++ b/zuul/cmd/__init__.py -@@ -26,8 +26,6 @@ import traceback - - yappi = extras.try_import('yappi') - --import zuul.lib.connections -- - # Do not import modules that will pull in paramiko which must not be - # imported until after the daemonization. - # https://github.com/paramiko/paramiko/issues/59 -@@ -91,5 +89,8 @@ class ZuulApp(object): - logging.basicConfig(level=logging.DEBUG) - - def configure_connections(self): -+ # See comment at top of file about zuul imports -+ import zuul.lib.connections -+ - self.connections = zuul.lib.connections.configure_connections( - self.config) diff --git a/debian/patches/0008-Gerrit-trailing-delay-is-now-configurable.patch b/debian/patches/0008-Gerrit-trailing-delay-is-now-configurable.patch new file mode 100644 index 0000000..3ec03c4 --- /dev/null +++ b/debian/patches/0008-Gerrit-trailing-delay-is-now-configurable.patch @@ -0,0 +1,148 @@ +From: Antoine Musso <has...@free.fr> +Date: Mon, 18 Jul 2016 12:11:13 +0200 +Subject: Gerrit trailing delay is now configurable + +Zuul handling of events is always behind Gerrit by 10 seconds due to +change dependencies tracking being off immediately after a patchset has +been uploaded. + +Make it a configuration option to easily fine tune the delay. On some +third parties setup the delay can be significantly reduced. + +GerritEventConnector() delay moved to a class parameter, default to 10. + +GerritConnection() now recognizes the optional configuration setting +'event_delay' in zuul.conf representing the amount of seconds to delay. +Default to 10 as well. + +Change the way test/base.py set the GerritEventConnector delay by +injecting the 'event_delay' setting when the test suite load the +configuration. + +Fix an unrelated typo in GerritConnection debug log. + +Update documentation inspired by James E. Blair code comment and commit +message summaries that introduced the delay: + + 5241b88 Delay Gerrit events by 5s + 490f4aa Increase the Gerrit trailing delay + +Change-Id: I0179d5bac6e0b3313e44e850823385345d28cb9d +--- + doc/source/connections.rst | 12 ++++++++++++ + etc/zuul.conf-sample | 1 + + tests/base.py | 2 +- + zuul/connection/gerrit.py | 12 ++++++++---- + 4 files changed, 22 insertions(+), 5 deletions(-) + +diff --git a/doc/source/connections.rst b/doc/source/connections.rst +index f0820a6..c1753da 100644 +--- a/doc/source/connections.rst ++++ b/doc/source/connections.rst +@@ -38,6 +38,18 @@ Create a connection with gerrit. + Path to SSH key to use when logging into above server. + ``sshkey=/home/zuul/.ssh/id_rsa`` + ++**event_delay** (optional) ++ ++ When querying a change immediately after a patchset upload, Gerrit may ++ return incorrect data about dependent changes. In order to avoid this, ++ the events are not delivered to Zuul until a constant number of ++ seconds has passed. ++ ++ Note that if we receive several events in succession, we will only ++ need to delay for the first event. ++ ++ Default: ``10`` (seconds) ++ + + Gerrit Configuration + ~~~~~~~~~~~~~~~~~~~~ +diff --git a/etc/zuul.conf-sample b/etc/zuul.conf-sample +index d7b8eae..5cd9674 100644 +--- a/etc/zuul.conf-sample ++++ b/etc/zuul.conf-sample +@@ -36,6 +36,7 @@ server=review.example.com + ;baseurl=https://review.example.com/r + user=jenkins + sshkey=/home/jenkins/.ssh/id_rsa ++event_delay=10 + + [connection smtp] + driver=smtp +diff --git a/tests/base.py b/tests/base.py +index 6d4f017..622de36 100755 +--- a/tests/base.py ++++ b/tests/base.py +@@ -952,7 +952,6 @@ class ZuulTestCase(BaseTestCase): + + zuul.source.gerrit.GerritSource.replication_timeout = 1.5 + zuul.source.gerrit.GerritSource.replication_retry_interval = 0.5 +- zuul.connection.gerrit.GerritEventConnector.delay = 0.0 + + self.sched = zuul.scheduler.Scheduler(self.config) + +@@ -1044,6 +1043,7 @@ class ZuulTestCase(BaseTestCase): + Queue.Queue() + self.event_queues.append( + self.gerrit_queues_dbs[con_config['server']]) ++ con_config['event_delay'] = 0.0 + self.connections[con_name] = FakeGerritConnection( + con_name, con_config, + changes_db=self.gerrit_changes_dbs[con_config['server']], +diff --git a/zuul/connection/gerrit.py b/zuul/connection/gerrit.py +index 62891cd..927aa11 100644 +--- a/zuul/connection/gerrit.py ++++ b/zuul/connection/gerrit.py +@@ -32,13 +32,13 @@ class GerritEventConnector(threading.Thread): + """Move events from Gerrit to the scheduler.""" + + log = logging.getLogger("zuul.GerritEventConnector") +- delay = 10.0 + +- def __init__(self, connection): ++ def __init__(self, connection, delay=10): + super(GerritEventConnector, self).__init__() + self.daemon = True + self.connection = connection + self._stopped = False ++ self.delay = delay + + def stop(self): + self._stopped = True +@@ -54,6 +54,8 @@ class GerritEventConnector(threading.Thread): + # that if we receive several events in succession, we will + # only need to delay for the first event. In essence, Zuul + # should always be a constant number of seconds behind Gerrit. ++ # ++ # Can be configured via the Gerrit driver setting 'event_delay'. + now = time.time() + time.sleep(max((ts + self.delay) - now, 0.0)) + event = TriggerEvent() +@@ -226,6 +228,7 @@ class GerritConnection(BaseConnection): + self.keyfile = self.connection_config.get('sshkey', None) + self.watcher_thread = None + self.event_queue = None ++ self.event_delay = int(self.connection_config.get('event_delay', 10)) + self.client = None + + self.baseurl = self.connection_config.get('baseurl', +@@ -440,7 +443,7 @@ class GerritConnection(BaseConnection): + return url + + def onLoad(self): +- self.log.debug("Starting Gerrit Conncetion/Watchers") ++ self.log.debug("Starting Gerrit Connection/Watchers") + self._start_watcher_thread() + self._start_event_connector() + +@@ -470,7 +473,8 @@ class GerritConnection(BaseConnection): + self.gerrit_event_connector.join() + + def _start_event_connector(self): +- self.gerrit_event_connector = GerritEventConnector(self) ++ self.gerrit_event_connector = GerritEventConnector( ++ self, delay=self.event_delay) + self.gerrit_event_connector.start() + + diff --git a/debian/patches/0009-Register-connections-when-testing-configuration.patch b/debian/patches/0009-Register-connections-when-testing-configuration.patch deleted file mode 100644 index 01d4bfb..0000000 --- a/debian/patches/0009-Register-connections-when-testing-configuration.patch +++ /dev/null @@ -1,50 +0,0 @@ -From: Joshua Hesketh <j...@nitrotech.org> -Date: Mon, 4 Apr 2016 13:38:51 +1000 -Subject: Register connections when testing configuration - -The layout validation requires the connections to be registered -with the scheduler to know what connection names are valid in the -layout.yaml. - -However avoid starting the connections so that things like the -gerrit connection don't start streaming from gerrit yet. - -Change-Id: Ie9a03287835c6966f5ac32cac020cf2642ce27d5 ---- - zuul/cmd/server.py | 1 + - zuul/scheduler.py | 7 +++++-- - 2 files changed, 6 insertions(+), 2 deletions(-) - -diff --git a/zuul/cmd/server.py b/zuul/cmd/server.py -index 8861972..e713a52 100755 ---- a/zuul/cmd/server.py -+++ b/zuul/cmd/server.py -@@ -88,6 +88,7 @@ class Server(zuul.cmd.ZuulApp): - logging.basicConfig(level=logging.DEBUG) - self.sched = zuul.scheduler.Scheduler(self.config) - self.configure_connections() -+ self.sched.registerConnections(self.connections, load=False) - layout = self.sched.testConfig(self.config.get('zuul', - 'layout_config'), - self.connections) -diff --git a/zuul/scheduler.py b/zuul/scheduler.py -index aea9a67..b631344 100644 ---- a/zuul/scheduler.py -+++ b/zuul/scheduler.py -@@ -313,11 +313,14 @@ class Scheduler(threading.Thread): - # Any skip-if predicate can be matched to trigger a skip - return cm.MatchAny(skip_matchers) - -- def registerConnections(self, connections): -+ def registerConnections(self, connections, load=True): -+ # load: whether or not to trigger the onLoad for the connection. This -+ # is useful for not doing a full load during layout validation. - self.connections = connections - for connection_name, connection in self.connections.items(): - connection.registerScheduler(self) -- connection.onLoad() -+ if load: -+ connection.onLoad() - - def stopConnections(self): - for connection_name, connection in self.connections.items(): diff --git a/debian/patches/0009-WMF-soften-pbr-requirement.patch b/debian/patches/0009-WMF-soften-pbr-requirement.patch new file mode 100644 index 0000000..c75362e --- /dev/null +++ b/debian/patches/0009-WMF-soften-pbr-requirement.patch @@ -0,0 +1,21 @@ +From: Antoine Musso <has...@free.fr> +Date: Fri, 22 Jul 2016 15:33:50 +0200 +Subject: WMF: soften pbr requirement + +The old >=0.5.21 should be good enough. + +Change-Id: Ia279bc09b886a3be86894ad13261c842bd9cb95b +--- + requirements.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/requirements.txt b/requirements.txt +index 69484c7..c85f242 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -1,4 +1,4 @@ +-pbr>=1.1.0 ++pbr>=0.5.21 + + PyYAML>=3.1.0 + Paste diff --git a/debian/patches/0010-WMF-constraint-apscheduler-to-3.1.0.patch b/debian/patches/0010-WMF-constraint-apscheduler-to-3.1.0.patch new file mode 100644 index 0000000..379f04b --- /dev/null +++ b/debian/patches/0010-WMF-constraint-apscheduler-to-3.1.0.patch @@ -0,0 +1,27 @@ +From: Antoine Musso <has...@free.fr> +Date: Fri, 22 Jul 2016 15:35:07 +0200 +Subject: WMF: constraint apscheduler to <3.1.0 + +apscheduler 3.1.0 fail with old setuptools +https://github.com/agronholm/apscheduler/issues/123 + +That is solved in 3.2.x apparently. + +Change-Id: I9a736379f9391a91219afa1e9e192e53ca8883fe +--- + requirements.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/requirements.txt b/requirements.txt +index c85f242..3ab3540 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -10,7 +10,7 @@ extras + statsd>=1.0.0,<3.0 + voluptuous>=0.7 + gear>=0.5.7,<1.0.0 +-apscheduler>=3.0 ++apscheduler>=3.0,<3.1.0 + PrettyTable>=0.6,<0.8 + babel>=1.0 + six>=1.6.0 diff --git a/debian/patches/series b/debian/patches/series index 5d0a3c0..a712b0e 100644 --- a/debian/patches/series +++ b/debian/patches/series @@ -1,9 +1,10 @@ -0001-wmf-soften-requirements.patch -0002-Ensure-the-repository-configuration-lock-is-released.patch -0003-Replace-python-shebang-with-python2.7.patch -0004-Cloner-Implement-cache-no-hardlinks-argument.patch -0005-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch -0006-Properly-shutdown-apscheduler-on-reconfigure.patch -0007-GerritWatcher-add-poll_timeout.patch -0008-Do-not-import-paramiko-before-daemonization.patch -0009-Register-connections-when-testing-configuration.patch +0001-Ensure-the-repository-configuration-lock-is-released.patch +0002-Replace-python-shebang-with-python2.7.patch +0003-Cloner-Implement-cache-no-hardlinks-argument.patch +0004-zuul-cloner-recognizes-bare-repos-in-cache-dir.patch +0005-WMF-soften-paramiko-requirement.patch +0006-WMF-soften-WebOb-requirement.patch +0007-WMF-drop-requirement-ordereddict.patch +0008-Gerrit-trailing-delay-is-now-configurable.patch +0009-WMF-soften-pbr-requirement.patch +0010-WMF-constraint-apscheduler-to-3.1.0.patch diff --git a/debian/rules b/debian/rules index a1fc9e9..573709f 100755 --- a/debian/rules +++ b/debian/rules @@ -29,7 +29,7 @@ override_dh_virtualenv: # Needs dh_virtualenv 0.9+ for --use-system-packages # The .pyc are generated in postinst - PIP_NO_COMPILE=1 PYTHONDONTWRITEBYTECODE=true VIRTUALENV_VERBOSE=true dh_virtualenv \ + DEB_BUILD_OPTIONS=nocheck GIT_PBUILDER_AUTOCONF=no DIST=precise WIKIMEDIA=yes git-buildpackage -us -uc --git-builder=git-pbuilder PIP_NO_COMPILE=1 PYTHONDONTWRITEBYTECODE=true VIRTUALENV_VERBOSE=true dh_virtualenv \ --python '/usr/bin/python2.7' \ --use-system-packages \ --no-package ordereddict \ -- To view, visit https://gerrit.wikimedia.org/r/300705 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I08c9b30da7f0d55d5ba0b436a5742c3af0a48d21 Gerrit-PatchSet: 1 Gerrit-Project: integration/zuul Gerrit-Branch: master Gerrit-Owner: Paladox <thomasmulhall...@yahoo.com> Gerrit-Reviewer: Hashar <has...@free.fr> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
