Re: [Openstack] [CI] Retriggering Jenkins from Gerrit
On Tue, Jul 03, 2012 at 04:48:32PM -0700, James E. Blair wrote: Hi, As mentioned in the thread Jenkins and transient failures, we've had an unusually high number of transient failures in Jenkins lately. We've done several things in response to that: 1) Monty identified a problem with our pypi mirror which was the cause of many of the errors, and corrected it. 2) Monty is continuing to work on the single dependency list which should allow us to switch to using our local pypi mirror exclusively, further reducing transient network errors, as well as significantly speeding up test run time. 3) Several transient errors were caused by failed fetches from Gerrit. While consulting with the Gerrit authors about tuning, they discovered a bug in Gerrit where a 5 minute timeout was being interpreted as a 5 millisecond timeout. I have updated our gerrit configuration to work around that. 4) Clark Boylan implemented automatic retrying for the git fetches that we use with Jenkins. I hope that we'll get to the point where we have almost no transient network errors when testing, but we know it will never be perfect, so at the CI meeting we discussed how best to implement retriggering with Zuul. Clark added a comment filter that will retrigger Jenkins if you leave a comment that matches a regex. We currently run two kinds of jobs in Jenkins, the check job and the gate job. The check jobs run immediately when a patchset is uploaded and vote +/-1. The gate jobs run on approval, queue up across all projects and vote +/-2 (if they fail, jobs behind them in the gate pipeline may need to run again). To retrigger the initial Jenkins check job, just leave a comment on the review in Gerrit with only the text recheck. To retrigger the Jenkins merge gate job, leave a comment with only the text reverify, or if you are a core reviewer, just leave another Approved vote. (Don't leave a reverify comment if the change hasn't been approved yet, it still won't be merged and will slow Jenkins down.) Thanks to the team who worked on resolving the problems from last also adding this retrigger support. The latter will bring a nice improvement to productivity should there be problems in the future. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [CI] Retriggering Jenkins from Gerrit
On Wed, Jul 4, 2012 at 1:48 AM, James E. Blair cor...@inaugust.com wrote: To retrigger the Jenkins merge gate job, leave a comment with only the text reverify, or if you are a core reviewer, just leave another Approved vote. (Don't leave a reverify comment if the change hasn't been approved yet, it still won't be merged and will slow Jenkins down.) Really cool thanks, would it be possible to have those things documented in the wiki or even better in the ? keyboard shortcuts help. Chmouel. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [CI] Retriggering Jenkins from Gerrit
Hi, As mentioned in the thread Jenkins and transient failures, we've had an unusually high number of transient failures in Jenkins lately. We've done several things in response to that: 1) Monty identified a problem with our pypi mirror which was the cause of many of the errors, and corrected it. 2) Monty is continuing to work on the single dependency list which should allow us to switch to using our local pypi mirror exclusively, further reducing transient network errors, as well as significantly speeding up test run time. 3) Several transient errors were caused by failed fetches from Gerrit. While consulting with the Gerrit authors about tuning, they discovered a bug in Gerrit where a 5 minute timeout was being interpreted as a 5 millisecond timeout. I have updated our gerrit configuration to work around that. 4) Clark Boylan implemented automatic retrying for the git fetches that we use with Jenkins. I hope that we'll get to the point where we have almost no transient network errors when testing, but we know it will never be perfect, so at the CI meeting we discussed how best to implement retriggering with Zuul. Clark added a comment filter that will retrigger Jenkins if you leave a comment that matches a regex. We currently run two kinds of jobs in Jenkins, the check job and the gate job. The check jobs run immediately when a patchset is uploaded and vote +/-1. The gate jobs run on approval, queue up across all projects and vote +/-2 (if they fail, jobs behind them in the gate pipeline may need to run again). To retrigger the initial Jenkins check job, just leave a comment on the review in Gerrit with only the text recheck. To retrigger the Jenkins merge gate job, leave a comment with only the text reverify, or if you are a core reviewer, just leave another Approved vote. (Don't leave a reverify comment if the change hasn't been approved yet, it still won't be merged and will slow Jenkins down.) -Jim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
