On Tue, Jul 03, 2012 at 04:48:32PM -0700, James E. Blair wrote: > Hi, > > As mentioned in the thread "Jenkins and transient failures", we've had > an unusually high number of transient failures in Jenkins lately. We've > done several things in response to that: > > 1) Monty identified a problem with our pypi mirror which was the cause > of many of the errors, and corrected it. > > 2) Monty is continuing to work on the single dependency list which > should allow us to switch to using our local pypi mirror exclusively, > further reducing transient network errors, as well as significantly > speeding up test run time. > > 3) Several transient errors were caused by failed fetches from Gerrit. > While consulting with the Gerrit authors about tuning, they discovered a > bug in Gerrit where a 5 minute timeout was being interpreted as a 5 > millisecond timeout. I have updated our gerrit configuration to work > around that. > > 4) Clark Boylan implemented automatic retrying for the git fetches that > we use with Jenkins. > > > I hope that we'll get to the point where we have almost no transient > network errors when testing, but we know it will never be perfect, so at > the CI meeting we discussed how best to implement retriggering with > Zuul. Clark added a comment filter that will retrigger Jenkins if you > leave a comment that matches a regex. > > We currently run two kinds of jobs in Jenkins, the check job and the > gate job. The check jobs run immediately when a patchset is uploaded > and vote +/-1. The gate jobs run on approval, queue up across all > projects and vote +/-2 (if they fail, jobs behind them in the gate > pipeline may need to run again). > > > To retrigger the initial Jenkins check job, just leave a comment on the > review in Gerrit with only the text "recheck". > > To retrigger the Jenkins merge gate job, leave a comment with only the > text "reverify", or if you are a core reviewer, just leave another > "Approved" vote. (Don't leave a "reverify" comment if the change hasn't > been approved yet, it still won't be merged and will slow Jenkins down.)
Thanks to the team who worked on resolving the problems from last & also adding this retrigger support. The latter will bring a nice improvement to productivity should there be problems in the future. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
