On 2015-04-27 11:38:09 -0600 (-0600), Chris Friesen wrote: [...] > In circumstances like this I wonder if it might make sense to have Jenkins > "abstain" rather than mark it -1. We could then have a job that went around > and re-ran Jenkins tests for cases where it "abstained" previously. [...]
In a Jenkinsless future, this may be possible. At the moment, we're using Jenkins and its status reporting is effectively limited to "SUCCESS" and "FAILURE." We're going to need something which can provide a finer-grained result (the current plans around non-Jenkins job workers leave us open to this possibility). Also, some of our projects actually ARE test frameworks, so for changes proposed to those we definitely want failures of that framework to be reported as failures of the jobs testing them. It may be a little tough to differentiate between these cases. Further, reviewers want to rely on job results to know whether the change works. If the CI merely abstained from reporting when it didn't get a chance to run the job to completion, there's no indication of whether that job would have worked. We could try to solve this by automatically requeuing the jobs which hit this condition, but it leaves us open to a rather nasty problem for which we'd need a separate release valve: if the test framework itself is broken and not simply experiencing an intermittent problem, we'd requeue and run all jobs using it in an infinite loop consuming all our resources and effectively starving out worker availability for other jobs which aren't broken. -- Jeremy Stanley __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
