Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset
https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? Obviously this was all made worse by the transient problems we've had with the tests suite infrastructure these past 2 days, but regardless it seems like we have a gap in our merge approval procedures here. IMHO, either SmokeStack needs to be made compulsory, or Jenkins needs to ensure tests are run on suitable hosts like SmokeStack does, or both. Regards, Daniel [1] -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp