Hey folks,
Today we found out that a patch[1] that was made against our lefthand driver caused the driver to fail. The 3rd party CI that we have setup to test our driver (hp_lefthand_rest_proxy.py) caught the CI failure and reported it correctly[2]. The patch that broke the driver was reviewed and
approved without a mention of 3rd Party CI failures.

This is a perfect example of 3rd Party CI working perfectly and catching a failure, and being completely ignored by everyone
involved in the review process for the patch

I know that 3rd party CI isn't perfect, and has been ripe with false failures, which is one of the reasons why they aren't voting today. But, that being said, if patch submitters aren't even looking at the failures for CI when they are touching drivers that they don't maintain, and reviewers
aren't looking at the CI failures, then why are we even doing 3rd party CI?

Our team is partially responsible for not seeing the failure as well. We should be watching the CI failures closely, but we are doing the best we can. There are enough patches for Cinder ongoing at any one time, that we simply can't watch every single one of them for failures. We did eventually see that every single patchset in gerrit was now failing against our driver, and this is how we caught it. Yes, it was a bit after the fact, but we did notice it and now have a new patch up that fixes it. So, in that regard 3rd party CI did eventually vet out a problem that our team caught.

How can we prevent this in the future?
1) Make 3rd party CI voting.  I don't believe this is ready yet.
2) Authors and reviews need to look at 3rd party CI failures when a patch touches a driver. If a failure is seen, contact the CI maintainer and work with them and see if the failure is related to the patch, if it's not obvious. In this case, the failure was obvious. The import changed, and now the package can't find the module. 3) CI maintainers watch every single patchset and report -1's on reviews? (ouch)
4) ?

Here is the patch that broke the lefthand driver[1]
Here is the reported failure in the c-vol log for the patch by our 3rd party CI system[2]
Here is my new patch that fixes the lefthand driver again.[3]

[1] https://review.openstack.org/#/c/145780/
[3] https://review.openstack.org/#/c/159586/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Reply via email to