Today we found out that a patch that was made against our
lefthand driver caused the driver to fail. The 3rd party CI that we
to test our driver (hp_lefthand_rest_proxy.py) caught the CI failure and
reported it correctly. The patch that broke the driver was reviewed and
approved without a mention of 3rd Party CI failures.
This is a perfect example of 3rd Party CI working perfectly and catching
a failure, and being completely ignored by everyone
involved in the review process for the patch
I know that 3rd party CI isn't perfect, and has been ripe with false
failures, which is one of the reasons why they aren't voting today.
But, that being said, if patch submitters aren't even looking at the
failures for CI when they are touching drivers that they don't maintain,
aren't looking at the CI failures, then why are we even doing 3rd party CI?
Our team is partially responsible for not seeing the failure as well.
We should be watching the CI failures closely, but we are doing the best we
can. There are enough patches for Cinder ongoing at any one time, that
we simply can't watch every single one of them for failures. We did
see that every single patchset in gerrit was now failing against our
driver, and this is how we caught it. Yes, it was a bit after the fact,
but we did notice
it and now have a new patch up that fixes it. So, in that regard 3rd
party CI did eventually vet out a problem that our team caught.
How can we prevent this in the future?
1) Make 3rd party CI voting. I don't believe this is ready yet.
2) Authors and reviews need to look at 3rd party CI failures when a
patch touches a driver. If a failure is seen, contact the CI maintainer
and work with them and
see if the failure is related to the patch, if it's not obvious. In this
case, the failure was obvious. The import changed, and now the package
can't find the module.
3) CI maintainers watch every single patchset and report -1's on
Here is the patch that broke the lefthand driver
Here is the reported failure in the c-vol log for the patch by our 3rd
party CI system
Here is my new patch that fixes the lefthand driver again.
OpenStack Development Mailing List (not for usage questions)