(adding [releng] topic and integration list) tl;dr: lots of troubles and debug cycles happening because of new dev and refactoring that we know can be gated with already existing tests (IT, CSIT)
Josh, I agree 100% with your sentiment and feel your pain. This is something I have lobbied for, for quite some time. As Anil notes, the biggest push back is that it would require too many resources to gate every patch set with our tests. We did spend some time on the whiteboard at the hackfest coming up with a plan. Andy and Thanh were bringing up a test repo where we can try this out and once we work out the quirks, OVSDB can try it first. The basic idea is that *no* tests (except normal verify job) will run on a gerrit patch, until a new category is given a +1 (I think we want to call it "approve", so we'll have verify, code-review, and approve now). That approve +1 would kick off all the tests (as defined by the projects in their job yamls) and if they come back clean, then the patch can be submitted. Someone can correct me if I got something wrong. Thanks, JamO On 07/06/2016 12:30 AM, Anil Vishnoi wrote: > I echo your thoughts. I was talking to Jamo & Luis about this during the > hackfest and we are planning to use CSIT to gate the > commits at least for OVSDB project to start with. We also discuss about when > and how to trigger CSIT jobs on patches so that > we can use the existing jenkins resources optimally. Currently the bottleneck > is Jenkins resources and we tried to come up > with a solution that will help us with better utilization of these jenkins > resources and hopefully we will have more space to > run CSIT jobs on patches before merge. > > Adding more CSIT test is something we need to bring into practise. We should > closely keep track of new features and how we > can test them through CSIT. Good news is that we already started it in > netvirt/ovsdb project, but i think we need more > contributors toward that. > > Thanks for explicitly sharing the known pain point :). Let's start it from > netvirt/ovsdb project and see if we can make it > better in coming days. > > On Wed, Jul 6, 2016 at 12:19 AM, Josh Hershberg <[email protected] > <mailto:[email protected]>> wrote: > > All, > > Over the past week+ we've been experiencing regressions of various > severities as a result of refactoring. This has eaten > up a lot of my time and I've also heard similar reports from others > inside and out of the company where I work. Some of > the regressions were in openflowplugin and a few as a result of the > transition to blueprint. It is not my intention to > blame storm at all, all the errors have been very understandable. > However, the fact that I hit these problems by running > automated tests THAT WE ALREADY HAVE (NetvirtIt) really underscores how > beneficial it would be to buff up our automated > testing and have the results gate the commits. I know that this is an > ongoing effort we are currently working on but > thought I would share this rather clear example of how it would help us. > > One of the issues with the blueprint transition resulted in netvirt > failing to initialize. Translate that back from > inversion-of-control to the way things used to be and that's more or less > comparable to compiling an executable, running > it, and then it doing nothing! > > Again, not looking to blame anyone, EVERYONE involved has been very > responsive and responsible, just to share some > thoughts on an issue I'm pretty sure we all agree on anyway ;-) > > -J > > > > > -- > Thanks > Anil > > > _______________________________________________ > openflowplugin-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev > _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
