Hi, I understand that 8099 has put us in a difficult position. Nevertheless, 8099 is a kind of special case and in my opinion ignoring failing tests can be a dangerous thing.
I am not against doing it for 3.0 and trunk but, if we do it, we should do it only until we have recover from 8099. [image: datastax_logo.png] <http://www.datastax.com/> Benjamin Lerer Software Engineer| +41 76 396 64 60 | benjamin.le...@datastax.com [image: linkedin.png] <http://ch.linkedin.com/pub/benjamin-lerer/2/303/784/> [image: facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] <https://twitter.com/datastax> [image: g+.png] <https://plus.google.com/+Datastax/about> <http://feeds.feedburner.com/datastax> <https://github.com/datastax/> On Fri, Aug 14, 2015 at 9:29 PM, Ariel Weisberg <ariel.weisb...@datastax.com > wrote: > Hi, > > Thanks for bring this up Michael. > > I want to elaborate on the impetus for this (or at least my take on it). > When 8099 merged we had a thing that must never happen for our process to > work. We introduced a large enough number of test failures that it was > difficult to tell if you introduced a regression. > > At the time we thought we could exclude the test failures prior to 8099 > and that the test failures introduced by 8099 would get addressed promptly. > What has happened instead is that the number of failures have snowballed to > the point that you can hardly tell if you broke anything even if you > compare test by test with trunk. You have to go into the history on trunk > for each test and go back several pages to really be sure. > > If you don’t have consistently passing CI you can’t avoid the addition of > test failures by ongoing work that slip in masked by known failures. > > The artery is severed, we’re bleeding out, and we’re going to have to lose > the leg. I’m sure the prosthetic when it comes will be just as good, but > the rehab is going to suck. There that’s my analogy. > > I think the utests are in pretty good shape but the pig tests are a > problem. They extend the job time a lot, cause aborts, and fail randomly. > > Ariel > > > On Aug 14, 2015, at 3:16 PM, Michael Shuler <mich...@pbandjelly.org> > wrote: > > > > This is a prompt for Cassandra developers to discuss the alternatives > and let Test Engineering know what you desire. > > > > As discussed a few times in person, on irc, etc., there are a couple > different ways we can run tests in Jenkins, particularly cassandra-dtest. > The Cassandra developers are the committers to unit tests, so Test > Engineering runs whatever is in the branch. If you'd like to make changes > to unit tests to make things blue, just commit those! > > > > Currently, we run dtests as 1), but we could do 2): > > > > 1) Run all dtests that don't catastrophically hang a server, pass or > fail, and report the results. > > 2) Run only known passing dtests, skipping anything that fails - make it > all blue on the main branch builds. > > > > The biggest benefit is that dev branch builds should be easily > recognizable as able to merge, if the dtest run is passing and blue. There > is no comparison with the main branch build needing interpretation. > > > > Test Eng has recently added the ability run *only* the skipped tests and > has a a prototype job, trunk_dtest-skipped-with-require, to dig through. > This could be set up for all main branch builds, moving anything that > doesn't pass 100% to the -skipped job. This is perhaps the drawback with 2) > above: we're simply not going to run all the dtests on your dev branch. I > don't think it makes sense to set up a -skipped dtest job on your dev > branches. In addition, there's another job result set to go look at to > properly evaluate the true state of a Cassandra branch or release. There > may be other side effects - feel free to chime in. > > > > I'm on a "disconnected" holiday until Monday Aug 24, so I won't have a > chance to check in until then - the Test Eng team can field questions or > clarifications, if needed. > > > > -- > > Warm regards, > > Michael > >
