It seems there has been a bit of a slip in testing as of recently, mostly due to the fact that there's no canonical testing environment that isn't flaky. We probably need to come up with some ideas and a plan on how we're going to do testing in the future, and how we're going to make testing accessible for all contributors. I think this is the only way we're really going to change behaviour. Having an incredibly tedious process and then being aggressive about it only leads to resentment and workarounds.
I'm completely unsure of where dtests are at since the conversion to pytest, and there's a lot of failing dtests on the ASF jenkins jobs (which appear to be running pytest). As there's currently not a lot of visibility into what people are doing with CircleCI for this it's hard to say if things are better over there. I'd like to help here if anyone wants to fill me in. On 15 February 2018 at 21:14, Josh McKenzie <jmcken...@apache.org> wrote: > > > > We’ve said in the past that we don’t release without green tests. The PMC > > gets to vote and enforce it. If you don’t vote yes without seeing the > test > > results, that enforces it. > > I think this is noble and ideal in theory. In practice, the tests take long > enough, hardware infra has proven flaky enough, and the tests *themselves* > flaky enough, that there's been a consistent low-level of test failure > noise that makes separating signal from noise in this context very time > consuming. Reference 3.11-test-all for example re:noise: > https://builds.apache.org/view/A-D/view/Cassandra/job/ > Cassandra-3.11-test-all/test/?width=1024&height=768 > > Having spearheaded burning test failures to 0 multiple times and have them > regress over time, my gut intuition is we should have one person as our > Source of Truth with a less-flaky source for release-vetting CI (dedicated > hardware, circle account, etc) we can use as a reference to vote on release > SHA's. > > We’ve declared this a requirement multiple times > > Declaring things != changed behavior, and thus != changed culture. The > culture on this project is one of having a constant low level of test > failure noise in our CI as a product of our working processes. Unless we > change those (actually block release w/out green board, actually > aggressively block merge w/any failing tests, aggressively retroactively > track down test failures on a daily basis and RCA), the situation won't > improve. Given that this is a volunteer organization / project, that kind > of daily time investment is a big ask. > > On Thu, Feb 15, 2018 at 1:10 PM, Jeff Jirsa <jji...@gmail.com> wrote: > > > Moving this to it’s own thread: > > > > We’ve declared this a requirement multiple times and then we occasionally > > get a critical issue and have to decide whether it’s worth the delay. I > > assume Jason’s earlier -1 on attempt 1 was an enforcement of that earlier > > stated goal. > > > > It’s up to the PMC. We’ve said in the past that we don’t release without > > green tests. The PMC gets to vote and enforce it. If you don’t vote yes > > without seeing the test results, that enforces it. > > > > -- > > Jeff Jirsa > > > > > > > On Feb 15, 2018, at 9:49 AM, Josh McKenzie <jmcken...@apache.org> > wrote: > > > > > > What would it take for us to get green utest/dtests as a blocking part > of > > > the release process? i.e. "for any given SHA, here's a link to the > tests > > > that passed" in the release vote email? > > > > > > That being said, +1. > > > > > >> On Wed, Feb 14, 2018 at 4:33 PM, Nate McCall <zznat...@gmail.com> > > wrote: > > >> > > >> +1 > > >> > > >> On Thu, Feb 15, 2018 at 9:40 AM, Michael Shuler < > mich...@pbandjelly.org > > > > > >> wrote: > > >>> I propose the following artifacts for release as 3.0.16. > > >>> > > >>> sha1: 890f319142ddd3cf2692ff45ff28e71001365e96 > > >>> Git: > > >>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= > > >> shortlog;h=refs/tags/3.0.16-tentative > > >>> Artifacts: > > >>> https://repository.apache.org/content/repositories/ > > >> orgapachecassandra-1157/org/apache/cassandra/apache-cassandra/3.0.16/ > > >>> Staging repository: > > >>> https://repository.apache.org/content/repositories/ > > >> orgapachecassandra-1157/ > > >>> > > >>> Debian and RPM packages are available here: > > >>> http://people.apache.org/~mshuler > > >>> > > >>> *** This release addresses an important fix for CASSANDRA-14092 *** > > >>> "Max ttl of 20 years will overflow localDeletionTime" > > >>> https://issues.apache.org/jira/browse/CASSANDRA-14092 > > >>> > > >>> The vote will be open for 72 hours (longer if needed). > > >>> > > >>> [1]: (CHANGES.txt) https://goo.gl/rLj59Z > > >>> [2]: (NEWS.txt) https://goo.gl/EkrT4G > > >>> > > >>> ------------------------------------------------------------ > --------- > > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org > > >>> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > > >> > > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > >
