On 2017-03-16 10:32 (-0700), François Deliège <franc...@instagram.com> wrote: > > To get this started, here is an initial proposal: > > Principles: > > 1. Tests always pass. This is the starting point. If we don't care about > test failures, then we should stop writing tests. A recurring failing test > carries no signal and is better deleted. > 2. The code is tested. > > Assuming we can align on these principles, here is a proposal for their > implementation. > > Rules: > > 1. Each new release passes all tests (no flakinesss). > 2. If a patch has a failing test (test touching the same code path), the code > or test should be fixed prior to being accepted. > 3. Bugs fixes should have one test that fails prior to the fix and passes > after fix. > 4. New code should have at least 90% test coverage. >
I agree with all of these and hope they become codified and followed. I don't know anyone who believes we should be committing code that breaks tests - but we should be more strict with requiring green test runs, and perhaps more strict with reverting patches that break tests (or cause them to be flakey). Ed also noted on the user list [0] that certain sections of the code itself are difficult to test because of singletons - I agree with the suggestion that it's time to revisit CASSANDRA-7837 and CASSANDRA-10283 Finally, we should also recall Jason's previous notes [1] that the actual test infrastructure available is limited - the system provided by Datastax is not generally open to everyone (and not guaranteed to be permanent), and the infrastructure currently available to the ASF is somewhat limited (much slower, at the very least). If we require tests passing (and I agree that we should), we need to define how we're going to be testing (or how we're going to be sharing test results), because the ASF hardware isn't going to be able to do dozens of dev branch dtest runs per day in its current form. 0: https://lists.apache.org/thread.html/f6f3fc6d0ad1bd54a6185ce7bd7a2f6f09759a02352ffc05df92eef6@%3Cuser.cassandra.apache.org%3E 1: https://lists.apache.org/thread.html/5fb8f0446ab97644100e4ef987f36e07f44e8dd6d38f5dc81ecb3cdd@%3Cdev.cassandra.apache.org%3E
