Yes, failed test results need to be looked at by someone. But this is already the case and won't change no matter if we run tests for each patch and branch, or just once a day for a single dev branch. Having to figure out which commit exactly causes the regression would take some additional effort, but I don't think that would be the hardest part dealing with failed test results. I'd be happy to discus other options, but I'm pretty sure all of them will come with a price and we eventually have to agree on something.
On 03/10/2017 03:43 PM, Josh McKenzie wrote: >> I think we'd be able to figure out the one of them causing a regression >> on the day after. > That sounds great in theory. In practice, that doesn't happen unless one > person steps up and makes themselves accountable for it. > > For reference, take a look at: https://cassci.datastax.com/view/trunk/, and > https://issues.apache.org/jira/issues/?jql=project%20%3D%20cassandra%20and%20resolution%20%3D%20unresolved%20and%20labels%20in%20(%27test-fail%27%2C%20%27test-failure%27%2C%20%27testall%27%2C%20%27dtest%27%2C%20%27unit-test%27%2C%20%27unittest%27)%20and%20assignee%20%3D%20null%20order%20by%20created%20ASC > > We're thankfully still in a place where these tickets are at least being > created, but unless there's a body of people that are digging in to fix > those test failures they're just going to keep growing. > > On Fri, Mar 10, 2017 at 5:03 AM, Stefan Podkowinski <s...@apache.org> wrote: > >> If I remember correctly, the requirement of providing test results along >> with each patch was because of tick-tock, where the goal was to have >> stable release branches at all times. Without CI for testing each >> individual commit on all branches, this just won't work anymore. But >> would that really be that bad? Can't we just get away with a single CI >> run per branch and day? >> >> E.g. in the future we could commit to dev branches that are used to run >> all tests automatically on Apache CI on daily basis, which is then >> exclusively used for that. We don't have that many commits on a single >> day, some of them rather trivial, and I think we'd be able to figure out >> the one of them causing a regression on the day after. If all tests >> pass, we can merge dev manually or even better automatically. If anyone >> wants to run tests on his own CI before committing to dev, that's fine >> too and will help analyzing any regressions if they happen, as we then >> don't have to look at those patches (and all commits before on dev). >> >> >> >> On 09.03.2017 19:51, Jason Brown wrote: >>> Hey all, >>> >>> A nice convention we've stumbled into wrt to patches submitted via Jira >> is >>> to post the results of unit test and dtest runs to the ticket (to show >> the >>> patch doesn't break things). Many contributors have used the >>> DataStax-provided cassci system, but that's not the best long term >>> solution. To that end, I'd like to start a conversation about what is the >>> best way to proceed going forward, and then add it to the "How to >>> contribute" docs. >>> >>> As an example, should contributors/committers run dtests and unit tests >> on >>> *some* machine (publicly available or otherwise), and then post those >>> results to the ticket? This could be a link to a build system, like what >> we >>> have with cassci, or just upload the output of the test run(s). >>> >>> I don't have any fixed notions, and am looking forward to hearing other's >>> ideas. >>> >>> Thanks, >>> >>> -Jason >>> >>> p.s. a big thank you to DataStax for providing the cassci system >>>
