I'll get this into a draft article on the wiki so we can collab on those 3 outstanding TBD's without further cluttering up the dev list. :)
On Fri, Dec 17, 2021 at 11:38 AM Ekaterina Dimitrova <e.dimitr...@gmail.com> wrote: > It’s indeed good call but I thought this will be addressed in a separate > document where we discuss required test suites to be run pre-commit. If not > - then I guess we should add those things here too? > > On Fri, 17 Dec 2021 at 11:36, Joshua McKenzie <jmcken...@apache.org> > wrote: > > > Good call; thanks for the reminder. > > > > So maybe add a > > > > 3.a: Run all new or modified tests through either local or remote > > multiplexer N (TBD - 50?) times (w/link to instructions, etc) > > 3.b Non-regressing is defined here... > > 3.c After merging tickets... > > > > On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams <dri...@gmail.com> > > wrote: > > > > > Could we also add something about running new tests through the > > > multiplexer? > > > > > > On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <jmcken...@apache.org > > > > > wrote: > > > > > > > > So to clarify it all in one place, the proposed new CI process we > > should > > > > test for consensus is: > > > > > > > > 1. Canonical CI for a release is ci-cassandra. We can optionally, and > > in > > > > practice will, run circle as well but don't codify blocking on that. > > > > 2. (NEW) We don't release unless we get a fully green run. > > > > 3. Before any merge, you need either a non-regressing (i.e. no new > > > > failures) run of circleci with a (specific suite of tests TBD) or of > > > > ci-cassandra. > > > > 3.a Non-regressing is defined here as "Doesn't introduce any new > > > test > > > > failures; any new failures in CI are clearly not attributable to this > > > diff" > > > > 3.b: (NEW) After merging tickets, ci-cassandra runs against the > > SHA > > > > and the author gets an advisory update on the related JIRA for any > new > > > > errors on CI. The author of the ticket will take point on triaging > this > > > new > > > > failure and either fixing (if clearly reproducible or related to > their > > > > work) or opening a JIRA for the intermittent failure and linking it > in > > > > butler (https://butler.cassandra.apache.org/#/) > > > > 4. (NEW) The Build Lead role + Butler catches and documents all > > failures > > > > and anything that slips through the procedural cracks in 3.b; > > resourcing > > > > for fixing flakey tests TBD > > > > > > > > Our two TBD we can tackle separately from consensus on the above: > > > > 1. Suite of tests on circle required to be considered ready for merge > > > > 2. How we resource fixing flakey tests that are functionally > impossible > > > to > > > > attribute without essentially fixing the flake > > > > > > > > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova < > > > e.dimitr...@gmail.com> > > > > wrote: > > > > > > > > > +1 (nb) on my end too, I second Mick > > > > > Thanks for putting this together Josh > > > > > > > > > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <m...@apache.org> > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already do > > > this) > > > > > and > > > > > > > get an advisory update on the related JIRA for any new errors > on > > > the > > > > > run > > > > > > of > > > > > > > the SHA > > > > > > > > > > > > > > I strongly prefer we amend our process with 3.c. > > > > > > > > > > > > > > > > > > > > > > > > +1 Yup, this is the most important missing piece for me. > > > > > > > > > > > > I also wouldn't mind we word the responsibility of the author at > > > > > > post-commit fault to be involved/leading in the fix. This > > > incentivises > > > > > > people to do 2+3 properly, and not push it onto the build role. > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > >
