Perhaps you could clarify what you personally hope we _should_ agree as a
project, and what you want us to _not_ agree (blossom in infinite variety)?
My view: We need to agree a shared framework for quality going forwards. This
will raise the bar to contributions, including above many that already exist.
So, we then need a roadmap to meeting the framework's requirements for past and
future contributions, so that feature development does not suffer too greatly
from the extra expectations imposed upon them. I hope the framework and
roadmap will be very specific and prescriptive in setting their minimum
standards, which can of course be further augmented as any contributor desires.
This seems to be the only way to come to an agreement about the point of
contention you raise: some people perceive an insufficient concern about
quality, others perceive a surplus of concern about quality. Until we agree
quite specifically what we mean, this tension will persist. I also think it's
a great way to improve project efficiency, if a contributor so cares: resources
can be focused on the shared requirements first, since they're the "table
stakes".
Could you elaborate what you would prefer to leave out of this in your
"Definition of Done"?
On 15/07/2020, 16:28, "Joshua McKenzie" <jmcken...@apache.org> wrote:
>
> This section reads as very anti-adding tests to test/unit; I am 100% in
> favor of improving/creating our smoke, integration, regression,
> performance, E2E, etc. testing, but don't think I am as negative to
> test/unit, these tests are still valuable and more are welcome.
I am a strong proponent of unit tests; upon re-reading the document I don't
draw the same conclusion you do about the implications of the
verbiage, however it's completely reasonable to have a point of view that's
skeptical of people on this project's dedication to rigor and quality. :) I
think it's critical to "name and tame" the current architectural
constraints that undermine our ability to thoroughly unit test, as well as
understand and mitigate the weaknesses of our current unit testing
capabilities. A discrete example - attempting to "unit test" anything in
the CommitLog largely leads to the entire CommitLog package spinning up,
which drags in other packages, and before you know it you have multiple
modules up and running thanks to the dependency tree. This is something
myself, Jason, Stupp, Branimir, and others have all repeatedly burned time
on trying to delicately walk through re: test spin up and tear down. This
has ramifications far beyond just the time lost by engineers; the
opportunity cost of that combined with the fragility of systems means that
what testing we *do* perform is going to be constrained in scope relative
to a traditional battery against a stand-alone, modularized artifact.
Any and all contribution to *any* testing is strongly welcomed by all of us
on the project. In terms of "where I and a few others are going to choose
to invest our efforts" right now, accepting the current shortcomings of the
system to make as much headway on the urgent + important is where we're
headed.
I think it's more important that we set a standard for the project (e.g.,
> fundamental conformance to properties of the database) rather than
> attempting to measure quality relative to other DBs
I'm sympathetic to this then the pragmatist in me hammers me down. In
general, the adage "Software is never done; it is only released" resonates
for me as the core of what we have to navigate here. We will never be able
to state with 100% certainty that there is fundamental conformance to the
availability and correctness properties of the database; this dissatisfying
reality is why you have multiple teams implementing the software for
spacecraft and then redundancies within redundancies in each system for
unexpected failure scenarios and the unknown-unknown. In my opinion, we
need a very clear articulation of our Definition of Done when it comes to
correctness guarantees (yes Ariel, you were right) as well as a more
skillful and deliberate articulated and implemented "failsafe" for catching
things and/or surfacing adverse conditions within the system upon failure.
It's tricky because in the past (in my opinion) we've been pretty remiss as
a project when it comes to a devotion to correctness and rigor. The danger
I'm anecdotally seeing is that if we let that pendulum swing too far in the
other direction without successfully clearly defining what "Done" looks
like from a quality perspective, that's an Everest we can all climb and die
on as a project.
On Wed, Jul 15, 2020 at 12:42 AM Scott Andreas <sc...@paradoxica.net> wrote:
> Thanks for starting discussion!
>
> Replying to the thread with what I would have left as comments.
>
> ––––––
>
> > As yet, we lack empirical evidence to quantify the relative stability or
> instability of our project compared to a peer cohort
>
> I think it's more important that we set a standard for the project (e.g.,
> fundamental conformance to properties of the database) rather than
> attempting to measure quality relative to other DBs. That might be a
useful
> measure, but I don't think it's the most important one. With regard to
> measuring against a common standard in the project, this is roughly what I
> had in mind when proposing "Release Quality Metrics" on the list in 2018.
I
> still think making progress on something like this is essential toward
> defining a quantitative bar for release:
> https://www.mail-archive.com/dev@cassandra.apache.org/msg13154.html
>
> > Conversely, the ability to repeatedly and thoroughly validate the
> correctness of both new and existing functionality in the system is vital
> to the speed with which we can evolve it's form and function.
>
> Strongly agreed.
>
> > Utopia (and following section)
>
> Some nods to great potential refactors to consider post-4.0 here. ^
>
> > We should productize a kubernetes-centric, infra agnostic tool that has
> the following available testing paradigms:
>
> This would be an excellent set of capabilities to have.
>
> > We need to empower our user community to participate in the testing
> process...
>
> I really like this point. I took as a thought experiment "what would feel
> great to be able to say" if one were to write a product announcement for
> 4.0 and landed on something like "Users of Apache Cassandra can preflight
> their 4.0 upgrade by runing $tool to clone, upgrade, and compare their
> clusters, ensuring that the upgrade will complete smoothly and correctly."
>
> > The less friction and less investment we can require from ecosystem
> participants, the more we can expect them to engage in desired behavior.
>
> +1
>
> ––––––
>
> I like the document and there's a lot that has me nodding. Toward the
> opening statement on "empirical evidence to quantify relative stability,"
> I'd love to revisit discussion on quantifying attributes like these here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93324430
>
> – Scott
>
> ________________________________________
> From: David Capwell <dcapw...@gmail.com>
> Sent: Tuesday, July 14, 2020 6:23 PM
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] A point of view on Testing Cassandra
>
> I am also not fully clear on the motives, but welcome anything which helps
> bring in better and more robust testing; thanks for starting this.
>
> Since I can not comment in the doc I have to copy/paste and put here... =(
>
> Reality
> > ...
> > investing in improving our smoke and integration testing as much as is
> > possible with our current constraints seems prudent.
>
>
> This section reads as very anti-adding tests to test/unit; I am 100% in
> favor of improving/creating our smoke, integration, regression,
> performance, E2E, etc. testing, but don't think I am as negative to
> test/unit, these tests are still valuable and more are welcome.
>
> To enumerate a punch list of traits we as engineers need from a testing
> > suite
>
>
> Would be good to speak about portability, accessibility, and version
> independents. If a new contributor wants to add tests to this suite they
> need to be able to run it, and it should run within a "reasonable" time
> frame; one of the big issuers with python dtests is that it takes 14+
hours
> to run, this makes it no longer accessible to new contributors.
>
>
> On Tue, Jul 14, 2020 at 11:47 AM Joshua McKenzie <jmcken...@apache.org>
> wrote:
>
> > The purpose is purely to signal a point of view on the state of testing
> in
> > the codebase, some shortcomings of the architecture, and what a few of
us
> > are doing and further planning to do about it. Kind of a "prompt
> discussion
> > if anyone has a wild allergic reaction to it, or encourage collaboration
> if
> > they have a wild positive reaction" sort of thing. Maybe a spiritual
> > "CEP-lite". :)
> >
> > I would advocate that we be very selective about the topics on which we
> > strive for a consistent shared point of view as a project. There are a
> lot
> > of us and we all have different experiences and different points of view
> > that lead to different perspectives and value systems. Agreeing on
> discrete
> > definitions of done, 100% - that's table stakes. But agreeing on how we
> get
> > there, my personal take is we'd all be well served to spend our energy
> > Doing the Work and expressing these complementary positions rather than
> > trying to bend everyone to one consistent point of view.
> >
> > Let a thousand flowers bloom, as someone wise recently told me. :)
> >
> > That said, this work will be happening in an open source repo with a
> > permissive license (almost certainly ASLv2), likely using github issues,
> so
> > anyone that wants to collaborate on it would be most welcome. I can make
> > sure Gianluca, Charles, Berenguer, and others bring that to this ML
> thread
> > once we've started open-sourcing things.
> >
> > On Tue, Jul 14, 2020 at 4:25 AM Benedict Elliott Smith <
> > bened...@apache.org>
> > wrote:
> >
> > > It does raise the bar to critiquing the document though, but perhaps
> > > that's also a feature.
> > >
> > > Perhaps we can first discuss the purpose of the document? It seems to
> be
> > a
> > > mix of mission statement for the project, as well as your own near
term
> > > roadmap? Should we interpret it only as an advertisement of your own
> > view
> > > of the problems the project faces, as a start to dialogue, or is the
> > > purpose to solicit feedback?
> > >
> > > Would it be helpful to work towards a similar document the whole
> > community
> > > endorses, with a shared mission statement, and a (perhaps loosely
> > defined)
> > > shared roadmap?
> > >
> > > I'd like to call out some specific things in the document that I am
> > > personally excited by: the project has long lacked a coherent,
> repeatable
> > > approach to performance testing and regressions; combined with easy
> > > visualisation tools this would be a huge win. The FQL sampling with
> data
> > > distribution inference is also something that has been discussed
> > privately
> > > elsewhere, and would be hugely advantageous to the former, so that we
> can
> > > discover representative workloads.
> > >
> > > Thanks for taking the time to put this together, and start this
> dialogue.
> > >
> > >
> > > On 13/07/2020, 23:41, "Joshua McKenzie" <jmcken...@apache.org> wrote:
> > >
> > > >
> > > > Can you please allow comments on the doc so we can leave
> feedback.
> > > >
> > >
> > >
> > > > Doc is view only; figured we could keep this to the ML.
> > > >
> > > That's a feature, not a bug.
> > >
> > > Happy to chat here or on slack w/anyone. This is a complex topic
so
> > > long-form or high bandwidth communication is a better fit than
gdoc
> > > comments. They rapidly become unwieldy.
> > >
> > > On Mon, Jul 13, 2020 at 6:17 PM sankalp kohli <
> > kohlisank...@gmail.com>
> > > wrote:
> > >
> > > > Can you please allow comments on the doc so we can leave
> feedback.
> > > >
> > > > On Mon, Jul 13, 2020 at 2:16 PM Joshua McKenzie <
> > > jmcken...@apache.org>
> > > > wrote:
> > > >
> > > > > Link:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://docs.google.com/document/d/1ktuBWpD2NLurB9PUvmbwGgrXsgnyU58koOseZAfaFBQ/edit#
> > > > >
> > > > >
> > > > > Myself and a few other contributors are working with this
point
> > of
> > > view
> > > > as
> > > > > our frame of where we're going to work on improving testing on
> > the
> > > > project.
> > > > > I figured it might be useful to foster collaboration more
> broadly
> > > in the
> > > > > community as well as provide people with the opportunity to
> > > discuss work
> > > > > they're doing they may not yet have had a chance to bring up
or
> > > open
> > > > > source. While fallout is already open-sourced, expect the
> schema
> > > > anonymizer
> > > > > and some of the cassandra-diff + nosqlbench framework effort
to
> > be
> > > > > open-sourced / openly worked on soon. Anyone that's interested
> in
> > > > > collaborating, that would be highly welcome.
> > > > >
> > > > > Doc is view only; figured we could keep this to the ML.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > ~Josh
> > > > >
> > > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org
