Re: QA signup

[Jay Zhuang]

+1 for publishing official snapshot artifacts for 4.0 and even other
branches.

We're publishing snapshot artifacts to our internal artifactory. One minor
bug we found is: currently build.xml won't publish any snapshot artifact:
https://issues.apache.org/jira/browse/CASSANDRA-12704

On Thu, Sep 20, 2018 at 11:35 PM Dinesh Joshi
 wrote:

> I favor versioned nightlies for testing so everyone is using the exact
> binary distribution.
>
> As far as actually building the packages go, I would prefer a Docker based
> solution like Jon mentioned. It provides a controlled, reproducible, clean
> room environment. Ideally the build script should ensure that the git
> branch is clean and that there aren't any local changes if the packages are
> being published to maven.
>
> Does anyone see a need to publish the git branch metadata in the build
> like the git-sha, branch and repo url? I am not sure if this is already
> captured somewhere. Its useful to trace a build's provenance.
>
> Dinesh
>
> > On Sep 20, 2018, at 2:26 PM, Jonathan Haddad  wrote:
> >
> > Sure - I'm not disagreeing with you that pre-built packages would be nice
> > to have.  That said, if someone's gone through the trouble of building an
> > entire testing infrastructure and has hundreds of machines available,
> > running `docker-compose up build-deb` is likely not a major issue.  If
> I'm
> > trying to decide between solving the 2 problems I'd prefer to make builds
> > easier as very few people actually know how to do it.  I'm also biased
> > because I'm working on a tool that does _exactly_ that (build arbitrary
> C*
> > debs and deploy them to AWS for perf testing with tlp-stress which we've
> > already open sourced https://github.com/thelastpickle/tlp-stress).
> >
> > I'll building it for internal TLP use but there's not much TLP specific
> > stuff, we'll be open sourcing it as soon as we can.
> >
> > TL;DR: we need both things
> >
> > On Thu, Sep 20, 2018 at 2:12 PM Scott Andreas 
> wrote:
> >
> >> Mick – Got it, thanks and sorry to have misunderstood. No fault in your
> >> writing at all; that was my misreading.
> >>
> >> Agreed with you and Kurt; I can’t think of a pressing need or immediate
> >> use for the Maven artifacts. As you mentioned, all of the examples I’d
> >> listed require binary artifacts only.
> >>
> >> Re: Jon’s question:
> >>> It seems to me that improving / simplifying the process of building the
> >> packages might solve this problem better.
> >>
> >> Agreed that making builds easy is important, and that manually-applied
> >> patches were involved in a couple cases I’d cited. My main motivation is
> >> toward making it easier for developers who’d like to produce
> >> fully-automated test pipelines to do so using common artifacts, rather
> than
> >> each replicating the build/packaging step for tarball artifacts
> themselves.
> >>
> >> Publishing binary artifacts in a common location would enable developers
> >> to configure testing and benchmarking pipelines to pick up those
> artifacts
> >> on a daily basis without intervention. In the case of a build landing
> DOA
> >> due to an issue with a commit, it’d be enough for zero-touch automation
> to
> >> pick up a new build with the fix the following day and run an extended
> >> suite across a large number of machines and publish results, for
> example.
> >>
> >>
> >> On September 19, 2018 at 8:17:05 PM, kurt greaves (k...@instaclustr.com
> >> ) wrote:
> >>
> >> It's pretty much only third party plugins. I need it for the LDAP
> >> authenticator, and StratIO's lucene plugin will also need it. I know
> there
> >> are users out there with their own custom plugins that would benefit
> from
> >> it as well (and various other open source projects). It would make it
> >> easier, however it certainly is feasible for these devs to just build
> the
> >> jars themselves (and I've done this so far). If it's going to be easy I
> >> think there's value in generating and hosting nightly jars, but if it's
> >> difficult I can just write some docs for DIY.
> >>
> >> On Thu, 20 Sep 2018 at 12:20, Mick Semb Wever  wrote:
> >>
> >>> Sorry about the terrible english in my last email.
> >>>
> >>>
>  On the target audience:
> 
>  [snip]
>  For developers building automation around testing and
>  validation, it’d be great to have a common build to work from rather
>  than each developer producing these builds themselves.
> >>>
> >>>
> >>> Sure. My question was only in context of maven artefacts.
> >>> It seems to me all the use-cases you highlight would be for the binary
> >>> artefacts.
> >>>
> >>> If that's the case we don't need to worry about publishing snapshots
> >> maven
> >>> artefacts, and can just focus on uploading nightly builds to
> >>> https://dist.apache.org/repos/dist/dev/cassandra/
> >>>
> >>> Or is there a use-case I'm missing that needs the maven artefacts?
> >>>
> >>> 

Re: QA signup

[Dinesh Joshi]

I favor versioned nightlies for testing so everyone is using the exact binary 
distribution.

As far as actually building the packages go, I would prefer a Docker based 
solution like Jon mentioned. It provides a controlled, reproducible, clean room 
environment. Ideally the build script should ensure that the git branch is 
clean and that there aren't any local changes if the packages are being 
published to maven.

Does anyone see a need to publish the git branch metadata in the build like the 
git-sha, branch and repo url? I am not sure if this is already captured 
somewhere. Its useful to trace a build's provenance.

Dinesh

> On Sep 20, 2018, at 2:26 PM, Jonathan Haddad  wrote:
> 
> Sure - I'm not disagreeing with you that pre-built packages would be nice
> to have.  That said, if someone's gone through the trouble of building an
> entire testing infrastructure and has hundreds of machines available,
> running `docker-compose up build-deb` is likely not a major issue.  If I'm
> trying to decide between solving the 2 problems I'd prefer to make builds
> easier as very few people actually know how to do it.  I'm also biased
> because I'm working on a tool that does _exactly_ that (build arbitrary C*
> debs and deploy them to AWS for perf testing with tlp-stress which we've
> already open sourced https://github.com/thelastpickle/tlp-stress).
> 
> I'll building it for internal TLP use but there's not much TLP specific
> stuff, we'll be open sourcing it as soon as we can.
> 
> TL;DR: we need both things
> 
> On Thu, Sep 20, 2018 at 2:12 PM Scott Andreas  wrote:
> 
>> Mick – Got it, thanks and sorry to have misunderstood. No fault in your
>> writing at all; that was my misreading.
>> 
>> Agreed with you and Kurt; I can’t think of a pressing need or immediate
>> use for the Maven artifacts. As you mentioned, all of the examples I’d
>> listed require binary artifacts only.
>> 
>> Re: Jon’s question:
>>> It seems to me that improving / simplifying the process of building the
>> packages might solve this problem better.
>> 
>> Agreed that making builds easy is important, and that manually-applied
>> patches were involved in a couple cases I’d cited. My main motivation is
>> toward making it easier for developers who’d like to produce
>> fully-automated test pipelines to do so using common artifacts, rather than
>> each replicating the build/packaging step for tarball artifacts themselves.
>> 
>> Publishing binary artifacts in a common location would enable developers
>> to configure testing and benchmarking pipelines to pick up those artifacts
>> on a daily basis without intervention. In the case of a build landing DOA
>> due to an issue with a commit, it’d be enough for zero-touch automation to
>> pick up a new build with the fix the following day and run an extended
>> suite across a large number of machines and publish results, for example.
>> 
>> 
>> On September 19, 2018 at 8:17:05 PM, kurt greaves (k...@instaclustr.com
>> ) wrote:
>> 
>> It's pretty much only third party plugins. I need it for the LDAP
>> authenticator, and StratIO's lucene plugin will also need it. I know there
>> are users out there with their own custom plugins that would benefit from
>> it as well (and various other open source projects). It would make it
>> easier, however it certainly is feasible for these devs to just build the
>> jars themselves (and I've done this so far). If it's going to be easy I
>> think there's value in generating and hosting nightly jars, but if it's
>> difficult I can just write some docs for DIY.
>> 
>> On Thu, 20 Sep 2018 at 12:20, Mick Semb Wever  wrote:
>> 
>>> Sorry about the terrible english in my last email.
>>> 
>>> 
 On the target audience:
 
 [snip]
 For developers building automation around testing and
 validation, it’d be great to have a common build to work from rather
 than each developer producing these builds themselves.
>>> 
>>> 
>>> Sure. My question was only in context of maven artefacts.
>>> It seems to me all the use-cases you highlight would be for the binary
>>> artefacts.
>>> 
>>> If that's the case we don't need to worry about publishing snapshots
>> maven
>>> artefacts, and can just focus on uploading nightly builds to
>>> https://dist.apache.org/repos/dist/dev/cassandra/
>>> 
>>> Or is there a use-case I'm missing that needs the maven artefacts?
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 
> 
> 
> -- 
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[Jonathan Haddad]

Sure - I'm not disagreeing with you that pre-built packages would be nice
to have.  That said, if someone's gone through the trouble of building an
entire testing infrastructure and has hundreds of machines available,
running `docker-compose up build-deb` is likely not a major issue.  If I'm
trying to decide between solving the 2 problems I'd prefer to make builds
easier as very few people actually know how to do it.  I'm also biased
because I'm working on a tool that does _exactly_ that (build arbitrary C*
debs and deploy them to AWS for perf testing with tlp-stress which we've
already open sourced https://github.com/thelastpickle/tlp-stress).

I'll building it for internal TLP use but there's not much TLP specific
stuff, we'll be open sourcing it as soon as we can.

TL;DR: we need both things

On Thu, Sep 20, 2018 at 2:12 PM Scott Andreas  wrote:

> Mick – Got it, thanks and sorry to have misunderstood. No fault in your
> writing at all; that was my misreading.
>
> Agreed with you and Kurt; I can’t think of a pressing need or immediate
> use for the Maven artifacts. As you mentioned, all of the examples I’d
> listed require binary artifacts only.
>
> Re: Jon’s question:
> > It seems to me that improving / simplifying the process of building the
> packages might solve this problem better.
>
> Agreed that making builds easy is important, and that manually-applied
> patches were involved in a couple cases I’d cited. My main motivation is
> toward making it easier for developers who’d like to produce
> fully-automated test pipelines to do so using common artifacts, rather than
> each replicating the build/packaging step for tarball artifacts themselves.
>
> Publishing binary artifacts in a common location would enable developers
> to configure testing and benchmarking pipelines to pick up those artifacts
> on a daily basis without intervention. In the case of a build landing DOA
> due to an issue with a commit, it’d be enough for zero-touch automation to
> pick up a new build with the fix the following day and run an extended
> suite across a large number of machines and publish results, for example.
>
>
> On September 19, 2018 at 8:17:05 PM, kurt greaves (k...@instaclustr.com
> ) wrote:
>
> It's pretty much only third party plugins. I need it for the LDAP
> authenticator, and StratIO's lucene plugin will also need it. I know there
> are users out there with their own custom plugins that would benefit from
> it as well (and various other open source projects). It would make it
> easier, however it certainly is feasible for these devs to just build the
> jars themselves (and I've done this so far). If it's going to be easy I
> think there's value in generating and hosting nightly jars, but if it's
> difficult I can just write some docs for DIY.
>
> On Thu, 20 Sep 2018 at 12:20, Mick Semb Wever  wrote:
>
> > Sorry about the terrible english in my last email.
> >
> >
> > > On the target audience:
> > >
> > > [snip]
> > > For developers building automation around testing and
> > > validation, it’d be great to have a common build to work from rather
> > > than each developer producing these builds themselves.
> >
> >
> > Sure. My question was only in context of maven artefacts.
> > It seems to me all the use-cases you highlight would be for the binary
> > artefacts.
> >
> > If that's the case we don't need to worry about publishing snapshots
> maven
> > artefacts, and can just focus on uploading nightly builds to
> > https://dist.apache.org/repos/dist/dev/cassandra/
> >
> > Or is there a use-case I'm missing that needs the maven artefacts?
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: QA signup

[Scott Andreas]

Mick – Got it, thanks and sorry to have misunderstood. No fault in your writing 
at all; that was my misreading.

Agreed with you and Kurt; I can’t think of a pressing need or immediate use for 
the Maven artifacts. As you mentioned, all of the examples I’d listed require 
binary artifacts only.

Re: Jon’s question:
> It seems to me that improving / simplifying the process of building the 
> packages might solve this problem better.

Agreed that making builds easy is important, and that manually-applied patches 
were involved in a couple cases I’d cited. My main motivation is toward making 
it easier for developers who’d like to produce fully-automated test pipelines 
to do so using common artifacts, rather than each replicating the 
build/packaging step for tarball artifacts themselves.

Publishing binary artifacts in a common location would enable developers to 
configure testing and benchmarking pipelines to pick up those artifacts on a 
daily basis without intervention. In the case of a build landing DOA due to an 
issue with a commit, it’d be enough for zero-touch automation to pick up a new 
build with the fix the following day and run an extended suite across a large 
number of machines and publish results, for example.


On September 19, 2018 at 8:17:05 PM, kurt greaves 
(k...@instaclustr.com) wrote:

It's pretty much only third party plugins. I need it for the LDAP
authenticator, and StratIO's lucene plugin will also need it. I know there
are users out there with their own custom plugins that would benefit from
it as well (and various other open source projects). It would make it
easier, however it certainly is feasible for these devs to just build the
jars themselves (and I've done this so far). If it's going to be easy I
think there's value in generating and hosting nightly jars, but if it's
difficult I can just write some docs for DIY.

On Thu, 20 Sep 2018 at 12:20, Mick Semb Wever  wrote:

> Sorry about the terrible english in my last email.
>
>
> > On the target audience:
> >
> > [snip]
> > For developers building automation around testing and
> > validation, it’d be great to have a common build to work from rather
> > than each developer producing these builds themselves.
>
>
> Sure. My question was only in context of maven artefacts.
> It seems to me all the use-cases you highlight would be for the binary
> artefacts.
>
> If that's the case we don't need to worry about publishing snapshots maven
> artefacts, and can just focus on uploading nightly builds to
> https://dist.apache.org/repos/dist/dev/cassandra/
>
> Or is there a use-case I'm missing that needs the maven artefacts?
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: QA signup

[kurt greaves]

It's pretty much only third party plugins. I need it for the LDAP
authenticator, and StratIO's lucene plugin will also need it. I know there
are users out there with their own custom plugins that would benefit from
it as well (and various other open source projects). It would make it
easier, however it certainly is feasible for these devs to just build the
jars themselves (and I've done this so far). If it's going to be easy I
think there's value in generating and hosting nightly jars, but if it's
difficult I can just write some docs for DIY.

On Thu, 20 Sep 2018 at 12:20, Mick Semb Wever  wrote:

> Sorry about the terrible english in my last email.
>
>
> > On the target audience:
> >
> > [snip]
> >  For developers building automation around testing and
> > validation, it’d be great to have a common build to work from rather
> > than each developer producing these builds themselves.
>
>
> Sure. My question was only in context of maven artefacts.
> It seems to me all the use-cases you highlight would be for the binary
> artefacts.
>
> If that's the case we don't need to worry about publishing snapshots maven
> artefacts, and can just focus on uploading nightly builds to
> https://dist.apache.org/repos/dist/dev/cassandra/
>
> Or is there a use-case I'm missing that needs the maven artefacts?
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: QA signup

[Mick Semb Wever]

Sorry about the terrible english in my last email.


> On the target audience:
> 
> [snip]
>  For developers building automation around testing and 
> validation, it’d be great to have a common build to work from rather 
> than each developer producing these builds themselves.


Sure. My question was only in context of maven artefacts.
It seems to me all the use-cases you highlight would be for the binary 
artefacts.

If that's the case we don't need to worry about publishing snapshots maven 
artefacts, and can just focus on uploading nightly builds to 
https://dist.apache.org/repos/dist/dev/cassandra/

Or is there a use-case I'm missing that needs the maven artefacts?

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[Jonathan Haddad]

It seems to me that improving / simplifying the process of building the
packages might solve this problem better.  For example, in the tests you
linked to, they were using a custom build that hadn't been rolled into
trunk.  I expect we're going to see a lot of that.

If we make building a deb package as easy as running `docker-compose run
build-deb` then that addresses the problem of testing all branches.

This sort of exists already in cassandra-builds, but it's doing a little
more than just building a basic deb package.  Might be easier if it was
directly in the Cassandra repo.

On Wed, Sep 19, 2018 at 5:00 PM Scott Andreas  wrote:

> Got it, thanks!
>
> On the target audience:
>
> This would primarily be C* developers who are working on development,
> testing, and validation of the release. The performance testing exercise
> Joey, Vinay, Sumanth, Jason, Jordan, and Dinesh completed yesterday is a
> good example [1]. For developers building automation around testing and
> validation, it’d be great to have a common build to work from rather than
> each developer producing these builds themselves.
>
> Some purposes that could serve:
> – Ensuring we’re testing the same artifact. While a build produced from a
> given SHA should be ~identical, we can make stronger statements about
> particular build artifacts produced by common infrastructure than local
> builds produced by individual developers.
> – Automation: the ability to automate test suite runs on publication of a
> new build (e.g., perf, replay, traffic shadowing, upgrade tests, etc).
> – Faster dev/test/validation cycles. The perf tests in [1] identified two
> issues whose fixes will land in C-14503. Being able to pick up these fixes
> on commit via automation provides quicker feedback than waiting for a new
> build to be manually cut.
> – Fewer developers experiencing blocked automation. In a case where a
> regression is identified in a build produced by a commit (e.g., a snapshot
> build is “DOA” for testing purposes), a quick fix could resolve the issue
> with a new testable artifact produced within a day.
> – Better delineation between developer builds and those we recommend to
> the user community. Our ability to produce snapshot/nightly artifacts
> reduces pressure to cut an alpha for testing, reducing pressure to nominate
> community-facing releases in order to further the developer-focused goals
> above.
>
> ––
> [1]
> https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Performance+Testing
>
>
> On September 18, 2018 at 5:47:18 PM, Mick Semb Wever (m...@apache.org
> ) wrote:
>
> Scott,
>
> > @Mick, thanks for your reply re: publishing snapshots/nightlies. In
> > terms of what’s needed to configure these, would it be automation around
> > building release artifacts, publishing jars to the Maven snapshots repo,
> …
>
>
> Maven artefacts are deployed to the ASF snapshot repository in Nexus.
> The short of it is to add credentials for `apache.snapshots.https` to
> ~/.m2/settings.xml and run `ant publish`.
>
> It looks like `ant publish` won't run when it's not a release, but
> otherwise the maven deploy properties in `build.xml` look correct for
> snapshots.
>
> I haven't looked into how to automate this in Jenkins in regards to the
> settings.xml credentials and the gpg signing.
>
> For info at: http://www.apache.org/dev/publishing-maven-artifacts.html
>
> I question I have is who are we targeting with maven snapshots? Is this an
> audience that can easily enough be building the jars themselves to test
> during the feature freeze period?
>
>
> > and to dist/dev/cassandra on dist.apache.org for binary artifacts?
>
>
> This is a simpler task, just upload (`svn commit`) the nightly binaries to
> https://dist.apache.org/repos/dist/dev/cassandra/
>
> See https://www.apache.org/legal/release-policy.html#host-rc
>
> regards,
> Mick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: QA signup

[Scott Andreas]

Got it, thanks!

On the target audience:

This would primarily be C* developers who are working on development, testing, 
and validation of the release. The performance testing exercise Joey, Vinay, 
Sumanth, Jason, Jordan, and Dinesh completed yesterday is a good example [1]. 
For developers building automation around testing and validation, it’d be great 
to have a common build to work from rather than each developer producing these 
builds themselves.

Some purposes that could serve:
– Ensuring we’re testing the same artifact. While a build produced from a given 
SHA should be ~identical, we can make stronger statements about particular 
build artifacts produced by common infrastructure than local builds produced by 
individual developers.
– Automation: the ability to automate test suite runs on publication of a new 
build (e.g., perf, replay, traffic shadowing, upgrade tests, etc).
– Faster dev/test/validation cycles. The perf tests in [1] identified two 
issues whose fixes will land in C-14503. Being able to pick up these fixes on 
commit via automation provides quicker feedback than waiting for a new build to 
be manually cut.
– Fewer developers experiencing blocked automation. In a case where a 
regression is identified in a build produced by a commit (e.g., a snapshot 
build is “DOA” for testing purposes), a quick fix could resolve the issue with 
a new testable artifact produced within a day.
– Better delineation between developer builds and those we recommend to the 
user community. Our ability to produce snapshot/nightly artifacts reduces 
pressure to cut an alpha for testing, reducing pressure to nominate 
community-facing releases in order to further the developer-focused goals above.

––
[1] 
https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Performance+Testing


On September 18, 2018 at 5:47:18 PM, Mick Semb Wever 
(m...@apache.org) wrote:

Scott,

> @Mick, thanks for your reply re: publishing snapshots/nightlies. In
> terms of what’s needed to configure these, would it be automation around
> building release artifacts, publishing jars to the Maven snapshots repo, …


Maven artefacts are deployed to the ASF snapshot repository in Nexus.
The short of it is to add credentials for `apache.snapshots.https` to 
~/.m2/settings.xml and run `ant publish`.

It looks like `ant publish` won't run when it's not a release, but otherwise 
the maven deploy properties in `build.xml` look correct for snapshots.

I haven't looked into how to automate this in Jenkins in regards to the 
settings.xml credentials and the gpg signing.

For info at: http://www.apache.org/dev/publishing-maven-artifacts.html

I question I have is who are we targeting with maven snapshots? Is this an 
audience that can easily enough be building the jars themselves to test during 
the feature freeze period?


> and to dist/dev/cassandra on dist.apache.org for binary artifacts?


This is a simpler task, just upload (`svn commit`) the nightly binaries to 
https://dist.apache.org/repos/dist/dev/cassandra/

See https://www.apache.org/legal/release-policy.html#host-rc

regards,
Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[Mick Semb Wever]

Scott,

> @Mick, thanks for your reply re: publishing snapshots/nightlies. In 
> terms of what’s needed to configure these, would it be automation around 
> building release artifacts, publishing jars to the Maven snapshots repo, …


Maven artefacts are deployed to the ASF snapshot repository in Nexus.
The short of it is to add credentials for `apache.snapshots.https` to 
~/.m2/settings.xml and run `ant publish`. 

It looks like `ant publish` won't run when it's not a release, but otherwise 
the maven deploy properties in `build.xml` look correct for snapshots.

I haven't looked into how to automate this in Jenkins in regards to the 
settings.xml credentials and the gpg signing. 

For info at: http://www.apache.org/dev/publishing-maven-artifacts.html

I question I have is who are we targeting with maven snapshots? Is this an 
audience that can easily enough be building the jars themselves to test during 
the feature freeze period?


> and to dist/dev/cassandra on dist.apache.org for  binary artifacts?


This is a simpler task, just upload (`svn commit`) the nightly binaries to 
https://dist.apache.org/repos/dist/dev/cassandra/  

See  https://www.apache.org/legal/release-policy.html#host-rc

regards,
Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[Scott Andreas]

@Mick, thanks for your reply re: publishing snapshots/nightlies. In terms of 
what’s needed to configure these, would it be automation around building 
release artifacts, publishing jars to the Maven snapshots repo, and to 
dist/dev/cassandra on dist.apache.org for binary 
artifacts?

@Joey Thanks! Jason’s shown me what I was missing in Confluence. Can you try 
creating a page and letting me know if you’re able? Feel free to create any 
pages you think appropriate.

I’ve added permissions for the "cassandra-committers” group, and for the PMC 
members and committers who have accounts on 
cwiki.apache.org [1]. For those who don’t have a cwiki 
account but would like write access, please sign up and send Jason or myself a 
message; we can enable write perms: 
https://cwiki.apache.org/confluence/signup.action

– Scott

–––

[1] PMC members and committers with write access added:
– Jake Luciani
– Jason Brown
– Jonathan Ellis
– Jake Farrell
– Jeff Jirsa
– Jun Rao
– Matthieu Riou
– Sylvain Lebresna
– Avinash Lakshman
– Johan Oskarsson
– Mick Semb Wever


On September 12, 2018 at 4:44:51 PM, Joseph Lynch 
(joe.e.ly...@gmail.com) wrote:

> In looking at the Confluence space restrictions, it appears the main page is 
> open for editing and I don't see restrictions on page creation; can you try 
> to sign in, create one, and let me know if that doesn't work?

I signed in and went to "Jira reports" and then tried to hit "Add Jira
Report" and it said "Sorry you don't have permission to create
content. Contact your space administrator to request access". Also if
I just go to the Cassandra space there is no "Create" button in the in
the top left where it normally is, all I see when logged on in the
Cassandra space is "Spaces" and "People".

> I'm planning to create a few more pages for a couple testing tracks in a few 
> days, some of which are described here [1]. Eager to collaborate on these as 
> well.

Please let me know if you create one for internode messaging, we have
(I think valuable) data/methods from last Wednesday we can share. I've
started leaving the results we have so far as well as the methodology
we used at https://issues.apache.org/jira/browse/CASSANDRA-14746 with
the label "4.0-QA" but I can copy and paste anywhere we decide to put
these things.

-Joey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[Mick Semb Wever]

> Periodic SNAPSHOT builds sounds great. I'd feel much better about builds 
> published as date- or SHA-stamped snapshots / nightlies rather than 
> calling them alphas at this point, as everyone's testing work is 
> beginning. Can someone offer details on what would need to be done to 
> publish snapshots or nightlies in the context of Apache build 
> infrastructure?


Yes.

For the maven artefacts, timestamped snapshots (or nightlies) can be made 
available through
  https://repository.apache.org/content/repositories/snapshots/

For the downloadable binary and source artefacts the staging area can be used 
for nightlies
  https://dist.apache.org/repos/dist/dev/cassandra/


(This staging area is now what is supposed to be used for staging releases, 
instead of people.apache.org which was the old ASF way of doing it. I wrote 
some of this up in the recent release docs, which are still wip: 
https://github.com/apache/cassandra/blob/trunk/doc/source/development/release_process.rst
 )

Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[Scott Andreas]

Thanks for getting started with performance testing - this is exciting to hear!

Periodic SNAPSHOT builds sounds great. I'd feel much better about builds 
published as date- or SHA-stamped snapshots / nightlies rather than calling 
them alphas at this point, as everyone's testing work is beginning. Can someone 
offer details on what would need to be done to publish snapshots or nightlies 
in the context of Apache build infrastructure?

In looking at the Confluence space restrictions, it appears the main page is 
open for editing and I don't see restrictions on page creation; can you try to 
sign in, create one, and let me know if that doesn't work?

I'm planning to create a few more pages for a couple testing tracks in a few 
days, some of which are described here [1]. Eager to collaborate on these as 
well.

–––
[1] http://cassandra.apache.org/blog/2018/08/21/testing_apache_cassandra.html.


From: Joseph Lynch 
Sent: Friday, September 7, 2018 1:20:19 PM
To: dev@cassandra.apache.org
Subject: Re: QA signup

I don't think anyone has mentioned this yet but we probably want to
consider releasing 4.0 alpha jars to maven central soon so the open
source ecosystem can start testing a consistent Cassandra 4.0; for
example I had to hack 4.0 into Priam's build [1] by manually building
a jar and checking it in which is ... not particularly good or
reproducible for others. I'm not sure how hard it would be but
supporting periodic SNAPSHOT releases would also at least allow
building against trunk and would be great too. It also might be a good
idea to have a document (confluence page?) of breaking changes that
are most likely to require a change from users. For example the
SeedProvider interface change is probably going to break almost
everyone's deployment (but is easy to fix), and having a central list
of removed yaml options might be helpful past the NEWs file.

Regarding testing areas, we deployed trunk in the Netflix testing
environment on Wednesday with the aim to test the netty internode
messaging subsystem on 200+ node clusters. We are working with Jason,
Dinesh, and Jordan and have already found some interesting results and
would like to write them down as well as working on establishing good
baselines and testing methodology for stressing that subsystem. Is the
consensus here to create Jira epics tagged with 4.0 blocker for each
subsystem, or confluence pages (if confluence I think we need to give
people permissions to add pages?)?

Other areas we can help test and are looking for collaborators on are
audit/full query logging and we are potentially interested in helping
to test repair, but our internal implementation doesn't support
Cassandra 4.x ... we can re-work the CASSANDRA-14346 patch without too
much effort I think to thoroughly test full/incremental repair at any
scale cluster (or maybe Reaper folks can test repair).

[1] 
https://github.com/Netflix/Priam/pull/713/files#diff-3c33bef9f0334cf724470d50eae8dd8b

-Joey

On Fri, Sep 7, 2018 at 9:57 AM Jonathan Haddad  wrote:
>
> Really good idea JD. Keeping all the tests under an umbrella ticket for the
> feature with everything linked back makes a lot of sense.
>
> On Thu, Sep 6, 2018 at 11:09 PM J. D. Jordan 
> wrote:
>
> > I would suggest that JIRA’s tagged as 4.0 blockers be created for the list
> > once it is fleshed out.  Test plans and results could be posted to said
> > JIRAs, to be closed once a given test passes. Any bugs found can also then
> > be related back to such a ticket for tracking them as well.
> >
> > -Jeremiah
> >
> > > On Sep 6, 2018, at 12:27 PM, Jonathan Haddad  wrote:
> > >
> > > I completely agree with you, Sankalp.  I didn't want to dig too deep into
> > > the underlying testing methodology (and I still think we shouldn't just
> > > yet) but if the goal is to have confidence in the release, our QA process
> > > needs to be comprehensive.
> > >
> > > I believe that having focused teams for each component with a team leader
> > > with support from committers & contributors gives us the best shot at
> > > defining large scale functional tests that can be used to form both
> > > progress and bug reports.  (A person could / hopefully will be on more
> > than
> > > one team).  Coming up with those comprehensive tests will be the jobs of
> > > the teams, getting frequent bidirectional feedback on the dev ML.  Bugs
> > go
> > > in JIRA as per usual.
> > >
> > > Hopefully we can continue this process after the release, giving the
> > > project more structure, and folding more people in over time as
> > > contributors and ideally committers / PMC.
> > >
> > > Jon
> > >
> > >
> > >> On Thu, Sep 6, 2018 at 1:15 PM sank

Re: QA signup

[Joseph Lynch]

I don't think anyone has mentioned this yet but we probably want to
consider releasing 4.0 alpha jars to maven central soon so the open
source ecosystem can start testing a consistent Cassandra 4.0; for
example I had to hack 4.0 into Priam's build [1] by manually building
a jar and checking it in which is ... not particularly good or
reproducible for others. I'm not sure how hard it would be but
supporting periodic SNAPSHOT releases would also at least allow
building against trunk and would be great too. It also might be a good
idea to have a document (confluence page?) of breaking changes that
are most likely to require a change from users. For example the
SeedProvider interface change is probably going to break almost
everyone's deployment (but is easy to fix), and having a central list
of removed yaml options might be helpful past the NEWs file.

Regarding testing areas, we deployed trunk in the Netflix testing
environment on Wednesday with the aim to test the netty internode
messaging subsystem on 200+ node clusters. We are working with Jason,
Dinesh, and Jordan and have already found some interesting results and
would like to write them down as well as working on establishing good
baselines and testing methodology for stressing that subsystem. Is the
consensus here to create Jira epics tagged with 4.0 blocker for each
subsystem, or confluence pages (if confluence I think we need to give
people permissions to add pages?)?

Other areas we can help test and are looking for collaborators on are
audit/full query logging and we are potentially interested in helping
to test repair, but our internal implementation doesn't support
Cassandra 4.x ... we can re-work the CASSANDRA-14346 patch without too
much effort I think to thoroughly test full/incremental repair at any
scale cluster (or maybe Reaper folks can test repair).

[1] 
https://github.com/Netflix/Priam/pull/713/files#diff-3c33bef9f0334cf724470d50eae8dd8b

-Joey

On Fri, Sep 7, 2018 at 9:57 AM Jonathan Haddad  wrote:
>
> Really good idea JD. Keeping all the tests under an umbrella ticket for the
> feature with everything linked back makes a lot of sense.
>
> On Thu, Sep 6, 2018 at 11:09 PM J. D. Jordan 
> wrote:
>
> > I would suggest that JIRA’s tagged as 4.0 blockers be created for the list
> > once it is fleshed out.  Test plans and results could be posted to said
> > JIRAs, to be closed once a given test passes. Any bugs found can also then
> > be related back to such a ticket for tracking them as well.
> >
> > -Jeremiah
> >
> > > On Sep 6, 2018, at 12:27 PM, Jonathan Haddad  wrote:
> > >
> > > I completely agree with you, Sankalp.  I didn't want to dig too deep into
> > > the underlying testing methodology (and I still think we shouldn't just
> > > yet) but if the goal is to have confidence in the release, our QA process
> > > needs to be comprehensive.
> > >
> > > I believe that having focused teams for each component with a team leader
> > > with support from committers & contributors gives us the best shot at
> > > defining large scale functional tests that can be used to form both
> > > progress and bug reports.  (A person could / hopefully will be on more
> > than
> > > one team).  Coming up with those comprehensive tests will be the jobs of
> > > the teams, getting frequent bidirectional feedback on the dev ML.  Bugs
> > go
> > > in JIRA as per usual.
> > >
> > > Hopefully we can continue this process after the release, giving the
> > > project more structure, and folding more people in over time as
> > > contributors and ideally committers / PMC.
> > >
> > > Jon
> > >
> > >
> > >> On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli 
> > wrote:
> > >>
> > >> Thanks for starting this Jon.
> > >> Instead of saying "I tested streaming", we should define what all was
> > >> tested like was all data transferred, what happened when stream failed,
> > >> etc.
> > >> Based on talking to a few users, looks like most testing is done by
> > doing
> > >> an operation or running a load and seeing if it "worked" and no errors
> > in
> > >> logs.
> > >>
> > >> Another important thing will be to fix bugs asap ahead of testing,  as
> > >> fixes can lead to more bugs :)
> > >>
> >  On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad 
> > wrote:
> > >>>
> > >>> I was thinking along the same lines.  For this to be successful I think
> > >>> either weekly or bi-weekly summary reports back to the mailing list by
> > >> the
> > >>> team lead for each subsection on what's been tested and how it's been
> > >>> tested will help keep things moving along.
> > >>>
> > >>> In my opinion the lead for each team should *not* be the contributor
> > that
> > >>> wrote the feature, but someone who's very interested in it and can use
> > >> the
> > >>> contributor as a resource.  I think it would be difficult for the
> > >>> contributor to poke holes in their own work - if they could do that it
> > >>> would have been done already.  This should be a verification process
> > >> that's
> 

Re: QA signup

[Jonathan Haddad]

Really good idea JD. Keeping all the tests under an umbrella ticket for the
feature with everything linked back makes a lot of sense.

On Thu, Sep 6, 2018 at 11:09 PM J. D. Jordan 
wrote:

> I would suggest that JIRA’s tagged as 4.0 blockers be created for the list
> once it is fleshed out.  Test plans and results could be posted to said
> JIRAs, to be closed once a given test passes. Any bugs found can also then
> be related back to such a ticket for tracking them as well.
>
> -Jeremiah
>
> > On Sep 6, 2018, at 12:27 PM, Jonathan Haddad  wrote:
> >
> > I completely agree with you, Sankalp.  I didn't want to dig too deep into
> > the underlying testing methodology (and I still think we shouldn't just
> > yet) but if the goal is to have confidence in the release, our QA process
> > needs to be comprehensive.
> >
> > I believe that having focused teams for each component with a team leader
> > with support from committers & contributors gives us the best shot at
> > defining large scale functional tests that can be used to form both
> > progress and bug reports.  (A person could / hopefully will be on more
> than
> > one team).  Coming up with those comprehensive tests will be the jobs of
> > the teams, getting frequent bidirectional feedback on the dev ML.  Bugs
> go
> > in JIRA as per usual.
> >
> > Hopefully we can continue this process after the release, giving the
> > project more structure, and folding more people in over time as
> > contributors and ideally committers / PMC.
> >
> > Jon
> >
> >
> >> On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli 
> wrote:
> >>
> >> Thanks for starting this Jon.
> >> Instead of saying "I tested streaming", we should define what all was
> >> tested like was all data transferred, what happened when stream failed,
> >> etc.
> >> Based on talking to a few users, looks like most testing is done by
> doing
> >> an operation or running a load and seeing if it "worked" and no errors
> in
> >> logs.
> >>
> >> Another important thing will be to fix bugs asap ahead of testing,  as
> >> fixes can lead to more bugs :)
> >>
>  On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad 
> wrote:
> >>>
> >>> I was thinking along the same lines.  For this to be successful I think
> >>> either weekly or bi-weekly summary reports back to the mailing list by
> >> the
> >>> team lead for each subsection on what's been tested and how it's been
> >>> tested will help keep things moving along.
> >>>
> >>> In my opinion the lead for each team should *not* be the contributor
> that
> >>> wrote the feature, but someone who's very interested in it and can use
> >> the
> >>> contributor as a resource.  I think it would be difficult for the
> >>> contributor to poke holes in their own work - if they could do that it
> >>> would have been done already.  This should be a verification process
> >> that's
> >>> independent as possible from the original work.
> >>>
> >>> In addition to the QA process, it would be great if we could get a docs
> >>> team together.  We've got quite a bit of undocumented features and
> nuance
> >>> still, I think hammering that out would be a good idea.  Mick brought
> up
> >>> updating the website docs in the thread on testing different JDK's [1],
> >> if
> >>> we could figure that out in the process we'd be in a really great
> >> position
> >>> from the user perspective.
> >>>
> >>> Jon
> >>>
> >>> [1]
> >>
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
> >>>
> > On Thu, Sep 6, 2018 at 10:35 AM Jordan West 
> wrote:
> 
>  Thanks for staring this thread Jon!
> 
> > On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
>  wrote:
> 
> > For 4.0, I'm thinking it would be a good idea to put together a list
> >> of
>  the
> > things that need testing and see if people are willing to help test /
>  break
> > those things.  My goal here is to get as much coverage as possible,
> >> and
>  let
> > folks focus on really hammering on specific things rather than just
>  firing
> > up a cluster and rubber stamping it.  If we're going to be able to
> > confidently deploy 4.0 quickly after it's release we're going to
> >> need a
> > high attention to detail.
>  +1 to a more coordinated effort. I think we could use the Confluence
> >> that
>  was set up a little bit ago since it was setup for this purpose, at
> >> least
>  for finalized plans and results:
>  https://cwiki.apache.org/confluence/display/CASSANDRA.
> 
> 
> > In addition to a signup sheet, I think providing some guidance on how
> >>> to
>  QA
> > each thing that's being tested would go a long way.  Throwing "hey
> >>> please
> > test sstable streaming" over the wall will only get quality feedback
> >>> from
> > folks that are already heavily involved in the development process.
> >> It
> > would be nice to bring some new faces into the project 

Re: QA signup

[Hyunsoo Lee]

I would like to contribute as well.

Best,
Hyunsoo

On Fri., Sep. 7, 2018, 7:22 a.m. Per Otterström, <
per.otterst...@ericsson.com> wrote:

> This is a great initiative! If we can create a structured verification
> approach on new and old features I think 4.0 will be up for a good start.
> Jon, you can add my team to that signup sheet.
>
> /pelle
>
> -Original Message-
> From: Varun Barala 
> Sent: den 6 september 2018 15:16
> To: dev@cassandra.apache.org
> Subject: Re: QA signup
>
> +1
> I personally would like to contribute.
>
> On Thu, Sep 6, 2018 at 8:51 PM Jonathan Haddad  wrote:
>
> > For 4.0, I'm thinking it would be a good idea to put together a list
> > of the things that need testing and see if people are willing to help
> > test / break those things.  My goal here is to get as much coverage as
> > possible, and let folks focus on really hammering on specific things
> > rather than just firing up a cluster and rubber stamping it.  If we're
> > going to be able to confidently deploy 4.0 quickly after it's release
> > we're going to need a high attention to detail.
> >
> > In addition to a signup sheet, I think providing some guidance on how
> > to QA each thing that's being tested would go a long way.  Throwing
> > "hey please test sstable streaming" over the wall will only get
> > quality feedback from folks that are already heavily involved in the
> > development process.  It would be nice to bring some new faces into
> > the project by providing a little guidance.
> >
> > We could help facilitate this even further by considering the people
> > signing up to test a particular feature as a team, with seasoned
> > Cassandra veterans acting as team leads.
> >
> > Any thoughts?  I'm happy to take the lead on this.
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

RE: QA signup

[Per Otterström]

This is a great initiative! If we can create a structured verification approach 
on new and old features I think 4.0 will be up for a good start. Jon, you can 
add my team to that signup sheet.

/pelle

-Original Message-
From: Varun Barala  
Sent: den 6 september 2018 15:16
To: dev@cassandra.apache.org
Subject: Re: QA signup

+1
I personally would like to contribute.

On Thu, Sep 6, 2018 at 8:51 PM Jonathan Haddad  wrote:

> For 4.0, I'm thinking it would be a good idea to put together a list 
> of the things that need testing and see if people are willing to help 
> test / break those things.  My goal here is to get as much coverage as 
> possible, and let folks focus on really hammering on specific things 
> rather than just firing up a cluster and rubber stamping it.  If we're 
> going to be able to confidently deploy 4.0 quickly after it's release 
> we're going to need a high attention to detail.
>
> In addition to a signup sheet, I think providing some guidance on how 
> to QA each thing that's being tested would go a long way.  Throwing 
> "hey please test sstable streaming" over the wall will only get 
> quality feedback from folks that are already heavily involved in the 
> development process.  It would be nice to bring some new faces into 
> the project by providing a little guidance.
>
> We could help facilitate this even further by considering the people 
> signing up to test a particular feature as a team, with seasoned 
> Cassandra veterans acting as team leads.
>
> Any thoughts?  I'm happy to take the lead on this.
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[Dinesh Joshi]

> On Sep 6, 2018, at 10:27 AM, Jonathan Haddad  wrote:
> 
> I completely agree with you, Sankalp.  I didn't want to dig too deep into
> the underlying testing methodology (and I still think we shouldn't just
> yet) but if the goal is to have confidence in the release, our QA process
> needs to be comprehensive.

I think it is critical to specify the parameters and methodology. We can start 
with the major areas that we'd like to test and later dig deeper into the 
specifics.

> I believe that having focused teams for each component with a team leader
> with support from committers & contributors gives us the best shot at
> defining large scale functional tests that can be used to form both
> progress and bug reports.  (A person could / hopefully will be on more than
> one team).  Coming up with those comprehensive tests will be the jobs of
> the teams, getting frequent bidirectional feedback on the dev ML.  Bugs go
> in JIRA as per usual.
> 
> Hopefully we can continue this process after the release, giving the
> project more structure, and folding more people in over time as
> contributors and ideally committers / PMC.

+1

Dinesh
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: QA signup

[J. D. Jordan]

I would suggest that JIRA’s tagged as 4.0 blockers be created for the list once 
it is fleshed out.  Test plans and results could be posted to said JIRAs, to be 
closed once a given test passes. Any bugs found can also then be related back 
to such a ticket for tracking them as well.

-Jeremiah

> On Sep 6, 2018, at 12:27 PM, Jonathan Haddad  wrote:
> 
> I completely agree with you, Sankalp.  I didn't want to dig too deep into
> the underlying testing methodology (and I still think we shouldn't just
> yet) but if the goal is to have confidence in the release, our QA process
> needs to be comprehensive.
> 
> I believe that having focused teams for each component with a team leader
> with support from committers & contributors gives us the best shot at
> defining large scale functional tests that can be used to form both
> progress and bug reports.  (A person could / hopefully will be on more than
> one team).  Coming up with those comprehensive tests will be the jobs of
> the teams, getting frequent bidirectional feedback on the dev ML.  Bugs go
> in JIRA as per usual.
> 
> Hopefully we can continue this process after the release, giving the
> project more structure, and folding more people in over time as
> contributors and ideally committers / PMC.
> 
> Jon
> 
> 
>> On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli  wrote:
>> 
>> Thanks for starting this Jon.
>> Instead of saying "I tested streaming", we should define what all was
>> tested like was all data transferred, what happened when stream failed,
>> etc.
>> Based on talking to a few users, looks like most testing is done by doing
>> an operation or running a load and seeing if it "worked" and no errors in
>> logs.
>> 
>> Another important thing will be to fix bugs asap ahead of testing,  as
>> fixes can lead to more bugs :)
>> 
 On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad  wrote:
>>> 
>>> I was thinking along the same lines.  For this to be successful I think
>>> either weekly or bi-weekly summary reports back to the mailing list by
>> the
>>> team lead for each subsection on what's been tested and how it's been
>>> tested will help keep things moving along.
>>> 
>>> In my opinion the lead for each team should *not* be the contributor that
>>> wrote the feature, but someone who's very interested in it and can use
>> the
>>> contributor as a resource.  I think it would be difficult for the
>>> contributor to poke holes in their own work - if they could do that it
>>> would have been done already.  This should be a verification process
>> that's
>>> independent as possible from the original work.
>>> 
>>> In addition to the QA process, it would be great if we could get a docs
>>> team together.  We've got quite a bit of undocumented features and nuance
>>> still, I think hammering that out would be a good idea.  Mick brought up
>>> updating the website docs in the thread on testing different JDK's [1],
>> if
>>> we could figure that out in the process we'd be in a really great
>> position
>>> from the user perspective.
>>> 
>>> Jon
>>> 
>>> [1]
>> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
>>> 
> On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:
 
 Thanks for staring this thread Jon!
 
> On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
 wrote:
 
> For 4.0, I'm thinking it would be a good idea to put together a list
>> of
 the
> things that need testing and see if people are willing to help test /
 break
> those things.  My goal here is to get as much coverage as possible,
>> and
 let
> folks focus on really hammering on specific things rather than just
 firing
> up a cluster and rubber stamping it.  If we're going to be able to
> confidently deploy 4.0 quickly after it's release we're going to
>> need a
> high attention to detail.
 +1 to a more coordinated effort. I think we could use the Confluence
>> that
 was set up a little bit ago since it was setup for this purpose, at
>> least
 for finalized plans and results:
 https://cwiki.apache.org/confluence/display/CASSANDRA.
 
 
> In addition to a signup sheet, I think providing some guidance on how
>>> to
 QA
> each thing that's being tested would go a long way.  Throwing "hey
>>> please
> test sstable streaming" over the wall will only get quality feedback
>>> from
> folks that are already heavily involved in the development process.
>> It
> would be nice to bring some new faces into the project by providing a
> little guidance.
 
> We could help facilitate this even further by considering the people
> signing up to test a particular feature as a team, with seasoned
 Cassandra
> veterans acting as team leads.
 
 +1 to this as well. I am always a fan of folks learning about a
 subsystem/project through testing. It can be challenging to get folks
>> new
 to a 

Re: QA signup

[Jonathan Haddad]

I completely agree with you, Sankalp.  I didn't want to dig too deep into
the underlying testing methodology (and I still think we shouldn't just
yet) but if the goal is to have confidence in the release, our QA process
needs to be comprehensive.

I believe that having focused teams for each component with a team leader
with support from committers & contributors gives us the best shot at
defining large scale functional tests that can be used to form both
progress and bug reports.  (A person could / hopefully will be on more than
one team).  Coming up with those comprehensive tests will be the jobs of
the teams, getting frequent bidirectional feedback on the dev ML.  Bugs go
in JIRA as per usual.

Hopefully we can continue this process after the release, giving the
project more structure, and folding more people in over time as
contributors and ideally committers / PMC.

Jon


On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli  wrote:

> Thanks for starting this Jon.
> Instead of saying "I tested streaming", we should define what all was
> tested like was all data transferred, what happened when stream failed,
> etc.
> Based on talking to a few users, looks like most testing is done by doing
> an operation or running a load and seeing if it "worked" and no errors in
> logs.
>
> Another important thing will be to fix bugs asap ahead of testing,  as
> fixes can lead to more bugs :)
>
> On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad  wrote:
>
> > I was thinking along the same lines.  For this to be successful I think
> > either weekly or bi-weekly summary reports back to the mailing list by
> the
> > team lead for each subsection on what's been tested and how it's been
> > tested will help keep things moving along.
> >
> > In my opinion the lead for each team should *not* be the contributor that
> > wrote the feature, but someone who's very interested in it and can use
> the
> > contributor as a resource.  I think it would be difficult for the
> > contributor to poke holes in their own work - if they could do that it
> > would have been done already.  This should be a verification process
> that's
> > independent as possible from the original work.
> >
> > In addition to the QA process, it would be great if we could get a docs
> > team together.  We've got quite a bit of undocumented features and nuance
> > still, I think hammering that out would be a good idea.  Mick brought up
> > updating the website docs in the thread on testing different JDK's [1],
> if
> > we could figure that out in the process we'd be in a really great
> position
> > from the user perspective.
> >
> > Jon
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
> >
> > On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:
> >
> > > Thanks for staring this thread Jon!
> > >
> > > On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
> > wrote:
> > >
> > > > For 4.0, I'm thinking it would be a good idea to put together a list
> of
> > > the
> > > > things that need testing and see if people are willing to help test /
> > > break
> > > > those things.  My goal here is to get as much coverage as possible,
> and
> > > let
> > > > folks focus on really hammering on specific things rather than just
> > > firing
> > > > up a cluster and rubber stamping it.  If we're going to be able to
> > > > confidently deploy 4.0 quickly after it's release we're going to
> need a
> > > > high attention to detail.
> > > >
> > > >
> > > +1 to a more coordinated effort. I think we could use the Confluence
> that
> > > was set up a little bit ago since it was setup for this purpose, at
> least
> > > for finalized plans and results:
> > > https://cwiki.apache.org/confluence/display/CASSANDRA.
> > >
> > >
> > > > In addition to a signup sheet, I think providing some guidance on how
> > to
> > > QA
> > > > each thing that's being tested would go a long way.  Throwing "hey
> > please
> > > > test sstable streaming" over the wall will only get quality feedback
> > from
> > > > folks that are already heavily involved in the development process.
> It
> > > > would be nice to bring some new faces into the project by providing a
> > > > little guidance.
> > > >
> > >
> > > > We could help facilitate this even further by considering the people
> > > > signing up to test a particular feature as a team, with seasoned
> > > Cassandra
> > > > veterans acting as team leads.
> > > >
> > >
> > > +1 to this as well. I am always a fan of folks learning about a
> > > subsystem/project through testing. It can be challenging to get folks
> new
> > > to a project excited about testing first but for those that do, or for
> > > committers who want to learn another part of the db, its a great way to
> > > learn.
> > >
> > > Another thing we can do here is make sure teams are writing about the
> > > testing they are doing and their results. This will help share
> knowledge
> > > about techniques and approaches 

Re: QA signup

[sankalp kohli]

Thanks for starting this Jon.
Instead of saying "I tested streaming", we should define what all was
tested like was all data transferred, what happened when stream failed,
etc.
Based on talking to a few users, looks like most testing is done by doing
an operation or running a load and seeing if it "worked" and no errors in
logs.

Another important thing will be to fix bugs asap ahead of testing,  as
fixes can lead to more bugs :)

On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad  wrote:

> I was thinking along the same lines.  For this to be successful I think
> either weekly or bi-weekly summary reports back to the mailing list by the
> team lead for each subsection on what's been tested and how it's been
> tested will help keep things moving along.
>
> In my opinion the lead for each team should *not* be the contributor that
> wrote the feature, but someone who's very interested in it and can use the
> contributor as a resource.  I think it would be difficult for the
> contributor to poke holes in their own work - if they could do that it
> would have been done already.  This should be a verification process that's
> independent as possible from the original work.
>
> In addition to the QA process, it would be great if we could get a docs
> team together.  We've got quite a bit of undocumented features and nuance
> still, I think hammering that out would be a good idea.  Mick brought up
> updating the website docs in the thread on testing different JDK's [1], if
> we could figure that out in the process we'd be in a really great position
> from the user perspective.
>
> Jon
>
> [1]
>
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
>
> On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:
>
> > Thanks for staring this thread Jon!
> >
> > On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
> wrote:
> >
> > > For 4.0, I'm thinking it would be a good idea to put together a list of
> > the
> > > things that need testing and see if people are willing to help test /
> > break
> > > those things.  My goal here is to get as much coverage as possible, and
> > let
> > > folks focus on really hammering on specific things rather than just
> > firing
> > > up a cluster and rubber stamping it.  If we're going to be able to
> > > confidently deploy 4.0 quickly after it's release we're going to need a
> > > high attention to detail.
> > >
> > >
> > +1 to a more coordinated effort. I think we could use the Confluence that
> > was set up a little bit ago since it was setup for this purpose, at least
> > for finalized plans and results:
> > https://cwiki.apache.org/confluence/display/CASSANDRA.
> >
> >
> > > In addition to a signup sheet, I think providing some guidance on how
> to
> > QA
> > > each thing that's being tested would go a long way.  Throwing "hey
> please
> > > test sstable streaming" over the wall will only get quality feedback
> from
> > > folks that are already heavily involved in the development process.  It
> > > would be nice to bring some new faces into the project by providing a
> > > little guidance.
> > >
> >
> > > We could help facilitate this even further by considering the people
> > > signing up to test a particular feature as a team, with seasoned
> > Cassandra
> > > veterans acting as team leads.
> > >
> >
> > +1 to this as well. I am always a fan of folks learning about a
> > subsystem/project through testing. It can be challenging to get folks new
> > to a project excited about testing first but for those that do, or for
> > committers who want to learn another part of the db, its a great way to
> > learn.
> >
> > Another thing we can do here is make sure teams are writing about the
> > testing they are doing and their results. This will help share knowledge
> > about techniques and approaches that others can then apply. This
> knowledge
> > can be shared on the mailing list, a blog post, or in JIRA.
> >
> >  Jordan
> >
> >
> > > Any thoughts?  I'm happy to take the lead on this.
> > > --
> > > Jon Haddad
> > > http://www.rustyrazorblade.com
> > > twitter: rustyrazorblade
> > >
> >
>
>
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>

Re: QA signup

[Jonathan Haddad]

I was thinking along the same lines.  For this to be successful I think
either weekly or bi-weekly summary reports back to the mailing list by the
team lead for each subsection on what's been tested and how it's been
tested will help keep things moving along.

In my opinion the lead for each team should *not* be the contributor that
wrote the feature, but someone who's very interested in it and can use the
contributor as a resource.  I think it would be difficult for the
contributor to poke holes in their own work - if they could do that it
would have been done already.  This should be a verification process that's
independent as possible from the original work.

In addition to the QA process, it would be great if we could get a docs
team together.  We've got quite a bit of undocumented features and nuance
still, I think hammering that out would be a good idea.  Mick brought up
updating the website docs in the thread on testing different JDK's [1], if
we could figure that out in the process we'd be in a really great position
from the user perspective.

Jon

[1]
https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E

On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:

> Thanks for staring this thread Jon!
>
> On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad  wrote:
>
> > For 4.0, I'm thinking it would be a good idea to put together a list of
> the
> > things that need testing and see if people are willing to help test /
> break
> > those things.  My goal here is to get as much coverage as possible, and
> let
> > folks focus on really hammering on specific things rather than just
> firing
> > up a cluster and rubber stamping it.  If we're going to be able to
> > confidently deploy 4.0 quickly after it's release we're going to need a
> > high attention to detail.
> >
> >
> +1 to a more coordinated effort. I think we could use the Confluence that
> was set up a little bit ago since it was setup for this purpose, at least
> for finalized plans and results:
> https://cwiki.apache.org/confluence/display/CASSANDRA.
>
>
> > In addition to a signup sheet, I think providing some guidance on how to
> QA
> > each thing that's being tested would go a long way.  Throwing "hey please
> > test sstable streaming" over the wall will only get quality feedback from
> > folks that are already heavily involved in the development process.  It
> > would be nice to bring some new faces into the project by providing a
> > little guidance.
> >
>
> > We could help facilitate this even further by considering the people
> > signing up to test a particular feature as a team, with seasoned
> Cassandra
> > veterans acting as team leads.
> >
>
> +1 to this as well. I am always a fan of folks learning about a
> subsystem/project through testing. It can be challenging to get folks new
> to a project excited about testing first but for those that do, or for
> committers who want to learn another part of the db, its a great way to
> learn.
>
> Another thing we can do here is make sure teams are writing about the
> testing they are doing and their results. This will help share knowledge
> about techniques and approaches that others can then apply. This knowledge
> can be shared on the mailing list, a blog post, or in JIRA.
>
>  Jordan
>
>
> > Any thoughts?  I'm happy to take the lead on this.
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: QA signup

[Jordan West]

Thanks for staring this thread Jon!

On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad  wrote:

> For 4.0, I'm thinking it would be a good idea to put together a list of the
> things that need testing and see if people are willing to help test / break
> those things.  My goal here is to get as much coverage as possible, and let
> folks focus on really hammering on specific things rather than just firing
> up a cluster and rubber stamping it.  If we're going to be able to
> confidently deploy 4.0 quickly after it's release we're going to need a
> high attention to detail.
>
>
+1 to a more coordinated effort. I think we could use the Confluence that
was set up a little bit ago since it was setup for this purpose, at least
for finalized plans and results:
https://cwiki.apache.org/confluence/display/CASSANDRA.


> In addition to a signup sheet, I think providing some guidance on how to QA
> each thing that's being tested would go a long way.  Throwing "hey please
> test sstable streaming" over the wall will only get quality feedback from
> folks that are already heavily involved in the development process.  It
> would be nice to bring some new faces into the project by providing a
> little guidance.
>

> We could help facilitate this even further by considering the people
> signing up to test a particular feature as a team, with seasoned Cassandra
> veterans acting as team leads.
>

+1 to this as well. I am always a fan of folks learning about a
subsystem/project through testing. It can be challenging to get folks new
to a project excited about testing first but for those that do, or for
committers who want to learn another part of the db, its a great way to
learn.

Another thing we can do here is make sure teams are writing about the
testing they are doing and their results. This will help share knowledge
about techniques and approaches that others can then apply. This knowledge
can be shared on the mailing list, a blog post, or in JIRA.

 Jordan


> Any thoughts?  I'm happy to take the lead on this.
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>

Re: QA signup

[Varun Barala]

+1
I personally would like to contribute.

On Thu, Sep 6, 2018 at 8:51 PM Jonathan Haddad  wrote:

> For 4.0, I'm thinking it would be a good idea to put together a list of the
> things that need testing and see if people are willing to help test / break
> those things.  My goal here is to get as much coverage as possible, and let
> folks focus on really hammering on specific things rather than just firing
> up a cluster and rubber stamping it.  If we're going to be able to
> confidently deploy 4.0 quickly after it's release we're going to need a
> high attention to detail.
>
> In addition to a signup sheet, I think providing some guidance on how to QA
> each thing that's being tested would go a long way.  Throwing "hey please
> test sstable streaming" over the wall will only get quality feedback from
> folks that are already heavily involved in the development process.  It
> would be nice to bring some new faces into the project by providing a
> little guidance.
>
> We could help facilitate this even further by considering the people
> signing up to test a particular feature as a team, with seasoned Cassandra
> veterans acting as team leads.
>
> Any thoughts?  I'm happy to take the lead on this.
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>