Re: [PROPOSAL] Migrate to pytest from nosetests for dtests

[Philip Thompson]

I don't have any objection to this, really. I know I rely on a handful of
nose plugins, and possibly others do, but those should be easy enough to
re-write. I am curious though, what's the impetus for this? Is there some
pytest feature we want that nose lacks? Is there some nosetest bug or
restriction getting in the way?

On Tue, Nov 28, 2017 at 8:34 PM, Jon Haddad  wrote:

> +1
>
> I stopped using nose a long time ago in favor of py.test.  It’s a
> significant improvement.
>
> > On Nov 28, 2017, at 10:49 AM, Michael Kjellman 
> wrote:
> >
> > I'd like to propose we move from nosetest to pytest for the dtests. It
> looks like nosetests is basically abandoned, the python community doesn't
> like it, it hasn't been updated since 2015, and pytest even has nosetests
> support which would help us greatly during migration (
> https://docs.pytest.org/en/latest/nose.html).
> >
> > Thoughts?
> >
> > best,
> > kjellman
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Failing tests 2016-10-10

[Philip Thompson]

trunk:

===
testall: 2 failures

org.apache.cassandra.service.RemoveTest
 .testLocalHostId
CASSANDRA-9541.

org.apache.cassandra.db.KeyspaceTest
 .testLimitSSTables
New failure. Needs a jira ticket.

===
dtest: All passed!

===
upgrade: Currently failing to complete a run. Looking into tthis.
===
novnode: 8 failures

Still the 8 paging failures from CASSANDRA-12666. Under review

Failing tests 2016-10-05

[Philip Thompson]

trunk:

===
testall: 2 failures

org.apache.cassandra.service.RemoveTest
 .testLocalHostId
CASSANDRA-9541.

org.apache.cassandra.dht.tokenallocator.ReplicationAwareTokenAllocatorTest
 .testNewClusterWithMurmur3Partitioner
This times out every time. Someone should take a look

===
dtest: All passed!

===
upgrade: All passed!
===
novnode: 8 failures

Still the 8 paging failures from CASSANDRA-12666. Seriously, someone review
this.

Failing tests 2016-09-30

[Philip Thompson]

Hi All,

3.9 was released, so I've dropped it from this email.

trunk:

===
testall: 4 failures

org.apache.cassandra.config.DatabaseDescriptorRefTest
 .testDatabaseDescriptorRef

org.apache.cassandra.config.DatabaseDescriptorRefTest
 .testDatabaseDescriptorRef-compression
CASSANDRA-12677 for the two failures above.

org.apache.cassandra.service.RemoveTest
 .testBadHostId
CASSANDRA-12714. Can someone look at this? It's been unassigned for a
while

org.apache.cassandra.dht.tokenallocator.ReplicationAwareTokenAllocatorTest
 .testNewClusterWithMurmur3Partitioner
Just a timeout

===
dtest: 133 failures

These are all from CASSANDRA-12736

===
upgrade:All passed!
===
novnode: 8 failures

Still the 8 paging failures from CASSANDRA-12666. This is Patch Available,
can someone please review?

Re: Failing tests 2016-09-28

[Philip Thompson]

That ticket was only supposed to be committed to 3.10 and 3.0.x. Was it
accidentally also merged into 3.9?

On Wed, Sep 28, 2016 at 12:40 PM, Oleksandr Petrov <
oleksandr.pet...@gmail.com> wrote:

> LWTTester issues are leftovers from the bad merge in
> https://github.com/riptano/cassandra-dtest/pull/1214
>
> I've rebased and currently re-running tests.
>
> On Wed, Sep 28, 2016 at 6:23 PM Philip Thompson <
> philip.thomp...@datastax.com> wrote:
>
> > Hi All,
> >
> > cassandra-3.9:
> > ===
> > testall: All passed!
> >
> > ===
> > dtest: 4 failures
> >
> > cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> >  .test_round_trip_with_authentication
> > Flaky test, needs a Jira ticket
> >
> >
> > cql_tests.LWTTester
> >  .conditional_deletes_on_static_columns_with_null_values_test
> >
> > cql_tests.LWTTeste
> >  r.conditional_deletes_on_static_columns_with_null_values_batch_test
> >
> > cql_tests.LWTTester
> >  .conditional_updates_on_static_columns_with_null_values_test
> > These three failures are all the same, and need a ticket
> > ===
> > upgrade: 661 failures
> >
> > I won't enumerate these here. 12697 should resolve them, and also
> explains
> > where they're all coming from
> >
> > ===
> > novnode: All passed!
> >
> > ===
> >
> > trunk:
> >
> > ===
> > testall: 13 failures
> >
> > org.apache.cassandra.config.DatabaseDescriptorRefTest
> >  .testDatabaseDescriptorRef
> >
> > org.apache.cassandra.config.DatabaseDescriptorRefTest
> >  .testDatabaseDescriptorRef-compression
> > CASSANDRA-12677 for the two failures above.
> >
> > org.apache.cassandra.service.RemoveTest
> >  .testLocalHostId
> > CASSANDRA-12714
> >
> > 10 unlisted timeouts
> >
> > ===
> > dtest: 4 failures
> >
> > materialized_views_test.TestMaterializedViews
> >  .clustering_column_test
> >
> > materialized_views_test.TestMaterializedViews
> >  .insert_test
> >
> > materialized_views_test.TestMaterializedViews
> >  .query_all_new_column_test
> >
> > materialized_views_test.TestMaterializedViews
> >  .query_new_column_test
> >
> > No need to file a jira for these, I've fixed them already.
> >
> > ===
> > upgrade: 662 failures
> >
> > I won't enumerate these here. 12697 should resolve them, and also
> explains
> > where they're all coming from
> >
> > ===
> > novnode: 9 failures
> >
> > Same 8 paging failures. CASSANDRA-12666
> >
> > compaction_test.TestCompaction_with_DateTieredCompactionStrategy
> >  .bloomfilter_size_test
> > CASSANDRA-12711
> >
> > Thanks,
> > Philip
> >
> --
> Alex Petrov
>

Failing tests 2016-09-26

[Philip Thompson]

Hi All,

cassandra-3.9:
===
testall: All passed!

===
dtest: All passed!

===
upgrade: All passed!

===
novnode: All passed!

===

trunk:

===
testall:

org.apache.cassandra.config.DatabaseDescriptorRefTest
 .testDatabaseDescriptorRef

org.apache.cassandra.config.DatabaseDescriptorRefTest
 .testDatabaseDescriptorRef-compression
CASSANDRA-12677 for the two failures above. New failure

org.apache.cassandra.dht.tokenallocator.ReplicationAwareTokenAllocatorTest
 .testNewClusterWithMurmur3Partitioner
Failed due to timeout

org.apache.cassandra.service.RemoveTest
 .testLocalHostId
Failing due to NPE. New Failure, filed 12714
===
dtest: All passed!

===
upgrade: 660 failures

I won't enumerate these here. 12697 should resolve them, and also explains
where they're all coming from

===
novnode:

Same 8 paging failures.

compaction_test.TestCompaction_with_DateTieredCompactionStrategy
 .bloomfilter_size_test
New failure. 12711 was filed

Thanks,
Philip

Failing tests 2016-09-22

[Philip Thompson]

Hi All,

cassandra-3.9:
===
testall: 5 failures
org.apache.cassandra.cql3.ViewFilteringTest
 .testMVCreationSelectRestrictions

org.apache.cassandra.cql3.ViewFilteringTest
 .testClusteringKeyFilteringRestrictions

org.apache.cassandra.cql3.ViewTest
 .ttlExpirationTest

org.apache.cassandra.cql3.validation.entities
 .UFTest.testDropKeyspaceContainingFunctionDropsPreparedStatementsWit
hDelayedValues

org.apache.cassandra.cql3.validation.entities
 .UFTest.testEmptyString
All five test failures are due to timeouts.

===
dtest: All passed!

===
upgrade: All passed!

===
novnode: All passed!

===

trunk:

===
testall: 3 failures

org.apache.cassandra.config.DatabaseDescriptorRefTest
 .testDatabaseDescriptorRef

org.apache.cassandra.config.DatabaseDescriptorRefTest
 .testDatabaseDescriptorRef-compression
CASSANDRA-12677 for the two failures above. New failure

org.apache.cassandra.dht.tokenallocator.ReplicationAwareTokenAllocatorTest
 .testNewClusterWithMurmur3Partitioner
Failed due to timeout
===
dtest: All passed!

===
upgrade: All passed!

===
novnode: 8 failures

All failures are in paging test, and are due to CASSANDRA-12666 which is
Patch Available.

Re: [VOTE] Release Apache Cassandra 3.6

[Philip Thompson]

I've run the upgrade tests against both 3.4 and 3.5, and I have not seen
the new issues filed. This doesn't guarantee they are 3.6 regressions, but
they may be.

On Fri, May 13, 2016 at 5:59 AM, Aleksey Yeschenko <alek...@apache.org>
wrote:

> Seconding the -1 (binding) for the same reasons.
>
> --
> AY
>
> On 12 May 2016 at 23:44:46, Tyler Hobbs (ty...@datastax.com) wrote:
>
> Based on CASSANDRA-11613 (and CASSANDRA-11760), I'm changing my vote to a
> (non-binding) -1. There is a legit regression in upgrading non-frozen UDTs
> that needs to be fixed before releasing 3.6.
>
> On Thu, May 12, 2016 at 12:44 PM, Philip Thompson <
> philip.thomp...@datastax.com> wrote:
>
> > I've updated the TE report with the results of the upgrade testing we
> did.
> > We experienced a higher than expected number of test failures, which
> > prompted the filing of:
> >
> > CASSANDRA-11760
> > CASSANDRA-11763
> > CASSANDRA-11765
> > CASSANDRA-11767
> >
> > Two errors were related to handling the legacy hint format after upgrades
> > to 3.6 from either the 2.1 or 2.2 series. This should not affect upgrades
> > from 3.0.x, or new 3.6 clusters. 11760 is an issue handling UDTs in
> > mixed-versions 2.2.5 / 3.6 clusters.
> >
> > These four bugs were accompanied by other existing, known upgrade
> failures.
> > We do not suspect that these four issues are 3.6 regressions, as we have
> > tested this release with a large number of new upgrade tests, that were
> not
> > run on previous tick-tock releases. I am re-running these tests against
> > 3.4, to confirm that suspicion.
> >
> > On Tue, May 10, 2016 at 9:54 PM, Jake Luciani <j...@apache.org> wrote:
> >
> > > I propose the following artifacts for release as 3.6.
> > >
> > > sha1: c17cbe1875a974a00822ffbfad716abde363c8da
> > > Git:
> > >
> > >
> >
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.6-tentative
> > > Artifacts:
> > >
> > >
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/org/apache/cassandra/apache-cassandra/3.6/
> > > Staging repository:
> > >
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/
> > >
> > > The artifacts as well as the debian package are also available here:
> > > http://people.apache.org/~jake
> > >
> > > The vote will be open for 72 hours (longer if needed).
> > >
> > > [1]: http://goo.gl/Yv15Qz (CHANGES.txt)
> > > [2]: http://goo.gl/VyR9EG (NEWS.txt)
> > > [3]: https://goo.gl/raz8ok (DataStax QA Report)
> > >
> >
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>

Re: [VOTE] Release Apache Cassandra 3.6

[Philip Thompson]

I've updated the TE report with the results of the upgrade testing we did.
We experienced a higher than expected number of test failures, which
prompted the filing of:

CASSANDRA-11760
CASSANDRA-11763
CASSANDRA-11765
CASSANDRA-11767

Two errors were related to handling the legacy hint format after upgrades
to 3.6 from either the 2.1 or 2.2 series. This should not affect upgrades
from 3.0.x, or new 3.6 clusters. 11760 is an issue handling UDTs in
mixed-versions 2.2.5 / 3.6 clusters.

These four bugs were accompanied by other existing, known upgrade failures.
We do not suspect that these four issues are 3.6 regressions, as we have
tested this release with a large number of new upgrade tests, that were not
run on previous tick-tock releases. I am re-running these tests against
3.4, to confirm that suspicion.

On Tue, May 10, 2016 at 9:54 PM, Jake Luciani  wrote:

> I propose the following artifacts for release as 3.6.
>
> sha1: c17cbe1875a974a00822ffbfad716abde363c8da
> Git:
>
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.6-tentative
> Artifacts:
>
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/org/apache/cassandra/apache-cassandra/3.6/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/
>
> The artifacts as well as the debian package are also available here:
> http://people.apache.org/~jake
>
> The vote will be open for 72 hours (longer if needed).
>
> [1]: http://goo.gl/Yv15Qz (CHANGES.txt)
> [2]: http://goo.gl/VyR9EG (NEWS.txt)
> [3]: https://goo.gl/raz8ok (DataStax QA Report)
>

Re: March 2015 QA retrospective

[Philip Thompson]

To add to this:


*Went well*
Tyler Hobbs has reduced failing dtests on trunk by ~90%. By next month,
test results should be at 100% pass.

*Went poorly*
We've failed to make progress on running the full test suite across all
contributor branches. By the end of this month, I assume we will at least
have limited functionality in this area.

On Wed, Apr 1, 2015 at 3:57 PM, Ariel Weisberg ariel.weisb...@datastax.com
wrote:

 Hi all,

 It’s time for the first retrospective. For those not familiar this is the
 part of the development process where we discuss what is and isn’t working
 when it comes to making reliable releases. We go over the things that
 worked, the things that didn’t work, and what changes we are going to make.

 This is not a forum for discussing individual bugs (or bugs fixed before
 release due to successful process) although you can cite one and we can
 discuss what we could have done differently to catch it. Even if a bug
 wasn’t released if it was caught the wrong way (blind luck) and you think
 our process wouldn’t have caught it you can bring that up as well.

 I don’t expect this retrospective to be the most productive because we
 already know we are far behind in several areas (passing utests, dtests,
 running utests and dtests for on each commit, running a larger black box
 system test) and many issues will circle back around to being addressed by
 one of those three.

 If your a developer you can review all things you have committed (or
 reviewed) in the past month and ask yourself if it met the criteria of done
 that we agreed on including adding tests for existing untested code
 (usually the thing missed). Better to do it now then after discovering your
 definition of done was flawed because it released a preventible bug.

 For this one retrospective you can reach back further to something already
 released that you feel passionate about, and if you can point to a utest or
 dtest that should have caught it that is still missing we can add that to
 the list of things to test. That would go under CASSANDRA-9012 (Triage
 missing test coverage) 
 https://issues.apache.org/jira/browse/CASSANDRA-9012.

 There is a root JIRA https://issues.apache.org/jira/browse/CASSANDRA-9042
 for making trunk always releasable. A lot falls under CASSANDRA-9007 ( Run
 stress nightly against trunk in a way that validates ) 
 https://issues.apache.org/jira/browse/CASSANDRA-9007 which is the root
 for a new kitchen sink style test that validates the entire feature set
 together in a black box fashion. Philip Thompson has a basic job running so
 we are close to (or at) the tipping point where the doneness criteria for
 every ticket needs to include making sure this job covers the thing you
 added/changed. If you aren’t going to add the coverage you need to justify
 (to yourself and your reviewer) breaking it out into something separate and
 file a JIRA indicating the coverage was missing (if one doesn’t already
 exist). Make sure to link it to 9007 so we can see what has already been
 reported.

 The reason I say we might not be at the tipping point is that while we
 have the job we haven’t ironed out how stress (or something new) will act
 as a container for validating multiple features. Especially in an
 environment where things like cluster/node failures and topology changes
 occur.

 Retrospectives aren’t supposed to include the preceding paragraphs we
 should funnel discussion about them into a separate email thread.

 On to the retrospective. This is more for me to solicit from information
 from you then for me to push information to you.

 Went well
 Positive response to the definition of done
 Lot’s of manpower from QA and progress on test infrastructure
 Went poorly
 Some wanting to add validation to a kitchen sink style test, but not being
 able to yet
 Not having a way to know if we are effectively implementing the definition
 of done without waiting for bugs as feedback
 Changes
 Coordinate with Philip Thompson to see how we can get to having developers
 able to add validation to the kitchen sink style test

 Regards,
 Ariel

Re: Cassandra-dtest versioning proposal

[Philip Thompson]

The change that prompted the idea is CASSANDRA-8454. As we attempt to move
cql_tests.py and other single node, CQL based dtests into unit tests in 2.1
and 3.0+, we will have a large number of tests, in the low hundreds, that
not only have minimum version restrictions, but will only be applicable for
a particular slice of C* versions, e.g. 2.0.X  x  2.1.3.

On Fri, Jan 9, 2015 at 12:01 PM, Tyler Hobbs ty...@datastax.com wrote:

 On Thu, Jan 8, 2015 at 2:23 PM, Philip Thompson 
 philip.thomp...@datastax.com wrote:

  I expect the benefits to grow as we make more radical changes
  to cassandra-dtest for cassandra 3.0.
 

 What kinds of changes are you planning?  Perhaps we can come up with good
 alternatives.


 --
 Tyler Hobbs
 DataStax http://datastax.com/

Cassandra-dtest versioning proposal

[Philip Thompson]

All,

I would like to propose splitting the cassandra-dtest code base into
versioned branches that match the structure of the C* repository. Each C*
version branch, e.g. cassandra-2.0, would have a corresponding dtest
branch. This will allow us to isolate version specific changes to their own
branches, at the cost of extra merging work on contributors to
cassandra-dtest. I have a WIP and the code has become significantly
cleaner, and I expect the benefits to grow as we make more radical changes
to cassandra-dtest for cassandra 3.0.

I am looking for feedback from the contributors to Cassandra that use or
contribute to cassandra-dtest. Are you in favor of this change?

Thanks,
Philip Thompson

https://github.com/riptano/cassandra-dtest/compare/cassandra-2.0
https://github.com/riptano/cassandra-dtest/compare/cassandra-2.1
https://github.com/riptano/cassandra-dtest/compare/trunk

CASSANDRA-6313

[Philip Thompson]

Hello all,

The Cassandra Test Engineering team has been working on #6313, refactoring
cassandra-dtests to use the DataStax python driver used by cqlsh instead of
cassandra-dbapi2. The work has all been done in a branch on the
riptano/cassandra-dtest Github repository. The refactored tests have been
running on cassci since June 9 on the cassandra-2.1 branch. You can view
the changes either in the patch attached to 6313, or as a pull request on
the Github repository.

Our current plan is to make the merge sometime shortly after 2.1.0 is
released. Going forward after 6313 is resolved, all new dtests should be
written using the python-driver instead of dbapi2. The only new dependency
is the driver, and installation instructions will be available in the
README and INSTALL.md in dtests.

While the C* Test Engineering team will be reviewing all the changes for
the ticket, we welcome any additional set of eyes. Feel free to leave any
comments on either Github or JIRA.

Notable changes:
- Dtests no longer support versions older than 1.2.
- The cursor no longer has a fetch command, results from statements are now
returned directly from the cursor.execute() function

Notable constants:
- The API for dtest.py remains the same.
- Some thrift bindings remain for tests that absolutely required them.

Relevant links:
- Pull Request: https://github.com/riptano/cassandra-dtest/pull/89
- JIRA Ticket: https://issues.apache.org/jira/browse/CASSANDRA-6313
- Cassci job: http://cassci.datastax.com/job/cassandra-2.1_dtest_pydriver/

Thanks,
Philip Thompson