[
https://issues.apache.org/jira/browse/CASSANDRA-10250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10250:
------------------------------------
Attachment: node3.log
node2.log
node1.log
concurrent_schema_changes.py
> Executing lots of schema alters concurrently can lead to dropped alters
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-10250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10250
> Project: Cassandra
> Issue Type: Bug
> Reporter: Andrew Hust
> Attachments: concurrent_schema_changes.py, node1.log, node2.log,
> node3.log
>
>
> A recently added
> [dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
> has been flapping on cassci and has exposed an issue with running lots of
> schema alterations concurrently. The failures occur on healthy clusters but
> seem to occur at higher rates when 1 node is down during the alters.
> The test executes the following – 440 total commands:
> - Create 20 new tables
> - Drop 7 columns one at time across 20 tables
> - Add 7 columns on at time across 20 tables
> - Add one column index on each of the 7 columns on 20 tables
> Outcome is random. Majority of the failures are dropped columns still being
> present, but new columns and indexes have been observed to be incorrect. The
> logs are don’t have exceptions and the columns/indexes that are incorrect
> don’t seem to follow a pattern. Running a {{nodetool describecluster}} on
> each node shows the same schema id on all nodes.
> Attached is a python script extracted from the dtest. Running against a
> local 3 node cluster will reproduce the issue (with enough runs – fails ~20%
> on my machine).
> Also attached is the node logs from a run with when a dropped column
> (alter_me_7 table, column s1) is still present. Checking the system_schema
> tables for this case shows the s1 column in both the columns and drop_columns
> tables.
> This has been flapping on cassci on versions 2+ and doesn’t seem to be
> related to changes in 3.0. More testing needs to be done though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
