[
https://issues.apache.org/jira/browse/CASSANDRA-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175324#comment-15175324
]
Oliver Lockwood edited comment on CASSANDRA-11288 at 3/2/16 9:30 AM:
---------------------------------------------------------------------
I'm going to bump the severity to Major since I've got fairly reliable
reproduction of this issue and it's manifested in 4-node environments as well
as simpler 2-node ones.
See the log output below for a fuller example. What's interesting is that the
"agreed" schema AFTER the DROP TABLE command has been issued is exactly the
same version ({c264a58b-a3a4-39f8-84ae-c0960b7e8b4c}) as that at the end of the
previous DDL change. So it looks like the nodes agree that nothing has changed
yet, even though the EVENT DROPPED TABLE has come through.
{noformat}
2016-03-02 09:11:13 DEBUG SchemaChanger:61 - Executing cql statement ALTER
TABLE table_8 WITH compaction = { 'class' : 'LeveledCompactionStrategy' };
2016-03-02 09:11:13 DEBUG Cluster:2256 - Received event EVENT UPDATED TABLE
testkeyspace.table_8, scheduling delivery
2016-03-02 09:11:13 DEBUG Cluster:2199 - Refreshing schema for
testkeyspace.table_8 (TABLE)
2016-03-02 09:11:13 DEBUG ControlConnection:676 - Checking for schema
agreement: versions are [d138c607-135d-3081-99ec-350661791061,
c264a58b-a3a4-39f8-84ae-c0960b7e8b4c]
2016-03-02 09:11:14 DEBUG ControlConnection:676 - Checking for schema
agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c]
2016-03-02 09:11:15 DEBUG ControlConnection:294 - [Control connection]
Refreshing schema for testkeyspace.table_8 (TABLE)
2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement DROP TABLE
table_1;
2016-03-02 09:11:15 DEBUG Cluster:2256 - Received event EVENT DROPPED TABLE
testkeyspace.table_1, scheduling delivery
2016-03-02 09:11:15 DEBUG ControlConnection:676 - Checking for schema
agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c]
2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement CREATE
TABLE table_1 (party_id text primary key, profile text) WITH compaction = {
'class' : 'LeveledCompactionStrategy' };
Exception in thread "main"
com.datastax.driver.core.exceptions.AlreadyExistsException: Table
testkeyspace.table_1 already exists
at
com.datastax.driver.core.exceptions.AlreadyExistsException.copy(AlreadyExistsException.java:111)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
{noformat}
was (Author: oliver.lockwood):
I'm going to bump the severity to Major since I've got fairly reliable
reproduction of this issue and it's manifested in 4-node environments as well
as simpler 2-node ones.
See the log output below for a fuller example. What's interesting is that the
"agreed" schema AFTER the DROP TABLE command has been issued is exactly the
same version ({c264a58b-a3a4-39f8-84ae-c0960b7e8b4c}) as that at the end of the
previous DDL change. So it looks like the nodes agree that nothing has changed
yet, even though the EVENT DROPPED TABLE has come through.
{quote}
2016-03-02 09:11:13 DEBUG SchemaChanger:61 - Executing cql statement ALTER
TABLE table_8 WITH compaction = { 'class' : 'LeveledCompactionStrategy' };
2016-03-02 09:11:13 DEBUG Cluster:2256 - Received event EVENT UPDATED TABLE
testkeyspace.table_8, scheduling delivery
2016-03-02 09:11:13 DEBUG Cluster:2199 - Refreshing schema for
testkeyspace.table_8 (TABLE)
2016-03-02 09:11:13 DEBUG ControlConnection:676 - Checking for schema
agreement: versions are [d138c607-135d-3081-99ec-350661791061,
c264a58b-a3a4-39f8-84ae-c0960b7e8b4c]
2016-03-02 09:11:14 DEBUG ControlConnection:676 - Checking for schema
agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c]
2016-03-02 09:11:15 DEBUG ControlConnection:294 - [Control connection]
Refreshing schema for testkeyspace.table_8 (TABLE)
2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement DROP TABLE
table_1;
2016-03-02 09:11:15 DEBUG Cluster:2256 - Received event EVENT DROPPED TABLE
testkeyspace.table_1, scheduling delivery
2016-03-02 09:11:15 DEBUG ControlConnection:676 - Checking for schema
agreement: versions are [c264a58b-a3a4-39f8-84ae-c0960b7e8b4c]
2016-03-02 09:11:15 DEBUG SchemaChanger:61 - Executing cql statement CREATE
TABLE table_1 (party_id text primary key, profile text) WITH compaction = {
'class' : 'LeveledCompactionStrategy' };
Exception in thread "main"
com.datastax.driver.core.exceptions.AlreadyExistsException: Table
testkeyspace.table_1 already exists
at
com.datastax.driver.core.exceptions.AlreadyExistsException.copy(AlreadyExistsException.java:111)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52){quote}
> Schema agreement appears to be false positive following a DROP TABLE command
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-11288
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11288
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.0.14.439 (DSE 4.6.7)
> 2 nodes OR 4 nodes
> Connecting with Datastax Java driver 2.1.8
> Reporter: Oliver Lockwood
> Priority: Minor
>
> As part of a schema migration operation, our application is calling the
> following operations on the Java driver consecutively:
> {quote}
> session.execute("DROP TABLE table_name");
> session.execute("CREATE TABLE table_name (...)");
> {quote}
> The second of these sometimes fails with a {{DriverException}} whose message
> is "Table keyspace.table_name already exists".
> In the schema migration operation, there's 4 of these drop/create pairings
> and, although it's random which exact one fails, we've never managed to get
> further than the third operation in approximately 10 attempts - so there's a
> reasonably high proportion of failure.
> I don't believe this is a driver issue because the driver is checking for
> schema agreement (as per
> https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/ControlConnection.java#L701)
> and we are seeing a log message to that effect.
> {quote}
> c.d.d.c.ControlConnection - [] [] [] [] [] [] [] [] Checking for schema
> agreement: versions are [02bce936-fddd-3bef-bb54-124d31bede57]
> {quote}
> This log message appears in between our own logs which say "Executing
> statement DROP TABLE..." and "Executing statement CREATE TABLE...", so we can
> be reasonably sure this log message refers to the DROP operation being viewed
> as "in agreement".
> Could this be a bug in the Cassandra server erroneously reporting that the
> schemas are in agreement across the 2 nodes when, in fact, they are not?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
