[
https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110435#comment-17110435
]
Zhuqi Jin commented on CASSANDRA-15794:
---------------------------------------
Thanks for you prompt reply.
I still think that users who have already used compact storage should be
provided with a solution (even assuming CASSANDRA-15811 is fixed). Otherwise,
users may be confused, especially when they roll it back but cannot start 3.11.
For example,
1) Instead of directly block cassandra startup, is it possible to correctly
drop existing compact storage in 4.x (with the fix from CASSANDRA-15811)?
2) Or, if we stick to ask users to downgrade to 3.x and drop compact storage,
can we not generate new commit log before we hit the error in 4.x, so that we
avoid blocking 3.x from starting?
3) Or, block or warn users when they try to introduce compact storage in 3.x?
Throw a warning telling them that compact storage is deprecated in 3.x and will
cause problem upgrading to 4.x, or simply block them from doing that, or at
least modify the manual of 3.X, indicating that you should not use compact
storage.
If you think any of these is reasonable, I can provide help.
> Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact
> tables in downgraded C* (3.11.4) introduces non-existent columns
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15794
> Project: Cassandra
> Issue Type: Bug
> Reporter: Zhuqi Jin
> Priority: Normal
>
> We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the
> following problems.
> * We started a single 3.11.4 C* node.
> * We ran cassandra-stress like this
> {code:java}
> ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code}
> * We stopped this node, and started a C* node running C* compiled from trunk
> (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb)
> * New C* failed to start with the following error message
> {code:java}
> ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while
> loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245
> - Error while loading schema: java.lang.IllegalArgumentException: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception
> (java.lang.IllegalArgumentException) encountered during startup: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520
> CassandraDaemon.java:792 - Exception encountered during
> startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in
> Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE`
> command supplied in 3.x/3.11 Cassandra in order to migrate off Compact
> Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code}
> * We stopped the trunk version C* and started the 3.11.4 version C*.
> * 3.11.4 C* failed to start with the following error messages:
> {code:java}
> Exception (java.lang.IllegalStateException) encountered during startup:
> Unknown commitlog version 7Exception (java.lang.IllegalStateException)
> encountered during startup: Unknown commitlog version 7ERROR [main]
> 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered
> during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at
> org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148)
> ~[main/:na] at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
> [main/:na] at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
> [main/:na] at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732)
> [main/:na]
> {code}
> We manually deleted all new version commit logs and it worked. However, it's
> not obvious whether manually deleting these commit logs are safe.
> * And when we ran the "ALTER ... DROP COMPACT STORAGE" command on old
> version node as the error message suggested, non-existent columns appeared
> with null value.
> original table:
> {code:java}
> key | C0
> | C1
> | C2
> | C3
> | C4
> ------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------
> 0x36384b30374f33314c30 |
> 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 |
> 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 |
> 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 |
> 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 |
> 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986
> 0x30374b37384e364c3531 |
> 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d |
> 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 |
> 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c |
> 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa |
> 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb{code}
> After dropping compact storage:
> {code:java}
> key | column1 | C0
> | C1
> | C2
> | C3
> | C4
> | value
> ------------------------+---------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+-------
> 0x36384b30374f33314c30 | null |
> 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 |
> 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 |
> 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 |
> 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 |
> 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986 | null
> 0x30374b37384e364c3531 | null |
> 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d |
> 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 |
> 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c |
> 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa |
> 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb | null
> {code}
> * When we ran stress tool to read, IOExceptions were thrown, while it
> worked well on old version nodes
> * The error message is shown below:
> {code:java}
> java.io.IOException: Operation x0 on key(s) [4d32304c33314e374c30]: Data
> returned was not validatedjava.io.IOException: Operation x0 on key(s)
> [4d32304c33314e374c30]: Data returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:127) at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105) at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:248)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:466)java.io.IOException:
> Operation x0 on key(s) [4d304f334b3432353231]: Data returned was not
> validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:127) at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105) at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:248)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:466)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
