[
https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111117#comment-17111117
]
Alex Petrov commented on CASSANDRA-15794:
-----------------------------------------
[~Zhuqi1108] unfortunately, I do not see an easy solution for this problem. If
we do [CASSANDRA-15811] "the right way", it is rather easy to get rid of the
{{value}} column, since it's just a regular column which we can drop. However,
getting rid of the {{column1}} column is harder, since it's a clustering key.
It gets even trickier to do this without disruptions in the cluster, since we
have to rewrite sstables.
bq. Instead of directly block cassandra startup, is it possible to correctly
drop existing compact storage in 4.x
We've made a decision to prevent 4.0 from starting since it was clearly
announced that there will be _no_ Thrift support in 4.0. There's no reason not
to cut it out in 4.0, since we'll have to drop it eventually, and now is a
great time to do so.
bq. if we stick to ask users to downgrade to 3.x and drop compact storage, can
we not generate new commit log before we hit the error in 4.x, so that we avoid
blocking 3.x from starting?
That's a good point, and I'll look into how to make sure we don't write any
commit log messages in such case. If you would like to take over and check how
to do this - you're more than welcome.
bq. block or warn users when they try to introduce compact storage in 3.x?
That's also a good point. We should add a log and a client warning. Would you
like to do that? I'm not sure if blocking them from doing that is legitimate,
even though I don't understand why anyone would use compact storage today.
> Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact
> tables in downgraded C* (3.11.4) introduces non-existent columns
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15794
> Project: Cassandra
> Issue Type: Bug
> Reporter: Zhuqi Jin
> Priority: Normal
>
> We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the
> following problems.
> * We started a single 3.11.4 C* node.
> * We ran cassandra-stress like this
> {code:java}
> ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code}
> * We stopped this node, and started a C* node running C* compiled from trunk
> (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb)
> * New C* failed to start with the following error message
> {code:java}
> ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while
> loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245
> - Error while loading schema: java.lang.IllegalArgumentException: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception
> (java.lang.IllegalArgumentException) encountered during startup: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520
> CassandraDaemon.java:792 - Exception encountered during
> startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in
> Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE`
> command supplied in 3.x/3.11 Cassandra in order to migrate off Compact
> Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code}
> * We stopped the trunk version C* and started the 3.11.4 version C*.
> * 3.11.4 C* failed to start with the following error messages:
> {code:java}
> Exception (java.lang.IllegalStateException) encountered during startup:
> Unknown commitlog version 7Exception (java.lang.IllegalStateException)
> encountered during startup: Unknown commitlog version 7ERROR [main]
> 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered
> during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at
> org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148)
> ~[main/:na] at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
> [main/:na] at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
> [main/:na] at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732)
> [main/:na]
> {code}
> We manually deleted all new version commit logs and it worked. However, it's
> not obvious whether manually deleting these commit logs are safe.
> * And when we ran the "ALTER ... DROP COMPACT STORAGE" command on old
> version node as the error message suggested, non-existent columns appeared
> with null value.
> original table:
> {code:java}
> key | C0
> | C1
> | C2
> | C3
> | C4
> ------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------
> 0x36384b30374f33314c30 |
> 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 |
> 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 |
> 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 |
> 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 |
> 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986
> 0x30374b37384e364c3531 |
> 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d |
> 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 |
> 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c |
> 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa |
> 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb{code}
> After dropping compact storage:
> {code:java}
> key | column1 | C0
> | C1
> | C2
> | C3
> | C4
> | value
> ------------------------+---------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+-------
> 0x36384b30374f33314c30 | null |
> 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 |
> 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 |
> 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 |
> 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 |
> 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986 | null
> 0x30374b37384e364c3531 | null |
> 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d |
> 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 |
> 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c |
> 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa |
> 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb | null
> {code}
> * When we ran stress tool to read, IOExceptions were thrown, while it
> worked well on old version nodes
> * The error message is shown below:
> {code:java}
> java.io.IOException: Operation x0 on key(s) [4d32304c33314e374c30]: Data
> returned was not validatedjava.io.IOException: Operation x0 on key(s)
> [4d32304c33314e374c30]: Data returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:127) at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105) at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:248)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:466)java.io.IOException:
> Operation x0 on key(s) [4d304f334b3432353231]: Data returned was not
> validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:127) at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105) at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:248)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:466)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
