[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhuqi Jin updated CASSANDRA-15794: ---------------------------------- Discovered By: User Report Severity: Normal Since Version: 3.11.4 Description: We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the following problems. * We started a single 3.11.4 C* node. * We ran cassandra-stress like this {code:java} ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} * We stopped this node, and started a C* node running C* compiled from trunk (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) * New C* failed to start with the following error message {code:java} ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: java.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception (java.lang.IllegalArgumentException) encountered during startup: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 CassandraDaemon.java:792 - Exception encountered during startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} * We stopped the trunk version C* and started the 3.11.4 version C*. * 3.11.4 C* failed to start with the following error messages: {code:java} Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7ERROR [main] 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [main/:na] {code} We manually deleted all new version commit logs and it worked. However, it's not obvious whether manually deleting these commit logs are safe. * And when we run the "ALTER ... DROP COMPACT STORAGE" command on old version node as the error message suggested, non-existent columns appeared with null value. original table: {code:java} key | C0 | C1 | C2 | C3 | C4 ------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------ 0x36384b30374f33314c30 | 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 | 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 | 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 | 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 | 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986 0x30374b37384e364c3531 | 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d | 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 | 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c | 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa | 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb{code} After dropping compact storage: {code:java} key | column1 | C0 | C1 | C2 | C3 | C4 | value ------------------------+---------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------- 0x36384b30374f33314c30 | null | 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 | 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 | 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 | 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 | 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986 | null 0x30374b37384e364c3531 | null | 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d | 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 | 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c | 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa | 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb | null {code} was: We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the following problems. * We started a single 3.11.4 C* node. * We ran cassandra-stress like this {code:java} ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} * We stopped this node, and started a C* node running C* compiled from trunk (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) * New C* failed to start with the following error message {code:java} ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: java.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception (java.lang.IllegalArgumentException) encountered during startup: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 CassandraDaemon.java:792 - Exception encountered during startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} * We stopped the trunk version C* and started the 3.11.4 version C*. * 3.11.4 C* failed to start with the following error messages: {code:java} Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7ERROR [main] 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [main/:na] {code} We manually deleted all new version commit logs and it worked. However, it's not obvious whether manually deleting these commit logs are safe. Summary: Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns (was: Upgraded C* (4.x) fail to start because of Compact Tables & downgraded C* (3.11.4) fail to start because of commit log) > Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact > tables in downgraded C* (3.11.4) introduces non-existent columns > --------------------------------------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-15794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15794 > Project: Cassandra > Issue Type: Bug > Reporter: Zhuqi Jin > Priority: Normal > > We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the > following problems. > * We started a single 3.11.4 C* node. > * We ran cassandra-stress like this > {code:java} > ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} > * We stopped this node, and started a C* node running C* compiled from trunk > (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) > * New C* failed to start with the following error message > {code:java} > ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while > loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 > - Error while loading schema: java.lang.IllegalArgumentException: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception > (java.lang.IllegalArgumentException) encountered during startup: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 > CassandraDaemon.java:792 - Exception encountered during > startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in > Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` > command supplied in 3.x/3.11 Cassandra in order to migrate off Compact > Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} > * We stopped the trunk version C* and started the 3.11.4 version C*. > * 3.11.4 C* failed to start with the following error messages: > {code:java} > Exception (java.lang.IllegalStateException) encountered during startup: > Unknown commitlog version 7Exception (java.lang.IllegalStateException) > encountered during startup: Unknown commitlog version 7ERROR [main] > 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered > during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at > org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) > ~[main/:na] at > org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276) > ~[main/:na] at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220) > ~[main/:na] at > org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132) > ~[main/:na] at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137) > ~[main/:na] at > org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167) > ~[main/:na] at > org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148) > ~[main/:na] at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) > [main/:na] at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) > [main/:na] at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) > [main/:na] > {code} > We manually deleted all new version commit logs and it worked. However, it's > not obvious whether manually deleting these commit logs are safe. > * And when we run the "ALTER ... DROP COMPACT STORAGE" command on old > version node as the error message suggested, non-existent columns appeared > with null value. > original table: > {code:java} > key | C0 > | C1 > | C2 > | C3 > | C4 > ------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------ > 0x36384b30374f33314c30 | > 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 | > 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 | > 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 | > 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 | > 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986 > 0x30374b37384e364c3531 | > 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d | > 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 | > 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c | > 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa | > 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb{code} > After dropping compact storage: > {code:java} > key | column1 | C0 > | C1 > | C2 > | C3 > | C4 > | value > ------------------------+---------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------- > 0x36384b30374f33314c30 | null | > 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 | > 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 | > 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 | > 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 | > 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986 | null > 0x30374b37384e364c3531 | null | > 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d | > 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 | > 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c | > 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa | > 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb | null > {code} > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org