Jeremiah Jordan created CASSANDRA-12023:
-------------------------------------------
Summary: Schema upgrade bug with super columns
Key: CASSANDRA-12023
URL: https://issues.apache.org/jira/browse/CASSANDRA-12023
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Jeremiah Jordan
Priority: Critical
Fix For: 3.0.x, 3.x
Doing some upgrade tests starting on 2.0 to 2.1 to 3.0 we hit the follow bug
that prevents 3.0 nodes from starting. Running the test a few times with
different waits and flushing sometimes or not I have seen the following errors:
{code}
ERROR [main] 2016-06-17 10:42:40,112 CassandraDaemon.java:698 - Exception
encountered during startup
org.apache.cassandra.serializers.MarshalException: cannot parse 'value' as hex
bytes
at
org.apache.cassandra.db.marshal.BytesType.fromString(BytesType.java:45)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.createColumnFromColumnRow(LegacySchemaMigrator.java:682)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.createColumnsFromColumnRows(LegacySchemaMigrator.java:641)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.decodeTableMetadata(LegacySchemaMigrator.java:316)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readTableMetadata(LegacySchemaMigrator.java:273)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:244)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$7(LegacySchemaMigrator.java:237)
~[apache-cassandra-3.0.7.jar:3.0.7]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:237)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:186)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$4(LegacySchemaMigrator.java:177)
~[apache-cassandra-3.0.7.jar:3.0.7]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:177)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:77)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
[apache-cassandra-3.0.7.jar:3.0.7]
Caused by: java.lang.NumberFormatException: An hex string representing bytes
must have an even length
at org.apache.cassandra.utils.Hex.hexToBytes(Hex.java:57)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.db.marshal.BytesType.fromString(BytesType.java:41)
~[apache-cassandra-3.0.7.jar:3.0.7]
... 16 common frames omitted
{code}
{code}
ERROR [main] 2016-06-17 10:49:21,326 CassandraDaemon.java:698 - Exception
encountered during startup
java.lang.RuntimeException: org.codehaus.jackson.JsonParseException: Unexpected
character ('K' (code 75)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: java.io.StringReader@60d4475f; line: 1, column: 2]
at
org.apache.cassandra.utils.FBUtilities.fromJsonMap(FBUtilities.java:561)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.decodeTableParams(LegacySchemaMigrator.java:442)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.decodeTableMetadata(LegacySchemaMigrator.java:365)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readTableMetadata(LegacySchemaMigrator.java:273)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:244)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$7(LegacySchemaMigrator.java:237)
~[apache-cassandra-3.0.7.jar:3.0.7]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:237)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:186)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$4(LegacySchemaMigrator.java:177)
~[apache-cassandra-3.0.7.jar:3.0.7]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_66]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:177)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:77)
~[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
[apache-cassandra-3.0.7.jar:3.0.7]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
[apache-cassandra-3.0.7.jar:3.0.7]
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('K'
(code 75)): expected a valid value (number, String, array, object, 'true',
'false' or 'null')
at [Source: java.io.StringReader@60d4475f; line: 1, column: 2]
at
org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432)
~[jackson-core-asl-1.9.2.jar:1.9.2]
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
~[jackson-core-asl-1.9.2.jar:1.9.2]
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
~[jackson-core-asl-1.9.2.jar:1.9.2]
at
org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:1192)
~[jackson-core-asl-1.9.2.jar:1.9.2]
at
org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:479)
~[jackson-core-asl-1.9.2.jar:1.9.2]
at
org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2761)
~[jackson-mapper-asl-1.9.2.jar:1.9.2]
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2709)
~[jackson-mapper-asl-1.9.2.jar:1.9.2]
at
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1854)
~[jackson-mapper-asl-1.9.2.jar:1.9.2]
at
org.apache.cassandra.utils.FBUtilities.fromJsonMap(FBUtilities.java:557)
~[apache-cassandra-3.0.7.jar:3.0.7]
... 15 common frames omitted
{code}
Reproduce with ccm:
{code}
ccm create -n 1 -v 2.0.17 upgrade
ccm start
sleep 30
~/.ccm/repository/2.0.17/tools/bin/cassandra-stress
ccm node1 flush
ccm stop
ccm node1 setdir --version 2.1.14
ccm start
sleep 30
ccm node1 flush
ccm stop
ccm node1 setdir --version 3.0.7
ccm start
sleep 60
cat ~/.ccm/upgrade/node1/logs/system.log
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
