Cannot restart server after making schema changes to composite CFs
------------------------------------------------------------------
Key: CASSANDRA-3819
URL: https://issues.apache.org/jira/browse/CASSANDRA-3819
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.7, 1.0.6
Environment: Ubuntu 11.0.4
Reporter: Huy Le
This JIRA is for issue discussed in this thread
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cannot-start-cassandra-node-anymore-tp7150978p7150978.html.
We were using version 1.0.6. We added new keyspace using built-in composite
data type. We then decided to change the schema, specifically just the CF
names, so we dropped the keyspace. We recreated the key space with different
CF names in the key space.
There were a lot of uncommitted data in commit logs. Data dated back before
the original key space was created. When we restarted the server, the server
failed when it read it the commit logs, and the server stopped. Here is
snippet of the stack trace:
{code}
-3881-11e1-ac7f-12313d23ead3:true:4@1326223353559001,])}
DEBUG 18:02:01,057 Reading mutation at 66336992
DEBUG 18:02:01,058 replaying mutation for
Springpad.696d6167652d7363616c65722d6d657461:
{ColumnFamily(CassandraOrderedQueue
[0,eb321490-3881-11e1-ac7f-12313d23ead3:true:4@1326223353560000,])}
DEBUG 18:02:01,058 Reading mutation at 66337118
DEBUG 18:02:01,058 replaying mutation for
Springpad.737072696e674d6f64656c44617461626173652d6d657461:
{ColumnFamily(CassandraOrderedQueue
[0,80dc0cd0-3bc0-11e1-83a8-12313d23ead3:false:8@1326223386668000,])}
DEBUG 18:02:01,058 Reading mutation at 66337255
DEBUG 18:02:01,058 replaying mutation for
system.38363233616337302d336263302d313165312d303030302d323366623834646463346633:
{ColumnFamily(Schema
[Avro/Schema:false:2725@1326223386807,Backups:false:431@1326223386807,Springpad:false:10814@1326223386807,SpringpadGraph:false:2931@1326223386807,])}
DEBUG 18:02:01,059 Reading mutation at 66354352
DEBUG 18:02:01,059 replaying mutation for system.4d6967726174696f6e73204b6579:
{ColumnFamily(Migrations
[8623ac70-3bc0-11e1-0000-23fb84ddc4f3:false:23728@1326223386812,])}
DEBUG 18:02:01,059 Reading mutation at 66378184
DEBUG 18:02:01,059 replaying mutation for system.4c617374204d6967726174696f6e:
{ColumnFamily(Schema [Last Migration:false:16@1326223386812,])}
DEBUG 18:02:01,059 Reading mutation at 66378302
INFO 18:02:01,060 Finished reading
/mnt/cassandra/commitlog/CommitLog-1325861435420.log
ERROR 18:02:01,061 Exception encountered during startup
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:247)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:57)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:66)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.getString(AbstractCompositeType.java:129)
at org.apache.cassandra.db.Column.getString(Column.java:250)
at
org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:137)
at org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:280)
at org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:301)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:215)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)
Exception encountered during startup: null
{code}
Sample original CF schema:
{code}
create column family InEdges
with column_type = 'Standard'
and comparator =
'CompositeType(org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type)'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and rows_cached = 0.0
and row_cache_save_period = 0
and row_cache_keys_to_save = 0
and keys_cached = 0.0
and key_cache_save_period = 0
and read_repair_chance = 0.0
and gc_grace = 0
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and row_cache_provider = 'ConcurrentLinkedHashCacheProvider'
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';
{code}
And the updated schema (just CF name change):
{code}
create column family NewInEdges
with column_type = 'Standard'
and comparator =
'CompositeType(org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type)'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and rows_cached = 0.0
and row_cache_save_period = 0
and row_cache_keys_to_save = 0
and keys_cached = 0.0
and key_cache_save_period = 0
and read_repair_chance = 0.0
and gc_grace = 0
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and row_cache_provider = 'ConcurrentLinkedHashCacheProvider'
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';
{code}
This issue happens every time composite type is used and workaround can be
found at
(http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cannot-start-cassandra-node-anymore-tp7150978p7226863.html).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
