[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140984#comment-15140984
]
Sylvain Lebresne commented on CASSANDRA-11050:
--
I took a few minutes to upgrade the patch (below) for this with what's above
and to write a quick dtest (PR
[here|https://github.com/riptano/cassandra-dtest/pull/800]), I hope you don't
mind [~amorton].
|| patch || utests || dtests ||
| [11050-3.0|https://github.com/pcmanus/cassandra/commits/11050-3.0] |
[utests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-3.0-testall/lastBuild/]
|
[dtests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-3.0-dtest/lastBuild/]
|
| [11050-trunk|https://github.com/pcmanus/cassandra/commits/11050-trunk] |
[utests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-trunk-testall/lastBuild/]
|
[dtests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-trunk-dtest/lastBuild/]
|
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
> [main/:na]
> {code}
> {{dropped_columns}} is not in the list of tables to flush,
>
[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141110#comment-15141110
]
amorton commented on CASSANDRA-11050:
-
Sorry I was on the road with a client last week, had it on the plan for this
week.
No problems, atleast now I can see what the dtest should look like :)
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
> [main/:na]
> {code}
> {{dropped_columns}} is not in the list of tables to flush,
> {{SchemaKeyspace.ALL}}.
> It's a simple patch to add it, attached. The fix will need to go to 3.0, 3.1
> and trunk AFAIK
> however this will change the way the schema hash is calculated in
> {{SchemaKeyspace.calculateSchemaDigest()}} It looks like this would cause the
> nodes to announce a new version of the schema on (first) restart.
> I currently donit understand all the implications of changing the schema
> hash, thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119646#comment-15119646
]
Sylvain Lebresne commented on CASSANDRA-11050:
--
bq. It will change simply b/c of a different set of columns in
{{system_schema.columns}}
You're right, that's definitively the best solution then.
bq. is there some docs or example I should check out ?
I suspect poking around in the tests at
https://github.com/pcmanus/cassandra-dtest should be relatively intuitive (we
don't have formal doc on that though I'm afraid). Look maybe at
https://github.com/pcmanus/cassandra-dtest/blob/master/deletion_test.py if you
want a trivial example (though the test for this is probably even simpler).
bq. what should I do to make the full change in 4.0
We actually don't have a branch for 4.0 yet, and tbh we've avoided creating one
cause merging is already painful enough without it. I suppose for now the
simplest solution would be to create a followup ticket tagged 4.0 saying "Let's
not forget to remove the special casing added by CASSANDRA-11050".
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [main/:na]
> at
>
[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116610#comment-15116610
]
amorton commented on CASSANDRA-11050:
-
bq. As a side note, we should really write a dtest for this issue before
committing (amorton would you be up for that?).
[~slebresne] sure thing, is there some docs or example I should check out ?
bq. So, your option 2 is safe, IMO (special case
SchemaKeyspace::calculateSchemaDigest to ignore dropped_columns).
Thanks [~iamaleksey] will create a new patch with a list for schema hash and a
list for everything else.
[~slebresne] what should I do to make the full change in 4.0, is the trunk at
4.0 ? (sorry am out of the loop a bit :) )
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
> [main/:na]
> {code}
> {{dropped_columns}} is not in the list of tables to flush,
> {{SchemaKeyspace.ALL}}.
> It's a simple patch to add it, attached. The fix will need to go to 3.0, 3.1
> and trunk AFAIK
> however this will change the way the schema hash is calculated in
> {{SchemaKeyspace.calculateSchemaDigest()}} It looks like this would cause
[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112935#comment-15112935
]
Aleksey Yeschenko commented on CASSANDRA-11050:
---
[~slebresne] The issue you mention with the second approach is actually not
that terrible. If one drops a column, the schema digest will change, even if
{{dropped_columns}} table itself is not involved in the computation. It will
change simply b/c of a different set of columns in {{system_schema.columns}} -
one fewer if we drop one.
Now, there could be an issue if you drop then recreate a column. Then again,
the timestamps in schema tables will be different in that case, so the change
will still be detected. Unless there is an extremely unlikely clock jump to
that precise previous timestamp?
So, your option 2 is safe, IMO (special case
{{SchemaKeyspace::calculateSchemaDigest}} to ignore {{dropped_columns}}).
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
> [main/:na]
> {code}
> {{dropped_columns}} is not in the list of tables to flush,
> {{SchemaKeyspace.ALL}}.
> It's a simple patch to add it, attached.
[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112735#comment-15112735
]
Sylvain Lebresne commented on CASSANDRA-11050:
--
bq. do you have any concerns about the schema hash changing ?
Actually I have (but I was too fast and committed before realizing this was
indeed a problem which is why it's been committed and then reverted).
Unless I'm missing something in our code, the change to the hash means that
during an upgrade to a version with this patch, old and new nodes will never
agree on the schema version (only if they have dropped columns but still) and
will thus continuously exchange schema. That's not ideal. But we still need to
fix that bug.
Right off the bat I can see the following options:
# we just accept that there will be disagreement during upgrade. Having node
exchange schema for no reason is not great but I believe we'll do that at most
once every minute which hopefully shouldn't be hugely noticeable (of course,
assuming you have a reasonable amount of keyspaces/tables). Though it's worth
noting this will disturb clients if they wait for schema agreement. And 3.x is
very young so we could count that as one of the bump in the road. It's scary
though.
# we special case the schema has code to ignore that table (but include it for
anything else) and this until 4.0 (where, since it's a major version, there
won't be schema hash comparison involved). The downside being that we could get
into a situation where a node doesn't know about some dropped column and it
doesn't get fixed. On the one side it's not ideal (to leave a known bug in),
but on the other side you'd have to be unlucky to have that be a real problem,
and if you do get there, there is an easy workaround as you can reset your
schema version and thus force nodes to re-synchronize on the schema.
Anway, no option is perfect but the 2nd one feels a tad less scary. Any other
opinion ([~iamaleksey] in particular since you've been deep in schema code
lately).
As a side note, we should really write a dtest for this issue before committing
([~amorton] would you be up for that?).
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
>
[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110604#comment-15110604
]
Sylvain Lebresne commented on CASSANDRA-11050:
--
Lgtm but I've pushed the patch to CI for good measure. I'll commit once I have
the results.
|| patch || utests || dtests ||
| [11050-3.0|https://github.com/pcmanus/cassandra/commits/11050-3.0] |
[utests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-3.0-testall/lastBuild/]
|
[dtests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-3.0-dtest/lastBuild/]
|
| [11050-3.3|https://github.com/pcmanus/cassandra/commits/11050-3.3] |
[utests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-3.3-testall/lastBuild/]
|
[dtests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-3.3-dtest/lastBuild/]
|
| [11050-trunk|https://github.com/pcmanus/cassandra/commits/11050-trunk] |
[utests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-trunk-testall/lastBuild/]
|
[dtests|http://cassci.datastax.com/view/Dev/view/pcmanus/job/pcmanus-11050-trunk-dtest/lastBuild/]
|
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [main/:na]
> at
>
[jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
[
https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111828#comment-15111828
]
amorton commented on CASSANDRA-11050:
-
Thanks [~slebresne] - do you have any concerns about the schema hash changing ?
> SSTables not loaded after dropping column
> -
>
> Key: CASSANDRA-11050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: amorton
>Assignee: amorton
> Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is
> updated, this can result in SSTables not being loaded at startup and failure
> to start if the commit log contains mutations with the column.
> Reproduce on cassandra-3.0 branch by starting a node and running following in
> cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':1};
> use dev;
> create table foo (
> foo text primary key,
> bar text,
> baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does
> not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> at
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wcgn/T/mutation5408885979635225676dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column baz during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
> [main/:na]
> {code}
> {{dropped_columns}} is not in the list of tables to flush,
> {{SchemaKeyspace.ALL}}.
> It's a simple patch to add it, attached. The fix will need to go to 3.0, 3.1
> and trunk AFAIK
> however this will change the way the schema hash is calculated in
> {{SchemaKeyspace.calculateSchemaDigest()}} It looks like this would cause the
> nodes to announce a new version of the schema on (first) restart.
> I currently donit understand all the implications of changing the schema
> hash, thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
