[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126130#comment-17126130 ] Alex Petrov commented on CASSANDRA-15778: - Thank you all for review and comments. Committed to 3.0 with [c4064dd80e427aec7c04e8e2e1e4630d6c8087b6|https://github.com/apache/cassandra/commit/c4064dd80e427aec7c04e8e2e1e4630d6c8087b6], and merged up to [3.11|https://github.com/apache/cassandra/commit/056c9eff4b0311d9bde526e0bc9f39e5fad21e39]. Merge to trunk was done with {{-s ours}}. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126122#comment-17126122 ] Alex Petrov commented on CASSANDRA-15778: - [~dcapwell] thank you for the detailed explanation. I've taken your suggestion in. [~sumanth.pasupuleti] thank you for checking this on the cluster and your feedback on the patch! I've added the test you have suggested. As regards keyspace/cf key/value pairs, I did think about this. I think if you have written to even a single cf, it makes sense to force {{BytesType}} for all cfs, in particular - to avoid folks forgetting about previous writes. We can open up a ticket with a follow-up/improvement, but I do not feel this is a necessary condition for the fix. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123353#comment-17123353 ] Sumanth Pasupuleti commented on CASSANDRA-15778: Thanks for the patch [~ifesdjeen]. I was able to validate both the flows of the patch on the actual data that I was able to repro the issue with. In either of the flows, the resulting table ended up with the following schema, and doing a select * on the table includes the third column. {code:java} cqlsh:ks> desc "cf"; CREATE TABLE ks."cf" ( key text, value text, value1 blob, PRIMARY KEY (key, value) ) WITH COMPACT STORAGE AND CLUSTERING ORDER BY (value ASC) ... {code} A couple of comments on the patch: 1. It maybe helpful to add the following tests to {{testAlterTableAlterType()}} method. {code:java} createTable("CREATE TABLE %s (a int, value int, PRIMARY KEY (a,value)) WITH COMPACT STORAGE"); assertInvalidMessage("Altering of types is not allowed", "ALTER TABLE %s ALTER value TYPE 'org.apache.cassandra.db.marshal.IntegerType'"); execute("ALTER TABLE %s ALTER value1 TYPE 'org.apache.cassandra.db.marshal.BytesType'"); {code} 2. For the system property {{cassandra.init_dense_table_compact_value_as_bytes}}, I am not sure if folks may find it useful to control what keyspace/cf this needs to be applied to. I personally feel it may be rare enough to ignore this, but wanted to bring it up nevertheless. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121290#comment-17121290 ] David Capwell commented on CASSANDRA-15778: --- bq. Could you elaborate a bit? Stack trace doesn't change in this case, and exception itself hasn't changed from the old version. Think code will speak loader. {code} if (meta.isDense() && meta.compactValueColumn().type instanceof EmptyType) { if (validator != null && validator.getType() instanceof BytesType & meta.compactValueColumn().equals(def)) { cfm = meta.copyWithNewCompactValueType(validator.getType()); break; } else { throw new InvalidRequestException("Altering of compaction value column types is only allowed if changing to bytes, but given " + validator.getType()); } } {code} If I know that the column has always been an integer and I attempt to set as a int32 I get an exception saying that alter isn't allowed; this exception can imply that there is no way to fix the issue. To make it more actionable, it would be better to say that alter is only allowed to be bytes and that the provided type is not supported. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120155#comment-17120155 ] Alex Petrov commented on CASSANDRA-15778: - Thank you for the review! bq. One thing I would like to test is mixed mode, alter half way, will legacy still migrate to empty? I did test all of the scenarios locally, including the one you describe. I believe current tests cover all code paths. bq. exception can be hard for someone who hasn't read the code. It would be good to move the validation part into a different if statement after the dense check, so the exception can be more actionable. Could you elaborate a bit? Stack trace doesn't change in this case, and exception itself hasn't changed from the old version. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119847#comment-17119847 ] David Capwell commented on CASSANDRA-15778: --- Spoke on slack, mostly held off reviewing as I was waiting on testing. Feedback: * https://github.com/apache/cassandra/compare/trunk...ifesdjeen:15778-3.0?expand=1#diff-43b2d530031e2f7ad4286bd05fed4ca0R127. exception can be hard for someone who hasn't read the code. It would be good to move the validation part into a different if statement after the dense check, so the exception can be more actionable. * https://github.com/apache/cassandra/compare/trunk...ifesdjeen:15778-3.0?expand=1#diff-0a4167e798cf91857b5d6995661cd421R398 can we add a assert message to show how much is remaining? Same for the other method as well Mostly good with this since alter unblocks a cluster. One thing I would like to test is mixed mode, alter half way, will legacy still migrate to empty? > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119759#comment-17119759 ] Alex Petrov commented on CASSANDRA-15778: - Thank you for the review. Made suggested changes and triggered CI again: |[circleci|https://circleci.com/gh/ifesdjeen/cassandra/tree/15778-3.0]|[3.0 patch|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:15778-3.0?expand=1]| |[circleci|https://circleci.com/gh/ifesdjeen/cassandra/tree/15778-3.11]|[3.11 patch|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:15778-3.11?expand=1]| > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117827#comment-17117827 ] Sylvain Lebresne commented on CASSANDRA-15778: -- That patch looks like a reasonable solution to me, at least from my understanding of the issue. Small comments on the code itself: * I'd put a comment in {{AlterTableStatement}} to point out to this ticket (may feel like a peculiar special case to future readers without context). * In {{AbstractType}}, the changes to {{writeValue}}/{{writtenLength}} feels confusing to me, and if the new code is ever triggered, this would mean we silently drop a value on the floor (we get a non-empty value, but the type say the value should be empty, so we'd write nothing), and that doesn't feel lik a good idea. Instead of specializing the 0 size case, I'd just add a {{assert valueLengthIfFixed < 0 || value.remaining() == valueLengthIfFixed}} to basically ensure we're not going to write something we don't know how to read (and effectively forbid the call of those method for {{EmptyType}} in conjunction with the existing assert). * Assuming we agree on the previous point, I'd prefer not overriding the methods in {{EmptyType}}. For the write ones, it wouldn't add anything, and overriding {{readValue}} feels confusing when the rest of the code ensures we can never write an empty value through those methods. * Nit: LegacySchemaMigrator has unused leftover imports ({{java.io.InvalidClassException}} and {{net.bytebuddy.implementation.bytecode.Throw}}). > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108618#comment-17108618 ] David Capwell commented on CASSANDRA-15778: --- Alex and I spoke offline, dumping here for context. There are a few cases we want to worry about, and some cases which we don't feel are important to worry about: Cases we feel are important to worry about 1) upgrade from 2.1 to 3.0 *or* 3.11 2) CQL Table with values written to the previous blob type (hidden type only exposed via Thrift) 3) Thrift tables with arbitrary types Cases we don't feel are important to worry about 1) upgrade from 3.0 to 3.0 or 3.11 Given this, we came up with a proposal that should address the above concerns. 1) while upgrading from 2.1 to 3.0 or 3.11, legacy schema migration should default to BytesType, but allow an override to switch to EmptyType; BytesType will cause the column to be no longer hidden [1][2]. 2) if override is set to EmptyType and data is found to be rejected in validation, then DROP COMPACT STORAGE will be needed and should add an option to expose this column. This means the hidden column will now be exposed and will allow alter statements to change the type or drop it completely. 3) Attempt to add a more meaningful exception which tells users about this fix. Right now the exception is "EmptyType only accept empty values", this is not actionable and can cause confusion. 4) For clusters already upgraded to 3.0 or 3.11, the only impact would be the options added to DROP COMPACT STORAGE which would allow the column to be exposed or not [3]; we have not worked out the default behaviors of this. [1] - This is actually an open disagreement. Alex's point is that this affects a small user base, but would expose the field to all users upgrading. My point is summarized in [2]. [2] - The reason to default to BytesTypes is because the rolling upgrade case. Since there may be data present, the default of EmptyType is unsafe as it would cause the token range impacted to be unavailable to the users. Schema migrations are not allowed in mixed mode so the only way to resolve this would be to complete the upgrade, which may cause a larger outage for users. If the user is aware of this behavior and know its safe, they may choose to switch to the EmptyType case. [3] - There is an assumption that all clusters are 3.0.20+, the reason for this assumption is that validation checks were in 3.0.20 (see CASSANDRA-15373) and would protect against corrupting the SSTable in the upgrade or compaction case. If the cluster is 3.0.19 or earlier, then corruption would have happened as the EmptyType was writing the bytes but was not reading them (see CASSANDRA-15790). As of this moment we are not aware of a generic way to repair this data as we don't know what those bytes were so couldn't account for them generically (they are written without length). > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108389#comment-17108389 ] Alex Petrov commented on CASSANDRA-15778: - After thinking about it for longer time and talking with [~slebresne] and [~aleksey], I think there're no good solution to this problem. At the moment {{EmptyType}} assumes that it's fixed-size, therefore doesn't write its size (only writes bytes). This means that anyone who has already written the value, won't be able to read it back, and writing a generic scrub tool for that will be non-trivial since we can have arbitrary bytes there. >From the alternatives, I think everything we have right now is risky to some >degree: * if we inherit {{EmptyType}} from {{BytesType}}, we change semantics of the empty type, which can have unforeseen consequences * if we change schema and migrate from {{EmptyType}} to {{BytesType}}, we'll end up showing the hidden column. To keep it hidden, we'll have to add a flag to table metadata * if we just migrate {{EmptyType}} from fixed-size to variable-size (basically by not overriding {{valueLengthIfFixed}}), it'll work in principle because of {{BufferCell#hasValue}}, but this also might be risky So the only reasonable thing to do seems to handle this particular case: people who still have their 2.1 sstables, and they haven't upgraded, and they're (somewhat) interested in the data they've written. People who're not interested in the data they're written receive the same solution, but can just drop the column. I suggest we do the following: * we keep the existing validation that forbids writes into {{EmptyType}} columns and make {{EmptyType}} write method no-op to be on an even safer side * we add a way to "unbreak" cql-created thrift-written tables by adding new {{ALTER TABLE FIX EMPTY TYPE}} (naming is just for example here) statement, which would _only_ work for cql-created dense tables and nothing else, and would turn hidden {{EmptyType}} column to {{BytesType}}. Main reason to ask for human intervention is that we'd like to avoid changes that can add new problems whie not bein fully satisfactory for the given problem. This will give anyone with a problem a clear path to a solution without risking already existing setups. This also implies that in [CASSANDRA-15811], we'll simply drop {{EmptyType}} column entirely when doing {{DROP COMPACT STORAGE}}, so no surprising column without data will show up. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107899#comment-17107899 ] David Capwell commented on CASSANDRA-15778: --- We spoke offline, dumping here. Splunking through GitHub I found projects which use the empty type and you can create a schema with the empty type. Even if we update all usage within Cassandra, we struggle to really understand the impact outside of Cassandra since it is modifying a public API; changing EmptyType to me is very high risk. I was also showing that CQL created tables in 2.1.20 used the blob type and allowed me to write bytes to the hidden column (the write was outside CQL), so this isn't localized to CQL vs Thrift tables, but more that 3.0 changed behavior to go from BytesType to EmptyType. {code} -- this table was created via CQL cqlsh> desc table test.kv; CREATE TABLE test.kv ( key text, value text, "" blob, PRIMARY KEY (key, value) ) WITH COMPACT STORAGE AND CLUSTERING ORDER BY (value ASC) AND bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'} AND compression = {} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; cqlsh> select * from system.schema_columns where keyspace_name='test' and columnfamily_name='kv'; keyspace_name | columnfamily_name | column_name | component_index | index_name | index_options | index_type | type | validator ---+---+-+-++---+++--- test |kv | |null | null | null | null | compact_value | org.apache.cassandra.db.marshal.BytesType test |kv | key |null | null | null | null | partition_key | org.apache.cassandra.db.marshal.UTF8Type test |kv | value |null | null | null | null | clustering_key | org.apache.cassandra.db.marshal.UTF8Type (3 rows) {code} After upgrading to 3.0, the schema was changed to empty type {code} cqlsh> cqlsh> select * from system_schema.columns where keyspace_name='test' and table_name='kv'; keyspace_name | table_name | column_name | clustering_order | column_name_bytes | kind | position | type ---++-+--+---+---+--+--- test | kv | key | none | 0x6b6579 | partition_key |0 | text test | kv | value | asc | 0x76616c7565 |clustering |0 | text test | kv | value1 | none | 0x76616c756531 | regular | -1 | empty (3 rows) {code} > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107654#comment-17107654 ] Alex Petrov commented on CASSANDRA-15778: - bq. Why not change the schema? First off, I'd like to highlight that I was (and still am) reluctant to the change to {{EmptyType}}. I did take a look at that option (changing schema), but was weary, as this would be potentially a much larger change that would probably achieve the same thing. The only problem I see is that "empty" type is not really empty anymore. However, it is _already_ not empty from compaction perspective, and from data perspective. The reason I thought it was valuable to keep a distinction between the thift-created table with a Blob (where this column is visible) and CQL-created dense table (where value is hidden). I did check all usages of {{EmptyType}} and, apart from {{Set}}, all usages seem to be compact-storage related. It is not possible to create a table with CQL that has an empty type, so we're only really changing semantics of something that I think is internal to compact storage... > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107598#comment-17107598 ] David Capwell commented on CASSANDRA-15778: --- I am -1 to this. This attempts to change the EmptyType to be the BytesType. Why not change the schema? As we talked about COMPACT STOAGE was BLOB in 2.1 but is now VOID, which means this is the wrong type to use; we should change the schema to match the 2.1 behavior, not change the semantics of empty. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105793#comment-17105793 ] David Capwell commented on CASSANDRA-15778: --- Thanks [~ifesdjeen] for taking a look at this. Most of the context is not in JIRA and the patch I posted was so Sumanth and I could test out if the value field being empty was the issue and if there were other issues; *no one should apply that patch to production*. The current state is that the value field silently corrupted the data because * CASSANDRA-15373 is 3.0.20 and not 3.0.19 (test cluster) * CASSANDRA-15790 is needed to not write data and instead fail. We currently write out data that has no logic to read, so corrupts the stream. Action items so far are to come up with a resolution to this case. [~bdeggleston] has the idea that we should migrate the hidden column out of VOID and to BLOB (2.1 logic), this will make the data readable via Thrift but not CQL. [~ifesdjeen] about your reproduction, skimming the method (and CassandraServer version) you sent I don't see us injecting anything, are you saying CassandraDaemon, ThriftClient, or client code added this data? If ThriftClient or CassandraDaemon do you know where? > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: David Capwell >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105653#comment-17105653 ] Alex Petrov commented on CASSANDRA-15778: - I've taken a short look and wanted to precaution against ignoring or even purging values from the hidden compact column, since there are scenarios when it can contain legitimate data. I've been able to reproduce an issue with a similar, albeit not exactly same stacktrace (leaving only the part from sstable iterator downward): {code} ERROR 18:08:11 Uncaught exception on thread Thread[SharedPool-Worker-2,10,main] java.lang.RuntimeException: org.apache.cassandra.serializers.MarshalException: EmptyType only accept empty values at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470) ~[main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) ~[main/:na] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) [main/:na] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [main/:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171] Caused by: org.apache.cassandra.serializers.MarshalException: EmptyType only accept empty values at org.apache.cassandra.serializers.EmptySerializer.validate(EmptySerializer.java:42) ~[main/:na] at org.apache.cassandra.db.marshal.AbstractType.validate(AbstractType.java:159) ~[main/:na] at org.apache.cassandra.db.marshal.AbstractType.validateIfFixedSize(AbstractType.java:390) ~[main/:na] at org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1457) ~[main/:na] at org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1377) ~[main/:na] at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:542) ~[main/:na] at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:519) ~[main/:na] at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:336) ~[main/:na] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:140) ~[main/:na] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:172) ~[main/:na] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:336) ~ {code} In short, this is a table created on 2.2 side with: {{CREATE TABLE table_0 (key text, value text, PRIMARY KEY (key, value)) WITH COMPACT STORAGE;}} If you write data into this table with thrift ({{ThriftClient#batch_mutate}}), you will end up with data in the hidden column, which looks like this on the 2.1 side: {code} select * from ks1.table_0; key | value ---+--- key22 | key1 key22 | key4 key11 | key1 key11 | key4 (4 rows) {code} And like this in sstabledump: {code} [ {"key": "key22", "cells": [["key1","76616c756531",1589306163593], ["key4","76616c756534",1589306163594]]}, {"key": "key11", "cells": [["key1","76616c756531",1589306163593], ["key4","76616c756534",1589306163594]]} ] {code} Of course, on 3.x side you wouldn't be able to see the values. One theory is that stack trace that Sumanth has posted above could have been a result of a read-repair, which has propagated the value to 3.0 sstable, since normally a 3.0 node wouldn't even start with such table. I'll continue tomorrow, will create a mixed mode cluster, and will see if a read-repair propagates this write and corrupts 3.0 sstable. Not saying that this all couldn't have been a result of some weird corruption resulting from anything else, but primary point of writing this is not to get to the bottom of the issue but caution against changing behaviour of {{EmptyType}} at least until we know what is going on. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Assignee: David Capwell >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100198#comment-17100198 ] David Capwell commented on CASSANDRA-15778: --- Created a branch to ignore value field for compact storage: https://github.com/dcapwell/cassandra/tree/corruption/CASSANDRA-15778. Running through testing now. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_231] at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097566#comment-17097566 ] Sumanth Pasupuleti commented on CASSANDRA-15778: Thanks correct [~dcapwell]. We have restored 2.1 sstables on this 3.0 cluster, and kicked off sstableupgrade, to eagerly rewrite all 2.1 sstables to 3.0 format. I was able to trace back the 2.1 sstable that got compacted into this 3.0 sstable. The data was originally written using [astyanax|https://github.com/Netflix/astyanax] driver. > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data// at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097500#comment-17097500 ] David Capwell commented on CASSANDRA-15778: --- Looking closer the file is md-10151-big-Data.db, so this was after the 2.1 file got upgraded to 3.0 > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data/payments/ExternalCidLookUp-fda511301fb311e7bd79fd24f2fcfb0d/md-10151-big-Data.db > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_231] at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) >
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[ https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097498#comment-17097498 ] David Capwell commented on CASSANDRA-15778: --- The stack trace points to https://github.com/apache/cassandra/blob/cassandra-3.0.19/src/java/org/apache/cassandra/db/ClusteringPrefix.java#L425. If you have the 2.1 sstable attempt to read it with the tool sstable2json, that will tell you if the file was valid in 2.1. How was the data written, was it always though CQL or some other system (like MapReduce)? > CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads > - > > Key: CASSANDRA-15778 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15778 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction, Local/SSTable >Reporter: Sumanth Pasupuleti >Priority: Normal > Fix For: 3.0.x > > > Below is the exception with stack trace. This issue is consistently > reproduce-able. > {code:java} > ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 > 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception > on thread > Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /mnt/data/cassandra/data/payments/ExternalCidLookUp-fda511301fb311e7bd79fd24f2fcfb0d/md-10151-big-Data.db > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) >