Horia Mocioi created CASSANDRA-14673:
----------------------------------------
Summary: Removing user defined type column results in ReadFailure
due to CorruptSSTableExceptio or IllegalStateException
Key: CASSANDRA-14673
URL: https://issues.apache.org/jira/browse/CASSANDRA-14673
Project: Cassandra
Issue Type: Bug
Components: Local Write-Read Paths
Reporter: Horia Mocioi
Attachments: script
Steps to reproduce:
# create keyspace
# create user defined type
# create a table that would use the udt
# insert some data
# drop the udt column
# restart cassandra
# query the table
See the attached script for steps 1-5.
When querying the table in cqlsh I got the following error:
{code:java}
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
message="Operation failed - received 0 responses and 1 failures"
info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
'consistency': 'ONE'}{code}
On system.log I get errors. The errors are different, meaning that some times I
get:
{code:java}
WARN [ReadStage-1] 2018-08-29 10:22:30,229
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-1,10,main]: {}
java.lang.RuntimeException:
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/.ccm/3113/node1/data0/my_ks/my_table-90f73700ab6411e8a72e45b164590124/mc-1-big-Data.db
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2601)
~[apache-cassandra-3.11.3.jar:3.11.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/.ccm/3113/node1/data0/my_ks/my_table-90f73700ab6411e8a72e45b164590124/mc-1-big-Data.db
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:391)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:258)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1889)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2597)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 5 common frames omitted
Caused by: java.io.EOFException: null
at
org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.Columns$Serializer.deserializeSubset(Columns.java:513)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:599)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:209)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:182)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:378)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 25 common frames omitted
{code}
and with some other occasions I get:
{code:java}
WARN [ReadStage-4] 2018-08-28 16:47:22,734
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-4,5,main]: {}
java.lang.RuntimeException: java.lang.IllegalStateException: [mt] is not a
subset of []
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2591)
~[apache-cassandra-3.10.jar:3.10]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.lang.IllegalStateException: [mt] is not a subset of []
at
org.apache.cassandra.db.Columns$Serializer.encodeBitmap(Columns.java:543)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.Columns$Serializer.serializeSubset(Columns.java:477)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:225)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:200)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:132)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:120)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:315)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:145)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:138)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:134)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:333)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1884)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2587)
~[apache-cassandra-3.10.jar:3.10]
... 5 common frames omitted
{code}
I was not able to figure out a scenario when the first or second error would
appear.
I tested it on ccm cluster with 1 node, with RF=1 and 3 node cluster with RF=3.
I was not able to reproduce it if instead of a udt column the table contains a
normal text column.
The issue appeared initially
[here|https://stackoverflow.com/questions/52025963/cassandra-corruption-due-to-udt-column-drop].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
