[ https://issues.apache.org/jira/browse/CASSANDRA-14673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-14673: ------------------------------------- Summary: Removing user defined type column results in ReadFailure due to CorruptSSTableException or IllegalStateException (was: Removing user defined type column results in ReadFailure due to CorruptSSTableExceptio or IllegalStateException) > Removing user defined type column results in ReadFailure due to > CorruptSSTableException or IllegalStateException > ---------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-14673 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14673 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Reporter: Horia Mocioi > Priority: Minor > Attachments: script > > > Steps to reproduce: > # create keyspace > # create user defined type > # create a table that would use the udt > # insert some data > # drop the udt column > # restart cassandra > # query the table > See the attached script for steps 1-5. > When querying the table in cqlsh I got the following error: > {code:java} > ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] > message="Operation failed - received 0 responses and 1 failures" > info={'failures': 1, 'received_responses': 0, 'required_responses': 1, > 'consistency': 'ONE'}{code} > On system.log I get errors. The errors are different, meaning that some times > I get: > {code:java} > WARN [ReadStage-1] 2018-08-29 10:22:30,229 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-1,10,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /.ccm/3113/node1/data0/my_ks/my_table-90f73700ab6411e8a72e45b164590124/mc-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2601) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_131] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) > [apache-cassandra-3.11.3.jar:3.11.3] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) > [apache-cassandra-3.11.3.jar:3.11.3] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /.ccm/3113/node1/data0/my_ks/my_table-90f73700ab6411e8a72e45b164590124/mc-1-big-Data.db > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:391) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:258) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1889) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2597) > ~[apache-cassandra-3.11.3.jar:3.11.3] > ... 5 common frames omitted > Caused by: java.io.EOFException: null > at > org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:65) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.Columns$Serializer.deserializeSubset(Columns.java:513) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:599) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:209) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:153) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:182) > ~[apache-cassandra-3.11.3.jar:3.11.3] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:378) > ~[apache-cassandra-3.11.3.jar:3.11.3] > ... 25 common frames omitted > {code} > and with some other occasions I get: > {code:java} > WARN [ReadStage-4] 2018-08-28 16:47:22,734 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-4,5,main]: {} > java.lang.RuntimeException: java.lang.IllegalStateException: [mt] is not a > subset of [] > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2591) > ~[apache-cassandra-3.10.jar:3.10] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_131] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) > [apache-cassandra-3.10.jar:3.10] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) > [apache-cassandra-3.10.jar:3.10] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] > Caused by: java.lang.IllegalStateException: [mt] is not a subset of [] > at > org.apache.cassandra.db.Columns$Serializer.encodeBitmap(Columns.java:543) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.Columns$Serializer.serializeSubset(Columns.java:477) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:225) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:200) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:132) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:120) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:315) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:145) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:138) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:134) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:333) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1884) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2587) > ~[apache-cassandra-3.10.jar:3.10] > ... 5 common frames omitted > {code} > I was not able to figure out a scenario when the first or second error would > appear. > I tested it on ccm cluster with 1 node, with RF=1 and 3 node cluster with > RF=3. > I was not able to reproduce it if instead of a udt column the table contains > a normal text column. > The issue appeared initially > [here|https://stackoverflow.com/questions/52025963/cassandra-corruption-due-to-udt-column-drop]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org