Hi,
Could it be CASSANDRA-12090
<https://issues.apache.org/jira/browse/CASSANDRA-12090>?
Regards,
/Tommy
On 2016-11-17 09:47, adeline....@thomsonreuters.com wrote:
Hi,
We are using Cassandra 3.7 and we have a single datacenter of 4 nodes,
RF =3, and the consistency level is quorum.
I found *lots* *of* DigestMismatchException in debug.log, as below:
DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066
ReadCallback.java:235 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(-7187709033646443438,
0008000001586f95380000000405f72c7a0000040000000300)
(c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641)
at
org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226)
~[apache-cassandra-3.7.jar:3.7]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
INFO [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 -
Deleted hint file
bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints
INFO [HintsDispatcher:5] 2016-11-17 08:25:06,962
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints to endpoint
bcd84017-6068-4a76-be89-96c0130599f3
DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456
- Ignoring interval time of 2003165297 for /10.52.153.107
DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034
CompactionTask.java:223 - Compacted
(3b3d7600-ac9f-11e6-8b55-03fd515998e4) 4 sstables to
[/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/mb-2256-big,]
to level=0. 85.011MiB to 53.580MiB (~63% of original) in 75,524ms.
Throughput = 726.471KiB/s. 0 total partitions merged to 15,552.
Partition merge counts were {1:11623, 2:3543, 3:385, 4:1, }
ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17
08:25:42,126 CassandraDaemon.java:217 - Exception in thread
Thread[MessagingService-Incoming-/10.52.153.109,5,main]
java.io.IOError: java.io.IOException: Corrupt empty row found in
unfiltered partition
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:331)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:287)
~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.7.jar:3.7]
Caused by: java.io.IOException: Corrupt empty row found in unfiltered
partition
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:388)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)
~[apache-cassandra-3.7.jar:3.7]
... 11 common frames omitted
Is this normal?
--
Regards, Adeline
