Re: lots of DigestMismatchException in cassandra3

[kurt Greaves]

dclocal_read_repair_chance and read_repair_chance are only really relevant
when using a consistency level  wrote:

> Hi Kurt,
>
> Thank you for the suggestion. I ran repair on all the 4 nodes, and after
> the repair, the error “Corrupt empty row found in unfiltered partition”
> disappeared, but the “Mismatch” stopped for a little while and came up
> again.
>
> When we changed both the “dclocal_read_repair_chance” and the
> “read_repair_chance” to 0.0, the “Mismatch” stopped. Is it OK to do that?
> Does it mean when the inconsistence found in reading data, Cassandra
> wouldn’t do the repair and we will just get the inconsistent data? And you
> said the cause is not all replicas receiving all the writes, I think it is
> reasonable but the strange thing is I didn’t notice any failed writing ,
> another cause I can think of is there are insert, update, delete on the
> same record at the same time , is it a possibility?
>
>
>
> --
>
> Regards, Adeline
>
>
>
>
>
>
>
> *From:* kurt Greaves [mailto:k...@instaclustr.com]
> *Sent:* Wednesday, November 23, 2016 6:51 AM
> *To:* Pan, Adeline (TR Technology & Ops)
> *Cc:* user@cassandra.apache.org
> *Subject:* Re: lots of DigestMismatchException in cassandra3
>
>
>
> Yes it could potentially impact performance if there are lots of them. The
> mismatch would occur on a read, the error occurs on a write which is why
> the times wouldn't line up. The cause for the messages as I mentioned is
> when there is a digest mismatch between replicas. The cause is inconsistent
> deta/not all replicas receiving all writes. You should run a repair and see
> if the number of mismatches is reduced.
>
>
> Kurt Greaves
>
> k...@instaclustr.com
>
> www.instaclustr.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instaclustr.com=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=8lqiPNb8HiRlBNyddnGZahh0KiP-7P0MfAnjUHI0c84=E6-7Hti1G8DXfJZttqNy6gwGb56o65eS5Zhjm4deFFk=>
>
>
>
> On 22 November 2016 at 06:30, <adeline@thomsonreuters.com> wrote:
>
> Hi Kurt,
>
> Thank you for the information, but the error “Corrupt empty row found in
> unfiltered partition” seems not related to the “Mismatch”; the time they
> occurred didn’t match. We use “QUORUM” consistency level for both read and
> write and I didn’t notice any failed writing in the log. Any other cause
> you can think of?  Would it cause performance issue when lots of this
> “Mismatch” happened?
>
>
>
> --
>
> Regards, Adeline
>
>
>
>
>
>
>
> *From:* kurt Greaves [mailto:k...@instaclustr.com]
> *Sent:* Monday, November 21, 2016 5:02 PM
> *To:* user@cassandra.apache.org
> *Cc:* tommy.stend...@ericsson.com
> *Subject:* Re: lots of DigestMismatchException in cassandra3
>
>
>
> Actually, just saw the error message in those logs and what you're looking
> at is probably https://issues.apache.org/jira/browse/CASSANDRA-12694
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D12694=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=tMf24yohd0jRGCBo_pzYdRMw52h3NCImPOGXjy1SAsc=>
>
>
> Kurt Greaves
>
> k...@instaclustr.com
>
> www.instaclustr.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instaclustr.com=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=TwJ80glB0cSS0rW6jU1MGnlLWUtVYL1J7061vp2e_rI=>
>
>
>
> On 21 November 2016 at 08:59, kurt Greaves <k...@instaclustr.com> wrote:
>
> That's a debug message. From the sound of it, it's triggered on read where
> there is a digest mismatch between replicas. As to whether it's normal,
> well that depends on your cluster. Are the nodes reporting lots of dropped
> mutations and are you writing at <QUORUM?
>
>
>
>
>

RE: lots of DigestMismatchException in cassandra3

[Adeline.Pan]

Hi Kurt,
Thank you for the suggestion. I ran repair on all the 4 nodes, and after the 
repair, the error “Corrupt empty row found in unfiltered partition” 
disappeared, but the “Mismatch” stopped for a little while and came up again.
When we changed both the “dclocal_read_repair_chance” and the 
“read_repair_chance” to 0.0, the “Mismatch” stopped. Is it OK to do that? Does 
it mean when the inconsistence found in reading data, Cassandra wouldn’t do the 
repair and we will just get the inconsistent data? And you said the cause is 
not all replicas receiving all the writes, I think it is reasonable but the 
strange thing is I didn’t notice any failed writing , another cause I can think 
of is there are insert, update, delete on the same record at the same time , is 
it a possibility?

--
Regards, Adeline



From: kurt Greaves [mailto:k...@instaclustr.com]
Sent: Wednesday, November 23, 2016 6:51 AM
To: Pan, Adeline (TR Technology & Ops)
Cc: user@cassandra.apache.org
Subject: Re: lots of DigestMismatchException in cassandra3

Yes it could potentially impact performance if there are lots of them. The 
mismatch would occur on a read, the error occurs on a write which is why the 
times wouldn't line up. The cause for the messages as I mentioned is when there 
is a digest mismatch between replicas. The cause is inconsistent deta/not all 
replicas receiving all writes. You should run a repair and see if the number of 
mismatches is reduced.

Kurt Greaves
k...@instaclustr.com<mailto:k...@instaclustr.com>
www.instaclustr.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instaclustr.com=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=8lqiPNb8HiRlBNyddnGZahh0KiP-7P0MfAnjUHI0c84=E6-7Hti1G8DXfJZttqNy6gwGb56o65eS5Zhjm4deFFk=>

On 22 November 2016 at 06:30, 
<adeline@thomsonreuters.com<mailto:adeline@thomsonreuters.com>> wrote:
Hi Kurt,
Thank you for the information, but the error “Corrupt empty row found in 
unfiltered partition” seems not related to the “Mismatch”; the time they 
occurred didn’t match. We use “QUORUM” consistency level for both read and 
write and I didn’t notice any failed writing in the log. Any other cause you 
can think of?  Would it cause performance issue when lots of this “Mismatch” 
happened?

--
Regards, Adeline



From: kurt Greaves [mailto:k...@instaclustr.com<mailto:k...@instaclustr.com>]
Sent: Monday, November 21, 2016 5:02 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Cc: tommy.stend...@ericsson.com<mailto:tommy.stend...@ericsson.com>
Subject: Re: lots of DigestMismatchException in cassandra3

Actually, just saw the error message in those logs and what you're looking at 
is probably 
https://issues.apache.org/jira/browse/CASSANDRA-12694<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D12694=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=tMf24yohd0jRGCBo_pzYdRMw52h3NCImPOGXjy1SAsc=>

Kurt Greaves
k...@instaclustr.com<mailto:k...@instaclustr.com>
www.instaclustr.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instaclustr.com=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=TwJ80glB0cSS0rW6jU1MGnlLWUtVYL1J7061vp2e_rI=>

On 21 November 2016 at 08:59, kurt Greaves 
<k...@instaclustr.com<mailto:k...@instaclustr.com>> wrote:

That's a debug message. From the sound of it, it's triggered on read where 
there is a digest mismatch between replicas. As to whether it's normal, well 
that depends on your cluster. Are the nodes reporting lots of dropped mutations 
and are you writing at <QUORUM?

Re: lots of DigestMismatchException in cassandra3

[kurt Greaves]

Yes it could potentially impact performance if there are lots of them. The
mismatch would occur on a read, the error occurs on a write which is why
the times wouldn't line up. The cause for the messages as I mentioned is
when there is a digest mismatch between replicas. The cause is inconsistent
deta/not all replicas receiving all writes. You should run a repair and see
if the number of mismatches is reduced.

Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

On 22 November 2016 at 06:30, <adeline@thomsonreuters.com> wrote:

> Hi Kurt,
>
> Thank you for the information, but the error “Corrupt empty row found in
> unfiltered partition” seems not related to the “Mismatch”; the time they
> occurred didn’t match. We use “QUORUM” consistency level for both read and
> write and I didn’t notice any failed writing in the log. Any other cause
> you can think of?  Would it cause performance issue when lots of this
> “Mismatch” happened?
>
>
>
> --
>
> Regards, Adeline
>
>
>
>
>
>
>
> *From:* kurt Greaves [mailto:k...@instaclustr.com]
> *Sent:* Monday, November 21, 2016 5:02 PM
> *To:* user@cassandra.apache.org
> *Cc:* tommy.stend...@ericsson.com
> *Subject:* Re: lots of DigestMismatchException in cassandra3
>
>
>
> Actually, just saw the error message in those logs and what you're looking
> at is probably https://issues.apache.org/jira/browse/CASSANDRA-12694
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D12694=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=tMf24yohd0jRGCBo_pzYdRMw52h3NCImPOGXjy1SAsc=>
>
>
> Kurt Greaves
>
> k...@instaclustr.com
>
> www.instaclustr.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instaclustr.com=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=TwJ80glB0cSS0rW6jU1MGnlLWUtVYL1J7061vp2e_rI=>
>
>
>
> On 21 November 2016 at 08:59, kurt Greaves <k...@instaclustr.com> wrote:
>
> That's a debug message. From the sound of it, it's triggered on read where
> there is a digest mismatch between replicas. As to whether it's normal,
> well that depends on your cluster. Are the nodes reporting lots of dropped
> mutations and are you writing at <QUORUM?
>
>
>

RE: lots of DigestMismatchException in cassandra3

[Adeline.Pan]

Hi Kurt,
Thank you for the information, but the error “Corrupt empty row found in 
unfiltered partition” seems not related to the “Mismatch”; the time they 
occurred didn’t match. We use “QUORUM” consistency level for both read and 
write and I didn’t notice any failed writing in the log. Any other cause you 
can think of?  Would it cause performance issue when lots of this “Mismatch” 
happened?

--
Regards, Adeline



From: kurt Greaves [mailto:k...@instaclustr.com]
Sent: Monday, November 21, 2016 5:02 PM
To: user@cassandra.apache.org
Cc: tommy.stend...@ericsson.com
Subject: Re: lots of DigestMismatchException in cassandra3

Actually, just saw the error message in those logs and what you're looking at 
is probably 
https://issues.apache.org/jira/browse/CASSANDRA-12694<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D12694=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=tMf24yohd0jRGCBo_pzYdRMw52h3NCImPOGXjy1SAsc=>


Kurt Greaves
k...@instaclustr.com<mailto:k...@instaclustr.com>
www.instaclustr.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instaclustr.com=CwMFaQ=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=Km5uRGlDf2EjQFx7dbIrLzNfL6khh5OKA2sJk59l8-w=TwJ80glB0cSS0rW6jU1MGnlLWUtVYL1J7061vp2e_rI=>

On 21 November 2016 at 08:59, kurt Greaves 
<k...@instaclustr.com<mailto:k...@instaclustr.com>> wrote:

That's a debug message. From the sound of it, it's triggered on read where 
there is a digest mismatch between replicas. As to whether it's normal, well 
that depends on your cluster. Are the nodes reporting lots of dropped mutations 
and are you writing at <QUORUM?

Re: lots of DigestMismatchException in cassandra3

[kurt Greaves]

Actually, just saw the error message in those logs and what you're looking
at is probably https://issues.apache.org/jira/browse/CASSANDRA-12694



Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

On 21 November 2016 at 08:59, kurt Greaves  wrote:

> That's a debug message. From the sound of it, it's triggered on read where
> there is a digest mismatch between replicas. As to whether it's normal,
> well that depends on your cluster. Are the nodes reporting lots of dropped
> mutations and are you writing at 

RE: lots of DigestMismatchException in cassandra3

[kurt Greaves]

That's a debug message. From the sound of it, it's triggered on read where
there is a digest mismatch between replicas. As to whether it's normal,
well that depends on your cluster. Are the nodes reporting lots of dropped
mutations and are you writing at 

RE: lots of DigestMismatchException in cassandra3

[Adeline.Pan]

Hi Tommy,
Thank you for the reply, but I don't think this is the reason, because we don't 
have any static column.

--
Regards, Adeline



From: Tommy Stendahl [mailto:tommy.stend...@ericsson.com]
Sent: Thursday, November 17, 2016 5:16 PM
To: user@cassandra.apache.org
Subject: Re: lots of DigestMismatchException in cassandra3


Hi,

Could it be 
CASSANDRA-12090<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D12090=CwMF-g=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q=552xSDXEzKpvsyZM5wpE0TGEUDzVsX35L-K72hRjpLc=-f-6NfSYZVngB5-JpViJQf8p87WqCGDVfLrX_wrbNIY=lJNXNXd0alPlegiAnSD-h3srxIrZH5Qcv4er75LnFB0=>?

Regards,
/Tommy
On 2016-11-17 09:47, 
adeline@thomsonreuters.com<mailto:adeline@thomsonreuters.com> wrote:
Hi,

We are using Cassandra 3.7 and we have a single datacenter of 4 nodes, RF =3, 
and the consistency level is quorum.
I found lots of DigestMismatchException in debug.log, as below:
DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066 ReadCallback.java:235 - 
Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key 
DecoratedKey(-7187709033646443438, 
000801586f9538000405f72c7a04000300) 
(c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641)
at 
org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) 
~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226)
 ~[apache-cassandra-3.7.jar:3.7]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_25]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
INFO  [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - Deleted 
hint file bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints
INFO  [HintsDispatcher:5] 2016-11-17 08:25:06,962 
HintsDispatchExecutor.java:263 - Finished hinted handoff of file 
bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints to endpoint 
bcd84017-6068-4a76-be89-96c0130599f3
DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456 - 
Ignoring interval time of 2003165297 for /10.52.153.107
DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034 CompactionTask.java:223 - 
Compacted (3b3d7600-ac9f-11e6-8b55-03fd515998e4) 4 sstables to 
[/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/mb-2256-big,]
 to level=0.  85.011MiB to 53.580MiB (~63% of original) in 75,524ms.  
Throughput = 726.471KiB/s.  0 total partitions merged to 15,552.  Partition 
merge counts were {1:11623, 2:3543, 3:385, 4:1, }
ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17 08:25:42,126 
CassandraDaemon.java:217 - Exception in thread 
Thread[MessagingService-Incoming-/10.52.153.109,5,main]
java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered 
partition
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:331)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:287)
 ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) 
~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
 ~[apache-cassandra-3.7.jar:3.7]
Caused by: java.io.IOException: Corrupt empty row found in unfiltered partition
at 
org.apache.cassandra.db.rows.UnfilteredSerializer

Re: lots of DigestMismatchException in cassandra3

[Tommy Stendahl]

Hi,

Could it be CASSANDRA-12090 
?


Regards,
/Tommy

On 2016-11-17 09:47, adeline@thomsonreuters.com wrote:


Hi,

We are using Cassandra 3.7 and we have a single datacenter of 4 nodes, 
RF =3, and the consistency level is quorum.


I found *lots* *of* DigestMismatchException in debug.log, as below:

DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066 
ReadCallback.java:235 - Digest mismatch:


org.apache.cassandra.service.DigestMismatchException: Mismatch for key 
DecoratedKey(-7187709033646443438, 
000801586f9538000405f72c7a04000300) 
(c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641)


at 
org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226) 
~[apache-cassandra-3.7.jar:3.7]


at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_25]


at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_25]


at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

INFO [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - 
Deleted hint file 
bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints


INFO [HintsDispatcher:5] 2016-11-17 08:25:06,962 
HintsDispatchExecutor.java:263 - Finished hinted handoff of file 
bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints to endpoint 
bcd84017-6068-4a76-be89-96c0130599f3


DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456 
- Ignoring interval time of 2003165297 for /10.52.153.107


DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034 
CompactionTask.java:223 - Compacted 
(3b3d7600-ac9f-11e6-8b55-03fd515998e4) 4 sstables to 
[/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/mb-2256-big,] 
to level=0.  85.011MiB to 53.580MiB (~63% of original) in 75,524ms.  
Throughput = 726.471KiB/s. 0 total partitions merged to 15,552.  
Partition merge counts were {1:11623, 2:3543, 3:385, 4:1, }


ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17 
08:25:42,126 CassandraDaemon.java:217 - Exception in thread 
Thread[MessagingService-Incoming-/10.52.153.109,5,main]


java.io.IOError: java.io.IOException: Corrupt empty row found in 
unfiltered partition


at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:331) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:287) 
~[apache-cassandra-3.7.jar:3.7]


at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) 
~[apache-cassandra-3.7.jar:3.7]


Caused by: java.io.IOException: Corrupt empty row found in unfiltered 
partition


at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:388) 
~[apache-cassandra-3.7.jar:3.7]


at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219) 
~[apache-cassandra-3.7.jar:3.7]


... 11 common frames omitted

Is this normal?

--

Regards, Adeline

lots of DigestMismatchException in cassandra3

[Adeline.Pan]

Hi,

We are using Cassandra 3.7 and we have a single datacenter of 4 nodes, RF =3, 
and the consistency level is quorum.
I found lots of DigestMismatchException in debug.log, as below:
DEBUG [ReadRepairStage:11] 2016-11-17 08:24:57,066 ReadCallback.java:235 - 
Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key 
DecoratedKey(-7187709033646443438, 
000801586f9538000405f72c7a04000300) 
(c8d67cfae7be8470ca1786eaa0d8968e vs 3aa974d3eca302682c6f31ca250ea641)
at 
org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) 
~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226)
 ~[apache-cassandra-3.7.jar:3.7]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_25]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
INFO  [HintsDispatcher:5] 2016-11-17 08:25:06,958 HintsStore.java:126 - Deleted 
hint file bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints
INFO  [HintsDispatcher:5] 2016-11-17 08:25:06,962 
HintsDispatchExecutor.java:263 - Finished hinted handoff of file 
bcd84017-6068-4a76-be89-96c0130599f3-1479371095876-1.hints to endpoint 
bcd84017-6068-4a76-be89-96c0130599f3
DEBUG [GossipStage:1] 2016-11-17 08:25:32,551 FailureDetector.java:456 - 
Ignoring interval time of 2003165297 for /10.52.153.107
DEBUG [CompactionExecutor:9] 2016-11-17 08:25:33,034 CompactionTask.java:223 - 
Compacted (3b3d7600-ac9f-11e6-8b55-03fd515998e4) 4 sstables to 
[/data/c878jrn/data/pc_object_data_beta/vsc_data-53935ff0a74811e69ab46393a19c464a/mb-2256-big,]
 to level=0.  85.011MiB to 53.580MiB (~63% of original) in 75,524ms.  
Throughput = 726.471KiB/s.  0 total partitions merged to 15,552.  Partition 
merge counts were {1:11623, 2:3543, 3:385, 4:1, }
ERROR [MessagingService-Incoming-/10.52.153.109] 2016-11-17 08:25:42,126 
CassandraDaemon.java:217 - Exception in thread 
Thread[MessagingService-Incoming-/10.52.153.109,5,main]
java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered 
partition
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:642)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:331)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:287)
 ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) 
~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
 ~[apache-cassandra-3.7.jar:3.7]
Caused by: java.io.IOException: Corrupt empty row found in unfiltered partition
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:388)
 ~[apache-cassandra-3.7.jar:3.7]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219)
 ~[apache-cassandra-3.7.jar:3.7]
... 11 common frames omitted


Is this normal?


--
Regards, Adeline