Eric - I understand, thing is repairs are causing issues and I would like
to have the option to only run them on the tables that really need that.
Kane - Thanks, good idea, I will check that metric.
On Fri, Feb 26, 2021 at 12:07 AM Kane Wilson wrote:
> You should be able to use the Table
You should be able to use the Table metric ReadRepairRequests to determine
which table has read repairs occuring (fairly sure it's present on 3.11.
See
https://cassandra.apache.org/doc/latest/operating/metrics.html#table-metrics
Cheers,
Kane
raft.so - Cassandra consulting, support, and managed
Unfortunately, you won't be able to work it out just based on that debug
message. The only suggestion I have is to run repairs regularly. Cheers!
>
Hey
I'm running cassandra 3.11.9 and I have a lot of messages like this:
DEBUG [ReadRepairStage:2] 2021-02-25 16:41:11,464 ReadCallback.java:244 -
Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(4059620144736691554
[ReadRepairStage:372] 2020-12-14 09:36:09,002
ReadCallback.java:244 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(-7287062361589376757,
44535f313034335f32353839305f323032302d31322d31325430302d31392d33312e3330335a
Thanks all for the help on this. I've changed all my writes to
LOCAL_QUORUM, and same with reads. Under a constant load of doing
writes to a table and reads from the same table, I'm still getting the:
DEBUG [ReadRepairStage:372] 2020-12-14 09:36:09,002
ReadCallback.java:244 - Digest
Thank you. OK - I can see from 'nodetool getendpoints keyspace table
key' that 3 nodes respond as one would expect. My theory is that once I
encounter the error, a read repair is triggered, and by the time I
execute nodetool, 3 nodes respond.
I tried a test with the same table, but with
>
> Thank you Steve - once I have the key, how do I get to a node?
>
Run this command to determine which replicas own the partition:
$ nodetool getendpoints
> So if the propagation has not taken place and a node doesn't have the data
> and is the first to 'be asked' the client will get no data?
Thank you Steve - once I have the key, how do I get to a node?
After reading some of the documentation, it looks like the
load-balancing-policy below *is* a token aware policy. Perhaps writes
need to be done with QUORUM; I don't know how long Cassandra will take
to make sure replicas are
If you can determine the key, then you can determine which nodes do and do
not have the data. You may be able to glean a bit more information like
that, maybe one node is having problems, versus entire cluster.
On Wed, Dec 2, 2020 at 9:32 AM Joe Obernberger
wrote:
> Clients are using an
Clients are using an application.conf like:
datastax-java-driver {
basic.request.timeout = 60 seconds
basic.request.consistency = ONE
basic.contact-points = ["172.16.110.3:9042", "172.16.110.4:9042",
"172.16.100.208:9042", "172.16.100.224:9042", "172.16.100.225:9042",
:16 PM, Steve Lacerda wrote:
The digest mismatch typically shows the partition key info, with
something like this:
DecoratedKey(-1671292413668442751, 48343732322d3838353032)
That refers to the partition key, which you can gather like so:
python
import binascii
binascii.unhexlify
Are you using token aware policy for the driver?
If your writes are one and your reads are one, the propagation may not have
happened depending on the coordinator that is used.
TokenAware will make that a bit better.
On Wed, Dec 2, 2020 at 11:12 AM Joe Obernberger <
The digest mismatch typically shows the partition key info, with something
like this:
DecoratedKey(-1671292413668442751, 48343732322d3838353032)
That refers to the partition key, which you can gather like so:
python
import binascii
binascii.unhexlify('48343732322d3838353032')
'H4722-88502'
My
Hi Carl - thank you for replying.
I am using Cassandra 3.11.9-1
Rows are not typically being deleted - I assume you're referring to
Tombstones. I don't think that should be the case here as I don't think
we've deleted anything here.
This is a test cluster and some of the machines are small
Why is one of your nodes only at 14.6% ownership? That's weird, unless you
have a small rowcount.
Are you frequently deleting rows? Are you frequently writing rows at ONE?
What version of cassandra?
On Wed, Dec 2, 2020 at 9:56 AM Joe Obernberger
wrote:
> Hi All - this is my first post here.
Hi All - this is my first post here. I've been using Cassandra for
several months now and am loving it. We are moving from Apache HBase to
Cassandra for a big data analytics platform.
I'm using java to get rows from Cassandra and very frequently get a
java.util.NoSuchElementException when
series tables. Can I get some
pointers on how to track this down?
DEBUG [ReadRepairStage:25] 2017-02-14 17:45:03,256 ReadCallback.java:235 -
Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(-1025043841859687448
)
After delete, I try to read all columns from the row, I found the
node
found Digest mismatch due to Quorum consistency configuration, but the
result is not correct.
From the log, I can see the delete mutation already accepted
by 192.168.0.6, 192.168.0.1, but when 192.168.0.5
NodeB NodeC
1. New New New
2. Update Update Update
3. Delete Delete
When try to read from NodeB and NodeC, Digest mismatch exception
triggered, so Cassandra try to resolve this version conflict.
But the result is value Update.
Here is the suspect root
.
Here is event sequence:
seqID NodeA NodeB NodeC
1. New New New
2. Update Update Update
3. Delete Delete
When try to read from NodeB and NodeC, Digest mismatch exception triggered,
so Cassandra try to resolve this version conflict
, replication_factor is 3.
Here is event sequence:
seqID NodeA NodeB NodeC
1. New New New
2. Update Update Update
3. Delete Delete
When try to read from NodeB and NodeC, Digest mismatch exception
triggered, so Cassandra try to resolve this version conflict
level is
read/write quorum, replication_factor is 3.
Here is event sequence:
seqID NodeA NodeB NodeC
1. New New New
2. Update Update Update
3. Delete Delete
When try to read from NodeB and NodeC, Digest mismatch exception
triggered, so
at 4:43 PM, sankalp kohli kohlisank...@gmail.com
wrote:
Hi,
Lets say I am reading with consistency TWO and my replication is 3.
The
read is eligible for global read repair. It will send a request to get
data
from one node and a digest request to two.
If there is a digest mismatch
, sankalp kohli kohlisank...@gmail.com
wrote:
Hi,
Lets say I am reading with consistency TWO and my replication is
3. The
read is eligible for global read repair. It will send a request to get
data
from one node and a digest request to two.
If there is a digest mismatch, what I am reading
From my understanding, if CL = 2, one read, one digest are sent. Only if it
is read repair, digest is sent to all replicates.
Thanks.
-Wei
Sent from my Samsung smartphone on ATT
Original message
Subject: Re: Read during digest mismatch
From: Edward Capriolo edlinuxg
message
Subject: Re: Read during digest mismatch
From: Edward Capriolo edlinuxg...@gmail.com
To: user@cassandra.apache.org user@cassandra.apache.org
CC:
I think the code base does not benefit from having too many different read
code paths. Logically what your suggesting is reasonable
repair. It will send a request to get data
from one node and a digest request to two.
If there is a digest mismatch, what I am reading from the code looks like it
will get the data from all three nodes and do a resolve of the data before
returning to the client.
Is it correct or I am readind
Hi,
Lets say I am reading with consistency TWO and my replication is 3. The
read is eligible for global read repair. It will send a request to get data
from one node and a digest request to two.
If there is a digest mismatch, what I am reading from the code looks like
it will get the data from
/write quorum.
Check the Cassandra log, the local node don't perform the delete
operation but send the mutation to other nodes (192.168.0.6, 192.168.0.1)
After delete, I try to read all columns from the row, I found the node
found Digest mismatch due to Quorum consistency configuration
, I try to read all columns from the row, I found the node
found Digest mismatch due to Quorum consistency configuration, but the
result is not correct.
From the log, I can see the delete mutation already accepted
by 192.168.0.6, 192.168.0.1, but when 192.168.0.5 read response from 0.6
perform the delete
operation but send the mutation to other nodes (192.168.0.6, 192.168.0.1)
After delete, I try to read all columns from the row, I found the node
found Digest mismatch due to Quorum consistency configuration, but the
result is not correct.
From the log, I can see
to other nodes (192.168.0.6, 192.168.0.1)
After delete, I try to read all columns from the row, I found the node
found Digest mismatch due to Quorum consistency configuration, but the
result is not correct.
From the log, I can see the delete mutation already accepted
by 192.168.0.6, 192.168.0.1
33 matches
Mail list logo
