Re: Why a READ REPAIR ?
You can check for the string "digest mismatch" in the logs. Similarly, you can track the RR stats in nodetool netstats and the dropped mutations in nodetool tpstats. To be clear though, RRs are a side-effect of nodes either dropping mutations or being unresponsive so they miss mutations. RRs do *not* cause nodes to be unresponsive or unavailable. RRs are Cassandra's way of automatically dealing with the "outages" -- it doesn't cause them. Cheers! >
Re: Why a READ REPAIR ?
Thanx Erick Is there a way to turn on tracing based on certain criteria, I would like to start tracing when there is some sort of failure, i.e. in this case when a READ REPAIR is triggered as I would like to know why we sometimes can’t reach one of the nodes -Tobias From: Erick Ramirez Reply to: "user@cassandra.apache.org" Date: Wednesday, 12 August 2020 at 01:30 To: "user@cassandra.apache.org" Subject: Re: Why a READ REPAIR ? Perhaps this is super obvious but I need a confirmation as you say “…not subsequent reads for other data unrelated to the read being repaired…” But this is subsequent reads to the _same_ partition key So to be more explicit READ 1 with Local Quorum : SELECT * FROM products WHERE id = ABC123 READ 2 with Local One : SELECT * FROM products WHERE id = ABC123 Would read (2) be blocked by the READ REPAIR that was done by read (1) As I understand that the read repair is working not on the whole table but on the partition key it had problems with Not at all. We (the community) are continuing to invest more on improving the docs because these things are not obvious or well-understood. The read request is blocked while the repair of replicas is in progress. To be clear, it is the request (singular) that is blocked and not some locking mechanism at the partition level (say like LWTs). Other requests (read 2 in your case) will continue to work. Cheers!
Re: Why a READ REPAIR ?
> > Perhaps this is super obvious but I need a confirmation as you say “…not > subsequent reads for other data unrelated to the read being repaired…” > > But this is subsequent reads to the _*same*_ partition key > > So to be more explicit > > READ 1 with Local Quorum : SELECT * FROM products WHERE id = ABC123 > > READ 2 with Local One : SELECT * FROM products WHERE id = ABC123 > > > > Would read (2) be blocked by the READ REPAIR that was done by read (1) > > As I understand that the read repair is working not on the whole table but > on the partition key it had problems with > Not at all. We (the community) are continuing to invest more on improving the docs because these things are not obvious or well-understood. The read *request* is blocked while the repair of replicas is in progress. To be clear, it is the *request* (singular) that is blocked and *not* some locking mechanism at the partition level (say like LWTs). Other requests (read 2 in your case) will continue to work. Cheers!
Re: Why a READ REPAIR ?
Hi Tobias READ2 will not be blocked by READ repair of READ1. Regards Manish On Tue, Aug 11, 2020 at 6:02 PM Tobias Eriksson wrote: > Thanx Erick, > > Perhaps this is super obvious but I need a confirmation as you say “…not > subsequent reads for other data unrelated to the read being repaired…” > > But this is subsequent reads to the _*same*_ partition key > > So to be more explicit > > READ 1 with Local Quorum : SELECT * FROM products WHERE id = ABC123 > > READ 2 with Local One : SELECT * FROM products WHERE id = ABC123 > > > > Would read (2) be blocked by the READ REPAIR that was done by read (1) > > As I understand that the read repair is working not on the whole table but > on the partition key it had problems with > > > > -Tobias > > > > > > *From: *Erick Ramirez > *Reply to: *"user@cassandra.apache.org" > *Date: *Tuesday, 11 August 2020 at 11:26 > *To: *"user@cassandra.apache.org" > *Subject: *Re: Why a READ REPAIR ? > > > > If a READ triggers a READ REPAIR, and then if we do an additional READ > would then that BLOCK until the “first” READ REPAIR would be done ? > > -Tobias > > > > Not all read repairs are blocking RRs (aka foreground RRs). There are also > background RRs which by definition are non-blocking because they happen in > the background. > > > > In response to your question, the "additional read" is not blocked. In a > blocking RR, if there is a mismatch in the data returned to the coordinator > from the replicas involved in the query (determined by the read consistency > level) then the coordinator sends a repair to the out-of-sync replica in > the foreground before sending the result back to the client so the read is > blocked until the RR is completed. To reiterate, only the read involved in > the RR is blocked -- not subsequent reads for other data unrelated to the > read being repaired. Cheers! > > >
Re: Why a READ REPAIR ?
Thanx Erick, Perhaps this is super obvious but I need a confirmation as you say “…not subsequent reads for other data unrelated to the read being repaired…” But this is subsequent reads to the _same_ partition key So to be more explicit READ 1 with Local Quorum : SELECT * FROM products WHERE id = ABC123 READ 2 with Local One : SELECT * FROM products WHERE id = ABC123 Would read (2) be blocked by the READ REPAIR that was done by read (1) As I understand that the read repair is working not on the whole table but on the partition key it had problems with -Tobias From: Erick Ramirez Reply to: "user@cassandra.apache.org" Date: Tuesday, 11 August 2020 at 11:26 To: "user@cassandra.apache.org" Subject: Re: Why a READ REPAIR ? If a READ triggers a READ REPAIR, and then if we do an additional READ would then that BLOCK until the “first” READ REPAIR would be done ? -Tobias Not all read repairs are blocking RRs (aka foreground RRs). There are also background RRs which by definition are non-blocking because they happen in the background. In response to your question, the "additional read" is not blocked. In a blocking RR, if there is a mismatch in the data returned to the coordinator from the replicas involved in the query (determined by the read consistency level) then the coordinator sends a repair to the out-of-sync replica in the foreground before sending the result back to the client so the read is blocked until the RR is completed. To reiterate, only the read involved in the RR is blocked -- not subsequent reads for other data unrelated to the read being repaired. Cheers!
Re: Why a READ REPAIR ?
> > If a READ triggers a READ REPAIR, and then if we do an additional READ > would then that BLOCK until the “first” READ REPAIR would be done ? > > -Tobias > Not all read repairs are blocking RRs (aka foreground RRs). There are also background RRs which by definition are non-blocking because they happen in the background. In response to your question, the "additional read" is not blocked. In a blocking RR, if there is a mismatch in the data returned to the coordinator from the replicas involved in the query (determined by the read consistency level) then the coordinator sends a repair to the out-of-sync replica in the foreground before sending the result back to the client so the read is blocked until the RR is completed. To reiterate, only the read involved in the RR is blocked -- not subsequent reads for other data unrelated to the read being repaired. Cheers!
Re: Why a READ REPAIR ?
If a READ triggers a READ REPAIR, and then if we do an additional READ would then that BLOCK until the “first” READ REPAIR would be done ? -Tobias From: Jeff Jirsa Reply to: "user@cassandra.apache.org" Date: Tuesday, 11 August 2020 at 07:30 To: cassandra Subject: Re: Why a READ REPAIR ? Your schema may have read repair (non-blocking, background) set to 10% (0.1, for dclocal). You may have GC pauses causing writes (or reads) to be delayed. You may be hitting a cassandra bug. Would need the `TRACING` output to know for sure. On Mon, Aug 10, 2020 at 10:10 PM Tobias Eriksson mailto:tobias.eriks...@qvantel.com>> wrote: Hi We have a Cassandra solution with 2 DCs where each DC has >30 nodes From time to time we see problems with READ REPAIR, but I am stuck with the analysis We have a pattern for these faults where we do 1. INSERT with Local Quorum (2 out of 3) 2. Wait for 0.5 - 1 seconds time window 3. READ with Local Quorum (2 out of 3) * Triggers a read repair 1. Then we do an UPDATE … The replication factor is 3 In my world in (1) we for sure store the data in 2 out of 3 places, and I would be surprised if we would not also reach the 3;rd node within 0.5 sec So how come in (3) the read can’t get a proper response from 2 out of 3 Some are saying the problem started occurring when we added DC2, but I can’t understand how it could be as our query is Local Quorum and will involve only DC1 How can I debug this fault ? How can I track if the data has reached all 3 nodes ? All ideas are welcome -Tobias
Re: Why a READ REPAIR ?
Your schema may have read repair (non-blocking, background) set to 10% (0.1, for dclocal). You may have GC pauses causing writes (or reads) to be delayed. You may be hitting a cassandra bug. Would need the `TRACING` output to know for sure. On Mon, Aug 10, 2020 at 10:10 PM Tobias Eriksson < tobias.eriks...@qvantel.com> wrote: > Hi > > We have a Cassandra solution with 2 DCs where each DC has >30 nodes > > From time to time we see problems with READ REPAIR, but I am stuck with > the analysis > > We have a pattern for these faults where we do > >1. INSERT with Local Quorum (2 out of 3) >2. Wait for 0.5 - 1 seconds time window >3. READ with Local Quorum (2 out of 3) > 1. Triggers a read repair >4. Then we do an UPDATE … > > > > The replication factor is 3 > > In my world in (1) we for sure store the data in 2 out of 3 places, and I > would be surprised if we would not also reach the 3;rd node within 0.5 sec > > So how come in (3) the read can’t get a proper response from 2 out of 3 > > Some are saying the problem started occurring when we added DC2, but I > can’t understand how it could be as our query is Local Quorum and will > involve only DC1 > > > > How can I debug this fault ? > > How can I track if the data has reached all 3 nodes ? > > > > All ideas are welcome > > -Tobias > > > > >