Re: Cassandra Clients: Stale Cluster Topology
What consistency level are you using on reads and writes? If either are less than LOCAL_QUORUM, this behavior is definitely expected. If you ARE using quorum/local_quorum, and you can correlate these issues to when a node scales in / out, then it's probably consistency violation on bootstrap. Unless you run repair before you re-bootstrap a node (assuming you're using ephemeral disks), the bootstrap process may choose a streaming source that is missing ~1 write, and then you end up with 2 replicas without the write, so reads dont see it until chance read repair or repair is run. On 2025/10/10 18:30:41 FMH wrote: > Thanks for taking the time to Respond, Jeff. > > I was kind expecting this reply the minute I included version 2.2.8. We are > in the process of upgrading to 5. > > As for the bootstrap automation we use. It has been in effect for more than > 10 years and we replaced 100's of nodes without ever having any issues > including this one. > > This automation we put in place based on the available documentation for > Apache and Datastax cassandra. We have also had it assessed several times > over the years by external consultants. > > Thanks for clarifying the getendpoints. This is why we paired it with the > SELECT statement validation test as well to verify data. > > On Fri, Oct 10, 2025 at 1:54 PM Jeff Jirsa wrote: > > > Also: nodetool getendpoints just hashes the key you provide against the > > cluster topology / schema definition, which tells you which nodes WOULD own > > the data if it exists. It does NOT guarantee that it exists. > > > > > > > > On 2025/10/10 17:48:32 Jeff Jirsa wrote: > > > You're using a 9 year old release. There have been literally hundreds of > > correctness fixes over those 9 years. You need to upgrade. > > > > > > The rest of your answers inline. > > > > > > > > > > > > On 2025/10/10 12:56:58 FMH wrote: > > > > Few times a week, our developers report that Cassandra retrieves are > > > > coming back with zero rows. No error messages. > > > > > > > > Using the same item ID's, a CQLSH SELECT statement returns a single > > row as > > > > expected. Furthermore, the NODETOOL GETENDPOINTS returns three IP's as > > we > > > > expect. > > > > > > > > This confirms these ItemID's do exist in Cassandra, it is just the Java > > > > clients are not retrieving it. > > > > > > > > We noticed this issue to present itself more when nodes are replaced > > in the > > > > cluster as a result of EC2 node deprecation. > > > > > > Are you using EBS or ephemeral disk? Don't use ephemeral disk unless you > > are much better at running cassandra and know how to replace a node without > > data loss (which you do not seem to know how to do). > > > > > > > > > > > > > > Once the developers restarted the Java client apps, it was now able to > > > > retrieve these ItemID's. > > > > > > That sounds weird. It may be that they read repaired or normal-repaired, > > or it may be that the java apps were pointing to the wrong thing/cluster. > > > > > > > > > > > 1- Is this what is called the 'empty' read' behavior? > > > > 2- Is this caused by clients topology metadata getting out of sync > > with the > > > > cluster? > > > > > > Could be cluster scaling unsafely due to ec2 events. > > > Could be low consistency level > > > Could be any number of hundreds of topology bugs fixed since 2016. > > > > > > If it's a client bug, I assume it's an old client bug I've never seen > > before. Well functioning cassandra clients shouldn't care about the > > topology, the coordinating server will forward the request anyway. > > > > > > > 3- How can this be detected? Should we have client drivers return > > 'metadata > > > > = cluster.metadata' and compare it to 'nodetool gossipinfo'? > > > > > > Upgrade your cluster. > > > Use EBS so when nodes change, they don't change data ownership. > > > > > > > 4- Other than restarting the clients, is there a way to have client > > apps to > > > > force to refresh their ring metadata? > > > > > > > > The client apps are using 'com.datastax.oss:java-driver-core:4.13.0' > > > > driver. > > > > > > > > Google returns little information about this and GenAI's chat model > > even > > > > though useful, they tend to hallucinate with confidence often. > > > > > > > > Thanks > > > > > > > > > > > > Thank you > > > > > > > > > > > > -- > > > Thank you >
Re: Cassandra Clients: Stale Cluster Topology
No. It’s actually almost a guarantee that what’s happening is that you violate consistency on node replacement and the read at ALL copies the data back to where it needed to be Here’s what’s happening Let’s pretend you have 6 nodes, A-F When you write at LQ for a key, let’s pretend it goes to B, C, and D 2 of the three of those have to ack a write - let’s say it goes to B and D, but misses C. Write succeeds, reads will see it because any read will be either B and D (easy), B and C (data on B, visible, copies to C on read) or C and D (data on D, visible, copies to C on read) Now imagine that before C gets the data by repair or read repair B fails and gets sent for maintenance by your cloud provider A new B’ gets added to the ring. B’ has to choose about to send it data. It’s going to choose either C or D. If it chooses D, no problem, reads still always see data If it chooses B, now D still has the data, but B and C don’t. So a read may choose: B/C - data missing B/D - repairs to B from D C/D - repairs to C from D If you do ALL, it repairs to B and C from D If you use EBS instead of re-bootstrapping the data, the data drive can get re-attached and B still has the data. Alternatively, if you had a newer version, you could just run incremental repair “often and this would happen much less often (or never, if you force incremental repair to run before the bootstrap happens). Incremental repair in 2.2 is not good enough - don’t try to use it until you upgrade On 2025/10/10 19:38:08 FMH wrote: > Data read/written with CL=LQ. > > To attempt to isolate the issue on the C* server vs. client, running a > SELECT statement with CL=ALL returned the row 200 times with zero misses. > > Retrieving the same ID through the java service had a 30% failure rate. > > Isn't this conclusive enough that data does exist & has three replicas. > The issue must be isolated to the client? > > On Fri, Oct 10, 2025 at 2:47 PM Jeff Jirsa wrote: > > > What consistency level are you using on reads and writes? > > > > If either are less than LOCAL_QUORUM, this behavior is definitely expected. > > > > If you ARE using quorum/local_quorum, and you can correlate these issues > > to when a node scales in / out, then it's probably consistency violation on > > bootstrap. Unless you run repair before you re-bootstrap a node (assuming > > you're using ephemeral disks), the bootstrap process may choose a streaming > > source that is missing ~1 write, and then you end up with 2 replicas > > without the write, so reads dont see it until chance read repair or repair > > is run. > > > > > > > > On 2025/10/10 18:30:41 FMH wrote: > > > Thanks for taking the time to Respond, Jeff. > > > > > > I was kind expecting this reply the minute I included version 2.2.8. We > > are > > > in the process of upgrading to 5. > > > > > > As for the bootstrap automation we use. It has been in effect for more > > than > > > 10 years and we replaced 100's of nodes without ever having any issues > > > including this one. > > > > > > This automation we put in place based on the available documentation for > > > Apache and Datastax cassandra. We have also had it assessed several times > > > over the years by external consultants. > > > > > > Thanks for clarifying the getendpoints. This is why we paired it with the > > > SELECT statement validation test as well to verify data. > > > > > > On Fri, Oct 10, 2025 at 1:54 PM Jeff Jirsa wrote: > > > > > > > Also: nodetool getendpoints just hashes the key you provide against the > > > > cluster topology / schema definition, which tells you which nodes > > WOULD own > > > > the data if it exists. It does NOT guarantee that it exists. > > > > > > > > > > > > > > > > On 2025/10/10 17:48:32 Jeff Jirsa wrote: > > > > > You're using a 9 year old release. There have been literally > > hundreds of > > > > correctness fixes over those 9 years. You need to upgrade. > > > > > > > > > > The rest of your answers inline. > > > > > > > > > > > > > > > > > > > > On 2025/10/10 12:56:58 FMH wrote: > > > > > > Few times a week, our developers report that Cassandra retrieves > > are > > > > > > coming back with zero rows. No error messages. > > > > > > > > > > > > Using the same item ID's, a CQLSH SELECT statement returns a single > > > > row as > > > > > > expected. Furthermore, the NODETOOL GETENDPOINTS returns three > > IP's as > > > > we > > > > > > expect. > > > > > > > > > > > > This confirms these ItemID's do exist in Cassandra, it is just the > > Java > > > > > > clients are not retrieving it. > > > > > > > > > > > > We noticed this issue to present itself more when nodes are > > replaced > > > > in the > > > > > > cluster as a result of EC2 node deprecation. > > > > > > > > > > Are you using EBS or ephemeral disk? Don't use ephemeral disk unless > > you > > > > are much better at running cassandra and know how to replace a node > > without > > > > data loss (which you do not seem to know how to do). > > > > >
Re: Cassandra Clients: Stale Cluster Topology
In your statement ... If it chooses B, now D still has the data, but B and C don’t. So a read may choose: I think, you meant to say If it chooses C, now D still has the data, but B and C don’t. So a read may choose: Thanks On Sat, Oct 11, 2025 at 12:04 AM Jeff Jirsa wrote: > No. It’s actually almost a guarantee that what’s happening is that you > violate consistency on node replacement and the read at ALL copies the data > back to where it needed to be > > Here’s what’s happening > > Let’s pretend you have 6 nodes, A-F > > When you write at LQ for a key, let’s pretend it goes to B, C, and D > > 2 of the three of those have to ack a write - let’s say it goes to B and > D, but misses C. Write succeeds, reads will see it because any read will be > either B and D (easy), B and C (data on B, visible, copies to C on read) or > C and D (data on D, visible, copies to C on read) > > Now imagine that before C gets the data by repair or read repair B fails > and gets sent for maintenance by your cloud provider > > A new B’ gets added to the ring. B’ has to choose about to send it data. > It’s going to choose either C or D. > > If it chooses D, no problem, reads still always see data > > If it chooses B, now D still has the data, but B and C don’t. So a read > may choose: > > B/C - data missing > B/D - repairs to B from D > C/D - repairs to C from D > > If you do ALL, it repairs to B and C from D > > If you use EBS instead of re-bootstrapping the data, the data drive can > get re-attached and B still has the data. > > Alternatively, if you had a newer version, you could just run incremental > repair “often and this would happen much less often (or never, if you force > incremental repair to run before the bootstrap happens). Incremental repair > in 2.2 is not good enough - don’t try to use it until you upgrade > > > > > On 2025/10/10 19:38:08 FMH wrote: > > Data read/written with CL=LQ. > > > > To attempt to isolate the issue on the C* server vs. client, running a > > SELECT statement with CL=ALL returned the row 200 times with zero misses. > > > > Retrieving the same ID through the java service had a 30% failure rate. > > > > Isn't this conclusive enough that data does exist & has three replicas. > > The issue must be isolated to the client? > > > > On Fri, Oct 10, 2025 at 2:47 PM Jeff Jirsa wrote: > > > > > What consistency level are you using on reads and writes? > > > > > > If either are less than LOCAL_QUORUM, this behavior is definitely > expected. > > > > > > If you ARE using quorum/local_quorum, and you can correlate these > issues > > > to when a node scales in / out, then it's probably consistency > violation on > > > bootstrap. Unless you run repair before you re-bootstrap a node > (assuming > > > you're using ephemeral disks), the bootstrap process may choose a > streaming > > > source that is missing ~1 write, and then you end up with 2 replicas > > > without the write, so reads dont see it until chance read repair or > repair > > > is run. > > > > > > > > > > > > On 2025/10/10 18:30:41 FMH wrote: > > > > Thanks for taking the time to Respond, Jeff. > > > > > > > > I was kind expecting this reply the minute I included version 2.2.8. > We > > > are > > > > in the process of upgrading to 5. > > > > > > > > As for the bootstrap automation we use. It has been in effect for > more > > > than > > > > 10 years and we replaced 100's of nodes without ever having any > issues > > > > including this one. > > > > > > > > This automation we put in place based on the available documentation > for > > > > Apache and Datastax cassandra. We have also had it assessed several > times > > > > over the years by external consultants. > > > > > > > > Thanks for clarifying the getendpoints. This is why we paired it > with the > > > > SELECT statement validation test as well to verify data. > > > > > > > > On Fri, Oct 10, 2025 at 1:54 PM Jeff Jirsa > wrote: > > > > > > > > > Also: nodetool getendpoints just hashes the key you provide > against the > > > > > cluster topology / schema definition, which tells you which nodes > > > WOULD own > > > > > the data if it exists. It does NOT guarantee that it exists. > > > > > > > > > > > > > > > > > > > > On 2025/10/10 17:48:32 Jeff Jirsa wrote: > > > > > > You're using a 9 year old release. There have been literally > > > hundreds of > > > > > correctness fixes over those 9 years. You need to upgrade. > > > > > > > > > > > > The rest of your answers inline. > > > > > > > > > > > > > > > > > > > > > > > > On 2025/10/10 12:56:58 FMH wrote: > > > > > > > Few times a week, our developers report that Cassandra > retrieves > > > are > > > > > > > coming back with zero rows. No error messages. > > > > > > > > > > > > > > Using the same item ID's, a CQLSH SELECT statement returns a > single > > > > > row as > > > > > > > expected. Furthermore, the NODETOOL GETENDPOINTS returns three > > > IP's as > > > > > we > > > > > > > expect. > > > > > > > > > > > > > > This confirms these ItemID's
Re: Cassandra Clients: Stale Cluster Topology
Thanks for taking the time to Respond, Jeff. I was kind expecting this reply the minute I included version 2.2.8. We are in the process of upgrading to 5. As for the bootstrap automation we use. It has been in effect for more than 10 years and we replaced 100's of nodes without ever having any issues including this one. This automation we put in place based on the available documentation for Apache and Datastax cassandra. We have also had it assessed several times over the years by external consultants. Thanks for clarifying the getendpoints. This is why we paired it with the SELECT statement validation test as well to verify data. On Fri, Oct 10, 2025 at 1:54 PM Jeff Jirsa wrote: > Also: nodetool getendpoints just hashes the key you provide against the > cluster topology / schema definition, which tells you which nodes WOULD own > the data if it exists. It does NOT guarantee that it exists. > > > > On 2025/10/10 17:48:32 Jeff Jirsa wrote: > > You're using a 9 year old release. There have been literally hundreds of > correctness fixes over those 9 years. You need to upgrade. > > > > The rest of your answers inline. > > > > > > > > On 2025/10/10 12:56:58 FMH wrote: > > > Few times a week, our developers report that Cassandra retrieves are > > > coming back with zero rows. No error messages. > > > > > > Using the same item ID's, a CQLSH SELECT statement returns a single > row as > > > expected. Furthermore, the NODETOOL GETENDPOINTS returns three IP's as > we > > > expect. > > > > > > This confirms these ItemID's do exist in Cassandra, it is just the Java > > > clients are not retrieving it. > > > > > > We noticed this issue to present itself more when nodes are replaced > in the > > > cluster as a result of EC2 node deprecation. > > > > Are you using EBS or ephemeral disk? Don't use ephemeral disk unless you > are much better at running cassandra and know how to replace a node without > data loss (which you do not seem to know how to do). > > > > > > > > > > Once the developers restarted the Java client apps, it was now able to > > > retrieve these ItemID's. > > > > That sounds weird. It may be that they read repaired or normal-repaired, > or it may be that the java apps were pointing to the wrong thing/cluster. > > > > > > > > 1- Is this what is called the 'empty' read' behavior? > > > 2- Is this caused by clients topology metadata getting out of sync > with the > > > cluster? > > > > Could be cluster scaling unsafely due to ec2 events. > > Could be low consistency level > > Could be any number of hundreds of topology bugs fixed since 2016. > > > > If it's a client bug, I assume it's an old client bug I've never seen > before. Well functioning cassandra clients shouldn't care about the > topology, the coordinating server will forward the request anyway. > > > > > 3- How can this be detected? Should we have client drivers return > 'metadata > > > = cluster.metadata' and compare it to 'nodetool gossipinfo'? > > > > Upgrade your cluster. > > Use EBS so when nodes change, they don't change data ownership. > > > > > 4- Other than restarting the clients, is there a way to have client > apps to > > > force to refresh their ring metadata? > > > > > > The client apps are using 'com.datastax.oss:java-driver-core:4.13.0' > > > driver. > > > > > > Google returns little information about this and GenAI's chat model > even > > > though useful, they tend to hallucinate with confidence often. > > > > > > Thanks > > > > > > > > > Thank you > > > > > > -- Thank you
Re: Cassandra Clients: Stale Cluster Topology
Data read/written with CL=LQ. To attempt to isolate the issue on the C* server vs. client, running a SELECT statement with CL=ALL returned the row 200 times with zero misses. Retrieving the same ID through the java service had a 30% failure rate. Isn't this conclusive enough that data does exist & has three replicas. The issue must be isolated to the client? On Fri, Oct 10, 2025 at 2:47 PM Jeff Jirsa wrote: > What consistency level are you using on reads and writes? > > If either are less than LOCAL_QUORUM, this behavior is definitely expected. > > If you ARE using quorum/local_quorum, and you can correlate these issues > to when a node scales in / out, then it's probably consistency violation on > bootstrap. Unless you run repair before you re-bootstrap a node (assuming > you're using ephemeral disks), the bootstrap process may choose a streaming > source that is missing ~1 write, and then you end up with 2 replicas > without the write, so reads dont see it until chance read repair or repair > is run. > > > > On 2025/10/10 18:30:41 FMH wrote: > > Thanks for taking the time to Respond, Jeff. > > > > I was kind expecting this reply the minute I included version 2.2.8. We > are > > in the process of upgrading to 5. > > > > As for the bootstrap automation we use. It has been in effect for more > than > > 10 years and we replaced 100's of nodes without ever having any issues > > including this one. > > > > This automation we put in place based on the available documentation for > > Apache and Datastax cassandra. We have also had it assessed several times > > over the years by external consultants. > > > > Thanks for clarifying the getendpoints. This is why we paired it with the > > SELECT statement validation test as well to verify data. > > > > On Fri, Oct 10, 2025 at 1:54 PM Jeff Jirsa wrote: > > > > > Also: nodetool getendpoints just hashes the key you provide against the > > > cluster topology / schema definition, which tells you which nodes > WOULD own > > > the data if it exists. It does NOT guarantee that it exists. > > > > > > > > > > > > On 2025/10/10 17:48:32 Jeff Jirsa wrote: > > > > You're using a 9 year old release. There have been literally > hundreds of > > > correctness fixes over those 9 years. You need to upgrade. > > > > > > > > The rest of your answers inline. > > > > > > > > > > > > > > > > On 2025/10/10 12:56:58 FMH wrote: > > > > > Few times a week, our developers report that Cassandra retrieves > are > > > > > coming back with zero rows. No error messages. > > > > > > > > > > Using the same item ID's, a CQLSH SELECT statement returns a single > > > row as > > > > > expected. Furthermore, the NODETOOL GETENDPOINTS returns three > IP's as > > > we > > > > > expect. > > > > > > > > > > This confirms these ItemID's do exist in Cassandra, it is just the > Java > > > > > clients are not retrieving it. > > > > > > > > > > We noticed this issue to present itself more when nodes are > replaced > > > in the > > > > > cluster as a result of EC2 node deprecation. > > > > > > > > Are you using EBS or ephemeral disk? Don't use ephemeral disk unless > you > > > are much better at running cassandra and know how to replace a node > without > > > data loss (which you do not seem to know how to do). > > > > > > > > > > > > > > > > > > Once the developers restarted the Java client apps, it was now > able to > > > > > retrieve these ItemID's. > > > > > > > > That sounds weird. It may be that they read repaired or > normal-repaired, > > > or it may be that the java apps were pointing to the wrong > thing/cluster. > > > > > > > > > > > > > > 1- Is this what is called the 'empty' read' behavior? > > > > > 2- Is this caused by clients topology metadata getting out of sync > > > with the > > > > > cluster? > > > > > > > > Could be cluster scaling unsafely due to ec2 events. > > > > Could be low consistency level > > > > Could be any number of hundreds of topology bugs fixed since 2016. > > > > > > > > If it's a client bug, I assume it's an old client bug I've never seen > > > before. Well functioning cassandra clients shouldn't care about the > > > topology, the coordinating server will forward the request anyway. > > > > > > > > > 3- How can this be detected? Should we have client drivers return > > > 'metadata > > > > > = cluster.metadata' and compare it to 'nodetool gossipinfo'? > > > > > > > > Upgrade your cluster. > > > > Use EBS so when nodes change, they don't change data ownership. > > > > > > > > > 4- Other than restarting the clients, is there a way to have client > > > apps to > > > > > force to refresh their ring metadata? > > > > > > > > > > The client apps are using > 'com.datastax.oss:java-driver-core:4.13.0' > > > > > driver. > > > > > > > > > > Google returns little information about this and GenAI's chat model > > > even > > > > > though useful, they tend to hallucinate with confidence often. > > > > > > > > > > Thanks > > > > > > > > > > > > >
Re: Cassandra Clients: Stale Cluster Topology
Also: nodetool getendpoints just hashes the key you provide against the cluster topology / schema definition, which tells you which nodes WOULD own the data if it exists. It does NOT guarantee that it exists. On 2025/10/10 17:48:32 Jeff Jirsa wrote: > You're using a 9 year old release. There have been literally hundreds of > correctness fixes over those 9 years. You need to upgrade. > > The rest of your answers inline. > > > > On 2025/10/10 12:56:58 FMH wrote: > > Few times a week, our developers report that Cassandra retrieves are > > coming back with zero rows. No error messages. > > > > Using the same item ID's, a CQLSH SELECT statement returns a single row as > > expected. Furthermore, the NODETOOL GETENDPOINTS returns three IP's as we > > expect. > > > > This confirms these ItemID's do exist in Cassandra, it is just the Java > > clients are not retrieving it. > > > > We noticed this issue to present itself more when nodes are replaced in the > > cluster as a result of EC2 node deprecation. > > Are you using EBS or ephemeral disk? Don't use ephemeral disk unless you are > much better at running cassandra and know how to replace a node without data > loss (which you do not seem to know how to do). > > > > > > Once the developers restarted the Java client apps, it was now able to > > retrieve these ItemID's. > > That sounds weird. It may be that they read repaired or normal-repaired, or > it may be that the java apps were pointing to the wrong thing/cluster. > > > > > 1- Is this what is called the 'empty' read' behavior? > > 2- Is this caused by clients topology metadata getting out of sync with the > > cluster? > > Could be cluster scaling unsafely due to ec2 events. > Could be low consistency level > Could be any number of hundreds of topology bugs fixed since 2016. > > If it's a client bug, I assume it's an old client bug I've never seen before. > Well functioning cassandra clients shouldn't care about the topology, the > coordinating server will forward the request anyway. > > > 3- How can this be detected? Should we have client drivers return 'metadata > > = cluster.metadata' and compare it to 'nodetool gossipinfo'? > > Upgrade your cluster. > Use EBS so when nodes change, they don't change data ownership. > > > 4- Other than restarting the clients, is there a way to have client apps to > > force to refresh their ring metadata? > > > > The client apps are using 'com.datastax.oss:java-driver-core:4.13.0' > > driver. > > > > Google returns little information about this and GenAI's chat model even > > though useful, they tend to hallucinate with confidence often. > > > > Thanks > > > > > > Thank you > > >
Re: Cassandra Clients: Stale Cluster Topology
Forgot to mention we are using C* 2.2.8. On Fri, Oct 10, 2025 at 8:56 AM FMH wrote: > Few times a week, our developers report that Cassandra retrieves are > coming back with zero rows. No error messages. > > Using the same item ID's, a CQLSH SELECT statement returns a single row as > expected. Furthermore, the NODETOOL GETENDPOINTS returns three IP's as we > expect. > > This confirms these ItemID's do exist in Cassandra, it is just the Java > clients are not retrieving it. > > We noticed this issue to present itself more when nodes are replaced in > the cluster as a result of EC2 node deprecation. > > Once the developers restarted the Java client apps, it was now able to > retrieve these ItemID's. > > 1- Is this what is called the 'empty' read' behavior? > 2- Is this caused by clients topology metadata getting out of sync with > the cluster? > 3- How can this be detected? Should we have client drivers return ' > metadata = cluster.metadata' and compare it to 'nodetool gossipinfo'? > 4- Other than restarting the clients, is there a way to have client apps > to force to refresh their ring metadata? > > The client apps are using 'com.datastax.oss:java-driver-core:4.13.0' > driver. > > Google returns little information about this and GenAI's chat model even > though useful, they tend to hallucinate with confidence often. > > Thanks > > > Thank you > > > -- Thank you
Re: Cassandra Clients: Stale Cluster Topology
Correct yes, apologies, I was typing this on my phone while watching TV. If the new, joining node chooses a node in the ring that missed a write, then 2 nodes are missing the write, and reads at quorum/local_quorum do not see the data. FWIW, the "data" they see may also be a tombstone / deletion marker, which means you could ALSO "resurrect" deleted data with this type of operation, just in case that isn't obvious. "Empty" reads are at least easy to detect, "partial" or "resurrected" reads are harder to detect. On 2025/10/11 13:36:08 dbms-tech wrote: > In your statement ... > If it chooses B, now D still has the data, but B and C don’t. So a read may > choose: > > I think, you meant to say > If it chooses C, now D still has the data, but B and C don’t. So a read may > choose: > > Thanks > > On Sat, Oct 11, 2025 at 12:04 AM Jeff Jirsa wrote: > > > No. It’s actually almost a guarantee that what’s happening is that you > > violate consistency on node replacement and the read at ALL copies the data > > back to where it needed to be > > > > Here’s what’s happening > > > > Let’s pretend you have 6 nodes, A-F > > > > When you write at LQ for a key, let’s pretend it goes to B, C, and D > > > > 2 of the three of those have to ack a write - let’s say it goes to B and > > D, but misses C. Write succeeds, reads will see it because any read will be > > either B and D (easy), B and C (data on B, visible, copies to C on read) or > > C and D (data on D, visible, copies to C on read) > > > > Now imagine that before C gets the data by repair or read repair B fails > > and gets sent for maintenance by your cloud provider > > > > A new B’ gets added to the ring. B’ has to choose about to send it data. > > It’s going to choose either C or D. > > > > If it chooses D, no problem, reads still always see data > > > > If it chooses B, now D still has the data, but B and C don’t. So a read > > may choose: > > > > B/C - data missing > > B/D - repairs to B from D > > C/D - repairs to C from D > > > > If you do ALL, it repairs to B and C from D > > > > If you use EBS instead of re-bootstrapping the data, the data drive can > > get re-attached and B still has the data. > > > > Alternatively, if you had a newer version, you could just run incremental > > repair “often and this would happen much less often (or never, if you force > > incremental repair to run before the bootstrap happens). Incremental repair > > in 2.2 is not good enough - don’t try to use it until you upgrade > > > > > > > > > > On 2025/10/10 19:38:08 FMH wrote: > > > Data read/written with CL=LQ. > > > > > > To attempt to isolate the issue on the C* server vs. client, running a > > > SELECT statement with CL=ALL returned the row 200 times with zero misses. > > > > > > Retrieving the same ID through the java service had a 30% failure rate. > > > > > > Isn't this conclusive enough that data does exist & has three replicas. > > > The issue must be isolated to the client? > > > > > > On Fri, Oct 10, 2025 at 2:47 PM Jeff Jirsa wrote: > > > > > > > What consistency level are you using on reads and writes? > > > > > > > > If either are less than LOCAL_QUORUM, this behavior is definitely > > expected. > > > > > > > > If you ARE using quorum/local_quorum, and you can correlate these > > issues > > > > to when a node scales in / out, then it's probably consistency > > violation on > > > > bootstrap. Unless you run repair before you re-bootstrap a node > > (assuming > > > > you're using ephemeral disks), the bootstrap process may choose a > > streaming > > > > source that is missing ~1 write, and then you end up with 2 replicas > > > > without the write, so reads dont see it until chance read repair or > > repair > > > > is run. > > > > > > > > > > > > > > > > On 2025/10/10 18:30:41 FMH wrote: > > > > > Thanks for taking the time to Respond, Jeff. > > > > > > > > > > I was kind expecting this reply the minute I included version 2.2.8. > > We > > > > are > > > > > in the process of upgrading to 5. > > > > > > > > > > As for the bootstrap automation we use. It has been in effect for > > more > > > > than > > > > > 10 years and we replaced 100's of nodes without ever having any > > issues > > > > > including this one. > > > > > > > > > > This automation we put in place based on the available documentation > > for > > > > > Apache and Datastax cassandra. We have also had it assessed several > > times > > > > > over the years by external consultants. > > > > > > > > > > Thanks for clarifying the getendpoints. This is why we paired it > > with the > > > > > SELECT statement validation test as well to verify data. > > > > > > > > > > On Fri, Oct 10, 2025 at 1:54 PM Jeff Jirsa > > wrote: > > > > > > > > > > > Also: nodetool getendpoints just hashes the key you provide > > against the > > > > > > cluster topology / schema definition, which tells you which nodes > > > > WOULD own > > > > > > the data if it exists. It does NOT guarantee that it exists. > > > > > > > > > >
Re: Cassandra Clients: Stale Cluster Topology
You're using a 9 year old release. There have been literally hundreds of correctness fixes over those 9 years. You need to upgrade. The rest of your answers inline. On 2025/10/10 12:56:58 FMH wrote: > Few times a week, our developers report that Cassandra retrieves are > coming back with zero rows. No error messages. > > Using the same item ID's, a CQLSH SELECT statement returns a single row as > expected. Furthermore, the NODETOOL GETENDPOINTS returns three IP's as we > expect. > > This confirms these ItemID's do exist in Cassandra, it is just the Java > clients are not retrieving it. > > We noticed this issue to present itself more when nodes are replaced in the > cluster as a result of EC2 node deprecation. Are you using EBS or ephemeral disk? Don't use ephemeral disk unless you are much better at running cassandra and know how to replace a node without data loss (which you do not seem to know how to do). > > Once the developers restarted the Java client apps, it was now able to > retrieve these ItemID's. That sounds weird. It may be that they read repaired or normal-repaired, or it may be that the java apps were pointing to the wrong thing/cluster. > > 1- Is this what is called the 'empty' read' behavior? > 2- Is this caused by clients topology metadata getting out of sync with the > cluster? Could be cluster scaling unsafely due to ec2 events. Could be low consistency level Could be any number of hundreds of topology bugs fixed since 2016. If it's a client bug, I assume it's an old client bug I've never seen before. Well functioning cassandra clients shouldn't care about the topology, the coordinating server will forward the request anyway. > 3- How can this be detected? Should we have client drivers return 'metadata > = cluster.metadata' and compare it to 'nodetool gossipinfo'? Upgrade your cluster. Use EBS so when nodes change, they don't change data ownership. > 4- Other than restarting the clients, is there a way to have client apps to > force to refresh their ring metadata? > > The client apps are using 'com.datastax.oss:java-driver-core:4.13.0' > driver. > > Google returns little information about this and GenAI's chat model even > though useful, they tend to hallucinate with confidence often. > > Thanks > > > Thank you >
