You're using a 9 year old release. There have been literally hundreds of correctness fixes over those 9 years. You need to upgrade.
The rest of your answers inline. On 2025/10/10 12:56:58 FMH wrote: > Few times a week, our developers report that Cassandra retrieves are > coming back with zero rows. No error messages. > > Using the same item ID's, a CQLSH SELECT statement returns a single row as > expected. Furthermore, the NODETOOL GETENDPOINTS returns three IP's as we > expect. > > This confirms these ItemID's do exist in Cassandra, it is just the Java > clients are not retrieving it. > > We noticed this issue to present itself more when nodes are replaced in the > cluster as a result of EC2 node deprecation. Are you using EBS or ephemeral disk? Don't use ephemeral disk unless you are much better at running cassandra and know how to replace a node without data loss (which you do not seem to know how to do). > > Once the developers restarted the Java client apps, it was now able to > retrieve these ItemID's. That sounds weird. It may be that they read repaired or normal-repaired, or it may be that the java apps were pointing to the wrong thing/cluster. > > 1- Is this what is called the 'empty' read' behavior? > 2- Is this caused by clients topology metadata getting out of sync with the > cluster? Could be cluster scaling unsafely due to ec2 events. Could be low consistency level Could be any number of hundreds of topology bugs fixed since 2016. If it's a client bug, I assume it's an old client bug I've never seen before. Well functioning cassandra clients shouldn't care about the topology, the coordinating server will forward the request anyway. > 3- How can this be detected? Should we have client drivers return 'metadata > = cluster.metadata' and compare it to 'nodetool gossipinfo'? Upgrade your cluster. Use EBS so when nodes change, they don't change data ownership. > 4- Other than restarting the clients, is there a way to have client apps to > force to refresh their ring metadata? > > The client apps are using 'com.datastax.oss:java-driver-core:4.13.0' > driver. > > Google returns little information about this and GenAI's chat model even > though useful, they tend to hallucinate with confidence often. > > Thanks > > ---------------------------------------- > Thank you >
