Few times a week, our developers report that Cassandra retrieves are coming back with zero rows. No error messages.
Using the same item ID's, a CQLSH SELECT statement returns a single row as expected. Furthermore, the NODETOOL GETENDPOINTS returns three IP's as we expect. This confirms these ItemID's do exist in Cassandra, it is just the Java clients are not retrieving it. We noticed this issue to present itself more when nodes are replaced in the cluster as a result of EC2 node deprecation. Once the developers restarted the Java client apps, it was now able to retrieve these ItemID's. 1- Is this what is called the 'empty' read' behavior? 2- Is this caused by clients topology metadata getting out of sync with the cluster? 3- How can this be detected? Should we have client drivers return 'metadata = cluster.metadata' and compare it to 'nodetool gossipinfo'? 4- Other than restarting the clients, is there a way to have client apps to force to refresh their ring metadata? The client apps are using 'com.datastax.oss:java-driver-core:4.13.0' driver. Google returns little information about this and GenAI's chat model even though useful, they tend to hallucinate with confidence often. Thanks ---------------------------------------- Thank you
