I think this is a regression introduced as part of CASSANDRA-17248 <https://issues.apache.org/jira/browse/CASSANDRA-17248>, the following code got introduced in QueryProcessor.java <https://github.com/apache/cassandra/commit/242f7f9b18db77bce36c9bba00b2acda4ff3209e#r137491766> since C* 3.0.26.
// Make sure the missing one is going to be eventually re-prepared evictPrepared(hashWithKeyspace); evictPrepared(hashWithoutKeyspace); The above-mentioned code was not present in C* 3.0.25 <https://github.com/apache/cassandra/blob/cassandra-3.0.25/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L391> . Alex, could you please take a look at it? Jaydeep On Fri, Jan 19, 2024 at 7:17 PM Jaydeep Chovatia <chovatia.jayd...@gmail.com> wrote: > Hi, > > Today, in our production, we came across the following scenario: > > 1. We have 100 nodes of the Cassandra cluster on 4.0.6, and our client > uses PreparedStatement, say, "*SELECT * FROM T1 WHERE PK=?*" > 2. We applied a schema change to add a *regular* column, "*ALTER TABLE > T1 ADD COLUMN c1 UUID"* > 3. Around 20 (out of 100) of the Cassandra nodes started throwing the > error: "*Successfully prepared, but could not find prepared statement > for* " (Code path: QueryEvents.java > > <https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryEvents.java#L225C80-L225C145> > ) > 4. The error continued for around 2 hours, and only restarting those > 20 nodes resolved the issue. > > Our current hypothesis is that there is a race condition in the > QueryProcessor::prepare > <https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L575> > API in that one thread is evicting the prepared statements while the other > is adding, which is never-ending. A similar hypothesis has been mentioned > in a ticket in 2022: https://issues.apache.org/jira/browse/CASSANDRA-17401 > > Has anyone ever experienced this? Are there any quick pointers on what > could have gone wrong? > > Thanks in advance! > > Jaydeep >
