Yuntong Qu created CASSJAVA-142:
-----------------------------------
Summary: MetadataManager: schema refresh never runs again after
one refresh fails with "Control channel not available"
Key: CASSJAVA-142
URL: https://issues.apache.org/jira/browse/CASSJAVA-142
Project: Apache Cassandra Java driver
Issue Type: Task
Reporter: Yuntong Qu
Assignee: Yuntong Qu
{{MetadataManager.SingleThreaded.startSchemaRequest}} sets
{{currentSchemaRefresh}} when a refresh starts, but only the success path
clears it. Two failure paths fail the caller's future and leave it set:
# {{schemaQueriesFactory.newInstance()}} throws
{{IllegalStateException("Control channel not available, aborting schema
refresh")}} (catch block from CASSJAVA-13).
# {{initControlConnectionForSchema()}} or {{checkSchemaAgreement()}} fails
({{agreementError}} branch).
After that, every later refresh waits in {{queuedSchemaRefresh}} until the
session closes. CASSJAVA-13 made the first caller fail fast; it did not clear
{{currentSchemaRefresh}}, so all later callers still hang.
h3. Impact
* {{refreshSchemaAsync()}} never completes, and schema metadata stays frozen.
* Every later DDL fails with {{DriverTimeoutException}}, although the server
applied it.
* Only a new session recovers. Spark Cassandra Connector apps hang, because the
connector calls a blocking {{refreshSchema()}} on each table lookup.
Seen in production with driver 4.18.1 (Spark Cassandra Connector 3.5.1) during
node maintenance.
h3. Reproduce
Single-node Cassandra 5.0: connect, stop the node, call {{refreshSchema()}},
start the node, call {{refreshSchema()}} again, run a {{CREATE TABLE}}.
||Driver||Refresh after node is back||DDL after node is back||
|4.18.1, 4.19.3 (3 runs each)|no completion in 30 s|{{DriverTimeoutException}};
table missing from driver metadata|
|4.18.1 + proposed fix (3 runs)|completes|completes|
h3. Proposed fix
Add one method that clears {{currentSchemaRefresh}} and starts the queued
refresh. Call it from all three completion paths as a separate task on the
admin executor ({{adminExecutor.execute}}). The failure can complete on an I/O
thread, and running the cleanup inline could recurse when a caller retries from
the failure callback. The PR includes unit tests for both failure paths.
h3. Related
* CASSJAVA-13: partial fix (first caller only).
* [JAVA-3059|https://datastax-oss.atlassian.net/browse/JAVA-3059]: original
report (legacy DataStax JIRA, open). Its 2023-05-19 comment points at the
refresh queue.
* [SPARKC-636|https://datastax-oss.atlassian.net/browse/SPARKC-636],
[SPARKC-669|https://datastax-oss.atlassian.net/browse/SPARKC-669]: Spark apps
hang after a node stops; likely the same bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
