[
https://issues.apache.org/jira/browse/CASSJAVA-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta updated CASSJAVA-142:
---------------------------------
Change Category: Semantic
Complexity: Normal
Status: Open (was: Triage Needed)
> MetadataManager: schema refresh never runs again after one refresh fails with
> "Control channel not available"
> -------------------------------------------------------------------------------------------------------------
>
> Key: CASSJAVA-142
> URL: https://issues.apache.org/jira/browse/CASSJAVA-142
> Project: Apache Cassandra Java driver
> Issue Type: Task
> Components: Core
> Reporter: Yuntong Qu
> Assignee: Yuntong Qu
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{MetadataManager.SingleThreaded.startSchemaRequest}} sets
> {{currentSchemaRefresh}} when a refresh starts, but only the success path
> clears it. Two failure paths fail the caller's future and leave it set:
> # {{schemaQueriesFactory.newInstance()}} throws
> {{IllegalStateException("Control channel not available, aborting schema
> refresh")}} (catch block from CASSJAVA-13).
> # {{initControlConnectionForSchema()}} or {{checkSchemaAgreement()}} fails
> ({{agreementError}} branch).
> After that, every later refresh waits in {{queuedSchemaRefresh}} until the
> session closes. CASSJAVA-13 made the first caller fail fast; it did not clear
> {{currentSchemaRefresh}}, so all later callers still hang.
> h3. Impact
> * {{refreshSchemaAsync()}} never completes, and schema metadata stays frozen.
> * Every later DDL fails with {{DriverTimeoutException}}, although the server
> applied it.
> * Only a new session recovers. Spark Cassandra Connector apps hang, because
> the connector calls a blocking {{refreshSchema()}} on each table lookup.
> Seen in production with driver 4.18.1 (Spark Cassandra Connector 3.5.1)
> during node maintenance.
> h3. Reproduce
> Single-node Cassandra 5.0: connect, stop the node, call {{refreshSchema()}},
> start the node, call {{refreshSchema()}} again, run a {{CREATE TABLE}}.
> ||Driver||Refresh after node is back||DDL after node is back||
> |4.18.1, 4.19.3 (3 runs each)|no completion in 30
> s|{{DriverTimeoutException}}; table missing from driver metadata|
> |4.18.1 + proposed fix (3 runs)|completes|completes|
> h3. Proposed fix
> Add one method that clears {{currentSchemaRefresh}} and starts the queued
> refresh. Call it from all three completion paths as a separate task on the
> admin executor ({{adminExecutor.execute}}). The failure can complete on an
> I/O thread, and running the cleanup inline could recurse when a caller
> retries from the failure callback. The PR includes unit tests for both
> failure paths.
> h3. Related
> * CASSJAVA-13: partial fix (first caller only).
> * [JAVA-3059|https://datastax-oss.atlassian.net/browse/JAVA-3059]: original
> report (legacy DataStax JIRA, open). Its 2023-05-19 comment points at the
> refresh queue.
> * [SPARKC-636|https://datastax-oss.atlassian.net/browse/SPARKC-636],
> [SPARKC-669|https://datastax-oss.atlassian.net/browse/SPARKC-669]: Spark apps
> hang after a node stops; likely the same bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
