Andrés Beck-Ruiz created CASSSIDECAR-485:
--------------------------------------------
Summary: Fail operational job nodes whose owning Sidecar is
permanently down
Key: CASSSIDECAR-485
URL: https://issues.apache.org/jira/browse/CASSSIDECAR-485
Project: Sidecar for Apache Cassandra
Issue Type: Improvement
Reporter: Andrés Beck-Ruiz
CASSSIDECAR-378 adds a per-node execution timeout so a {{RUNNING}} node always
reaches a terminal state and cannot wedge an operation. That timeout is
enforced by the node's co-located Sidecar (only it holds the local job handle
and drives the node), so it only covers the case where that Sidecar is alive.
If the owning Sidecar goes down and does not return, nothing enforces the
bound: a node that never started stays CREATED, and a node that was
mid-execution stays RUNNING, indefinitely. Because finalization requires every
node in the execution order to be terminal, the operation never finalizes and
the datacenter-scoped active operation lock for that operation type is never
released, wedging all future operations of that type.
This is not solvable with a timeout, as a peer Sidecar cannot safely assume
another node's work has stopped just because time has passed (the owner may be
slow, GC-paused, or network-partitioned rather than dead). Closing the gap
requires a node-liveness mechanism so that some other participant can determine
an owning Sidecar is genuinely gone and mark its node FAILED, allowing the
operation to finalize and release the lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
