[
https://issues.apache.org/jira/browse/AMBARI-26640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jialiang Cai reassigned AMBARI-26640:
-------------------------------------
Assignee: Jialiang Cai
> Prevent auto-recovery from blocking cluster start operations
> ------------------------------------------------------------
>
> Key: AMBARI-26640
> URL: https://issues.apache.org/jira/browse/AMBARI-26640
> Project: Ambari
> Issue Type: Bug
> Reporter: Jialiang Cai
> Assignee: Jialiang Cai
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h2. Problem
> Ambari Agent auto-recovery commands use the same {{ActionQueue}} as
> server-issued execution commands, but recovery work is generated locally and
> is not represented in the server request graph or shown in the UI. A queued
> or running recovery command can therefore occupy the Agent while an operator
> starts the cluster or uses {{Start All Services}}. The page appears stuck
> even though the server operation is waiting behind hidden Agent work.
> Auto-recovery currently decides whether to start or restart a component from
> only that component local desired and current state. It does not honor the
> cluster {{RoleCommandOrder}} used by normal cluster start requests. After a
> server restart, Agent reconnect, heartbeat loss, or concurrent component
> transition, cached state can also be stale or delivered out of order. This
> can start downstream components before their dependencies are ready, or
> repeatedly schedule recovery while an operator is trying to control the
> cluster.
> h2. Reproduction Scenario
> # Enable auto-recovery for one or more components.
> # Stop or fail a component so the Agent queues or starts a local recovery
> command.
> # From Ambari, run {{Start All Services}} or another server-issued command on
> the same host.
> # Observe that the visible request waits while the local recovery task is not
> visible in the request graph.
> # Restart Ambari Server or reconnect an Agent while component states are
> changing. The Agent can make recovery decisions from a snapshot that does not
> identify its server lifecycle, version, completeness, or reporting Agent
> session.
> h2. Root Causes
> * Server-issued execution commands do not preempt queued or active
> auto-recovery commands. There is also a dequeue-to-active race where recovery
> can start immediately before a server command reaches the queue.
> * Recovery configuration contains enabled components and desired state, but
> no dependency graph or runtime dependency state.
> * The Agent cannot distinguish a complete current-session component snapshot
> from cached status left by an earlier connection.
> * Recovery topology updates have no server epoch or monotonic version, so
> delayed updates can overwrite newer state.
> * Status acknowledgements and cached reports can cross a reconnect boundary
> unless they are associated with a snapshot generation.
> h2. Proposed Behavior
> * A server-issued {{EXECUTION_COMMAND}} must take priority over Agent
> auto-recovery. Queued recovery commands should be removed and active recovery
> commands should be cancelled before the server command executes. Recovery
> must also yield when a server command wins the dequeue-to-active race.
> * Ambari Server should derive recovery dependencies from the same transitive
> {{RoleCommandOrder}} used by the stage planner. Each dependency must be
> expanded to real component instances and include service, component, host,
> current state, desired state, required state, and freshness.
> * The topology sent to Agents must include a server epoch, a monotonic
> version, and a completeness flag. Cached server topology must not regress to
> an older version in the same epoch.
> * Component state freshness must be scoped to the active Agent session.
> Registration invalidates the previous snapshot, a complete component-status
> scan makes the new session fresh, heartbeat loss invalidates it again, and
> reports from superseded sessions are ignored.
> * Agent {{START}} and {{RESTART}} recovery must remain blocked until the
> topology is complete and every dependency is fresh and satisfies its required
> state. A {{STARTED}} component also satisfies an {{INSTALLED}} dependency.
> Non-start recovery behavior should remain compatible with the existing
> recovery mode.
> * Agent reconnect must clear the old reported-status cache before exposing
> the new registered connection. Acknowledgements from an earlier snapshot
> generation must not complete or repopulate the new snapshot.
> h2. Acceptance Criteria
> # {{Start All Services}} and other server-issued operations are not blocked
> behind queued or running auto-recovery work on an Agent.
> # Auto-recovery {{START}} and {{RESTART}} follow transitive cluster startup
> ordering across hosts and services.
> # Recovery does not run from incomplete, stale, superseded-session, or
> out-of-order topology data.
> # Dependency state changes are republished and eventually unblock downstream
> recovery after upstream components become ready.
> # Server restart, Agent reconnect, and heartbeat loss conservatively block
> recovery until a fresh complete status snapshot is available.
> # Existing Agents ignore the additive topology fields, and updated Agents
> retain compatibility with recovery configuration from an older server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
