[
https://issues.apache.org/jira/browse/AMBARI-26640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jialiang Cai updated AMBARI-26640:
----------------------------------
Description:
h2. Problem
Ambari Agent auto-recovery commands use the same ActionQueue as server-issued
execution commands, but recovery work is generated locally and is not
represented in the server request graph or shown in the UI. A queued or running
recovery command can therefore occupy the Agent while an operator starts the
cluster or uses Start All Services. The page appears stuck even though the
server operation is waiting behind hidden Agent work.
Auto-recovery currently decides whether to start or restart a component from
that component's local desired and current state only. It does not honor the
cluster RoleCommandOrder used by normal cluster start requests. After a server
restart, Agent reconnect, heartbeat loss, or concurrent component transition,
cached state can also be stale or delivered out of order. This can start
downstream components before their dependencies are ready, or repeatedly
schedule recovery while an operator is trying to control the cluster.
h2. Reproduction scenario
h1. Enable auto-recovery for one or more components.
h1. Stop or fail a component so the Agent queues or starts a local recovery
command.
h1. From Ambari, run Start All Services or another server-issued command on the
same host.
h1. Observe that the visible request waits while the local recovery task is not
visible in the request graph.
h1. Restart Ambari Server or reconnect an Agent while component states are
changing. The Agent can make recovery decisions from a snapshot that does not
identify its server lifecycle, version, completeness, or reporting Agent
session.
h2. Root causes
* Server-issued execution commands do not preempt queued or active
auto-recovery commands. There is also a dequeue-to-active race where recovery
can start immediately before a server command reaches the queue.
* Recovery configuration contains enabled components and desired state, but no
dependency graph or runtime dependency state.
* The Agent cannot distinguish a complete current-session component snapshot
from cached status left by an earlier connection.
* Recovery topology updates have no server epoch or monotonic version, so
delayed updates can overwrite newer state.
* Status acknowledgements and cached reports can cross a reconnect boundary
unless they are associated with a snapshot generation.
h2. Proposed behavior
* A server-issued EXECUTION_COMMAND must take priority over Agent
auto-recovery. Queued recovery commands should be removed and active recovery
commands should be cancelled before the server command executes. Recovery must
also yield when a server command wins the dequeue-to-active race.
* Ambari Server should derive recovery dependencies from the same transitive
RoleCommandOrder used by the stage planner. Each dependency must be expanded to
real component instances and include service, component, host, current state,
desired state, required state, and freshness.
* The topology sent to Agents must include a server epoch, a monotonic version,
and a completeness flag. Cached server topology must not regress to an older
version in the same epoch.
* Component state freshness must be scoped to the active Agent session.
Registration invalidates the previous snapshot, a complete component-status
scan makes the new session fresh, heartbeat loss invalidates it again, and
reports from superseded sessions are ignored.
* Agent START and RESTART recovery must remain blocked until the topology is
complete and every dependency is fresh and satisfies its required state. A
STARTED component also satisfies an INSTALLED dependency. Non-start recovery
behavior should remain compatible with the existing recovery mode.
* Agent reconnect must clear the old reported-status cache before exposing the
new registered connection. Acknowledgements from an earlier snapshot generation
must not complete or repopulate the new snapshot.
h2. Acceptance criteria
h1. Start All Services and other server-issued operations are not blocked
behind queued or running auto-recovery work on an Agent.
h1. Auto-recovery START and RESTART follow transitive cluster startup ordering
across hosts and services.
h1. Recovery does not run from incomplete, stale, superseded-session, or
out-of-order topology data.
h1. Dependency state changes are republished and eventually unblock downstream
recovery after upstream components become ready.
h1. Server restart, Agent reconnect, and heartbeat loss conservatively block
recovery until a fresh complete status snapshot is available.
h1. Existing Agents ignore the additive topology fields, and updated Agents
retain compatibility with recovery configuration from an older server.
was:
Agent-side auto-recovery commands are hidden from users and share the action
queue with server-issued operations. When recovery work is queued or running,
Start All Services can appear stuck without showing the blocking task.
Auto-recovery also evaluates only local component state. It can restart a
component before the transitive RoleCommandOrder dependencies are ready,
particularly after server or agent restarts when cached component status is
stale.
Auto-recovery should yield to server-issued commands and follow a versioned,
stateful cluster startup topology. Component state freshness must be tied to
the active agent session and invalidated on registration or heartbeat loss so
stale or out-of-order topology snapshots cannot trigger recovery.
> Prevent auto-recovery from blocking cluster start operations
> ------------------------------------------------------------
>
> Key: AMBARI-26640
> URL: https://issues.apache.org/jira/browse/AMBARI-26640
> Project: Ambari
> Issue Type: Bug
> Reporter: Jialiang Cai
> Priority: Major
>
> h2. Problem
> Ambari Agent auto-recovery commands use the same ActionQueue as server-issued
> execution commands, but recovery work is generated locally and is not
> represented in the server request graph or shown in the UI. A queued or
> running recovery command can therefore occupy the Agent while an operator
> starts the cluster or uses Start All Services. The page appears stuck even
> though the server operation is waiting behind hidden Agent work.
> Auto-recovery currently decides whether to start or restart a component from
> that component's local desired and current state only. It does not honor the
> cluster RoleCommandOrder used by normal cluster start requests. After a
> server restart, Agent reconnect, heartbeat loss, or concurrent component
> transition, cached state can also be stale or delivered out of order. This
> can start downstream components before their dependencies are ready, or
> repeatedly schedule recovery while an operator is trying to control the
> cluster.
> h2. Reproduction scenario
> h1. Enable auto-recovery for one or more components.
> h1. Stop or fail a component so the Agent queues or starts a local recovery
> command.
> h1. From Ambari, run Start All Services or another server-issued command on
> the same host.
> h1. Observe that the visible request waits while the local recovery task is
> not visible in the request graph.
> h1. Restart Ambari Server or reconnect an Agent while component states are
> changing. The Agent can make recovery decisions from a snapshot that does not
> identify its server lifecycle, version, completeness, or reporting Agent
> session.
> h2. Root causes
> * Server-issued execution commands do not preempt queued or active
> auto-recovery commands. There is also a dequeue-to-active race where recovery
> can start immediately before a server command reaches the queue.
> * Recovery configuration contains enabled components and desired state, but
> no dependency graph or runtime dependency state.
> * The Agent cannot distinguish a complete current-session component snapshot
> from cached status left by an earlier connection.
> * Recovery topology updates have no server epoch or monotonic version, so
> delayed updates can overwrite newer state.
> * Status acknowledgements and cached reports can cross a reconnect boundary
> unless they are associated with a snapshot generation.
> h2. Proposed behavior
> * A server-issued EXECUTION_COMMAND must take priority over Agent
> auto-recovery. Queued recovery commands should be removed and active recovery
> commands should be cancelled before the server command executes. Recovery
> must also yield when a server command wins the dequeue-to-active race.
> * Ambari Server should derive recovery dependencies from the same transitive
> RoleCommandOrder used by the stage planner. Each dependency must be expanded
> to real component instances and include service, component, host, current
> state, desired state, required state, and freshness.
> * The topology sent to Agents must include a server epoch, a monotonic
> version, and a completeness flag. Cached server topology must not regress to
> an older version in the same epoch.
> * Component state freshness must be scoped to the active Agent session.
> Registration invalidates the previous snapshot, a complete component-status
> scan makes the new session fresh, heartbeat loss invalidates it again, and
> reports from superseded sessions are ignored.
> * Agent START and RESTART recovery must remain blocked until the topology is
> complete and every dependency is fresh and satisfies its required state. A
> STARTED component also satisfies an INSTALLED dependency. Non-start recovery
> behavior should remain compatible with the existing recovery mode.
> * Agent reconnect must clear the old reported-status cache before exposing
> the new registered connection. Acknowledgements from an earlier snapshot
> generation must not complete or repopulate the new snapshot.
> h2. Acceptance criteria
> h1. Start All Services and other server-issued operations are not blocked
> behind queued or running auto-recovery work on an Agent.
> h1. Auto-recovery START and RESTART follow transitive cluster startup
> ordering across hosts and services.
> h1. Recovery does not run from incomplete, stale, superseded-session, or
> out-of-order topology data.
> h1. Dependency state changes are republished and eventually unblock
> downstream recovery after upstream components become ready.
> h1. Server restart, Agent reconnect, and heartbeat loss conservatively block
> recovery until a fresh complete status snapshot is available.
> h1. Existing Agents ignore the additive topology fields, and updated Agents
> retain compatibility with recovery configuration from an older server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
