[
https://issues.apache.org/jira/browse/YARN-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882510#comment-17882510
]
ASF GitHub Bot commented on YARN-11730:
---------------------------------------
arjunmohnot commented on code in PR #7049:
URL: https://github.com/apache/hadoop/pull/7049#discussion_r1763997947
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java:
##########
@@ -1608,9 +1608,19 @@ protected void serviceStart() throws Exception {
// Non HA case, start after RM services are started.
if (!this.rmContext.isHAEnabled()) {
transitionToActive();
+
+ // Refresh node state at the service startup to reflect the unregistered
+ // nodemanagers as LOST if the tracking for unregistered nodes flag is
enabled.
+ // For HA setup, refreshNodes is already being called during the
transition.
+ Configuration yarnConf = getConfig();
+ if (yarnConf.getBoolean(
+ YarnConfiguration.ENABLE_TRACKING_FOR_UNREGISTERED_NODES,
+ YarnConfiguration.DEFAULT_ENABLE_TRACKING_FOR_UNREGISTERED_NODES)) {
+ this.rmContext.getNodesListManager().refreshNodes(yarnConf);
Review Comment:
Hey @zeekling, thanks for your question! After reviewing potential edge
cases and comparing the existing implementation, here’s a summary of the
different scenarios:
### Unregistered Lost Node Definition
- A node is marked as _LOST_ when listed in the _"include"_ file but not
registered in the ResourceManager's active node context, and is also not part
of the _"exclude"_ file during startup or HA failover. The unregistered lost
node is indicated by a port value of -2.
### **Case 1**: Node Marked as LOST, Heartbeat Received
- When a node is marked as **LOST**, a lost event is dispatched, adding the
node to the **active** and **inactive** node maps of the RMContext.
- If the node sends a heartbeat afterward, the transition method in
`RMNodeImpl` checks for the same hostname of that node with **port -2** (LOST
state).
- If found, the nodeID is removed from the `rmContext` and re-registered
with the desired port of the NM.
- It also decrements the LOST node counter and increments the ACTIVE node
counter, ensuring a clean state of transitions.
### **Case 2**: **Rare Scenario** - Race Condition
- A race condition may occur if the **ResourceTrackerService** starts before
the RM starts processing the unregistered lost nodes, and the NodeManager (NM)
sends its heartbeat **quickly** in parallel.
- **Example**:
- When fetching nodes from `rmContext`, an NM (say **host1**) may not
initially be present in the context.
- Before this operation completes, **host1** may send a heartbeat and
get registered with a valid port.
- Meanwhile, the RM could still attempt to mark the same NM (host1) as
LOST with port -2 as it was not registered while querying the context,
resulting in two entries for the same host: one as ACTIVE and another as LOST.
### Details on Case-2
- I reviewed the code and found that for a node to register, the
`ResourceTracker` service must start during service startup. In HA mode, nodes
only register once the RM becomes active.
- The current implementation for HA calls the `refreshNodes` function before
the `transitionToActive` method, which rules out the race condition for HA
setup since all unregistered nodes are dispatched first.
For standalone setups, there was a slight oversight. However, during
testing, I did not encounter or replicate this issue, as the NM heartbeat can
take time to register while the nodes were marked as LOST beforehand. With the
recent changes, I am making the `refreshNodes` operation before the service
starts. This change ensures that unregistered nodes are consistently marked as
LOST with **port -2** first. Only afterwards NMs can register themselves with a
proper port during heartbeat reception when the `ResourceTrackerService` starts
the server (Triggered during RM service start).
### Conclusion
The updated change should guarantee that the nodes are properly marked as
LOST **before** any heartbeats are processed. This eliminates the chance of
falsely reporting nodes as LOST. I’ve validated this behavior through logs,
which show that the lost event is dispatched first, and then the heartbeats
from NMs received after service start.
Let me know if this clears things up! 😊
> Resourcemanager node reporting enhancement for unregistered hosts
> -----------------------------------------------------------------
>
> Key: YARN-11730
> URL: https://issues.apache.org/jira/browse/YARN-11730
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, yarn
> Affects Versions: 3.4.0
> Environment: Tested on multiple environments:
> A. Docker Environment{*}:{*}
> * Base OS: *Ubuntu 20.04*
> * *Java 8* installed from OpenJDK.
> * Docker image includes Hadoop binaries, user configurations, and ports for
> YARN services.
> * Verified behavior using a Hadoop snapshot in a containerized environment.
> * Performed Namenode formatting and validated service interactions through
> exposed ports.
> * Repo reference:
> [arjunmohnot/hadoop-yarn-docker|https://github.com/arjunmohnot/hadoop-yarn-docker/tree/main]
> B. Bare-metal Distributed Setup (RedHat Linux){*}:{*}
> * Running *Java 8* in a High-Availability (HA) configuration with
> *Zookeeper* for locking mechanism.
> * Two ResourceManagers (RM) in HA: Failover tested between HA1 and HA2 RM
> node, including state retention and proper node state transitions.
> * Verified node state transitions during RM failover, ensuring nodes moved
> between LOST, ACTIVE, and other states as expected.
> Reporter: Arjun Mohnot
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> h3. Issue Overview
> When the ResourceManager (RM) starts, nodes listed in the _"include"_ file
> are not immediately reported until their corresponding NodeManagers (NMs)
> send their first heartbeat. However, nodes in the _"exclude"_ file are
> instantly reflected in the _"Decommissioned Hosts"_ section with a port value
> -1.
> This design creates several challenges:
> * {*}Untracked Nodemanagers{*}: During Resourcemanager HA failover or RM
> standalone restart, some nodes may not report back, even though they are
> listed in the _"include"_ file. These nodes neither appear in the _LOST_
> state nor are they represented in the RM's JMX metrics. This results in an
> untracked state, making it difficult to monitor their status. While in HDFS
> similar behaviour exists and is marked as {_}"DEAD"{_}.
> * {*}Monitoring Gaps{*}: Nodes in the _"include"_ file are not visible until
> they send their first heartbeat. This delay impacts real-time cluster
> monitoring, leading to a lack of immediate visibility for these nodes in
> Resourcemanager's state on the total no. of nodes.
> * {*}Operational Impact{*}: These unreported nodes cause operational
> difficulties, particularly in automated workflows such as OS Upgrade
> Automation (OSUA), node recovery automation, and others where validation
> depends on nodes being reflected in JMX as {_}LOST{_}, {_}UNHEALTHY{_}, or
> {_}DECOMMISSIONED, etc{_}. Nodes that don't report, however, require hacky
> workarounds to determine their accurate status.
> h3. Proposed Solution
> To address these issues, we propose automatically assigning the _LOST_ state
> to any node listed in the _"include"_ file that are not registered and not
> part of the exclude file by default at the RM startup or HA failover. This
> can be done by marking the node with a special port value {_}-2{_}, signaling
> that the node is considered LOST but has not yet been reported. Whenever a
> heartbeat is received for that {color:#de350b}nodeID{color}, it will be
> transitioned from _LOST_ to {_}RUNNING{_}, {_}UNHEALTHY{_}, or any other
> required desired state.
> h3. Key implementation points
> * Mark Unreported Nodes as LOST: Nodes in the _"include"_ file not part of
> the RM active node context should be automatically marked as {_}LOST{_}. This
> can be achieved by modifying the _NodesListManager_ under the
> {color:#de350b}refreshHostsReader{color} method, invoked during failover, or
> manual node refresh operations. This logic should ensure that all
> unregistered nodes are moved to the _LOST_ state, with port _-2_ indicating
> the node is untracked.
> * For non-HA setups, this process can be triggered during RM service startup
> to mark nodes as _LOST_ initially, and they will gradually transition to
> their desired state when the heartbeat is received.
> * Handle Node Heartbeat and Transition: When a node sends its first
> heartbeat, the system should verify if the node is listed in
> {color:#de350b}getInactiveRMNodes(){color}. If the node exists in the _LOST_
> state, the RM should remove it from the inactive list, decrement the _LOST_
> node count, and handle the transition back to the active node set.
> * This logic can be placed in the state transition method within
> {color:#de350b}RMNodeImpl.java{color}, ensuring that nodes transitioned from
> _NEW_ to _LOST_ state, and recover gracefully from the _LOST_ state upon
> receiving their heartbeat.
> h3. Benefits
> * {*}Improved Cluster Monitoring{*}: Automatically assigning a _LOST_ state
> to nodes listed in the _"include"_ file but not reporting ensures that every
> node in the cluster has a well-defined state ({_}ACTIVE{_}, {_}LOST{_},
> {_}DECOMMISSIONED{_}, {_}UNHEALTHY, etc{_}). This eliminates any potential
> gaps in cluster node visibility and simplifies operational monitoring.
> * {*}Better Recovery Management{*}: By marking unreported nodes as
> {_}LOST{_}, automation can quickly identify which nodes require attention
> during recovery efforts to restore cluster health. This prevents confusion
> between unreachable nodes and untracked nodes, improving recovery accuracy.
> * {*}Enhanced Cluster Stability{*}: This approach improves overall stability
> by preventing nodes from slipping into an untracked or unknown state. It
> guarantees that the system remains aware of all nodes, reducing issues during
> RM failover or restart scenarios.
> h3. Additional Considerations
> * Feature Flag Control: This feature will be enabled/disabled via a
> configuration flag, allowing users to adjust behavior based on their
> requirements. By default, it is marked as {_}False{_}.
> * Enough Validations: The approach has been well-tested on non-HA and HA
> setups, and a dummy docker-based
> [setup|https://github.com/arjunmohnot/hadoop-yarn-docker/tree/main] has been
> created to replicate the behavior. Added the required unit test cases to
> validate the code behavior. Demo
> [video|https://drive.google.com/file/d/1okiPe7uMNVMRUnNYtz-B8Igf8FMGr-SJ/view?usp=sharing]
> for this change.
>
> Any thoughts/suggestions/feedback are welcome!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
