[
https://issues.apache.org/jira/browse/YARN-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882304#comment-17882304
]
ASF GitHub Bot commented on YARN-11730:
---------------------------------------
slfan1989 commented on code in PR #7049:
URL: https://github.com/apache/hadoop/pull/7049#discussion_r1762632209
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java:
##########
@@ -387,6 +388,115 @@ private void handleExcludeNodeList(boolean graceful, int
timeout) {
updateInactiveNodes();
}
+ /**
+ * Marks the unregistered nodes as LOST
+ * if the feature is enabled via a configuration flag.
+ *
+ * This method finds nodes that are present in the include list but are not
+ * registered with the ResourceManager. Such nodes are then marked as LOST.
+ *
+ * The steps are as follows:
+ * 1. Retrieve all hostnames of registered nodes from RM.
+ * 2. Identify the nodes present in the include list but are not registered
+ * 3. Remove nodes from the exclude list
+ * 4. Dispatch LOST events for filtered nodes to mark them as LOST.
+ *
+ * @param yarnConf Configuration object that holds the YARN configurations.
+ */
+ private void markUnregisteredNodesAsLost(Configuration yarnConf) {
+ // Check if tracking unregistered nodes is enabled in the configuration
+ if
(!yarnConf.getBoolean(YarnConfiguration.ENABLE_TRACKING_FOR_UNREGISTERED_NODES,
+ YarnConfiguration.DEFAULT_ENABLE_TRACKING_FOR_UNREGISTERED_NODES)) {
+ LOG.debug("Unregistered node tracking is disabled. " +
+ "Skipping marking unregistered nodes as LOST.");
+ return;
+ }
+
+ // Set to store all registered hostnames from both active and inactive
lists
+ Set<String> registeredHostNames = gatherRegisteredHostNames();
+ // Event handler to dispatch LOST events
+ EventHandler eventHandler =
this.rmContext.getDispatcher().getEventHandler();
+
+ // Identify nodes that are in the include list but are not registered
+ // and are not in the exclude list
+ List<String> nodesToMarkLost = new ArrayList<>();
+ HostDetails hostDetails = hostsReader.getHostDetails();
+ Set<String> includes = hostDetails.getIncludedHosts();
+ Set<String> excludes = hostDetails.getExcludedHosts();
+
+ for (String includedNode : includes) {
+ if (!registeredHostNames.contains(includedNode) &&
!excludes.contains(includedNode)) {
+ LOG.info("Lost node: " + includedNode);
+ nodesToMarkLost.add(includedNode);
+ }
+ }
+
+ // Dispatch LOST events for the identified lost nodes
+ for (String lostNode : nodesToMarkLost) {
+ dispatchLostEvent(eventHandler, lostNode);
+ }
+
+ // Log successful completion of marking unregistered nodes as LOST
+ LOG.info("Successfully marked unregistered nodes as LOST");
+ }
+
+ /**
+ * Gathers all registered hostnames from both active and inactive RMNodes.
+ *
+ * @return A set of registered hostnames.
+ */
+ private Set<String> gatherRegisteredHostNames() {
+ Set<String> registeredHostNames = new HashSet<>();
+ LOG.info("Getting all the registered hostnames");
+
+ // Gather all registered nodes (active) from RM into the set
+ for (RMNode node : this.rmContext.getRMNodes().values()) {
+ registeredHostNames.add(node.getHostName());
+ }
+
+ // Gather all inactive nodes from RM into the set
+ for (RMNode node : this.rmContext.getInactiveRMNodes().values()) {
+ registeredHostNames.add(node.getHostName());
+ }
+
+ return registeredHostNames;
+ }
+
+ /**
+ * Dispatches a LOST event for a specified lost node.
+ *
+ * @param eventHandler The EventHandler used to dispatch the LOST event.
+ * @param lostNode The hostname of the lost node for which the event is
+ * being dispatched.
+ */
+ private void dispatchLostEvent(EventHandler eventHandler, String lostNode) {
+ // Generate a NodeId for the lost node with a special port -2
+ NodeId nodeId = createLostNodeId(lostNode);
+ RMNodeEvent lostEvent = new RMNodeEvent(nodeId, RMNodeEventType.EXPIRE);
+ RMNodeImpl rmNode = new RMNodeImpl(nodeId, this.rmContext, lostNode, -2,
-2,
+ new UnknownNode(lostNode), Resource.newInstance(0, 0), "unknown");
+
+ try {
+ // Dispatch the LOST event to signal the node is no longer active
+ eventHandler.handle(lostEvent);
+
+ // After successful dispatch, update the node status in RMContext
+ // Set the node's timestamp for when it became untracked
+ rmNode.setUntrackedTimeStamp(Time.monotonicNow());
+
+ // Add the node to the active and inactive node maps in RMContext
+ this.rmContext.getRMNodes().put(nodeId, rmNode);
+ this.rmContext.getInactiveRMNodes().put(nodeId, rmNode);
+
+ LOG.info("Successfully dispatched LOST event and deactivated node: "
Review Comment:
We use {} to record logs.
```
LOG.info("Successfully dispatched LOST event and deactivated node: {}, Node
ID: {}",
lostNode, nodeId);
```
> Resourcemanager node reporting enhancement for unregistered hosts
> -----------------------------------------------------------------
>
> Key: YARN-11730
> URL: https://issues.apache.org/jira/browse/YARN-11730
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, yarn
> Affects Versions: 3.4.0
> Environment: Tested on multiple environments:
> A. Docker Environment{*}:{*}
> * Base OS: *Ubuntu 20.04*
> * *Java 8* installed from OpenJDK.
> * Docker image includes Hadoop binaries, user configurations, and ports for
> YARN services.
> * Verified behavior using a Hadoop snapshot in a containerized environment.
> * Performed Namenode formatting and validated service interactions through
> exposed ports.
> * Repo reference:
> [arjunmohnot/hadoop-yarn-docker|https://github.com/arjunmohnot/hadoop-yarn-docker/tree/main]
> B. Bare-metal Distributed Setup (RedHat Linux){*}:{*}
> * Running *Java 8* in a High-Availability (HA) configuration with
> *Zookeeper* for locking mechanism.
> * Two ResourceManagers (RM) in HA: Failover tested between HA1 and HA2 RM
> node, including state retention and proper node state transitions.
> * Verified node state transitions during RM failover, ensuring nodes moved
> between LOST, ACTIVE, and other states as expected.
> Reporter: Arjun Mohnot
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> h3. Issue Overview
> When the ResourceManager (RM) starts, nodes listed in the _"include"_ file
> are not immediately reported until their corresponding NodeManagers (NMs)
> send their first heartbeat. However, nodes in the _"exclude"_ file are
> instantly reflected in the _"Decommissioned Hosts"_ section with a port value
> -1.
> This design creates several challenges:
> * {*}Untracked Nodemanagers{*}: During Resourcemanager HA failover or RM
> standalone restart, some nodes may not report back, even though they are
> listed in the _"include"_ file. These nodes neither appear in the _LOST_
> state nor are they represented in the RM's JMX metrics. This results in an
> untracked state, making it difficult to monitor their status. While in HDFS
> similar behaviour exists and is marked as {_}"DEAD"{_}.
> * {*}Monitoring Gaps{*}: Nodes in the _"include"_ file are not visible until
> they send their first heartbeat. This delay impacts real-time cluster
> monitoring, leading to a lack of immediate visibility for these nodes in
> Resourcemanager's state on the total no. of nodes.
> * {*}Operational Impact{*}: These unreported nodes cause operational
> difficulties, particularly in automated workflows such as OS Upgrade
> Automation (OSUA), node recovery automation, and others where validation
> depends on nodes being reflected in JMX as {_}LOST{_}, {_}UNHEALTHY{_}, or
> {_}DECOMMISSIONED, etc{_}. Nodes that don't report, however, require hacky
> workarounds to determine their accurate status.
> h3. Proposed Solution
> To address these issues, we propose automatically assigning the _LOST_ state
> to any node listed in the _"include"_ file that are not registered and not
> part of the exclude file by default at the RM startup or HA failover. This
> can be done by marking the node with a special port value {_}-2{_}, signaling
> that the node is considered LOST but has not yet been reported. Whenever a
> heartbeat is received for that {color:#de350b}nodeID{color}, it will be
> transitioned from _LOST_ to {_}RUNNING{_}, {_}UNHEALTHY{_}, or any other
> required desired state.
> h3. Key implementation points
> * Mark Unreported Nodes as LOST: Nodes in the _"include"_ file not part of
> the RM active node context should be automatically marked as {_}LOST{_}. This
> can be achieved by modifying the _NodesListManager_ under the
> {color:#de350b}refreshHostsReader{color} method, invoked during failover, or
> manual node refresh operations. This logic should ensure that all
> unregistered nodes are moved to the _LOST_ state, with port _-2_ indicating
> the node is untracked.
> * For non-HA setups, this process can be triggered during RM service startup
> to mark nodes as _LOST_ initially, and they will gradually transition to
> their desired state when the heartbeat is received.
> * Handle Node Heartbeat and Transition: When a node sends its first
> heartbeat, the system should verify if the node is listed in
> {color:#de350b}getInactiveRMNodes(){color}. If the node exists in the _LOST_
> state, the RM should remove it from the inactive list, decrement the _LOST_
> node count, and handle the transition back to the active node set.
> * This logic can be placed in the state transition method within
> {color:#de350b}RMNodeImpl.java{color}, ensuring that nodes transitioned from
> _NEW_ to _LOST_ state, and recover gracefully from the _LOST_ state upon
> receiving their heartbeat.
> h3. Benefits
> * {*}Improved Cluster Monitoring{*}: Automatically assigning a _LOST_ state
> to nodes listed in the _"include"_ file but not reporting ensures that every
> node in the cluster has a well-defined state ({_}ACTIVE{_}, {_}LOST{_},
> {_}DECOMMISSIONED{_}, {_}UNHEALTHY, etc{_}). This eliminates any potential
> gaps in cluster node visibility and simplifies operational monitoring.
> * {*}Better Recovery Management{*}: By marking unreported nodes as
> {_}LOST{_}, automation can quickly identify which nodes require attention
> during recovery efforts to restore cluster health. This prevents confusion
> between unreachable nodes and untracked nodes, improving recovery accuracy.
> * {*}Enhanced Cluster Stability{*}: This approach improves overall stability
> by preventing nodes from slipping into an untracked or unknown state. It
> guarantees that the system remains aware of all nodes, reducing issues during
> RM failover or restart scenarios.
> h3. Additional Considerations
> * Feature Flag Control: This feature will be enabled/disabled via a
> configuration flag, allowing users to adjust behavior based on their
> requirements. By default, it is marked as {_}False{_}.
> * Enough Validations: The approach has been well-tested on non-HA and HA
> setups, and a dummy docker-based
> [setup|https://github.com/arjunmohnot/hadoop-yarn-docker/tree/main] has been
> created to replicate the behavior. Added the required unit test cases to
> validate the code behavior. Demo
> [video|https://drive.google.com/file/d/1okiPe7uMNVMRUnNYtz-B8Igf8FMGr-SJ/view?usp=sharing]
> for this change.
>
> Any thoughts/suggestions/feedback are welcome!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
