[
https://issues.apache.org/jira/browse/YARN-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18010304#comment-18010304
]
ASF GitHub Bot commented on YARN-11838:
---------------------------------------
shameersss1 commented on code in PR #7828:
URL: https://github.com/apache/hadoop/pull/7828#discussion_r2235400661
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java:
##########
@@ -741,7 +741,15 @@ public void refreshNodeAttributesToScheduler(NodeId
nodeId) {
if (host == null || host.attributes == null) {
return;
}
- newNodeToAttributesMap.put(hostName, host.attributes.keySet());
+
+ // Use read lock and create defensive copy since
+ // other threads might access host.attributes
+ readLock.lock();
+ try {
+ newNodeToAttributesMap.put(hostName, new
HashSet<>(host.attributes.keySet()));
Review Comment:
The issue is that.
1. The LOG statement which prints `newNodeToAttributesMap` tries to iterate
`host.attribute`
2. `host.attribute` gets modified by some other thread - leading to
concurrent modification exception.
There are two ways to solve this
1. As you said to readLock before LOG statement so that `host.attribute`
does not get modified during LOG statement
2. Create a defensive copy of `host.attribute` (under read lock because the
modification can happen at that time as well).
The rationale behind using option 2 to avoid logging inconsistency- Assume
that we readLock before LOG statement. Once the LOG statement is executed, some
other thread modifies the `host.attribute` this will lead to we logging
something and processing something else.
Creating a defensive copy make sure that we don't change value. i.e what is
LOGed gets processed as well.
> YARN ConcurrentModificationException When Refreshing Node Attributes
> --------------------------------------------------------------------
>
> Key: YARN-11838
> URL: https://issues.apache.org/jira/browse/YARN-11838
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodeattibute, yarn
> Affects Versions: 3.3.0, 3.2.1
> Reporter: Syed Shameerur Rahman
> Assignee: Syed Shameerur Rahman
> Priority: Major
> Labels: pull-request-available
>
> h2. The Problem Flow
> # A new node is being added to the cluster (NODE_ADDED event)
> # The CapacityScheduler calls addNode() method
> # This triggers refreshNodeAttributesToScheduler() in
> NodeAttributesManagerImpl
> # During this process, the code attempts to convert a HashMap to a string
> for logging
> # While iterating through the HashMap for string conversion, another thread
> modifies the same HashMap :
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java#L748
> # This causes the ConcurrentModificationException
>
> {code:java}
> 025-07-17 19:23:37,166 ERROR org.apache.hadoop.yarn.event.EventDispatcher
> (SchedulerEventDispatcher:Event Processor): Error in handling event type
> NODE_ADDED to the Event Dispatcher
> java.util.ConcurrentModificationException
> at
> java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1597)
> at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1620)
> at
> java.base/java.util.AbstractCollection.toString(AbstractCollection.java:456)
> at java.base/java.lang.String.valueOf(String.java:4220)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
> at java.base/java.util.AbstractMap.toString(AbstractMap.java:555)
> at java.base/java.lang.String.valueOf(String.java:4220)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
> at
> org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NodeAttributesManagerImpl.refreshNodeAttributesToScheduler(NodeAttributesManagerImpl.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
