[
https://issues.apache.org/jira/browse/YARN-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539421#comment-14539421
]
zhihai xu commented on YARN-3619:
---------------------------------
thanks [~kasha] for assigning this JIRA to me.
The root cause is exactly what [~jlowe] said. I just added a little more
details based on [~jlowe] succinct comment.
{{sampleMetrics}} will be called periodically in MetricsSystemImpl.
{{sampleMetrics}} will iterate the {{sources}} in the following code:
{code}
for (Entry<String, MetricsSourceAdapter> entry : sources.entrySet()) {
if (sourceFilter == null || sourceFilter.accepts(entry.getKey())) {
snapshotMetrics(entry.getValue(), bufferBuilder);
}
}
{code}
{{snapshotMetrics}} will be called to process every entry from {{sources}}
The calling sequence which leads to a ConcurrentModificationException is
snapshotMetrics => MetricsSourceAdapter#getMetrics =>
ContainerMetrics#getMetrics => MetricsSystemImpl#unregisterSource =>
sources.remove(name)
the entry in the {{sources}} is removed when iterate the {{sources}}. So
unregisterSource can't be called from getMetrics.
I will prepare a patch for review.
> ContainerMetrics unregisters during getMetrics and leads to
> ConcurrentModificationException
> -------------------------------------------------------------------------------------------
>
> Key: YARN-3619
> URL: https://issues.apache.org/jira/browse/YARN-3619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.7.0
> Reporter: Jason Lowe
> Assignee: zhihai xu
>
> ContainerMetrics is able to unregister itself during the getMetrics method,
> but that method can be called by MetricsSystemImpl.sampleMetrics which is
> trying to iterate the sources. This leads to a
> ConcurrentModificationException log like this:
> {noformat}
> 2015-05-11 14:00:20,360 [Timer for 'NodeManager' metrics system] WARN
> impl.MetricsSystemImpl: java.util.ConcurrentModificationException
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
