[ https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335804#comment-17335804 ]
Eric Badger commented on YARN-9927: ----------------------------------- {noformat} + // Test multi thread dispatcher + conf.setBoolean(YarnConfiguration. + MULTI_THREAD_DISPATCHER_ENABLED, true); {noformat} If this is a feature that is disabled by default, I don't think we should have it enabled by default in all of the RM tests. I would be happier running it as a parameterized test with both multi and single thread dispatchers. In general I think the patch looks reasonable, but I would like to see testing done to see if this makes the problem better or worse. I would think it would make things better, but until we run some real tests on it, we won't really know. So getting something similar to what [~hcarrot] provided originally would be good. That way we can merge this with confidence. > RM multi-thread event processing mechanism > ------------------------------------------ > > Key: YARN-9927 > URL: https://issues.apache.org/jira/browse/YARN-9927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn > Affects Versions: 3.0.0, 2.9.2 > Reporter: hcarrot > Assignee: Qi Zhu > Priority: Major > Attachments: RM multi-thread event processing mechanism.pdf, > YARN-9927.001.patch, YARN-9927.002.patch, YARN-9927.003.patch, > YARN-9927.004.patch, YARN-9927.005.patch > > > Recently, we have observed serious event blocking in RM event dispatcher > queue. After analysis of RM event monitoring data and RM event processing > logic, we found that > 1) environment: a cluster with thousands of nodes > 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler > 3) Meanwhile, RM event processing is in a single-thread mode, and It results > in the low headroom of RM event scheduler, thus performance of RM. > So we proposed a RM multi-thread event processing mechanism to improve RM > performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org