[
https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463364#comment-16463364
]
genericqa commented on YARN-8234:
---------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.8.3 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
18s{color} | {color:green} branch-2.8.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
6s{color} | {color:green} branch-2.8.3 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} branch-2.8.3 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} branch-2.8.3 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} branch-2.8.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} branch-2.8.3 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new + 218 unchanged - 3 fixed = 227 total (was 221) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
21s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 49s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 45s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Synchronization performed on java.util.concurrent.LinkedBlockingQueue in
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.putEntity(TimelineEntity)
At
SystemMetricsPublisher.java:org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.putEntity(TimelineEntity)
At SystemMetricsPublisher.java:[line 599] |
| | Possible null pointer dereference of response in
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$SendEntity.run()
on exception path Dereferenced at SystemMetricsPublisher.java:response in
org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$SendEntity.run()
on exception path Dereferenced at SystemMetricsPublisher.java:[line 631] |
| Unreaped Processes | hadoop-yarn-server-resourcemanager:1 |
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| | hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens |
| | hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens |
| | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog |
| | hadoop.yarn.server.resourcemanager.security.TestAMRMTokens |
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.TestApplicationCleanup |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId |
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesHttpStaticUserPermissions
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| | hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor |
| | hadoop.yarn.server.resourcemanager.TestResourceManager |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
| | hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager |
| | hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels |
| | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils |
| | hadoop.yarn.server.resourcemanager.TestLeaderElectorService |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition
|
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher |
| | hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
| |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt |
| | hadoop.yarn.server.resourcemanager.TestSignalContainer |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
|
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
| | hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf |
| |
hadoop.yarn.server.resourcemanager.nodelabels.TestRMDelegatedNodeLabelsUpdater |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.policy.TestFairOrderingPolicy |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
|
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| | hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| | hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
|
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| | hadoop.yarn.server.resourcemanager.TestMoveApplication |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| | hadoop.yarn.server.resourcemanager.TestRM |
| | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability |
| | hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA |
| | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate |
| | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:c2d96dd |
| JIRA Issue | YARN-8234 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921224/YARN-8234-branch-2.8.3.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8f53c9e68220 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.8.3 / b3fe564 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_171 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
|
| Unreaped Processes Log |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-reaper.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 844 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/20592/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve RM system metrics publisher's performance by pushing events to
> timeline server in batch
> -----------------------------------------------------------------------------------------------
>
> Key: YARN-8234
> URL: https://issues.apache.org/jira/browse/YARN-8234
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, timelineserver
> Affects Versions: 2.8.3
> Reporter: Hu Ziqian
> Assignee: Hu Ziqian
> Priority: Major
> Attachments: YARN-8234-branch-2.8.3.001.patch
>
>
> When system metrics publisher is enabled, RM will push events to timeline
> server via restful api. If the cluster load is heavy, many events are sent to
> timeline server and the timeline server's event handler thread locked.
> YARN-7266 talked about the detail of this problem. Because of the lock,
> timeline server can't receive event as fast as it generated in RM and lots of
> timeline event stays in RM's memory. Finally, those events will consume all
> RM's memory and RM will start a full gc (which cause an JVM stop-world and
> cause a timeout from rm to zookeeper) or even get an OOM.
> The main problem here is that timeline can't receive timeline server's event
> as fast as it generated. Now, RM system metrics publisher put only one event
> in a request, and most time costs on handling http header or some thing about
> the net connection on timeline side. Only few time is spent on dealing with
> the timeline event which is truly valuable.
> In this issue, we add a buffer in system metrics publisher and let publisher
> send events to timeline server in batch via one request. When sets the batch
> size to 1000, in out experiment the speed of the timeline server receives
> events has 100x improvement. We have implement this function int our product
> environment which accepts 20000 app's in one hour and it works fine.
> We add following configuration:
> * yarn.resourcemanager.system-metrics-publisher.batch-size: the size of
> system metrics publisher sending events in one request. Default value is 1000
> * yarn.resourcemanager.system-metrics-publisher.buffer-size: the size of the
> event buffer in system metrics publisher.
> * yarn.resourcemanager.system-metrics-publisher.interval-seconds: When
> enable batch publishing, we must avoid that the publisher waits for a batch
> to be filled up and hold events in buffer for long time. So we add another
> thread which send event's in the buffer periodically. This config sets the
> interval of the cyclical sending thread. The default value is 60s.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
