[jira] [Commented] (YARN-4325) purge app state from NM state-store should be independent of log aggregation

[ https://issues.apache.org/jira/browse/YARN-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997932#comment-14997932 ] zhangshilong commented on YARN-4325: If permissions with hdfs is right, is there any other problem? If

[jira] [Updated] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in securt clusters

[ https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-4327: --- Description: in hadoop 2.7.1 ATS conf like this: yarn.timeline-service.http-authentication.type

[jira] [Created] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in securt clusters

zhangshilong created YARN-4327: -- Summary: RM can not renew TIMELINE_DELEGATION_TOKEN in securt clusters Key: YARN-4327 URL: https://issues.apache.org/jira/browse/YARN-4327 Project: Hadoop YARN

[jira] [Updated] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters

[ https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-4327: --- Summary: RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters (was: RM can not renew

[jira] [Commented] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in securt clusters

[ https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987054#comment-14987054 ] zhangshilong commented on YARN-4327: yarn.timeline-service.http-authentication.type simple, ATS will

[jira] [Commented] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters

[ https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069121#comment-15069121 ] zhangshilong commented on YARN-4327: yeah，I tried

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855267#comment-15855267 ] zhangshilong commented on YARN-4090: so sorry for whitespace.. A new patch will be submitted. I think

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-4090: --- Attachment: YARN-4090.006.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859019#comment-15859019 ] zhangshilong commented on YARN-4090: Thanks [~yufeigu]. when application finishes or its tasks

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-4090: --- Attachment: YARN-4090.005.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851288#comment-15851288 ] zhangshilong commented on YARN-4090: Thanks [~yufeigu]. I will submit the new patch as soon as

[jira] [Commented] (YARN-5188) FairScheduler performance bug

[ https://issues.apache.org/jira/browse/YARN-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851286#comment-15851286 ] zhangshilong commented on YARN-5188: [~chenfolin] Good idea! and how is the performance of this patch？

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863277#comment-15863277 ] zhangshilong commented on YARN-4090: Thanks [~yufeigu] for remind for more information. YARN-4691 is

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885072#comment-15885072 ] zhangshilong commented on YARN-4090: [~yufeigu] I found one problem when doing this issue. In

[jira] [Commented] (YARN-4752) FairScheduler should preempt for a ResourceRequest and all preempted containers should be on the same node

[ https://issues.apache.org/jira/browse/YARN-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885081#comment-15885081 ] zhangshilong commented on YARN-4752: [~kasha] I found one problem. In FSLeafQueue: I think

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage pool performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Attachment: 20161206.patch > FairShareComparator getResourceUsage pool performance >

[jira] [Created] (YARN-5969) FairShareComparator getResourceUsage pool performance

zhangshilong created YARN-5969: -- Summary: FairShareComparator getResourceUsage pool performance Key: YARN-5969 URL: https://issues.apache.org/jira/browse/YARN-5969 Project: Hadoop YARN Issue

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Summary: FairShareComparator getResourceUsage poor performance (was: FairShareComparator

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Description: in FairShareComparator class, the performance of function getResourceUsage() is very

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Description: in FairShareComparator.java, the performance of function getResourceUsage() is very

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784392#comment-15784392 ] zhangshilong commented on YARN-4090: [~xinxianyin] [~yufeigu] This optimization works in our

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784417#comment-15784417 ] zhangshilong commented on YARN-4090: would you please tell me yarn version you used? In trunk:

[jira] [Updated] (YARN-6045) apps/queues that have no pending containers will still affect the efficiency of scheduling

[ https://issues.apache.org/jira/browse/YARN-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-6045: --- Environment: yarn: 2.7.1 release jdk 1.7 kernel:2.6.32-431.20.3.el6 was: jdk 1.7

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787258#comment-15787258 ] zhangshilong commented on YARN-4090: I see. In branch-2.6, FSParentQueue.java: will lock FSQueue. But

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-4090: --- Attachment: YARN-4090.004.patch fix 2.6 deadlock in FSParentQueue.java > Make Collections.sort() more

[jira] [Created] (YARN-6045) apps/queues that have no pending containers will still affect the efficiency of scheduling

zhangshilong created YARN-6045: -- Summary: apps/queues that have no pending containers will still affect the efficiency of scheduling Key: YARN-6045 URL: https://issues.apache.org/jira/browse/YARN-6045

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787353#comment-15787353 ] zhangshilong commented on YARN-4090: no problem, thank you very much for your patch, a great help to

[jira] [Assigned] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong reassigned YARN-4090: -- Assignee: zhangshilong > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766437#comment-15766437 ] zhangshilong commented on YARN-5969: Test case： 500 app，3000 nm nodes queue: parent queue number:

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Attachment: containerAllocated_after.png apprunning_after.png > FairShareComparator

[jira] [Comment Edited] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766471#comment-15766471 ] zhangshilong edited comment on YARN-5969 at 12/21/16 8:34 AM: --

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Attachment: pending_before.png pending_after.png

[jira] [Commented] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766471#comment-15766471 ] zhangshilong commented on YARN-5969: ContainerAllocated picture means container allocation per minute.

[jira] [Commented] (YARN-5969) FairShareComparator: Cache value of getResourceUsage for better performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782248#comment-15782248 ] zhangshilong commented on YARN-5969: Thanks [~yufeigu] for advice and review and [~kasha] for commit.

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Attachment: YARN-5969.patch > FairShareComparator getResourceUsage poor performance >

[jira] [Commented] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769268#comment-15769268 ] zhangshilong commented on YARN-5969: Thanks yufei Gu for the reminder, I will improve my patch soon.

[jira] [Updated] (YARN-5969) FairShareComparator getResourceUsage poor performance

[ https://issues.apache.org/jira/browse/YARN-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-5969: --- Attachment: 20161222.patch > FairShareComparator getResourceUsage poor performance >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient by caching resource usage

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131643#comment-16131643 ] zhangshilong commented on YARN-4090: I am very sorry. I am always working on YARN project.But Job is

[jira] [Comment Edited] (YARN-4090) Make Collections.sort() more efficient by caching resource usage

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137944#comment-16137944 ] zhangshilong edited comment on YARN-4090 at 8/23/17 6:17 AM: -

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient by caching resource usage

[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137944#comment-16137944 ] zhangshilong commented on YARN-4090: [~dan...@cloudera.com] [~yufeigu] Never mind. It is my fault. I

[jira] [Commented] (YARN-7214) duplicated container completed To AM

[ https://issues.apache.org/jira/browse/YARN-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171270#comment-16171270 ] zhangshilong commented on YARN-7214: 3. {code:java} public static class AddNodeTransition implements

[jira] [Commented] (YARN-7214) duplicated container completed To AM

[ https://issues.apache.org/jira/browse/YARN-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171322#comment-16171322 ] zhangshilong commented on YARN-7214: in my thought, containers in recentlyStoppedContainers can be

[jira] [Commented] (YARN-7214) duplicated container completed To AM

[ https://issues.apache.org/jira/browse/YARN-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171396#comment-16171396 ] zhangshilong commented on YARN-7214: !screenshot-1.png! generally, 1、 NM complete one container(c) and

[jira] [Updated] (YARN-7214) duplicated container completed To AM

[ https://issues.apache.org/jira/browse/YARN-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-7214: --- Attachment: screenshot-1.png > duplicated container completed To AM >

[jira] [Comment Edited] (YARN-7214) duplicated container completed To AM

[ https://issues.apache.org/jira/browse/YARN-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171396#comment-16171396 ] zhangshilong edited comment on YARN-7214 at 9/19/17 9:38 AM: -

[jira] [Created] (YARN-7214) duplicated container completed To AM

zhangshilong created YARN-7214: -- Summary: duplicated container completed To AM Key: YARN-7214 URL: https://issues.apache.org/jira/browse/YARN-7214 Project: Hadoop YARN Issue Type: Bug

[jira] [Created] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

zhangshilong created YARN-7672: -- Summary: hadoop-sls can not simulate huge scale of YARN Key: YARN-7672 URL: https://issues.apache.org/jira/browse/YARN-7672 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

[ https://issues.apache.org/jira/browse/YARN-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-7672: --- Description: Our YARN cluster scale to nearly 10 thousands nodes. We need to do scheduler pressure

[jira] [Comment Edited] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

[ https://issues.apache.org/jira/browse/YARN-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298192#comment-16298192 ] zhangshilong edited comment on YARN-7672 at 12/20/17 9:55 AM: -- [~cxcw] I use

[jira] [Commented] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

[ https://issues.apache.org/jira/browse/YARN-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298192#comment-16298192 ] zhangshilong commented on YARN-7672: [~cxcw] I use two daemons deployed on different two hosts. I

[jira] [Updated] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

[ https://issues.apache.org/jira/browse/YARN-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-7672: --- Attachment: YARN-7672.patch > hadoop-sls can not simulate huge scale of YARN >