[jira] [Commented] (YARN-4198) CapacityScheduler locking / synchronization improvements

2015-12-16 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059730#comment-15059730 ] Xianyin Xin commented on YARN-4198: --- Hi [~curino], does this improvement only considers t

[jira] [Commented] (YARN-4477) FairScheduler: encounter infinite loop in attemptScheduling

2015-12-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063538#comment-15063538 ] Xianyin Xin commented on YARN-4477: --- cc [~asuresh]. > FairScheduler: encounter infinite

[jira] [Commented] (YARN-3870) Providing raw container request information for fine scheduling

2015-12-29 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074755#comment-15074755 ] Xianyin Xin commented on YARN-3870: --- +1 for adding an unique id for a resource request, b

[jira] [Commented] (YARN-4414) Nodemanager connection errors are retried at multiple levels

2016-01-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088804#comment-15088804 ] Xianyin Xin commented on YARN-4414: --- Hi [~lichangleo], need we also revisit the two layer

[jira] [Commented] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2016-03-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182564#comment-15182564 ] Xianyin Xin commented on YARN-4134: --- [~kasha], fine. :) > FairScheduler preemption stops

[jira] [Commented] (YARN-4752) [Umbrella] FairScheduler: Improve preemption

2016-03-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182566#comment-15182566 ] Xianyin Xin commented on YARN-4752: --- hi [~kasha], can we consider to decouple the schedul

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-03-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190506#comment-15190506 ] Xianyin Xin commented on YARN-4090: --- Sorry [~kasha], i didn't notice that, and thanks for

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-03-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190514#comment-15190514 ] Xianyin Xin commented on YARN-4090: --- Let's move to YARN-1297 to continue the discussion.

[jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups

2016-03-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190517#comment-15190517 ] Xianyin Xin commented on YARN-1297: --- Hi [~asuresh], could you please update the patch bas

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257960#comment-15257960 ] Xianyin Xin commented on YARN-4090: --- Thanks, [~kasha], will upload a new patch as soon as

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: YARN-4090.001.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261546#comment-15261546 ] Xianyin Xin commented on YARN-4090: --- Sorry for the delay, [~kasha] [~yufeigu]. Just uploa

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261837#comment-15261837 ] Xianyin Xin commented on YARN-4090: --- It seems the failed UT with JDK1.8.0_92 and JDK1.7.0

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-05-22 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295948#comment-15295948 ] Xianyin Xin commented on YARN-4090: --- Sorry for the delay [~yufeigu]. {quote} do you mean

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-05-22 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: YARN-4090.002.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Created] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-17 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-4177: - Summary: yarn.util.Clock should not be used to time a duration or time interval Key: YARN-4177 URL: https://issues.apache.org/jira/browse/YARN-4177 Project: Hadoop YARN

[jira] [Updated] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4177: -- Attachment: YARN-4177.001.patch Provide a MonotonicClock. > yarn.util.Clock should not be used to time a

[jira] [Commented] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14802911#comment-14802911 ] Xianyin Xin commented on YARN-4177: --- In a following steps I would try to fix several impo

[jira] [Updated] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4177: -- Attachment: YARN-4177.002.patch Upload patch ver2. > yarn.util.Clock should not be used to time a durati

[jira] [Assigned] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin reassigned YARN-4177: - Assignee: Xianyin Xin > yarn.util.Clock should not be used to time a duration or time interval > -

[jira] [Commented] (YARN-4190) missing container information in FairScheduler preemption log.

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876762#comment-14876762 ] Xianyin Xin commented on YARN-4190: --- Hi [~zxu], will we consider to fix the issue in YARN

[jira] [Commented] (YARN-4189) Capacity Scheduler : Improve location preference waiting mechanism

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876871#comment-14876871 ] Xianyin Xin commented on YARN-4189: --- Hi [~leftnoteasy], just go through the doc. Please c

[jira] [Commented] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-21 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900450#comment-14900450 ] Xianyin Xin commented on YARN-4177: --- Hi [~ste...@apache.org], thanks for your comment. I'

[jira] [Commented] (YARN-4189) Capacity Scheduler : Improve location preference waiting mechanism

2015-09-21 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901736#comment-14901736 ] Xianyin Xin commented on YARN-4189: --- [~leftnoteasy], convincing analysis. It's fine X <<

[jira] [Resolved] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and NN go down at the same time.

2015-10-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin resolved YARN-3639. --- Resolution: Fixed This has been resolved by YARN-4041, so close it. > It takes too long time for RM to

[jira] [Commented] (YARN-4403) (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period

2015-12-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037180#comment-15037180 ] Xianyin Xin commented on YARN-4403: --- hi [~djp], this is a good suggestion, and YARN-4177

[jira] [Commented] (YARN-4403) (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period

2015-12-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037185#comment-15037185 ] Xianyin Xin commented on YARN-4403: --- And will provide new patch of YARN-4177 once this is

[jira] [Commented] (YARN-4403) (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period

2015-12-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037284#comment-15037284 ] Xianyin Xin commented on YARN-4403: --- Thanks, [~sunilg]. > (AM/NM/Container)LivelinessMon

[jira] [Commented] (YARN-4415) Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned

2015-12-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044498#comment-15044498 ] Xianyin Xin commented on YARN-4415: --- Sorry for the late, [~Naganarasimha]. I don't know

[jira] [Commented] (YARN-4415) Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned

2015-12-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046147#comment-15046147 ] Xianyin Xin commented on YARN-4415: --- sorry [~Naganarasimha], i missed that, thanks for yo

[jira] [Commented] (YARN-4415) Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned

2015-12-08 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047909#comment-15047909 ] Xianyin Xin commented on YARN-4415: --- +1 for the general idea from [~Naganarasimha]. I thi

[jira] [Commented] (YARN-4415) Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned

2015-12-14 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057134#comment-15057134 ] Xianyin Xin commented on YARN-4415: --- Hi [~leftnoteasy], thanks for you comments. {quote}

[jira] [Updated] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-05 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3547: -- Attachment: YARN-3547.004.patch Fixed test failures. New test is not need. > FairScheduler: Apps that ha

[jira] [Commented] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler

2015-05-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530272#comment-14530272 ] Xianyin Xin commented on YARN-3139: --- Any progress? or any plans? > Improve locks in Abst

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532131#comment-14532131 ] Xianyin Xin commented on YARN-3547: --- Thanks for your feedback [~vinodkv]. We use Schedul

[jira] [Commented] (YARN-3590) Change the attribute "maxRunningApps" of FairSchudeler queue,it should be take effect for application in queue,but not

2015-05-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532538#comment-14532538 ] Xianyin Xin commented on YARN-3590: --- It seems duplicate YARN-3057, please check. > Chang

[jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539662#comment-14539662 ] Xianyin Xin commented on YARN-3630: --- I think an adaptive heartbeat interval is need, espe

[jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539666#comment-14539666 ] Xianyin Xin commented on YARN-3630: --- Hi [~Ehnalis], would you mind if i assign this jira

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539692#comment-14539692 ] Xianyin Xin commented on YARN-3547: --- Sorry for late response, [~kasha]. {{getDemand() - g

[jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539693#comment-14539693 ] Xianyin Xin commented on YARN-3630: --- Agree, [~Ehnalis]. > YARN should suggest a heartbea

[jira] [Assigned] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin reassigned YARN-3630: - Assignee: Xianyin Xin > YARN should suggest a heartbeat interval for applications > --

[jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539725#comment-14539725 ] Xianyin Xin commented on YARN-3630: --- It's true [~ste...@apache.org]. But I think comparin

[jira] [Created] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and namenode is deployed on the same node.

2015-05-13 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-3639: - Summary: It takes too long time for RM to recover all apps if the original active RM and namenode is deployed on the same node. Key: YARN-3639 URL: https://issues.apache.org/jira/browse

[jira] [Updated] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and namenode is deployed on the same node.

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3639: -- Assignee: (was: Xianyin Xin) > It takes too long time for RM to recover all apps if the original acti

[jira] [Updated] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and namenode is deployed on the same node.

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3639: -- Attachment: YARN-3639-recovery_log_1_app.txt Attach recovery log for 1 app. In the log, there's a gap(15s

[jira] [Commented] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and namenode is deployed on the same node.

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543040#comment-14543040 ] Xianyin Xin commented on YARN-3639: --- Sorry [~aw], i didn't make it clearly. "On the same

[jira] [Commented] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and namenode is deployed on the same node.

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543053#comment-14543053 ] Xianyin Xin commented on YARN-3639: --- Maybe we can fix this problem in the following way:

[jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543071#comment-14543071 ] Xianyin Xin commented on YARN-3630: --- Thanks [~kasha] and [~leftnoteasy]. Indeed the waiti

[jira] [Commented] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and namenode is deployed on the same node.

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543187#comment-14543187 ] Xianyin Xin commented on YARN-3639: --- Yes, you're right [~aw]. > It takes too long time

[jira] [Updated] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and NN go down at the same time.

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3639: -- Summary: It takes too long time for RM to recover all apps if the original active RM and NN go down at th

[jira] [Updated] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and NN go down at the same time.

2015-05-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3639: -- Description: If the active RM and NN go down at the same time, the new RM will take long time to recover

[jira] [Updated] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-14 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3547: -- Attachment: YARN-3547.005.patch A patch using {{getDemand() - getResourceUsage()}}. > FairScheduler: App

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-14 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544995#comment-14544995 ] Xianyin Xin commented on YARN-3547: --- [~kasha], can you please have a look? > FairSchedul

[jira] [Created] (YARN-3652) A {{SchedulerMetrics}} may be need for evaluating the scheduler's performance

2015-05-15 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-3652: - Summary: A {{SchedulerMetrics}} may be need for evaluating the scheduler's performance Key: YARN-3652 URL: https://issues.apache.org/jira/browse/YARN-3652 Project: Hadoop Y

[jira] [Updated] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-05-15 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3652: -- Summary: A SchedulerMetrics may be need for evaluating the scheduler's performance (was: A {{SchedulerMe

[jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-15 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545188#comment-14545188 ] Xianyin Xin commented on YARN-3630: --- Thanks, [~leftnoteasy] and [~kasha]. [~leftnoteasy],

[jira] [Created] (YARN-3653) Expose the scheduler's KPI in web UI

2015-05-15 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-3653: - Summary: Expose the scheduler's KPI in web UI Key: YARN-3653 URL: https://issues.apache.org/jira/browse/YARN-3653 Project: Hadoop YARN Issue Type: Improvement

[jira] [Updated] (YARN-3653) Expose the scheduler's KPI in web UI

2015-05-15 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3653: -- Component/s: webapp > Expose the scheduler's KPI in web UI > > >

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-05-15 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545607#comment-14545607 ] Xianyin Xin commented on YARN-3652: --- Thanks for comments, [~sunilg]. {quote} 1. *Throughp

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-05-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547435#comment-14547435 ] Xianyin Xin commented on YARN-3652: --- Thanks [~vinodkv], that's very helpful. > A Schedul

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547461#comment-14547461 ] Xianyin Xin commented on YARN-3547: --- Agree, [~leftnoteasy]. Now we have YARN-3547.004.pat

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-05-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549600#comment-14549600 ] Xianyin Xin commented on YARN-3652: --- Hi [~vinodkv], YARN-3293 and {{SchedulerMetrics}} ar

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-05-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549626#comment-14549626 ] Xianyin Xin commented on YARN-3652: --- Or just use {{SchedulerMetrics}} at the perspective

[jira] [Updated] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-19 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3630: -- Attachment: YARN-3630.001.patch.patch Initial patch with adaptive heartbeat policy unimplemented. If we d

[jira] [Created] (YARN-3716) Output node label expression in ResourceRequestPBImpl.toString

2015-05-26 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-3716: - Summary: Output node label expression in ResourceRequestPBImpl.toString Key: YARN-3716 URL: https://issues.apache.org/jira/browse/YARN-3716 Project: Hadoop YARN I

[jira] [Updated] (YARN-3716) Output node label expression in ResourceRequestPBImpl.toString

2015-05-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3716: -- Attachment: YARN-3716.001.patch > Output node label expression in ResourceRequestPBImpl.toString > --

[jira] [Updated] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3630: -- Attachment: Notes_for_adaptive_heartbeat_policy.pdf > YARN should suggest a heartbeat interval for applic

[jira] [Updated] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3630: -- Attachment: YARN-3630.002.patch Upload a patch for review. Currently, we can choose to turn off this feat

[jira] [Commented] (YARN-3716) Output node label expression in ResourceRequestPBImpl.toString

2015-05-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560224#comment-14560224 ] Xianyin Xin commented on YARN-3716: --- Thanks, [~leftnoteasy]. > Output node label express

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-05-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560252#comment-14560252 ] Xianyin Xin commented on YARN-3652: --- Thanks [~vinodkv]. When i said YARN-3293 and {{Sche

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560255#comment-14560255 ] Xianyin Xin commented on YARN-3547: --- Hi [~kasha], [~leftnoteasy], can we reach a consensu

[jira] [Commented] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562211#comment-14562211 ] Xianyin Xin commented on YARN-3630: --- Thanks for your comments, [~vvasudev]! {quote} your

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-05-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562221#comment-14562221 ] Xianyin Xin commented on YARN-3652: --- Thanks for your comments, [~vvasudev]. I agree with

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564039#comment-14564039 ] Xianyin Xin commented on YARN-3547: --- Thanks [~kasha], [~leftnoteasy]. [~kasha], do you h

[jira] [Commented] (YARN-3558) Additional containers getting reserved from RM in case of Fair scheduler

2015-05-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564112#comment-14564112 ] Xianyin Xin commented on YARN-3558: --- By check the log, i think i found the cause. First

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564204#comment-14564204 ] Xianyin Xin commented on YARN-3547: --- Sorry [~kasha], i misunderstood the discussion. Anyw

[jira] [Updated] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3547: -- Attachment: YARN-3547.006.patch Upload YARN-3547.006.patch. > FairScheduler: Apps that have no resource

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564211#comment-14564211 ] Xianyin Xin commented on YARN-3547: --- Sorry [~kasha], i correct the comment in https://is

[jira] [Commented] (YARN-314) Schedulers should allow resource requests of different sizes at the same priority and location

2015-05-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564274#comment-14564274 ] Xianyin Xin commented on YARN-314: -- How about use a form to organize the request, just like

[jira] [Commented] (YARN-314) Schedulers should allow resource requests of different sizes at the same priority and location

2015-05-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564301#comment-14564301 ] Xianyin Xin commented on YARN-314: -- A little more: we can limit a form only specify one typ

[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling

2015-05-29 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565733#comment-14565733 ] Xianyin Xin commented on YARN-3547: --- Thanks [~kasha]! Thanks [~kasha], [~leftnoteasy] and

[jira] [Updated] (YARN-3630) YARN should suggest a heartbeat interval for applications

2015-05-30 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3630: -- Attachment: YARN-3630.003.patch Upload a patch, adding a configured hard heartbeat interval upper limit,

[jira] [Commented] (YARN-3409) Add constraint node labels

2015-06-25 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602309#comment-14602309 ] Xianyin Xin commented on YARN-3409: --- According to my superficial understanding, rack/node

[jira] [Commented] (YARN-3409) Add constraint node labels

2015-06-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603869#comment-14603869 ] Xianyin Xin commented on YARN-3409: --- Thanks for comments, [~grey]. IMO, topology may be h

[jira] [Commented] (YARN-5188) FairScheduler performance bug

2016-05-31 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309341#comment-15309341 ] Xianyin Xin commented on YARN-5188: --- hi [~chenfolin], thanks for reporting this. {quote}

[jira] [Commented] (YARN-5241) FairScheduler repeat container completed

2016-06-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327315#comment-15327315 ] Xianyin Xin commented on YARN-5241: --- Thank you for reporting this, [~chenfolin]. So if th

[jira] [Comment Edited] (YARN-5241) FairScheduler repeat container completed

2016-06-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327315#comment-15327315 ] Xianyin Xin edited comment on YARN-5241 at 6/13/16 1:06 PM: Tha

[jira] [Created] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-06-30 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-5302: - Summary: Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II Key: YARN-5302 URL: https://issues.apache.org/jira/browse/YARN-5302

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-06-30 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Description: Different with YARN-5098, this happens at NM side. When NM recovers, credentials are read f

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-06-30 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358491#comment-15358491 ] Xianyin Xin commented on YARN-5302: --- Thanks varun. Maybe you are mentioning YARN-4783. Bu

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-06-30 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Description: Different with YARN-5098, this happens at NM side. When NM recovers, credentials are read f

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358518#comment-15358518 ] Xianyin Xin commented on YARN-5302: --- I think we have 2 options. One is persist the new to

[jira] [Created] (YARN-5305) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token III

2016-07-01 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-5305: - Summary: Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token III Key: YARN-5305 URL: https://issues.apache.org/jira/browse/YARN-5305

[jira] [Assigned] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin reassigned YARN-5302: - Assignee: Xianyin Xin > Yarn Application log Aggreagation fails due to NM can not get correct HDFS

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Attachment: YARN-5032.001.patch Upload a preview patch which update the new token to NMStateStore. > Yar

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358718#comment-15358718 ] Xianyin Xin commented on YARN-5302: --- Hi [~varun_saxena], when initAppAggregator, it uses

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-03 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360805#comment-15360805 ] Xianyin Xin commented on YARN-5302: --- Thanks [~Naganarasimha]. YARN-2704 gives fundamental

[jira] [Comment Edited] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-03 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360805#comment-15360805 ] Xianyin Xin edited comment on YARN-5302 at 7/4/16 3:51 AM: --- Thank

[jira] [Commented] (YARN-5305) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token III

2016-07-03 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360808#comment-15360808 ] Xianyin Xin commented on YARN-5305: --- Thanks [~Naganarasimha]. YARN-5175 may solve this, b

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-04 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Attachment: YARN-5032.002.patch > Yarn Application log Aggreagation fails due to NM can not get correct H

  1   2   >