[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-12-30 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787286#comment-15787286 ] Xianyin Xin commented on YARN-4090: --- Hi [~zsl2007], sorry i have moved to another project, and don't have

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-12-30 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Assignee: (was: Xianyin Xin) > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-11-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15700910#comment-15700910 ] Xianyin Xin commented on YARN-4090: --- Hi [~zhengchenyu], I've moved to another project so i don't have

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-11-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646181#comment-15646181 ] Xianyin Xin commented on YARN-4090: --- Sorry for the delay [~gsaha]. I attached the new patch based on the

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-11-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: YARN-4090.003.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-11-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646171#comment-15646171 ] Xianyin Xin commented on YARN-4090: --- Sorry for the delay [~He Tianyi]. and that i didn't examine the

[jira] [Commented] (YARN-1558) After apps are moved across queues, store new queue info in the RM state store

2016-09-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485925#comment-15485925 ] Xianyin Xin commented on YARN-1558: --- is there any update about this? > After apps are moved across

[jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement

2016-08-15 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422002#comment-15422002 ] Xianyin Xin commented on YARN-5479: --- [~He Tianyi], hope YARN-4090 can provide some information, in which

[jira] [Commented] (YARN-5310) AM restart failed because of the expired HDFS delegation tokens

2016-08-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403481#comment-15403481 ] Xianyin Xin commented on YARN-5310: --- Thanks [~aw]. Then do we have any good idea on this problem? > AM

[jira] [Updated] (YARN-5305) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token III

2016-07-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5305: -- Description: Different with YARN-5098 and YARN-5302, this problem happens when AM submits a

[jira] [Updated] (YARN-5367) HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials

2016-07-12 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5367: -- Attachment: YARN-5367.001.patch > HDFS delegation tokens in ApplicationSubmissionContext should be added

[jira] [Created] (YARN-5367) HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials

2016-07-12 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-5367: - Summary: HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials Key: YARN-5367 URL: https://issues.apache.org/jira/browse/YARN-5367

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365461#comment-15365461 ] Xianyin Xin commented on YARN-5302: --- Thanks [~varun_saxena]. Will upload a new patch as soon as possible.

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-05 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363625#comment-15363625 ] Xianyin Xin commented on YARN-5302: --- Thanks [~jianhe]. I'm also in favor of the second approach. Adopting

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-05 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362137#comment-15362137 ] Xianyin Xin commented on YARN-5302: --- Thanks [~Naganarasimha]. *we simply always manage our own tokens for

[jira] [Commented] (YARN-5175) Simplify the delegation token renewal management for YARN localization and log-aggregation

2016-07-04 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361936#comment-15361936 ] Xianyin Xin commented on YARN-5175: --- Hi [~jianhe], what's the progress on this? Is there any design or

[jira] [Created] (YARN-5310) AM restart failed because of the expired HDFS delegation tokens

2016-07-04 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-5310: - Summary: AM restart failed because of the expired HDFS delegation tokens Key: YARN-5310 URL: https://issues.apache.org/jira/browse/YARN-5310 Project: Hadoop YARN

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-04 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Attachment: YARN-5302.004.patch > Yarn Application log Aggreagation fails due to NM can not get correct

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-04 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Attachment: YARN-5302.003.patch Upload a new patch to fix the checkstyles. The javadoc warning is not

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-04 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Attachment: YARN-5032.002.patch > Yarn Application log Aggreagation fails due to NM can not get correct

[jira] [Commented] (YARN-5305) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token III

2016-07-03 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360808#comment-15360808 ] Xianyin Xin commented on YARN-5305: --- Thanks [~Naganarasimha]. YARN-5175 may solve this, but the solution

[jira] [Comment Edited] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-03 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360805#comment-15360805 ] Xianyin Xin edited comment on YARN-5302 at 7/4/16 3:51 AM: --- Thanks

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-03 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360805#comment-15360805 ] Xianyin Xin commented on YARN-5302: --- Thanks [~Naganarasimha]. YARN-2704 gives fundamental ability to

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358718#comment-15358718 ] Xianyin Xin commented on YARN-5302: --- Hi [~varun_saxena], when initAppAggregator, it uses the credentials

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Attachment: YARN-5032.001.patch Upload a preview patch which update the new token to NMStateStore. >

[jira] [Assigned] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin reassigned YARN-5302: - Assignee: Xianyin Xin > Yarn Application log Aggreagation fails due to NM can not get correct

[jira] [Created] (YARN-5305) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token III

2016-07-01 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-5305: - Summary: Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token III Key: YARN-5305 URL: https://issues.apache.org/jira/browse/YARN-5305

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358518#comment-15358518 ] Xianyin Xin commented on YARN-5302: --- I think we have 2 options. One is persist the new token to the

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Description: Different with YARN-5098, this happens at NM side. When NM recovers, credentials are read

[jira] [Commented] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358491#comment-15358491 ] Xianyin Xin commented on YARN-5302: --- Thanks varun. Maybe you are mentioning YARN-4783. But from the

[jira] [Updated] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-07-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5302: -- Description: Different with YARN-5098, this happens at NM side. When NM recovers, credentials are read

[jira] [Created] (YARN-5302) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II

2016-06-30 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-5302: - Summary: Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token II Key: YARN-5302 URL: https://issues.apache.org/jira/browse/YARN-5302

[jira] [Comment Edited] (YARN-5241) FairScheduler repeat container completed

2016-06-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327315#comment-15327315 ] Xianyin Xin edited comment on YARN-5241 at 6/13/16 1:06 PM: Thank you for

[jira] [Commented] (YARN-5241) FairScheduler repeat container completed

2016-06-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327315#comment-15327315 ] Xianyin Xin commented on YARN-5241: --- Thank you for reporting this, [~chenfolin]. So if the containers

[jira] [Commented] (YARN-5188) FairScheduler performance bug

2016-06-01 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309341#comment-15309341 ] Xianyin Xin commented on YARN-5188: --- hi [~chenfolin], thanks for reporting this. {quote} 1: application

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-05-23 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: YARN-4090.002.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-05-22 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295948#comment-15295948 ] Xianyin Xin commented on YARN-4090: --- Sorry for the delay [~yufeigu]. {quote} do you mean "bring down" or

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261837#comment-15261837 ] Xianyin Xin commented on YARN-4090: --- It seems the failed UT with JDK1.8.0_92 and JDK1.7.0_95 is not

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261546#comment-15261546 ] Xianyin Xin commented on YARN-4090: --- Sorry for the delay, [~kasha] [~yufeigu]. Just uploaded the patch

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: YARN-4090.001.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-04-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257960#comment-15257960 ] Xianyin Xin commented on YARN-4090: --- Thanks, [~kasha], will upload a new patch as soon as possible. >

[jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups

2016-03-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190517#comment-15190517 ] Xianyin Xin commented on YARN-1297: --- Hi [~asuresh], could you please update the patch based on the latest

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-03-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190514#comment-15190514 ] Xianyin Xin commented on YARN-4090: --- Let's move to YARN-1297 to continue the discussion. > Make

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2016-03-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190506#comment-15190506 ] Xianyin Xin commented on YARN-4090: --- Sorry [~kasha], i didn't notice that, and thanks for reminding. Yes,

[jira] [Commented] (YARN-4752) [Umbrella] FairScheduler: Improve preemption

2016-03-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182566#comment-15182566 ] Xianyin Xin commented on YARN-4752: --- hi [~kasha], can we consider to decouple the scheduling and

[jira] [Commented] (YARN-4414) Nodemanager connection errors are retried at multiple levels

2016-01-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088804#comment-15088804 ] Xianyin Xin commented on YARN-4414: --- Hi [~lichangleo], need we also revisit the two layer retries in

[jira] [Commented] (YARN-3870) Providing raw container request information for fine scheduling

2015-12-29 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074755#comment-15074755 ] Xianyin Xin commented on YARN-3870: --- +1 for adding an unique id for a resource request, but i would

[jira] [Commented] (YARN-4477) FairScheduler: encounter infinite loop in attemptScheduling

2015-12-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063538#comment-15063538 ] Xianyin Xin commented on YARN-4477: --- cc [~asuresh]. > FairScheduler: encounter infinite loop in

[jira] [Commented] (YARN-4198) CapacityScheduler locking / synchronization improvements

2015-12-16 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059730#comment-15059730 ] Xianyin Xin commented on YARN-4198: --- Hi [~curino], does this improvement only considers the locking /

[jira] [Commented] (YARN-4415) Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned

2015-12-14 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057134#comment-15057134 ] Xianyin Xin commented on YARN-4415: --- Hi [~leftnoteasy], thanks for you comments. {quote} You can see that

[jira] [Commented] (YARN-4415) Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned

2015-12-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046147#comment-15046147 ] Xianyin Xin commented on YARN-4415: --- sorry [~Naganarasimha], i missed that, thanks for you correction.

[jira] [Commented] (YARN-4415) Scheduler Web Ui shows max capacity for the queue is 100% but when we submit application doesnt get assigned

2015-12-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044498#comment-15044498 ] Xianyin Xin commented on YARN-4415: --- Sorry for the late, [~Naganarasimha]. I don't know i understand

[jira] [Commented] (YARN-4403) (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period

2015-12-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037185#comment-15037185 ] Xianyin Xin commented on YARN-4403: --- And will provide new patch of YARN-4177 once this is in. >

[jira] [Commented] (YARN-4403) (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period

2015-12-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037284#comment-15037284 ] Xianyin Xin commented on YARN-4403: --- Thanks, [~sunilg]. > (AM/NM/Container)LivelinessMonitor should use

[jira] [Commented] (YARN-4403) (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period

2015-12-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037180#comment-15037180 ] Xianyin Xin commented on YARN-4403: --- hi [~djp], this is a good suggestion, and YARN-4177 provides some

[jira] [Resolved] (YARN-3639) It takes too long time for RM to recover all apps if the original active RM and NN go down at the same time.

2015-10-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin resolved YARN-3639. --- Resolution: Fixed This has been resolved by YARN-4041, so close it. > It takes too long time for RM

[jira] [Commented] (YARN-4189) Capacity Scheduler : Improve location preference waiting mechanism

2015-09-21 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901736#comment-14901736 ] Xianyin Xin commented on YARN-4189: --- [~leftnoteasy], convincing analysis. It's fine X << Y and X is close

[jira] [Commented] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-21 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900450#comment-14900450 ] Xianyin Xin commented on YARN-4177: --- Hi [~ste...@apache.org], thanks for your comment. I've read your

[jira] [Commented] (YARN-4190) missing container information in FairScheduler preemption log.

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876762#comment-14876762 ] Xianyin Xin commented on YARN-4190: --- Hi [~zxu], will we consider to fix the issue in YARN-4134 or

[jira] [Updated] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4177: -- Attachment: YARN-4177.002.patch Upload patch ver2. > yarn.util.Clock should not be used to time a

[jira] [Commented] (YARN-4189) Capacity Scheduler : Improve location preference waiting mechanism

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876871#comment-14876871 ] Xianyin Xin commented on YARN-4189: --- Hi [~leftnoteasy], just go through the doc. Please correct me if i

[jira] [Assigned] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin reassigned YARN-4177: - Assignee: Xianyin Xin > yarn.util.Clock should not be used to time a duration or time interval >

[jira] [Created] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-17 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-4177: - Summary: yarn.util.Clock should not be used to time a duration or time interval Key: YARN-4177 URL: https://issues.apache.org/jira/browse/YARN-4177 Project: Hadoop YARN

[jira] [Updated] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4177: -- Attachment: YARN-4177.001.patch Provide a MonotonicClock. > yarn.util.Clock should not be used to time

[jira] [Commented] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval

2015-09-17 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14802911#comment-14802911 ] Xianyin Xin commented on YARN-4177: --- In a following steps I would try to fix several important places

[jira] [Commented] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2015-09-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742795#comment-14742795 ] Xianyin Xin commented on YARN-4134: --- Just found this issue duplicated YARN-3405, but the two has

[jira] [Commented] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account

2015-09-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742793#comment-14742793 ] Xianyin Xin commented on YARN-4120: --- hi [~asuresh], thanks for your comment. I've go through YARN-2154, i

[jira] [Commented] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account

2015-09-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742812#comment-14742812 ] Xianyin Xin commented on YARN-4120: --- Hi [~kasha], [~asuresh], [~ashwinshankar77], now both the preemption

[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request

2015-09-13 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742797#comment-14742797 ] Xianyin Xin commented on YARN-2154: --- The new logic would solve the issue raised in YARN-3405 and

[jira] [Updated] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2015-09-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4134: -- Attachment: YARN-4134.002.patch Remove testing remnant. > FairScheduler preemption stops at queue level

[jira] [Updated] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2015-09-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4134: -- Attachment: YARN-4134.001.patch Upload a patch for preview. > FairScheduler preemption stops at queue

[jira] [Updated] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2015-09-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4134: -- Attachment: YARN-4134.003.patch A tiny fix. > FairScheduler preemption stops at queue level that all

[jira] [Commented] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account

2015-09-10 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739974#comment-14739974 ] Xianyin Xin commented on YARN-4120: --- Link to YARN-4134, the two can be solved together. >

[jira] [Updated] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2015-09-09 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4134: -- Description: Now FairScheudler uses a choose-a-candidate method to select a container from leaf queues

[jira] [Assigned] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2015-09-09 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin reassigned YARN-4134: - Assignee: Xianyin Xin > FairScheduler preemption stops at queue level that all child queues are

[jira] [Commented] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account

2015-09-08 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735952#comment-14735952 ] Xianyin Xin commented on YARN-4120: --- Hi [~kasha], there's another issue in the current preemption logic,

[jira] [Commented] (YARN-4133) Containers to be preempted leaks in FairScheduler preemption logic.

2015-09-08 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735987#comment-14735987 ] Xianyin Xin commented on YARN-4133: --- Of course we can also address these problems one by one in different

[jira] [Commented] (YARN-4133) Containers to be preempted leaks in FairScheduler preemption logic.

2015-09-08 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735959#comment-14735959 ] Xianyin Xin commented on YARN-4133: --- Hi [~zxu], it seems the current preemption logic has many problems.

[jira] [Created] (YARN-4134) FairScheduler preemption stops at queue level that all child queues are not over their fairshare

2015-09-08 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-4134: - Summary: FairScheduler preemption stops at queue level that all child queues are not over their fairshare Key: YARN-4134 URL: https://issues.apache.org/jira/browse/YARN-4134

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2015-09-08 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736068#comment-14736068 ] Xianyin Xin commented on YARN-4090: --- Hi [~leftnoteasy], [~kasha], would you please take a look? since

[jira] [Commented] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account

2015-09-08 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736130#comment-14736130 ] Xianyin Xin commented on YARN-4120: --- Create YARN-4134 to track it. > FSAppAttempt.getResourceUsage()

[jira] [Commented] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account

2015-09-06 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14733235#comment-14733235 ] Xianyin Xin commented on YARN-4120: --- Thanks [~kasha]. How about distinguishing getResourceUsage() (the

[jira] [Created] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account

2015-09-06 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-4120: - Summary: FSAppAttempt.getResourceUsage() should not take preemptedResource into account Key: YARN-4120 URL: https://issues.apache.org/jira/browse/YARN-4120 Project: Hadoop

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2015-09-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: YARN-4090-preview.patch > Make Collections.sort() more efficient in FSParentQueue.java >

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2015-09-02 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: YARN-4090-TestResult.pdf A simple fix and the correspond test result are submitted. The

[jira] [Assigned] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2015-08-28 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin reassigned YARN-4090: - Assignee: Xianyin Xin Make Collections.sort() more efficient in FSParentQueue.java

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2015-08-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Summary: Make Collections.sort() more efficient in FSParentQueue.java (was: Make Collections.sort()

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2015-08-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716785#comment-14716785 ] Xianyin Xin commented on YARN-4090: --- We should also pay attention to the ReadLock.lock()

[jira] [Created] (YARN-4090) Make Collections.sort() more efficient in FSParent.java

2015-08-27 Thread Xianyin Xin (JIRA)
Xianyin Xin created YARN-4090: - Summary: Make Collections.sort() more efficient in FSParent.java Key: YARN-4090 URL: https://issues.apache.org/jira/browse/YARN-4090 Project: Hadoop YARN Issue

[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

2015-08-27 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Attachment: sampling2.jpg sampling1.jpg I construct a queue hierarchy with 3 levels,

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-08-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701291#comment-14701291 ] Xianyin Xin commented on YARN-3652: --- A simple introduction of the preview patch:

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-08-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701267#comment-14701267 ] Xianyin Xin commented on YARN-3652: --- In the patch i used functions from HADOOP-12338. A

[jira] [Commented] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-08-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701296#comment-14701296 ] Xianyin Xin commented on YARN-3652: --- Hi [~sunilg], [~vvasudev], would you please have a

[jira] [Updated] (YARN-3652) A SchedulerMetrics may be need for evaluating the scheduler's performance

2015-08-18 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-3652: -- Attachment: YARN-3652-preview.patch A preview patch submitted. A SchedulerMetrics may be need for

[jira] [Commented] (YARN-4002) make ResourceTrackerService.nodeHeartbeat more concurrent

2015-08-07 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662744#comment-14662744 ] Xianyin Xin commented on YARN-4002: --- +1 for the proposal. [~leftnoteasy], I think

[jira] [Commented] (YARN-3931) default-node-label-expression doesn’t apply when an application is submitted by RM rest api

2015-07-16 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630659#comment-14630659 ] Xianyin Xin commented on YARN-3931: --- This reminds me an earlier trouble i have met. Hi

[jira] [Commented] (YARN-3922) Introduce adaptive heartbeat between RM and NM

2015-07-14 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627374#comment-14627374 ] Xianyin Xin commented on YARN-3922: --- Maybe we should reconsider to rename the feature

[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level

2015-07-14 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627404#comment-14627404 ] Xianyin Xin commented on YARN-3885: --- Agree [~leftnoteasy]. suppose we have a case

[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level

2015-07-05 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614194#comment-14614194 ] Xianyin Xin commented on YARN-3885: --- Hi Ajith, CHILDRENPREEMPTABLE may be over counted.

[jira] [Commented] (YARN-3409) Add constraint node labels

2015-06-26 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603869#comment-14603869 ] Xianyin Xin commented on YARN-3409: --- Thanks for comments, [~grey]. IMO, topology may be

  1   2   >