[jira] [Commented] (YARN-4491) yarn list command to support filtering by tags
[
https://issues.apache.org/jira/browse/YARN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125483#comment-15125483
]
Varun Saxena commented on YARN-4491:
Sample Output :
{noformat}
varun@varun-Inspiron-5558:~/hadoop/bin$ ./yarn application -list --appStates
FINISHED --appTags tag1
Total number of applications (application-types: [], states: [FINISHED] and
tags: [tag1]):1
Application-Id Application-NameApplication-Type
User Queue State Final-State
ProgressTracking-URL
application_1454226137194_0003 Sleep job MAPREDUCE
varun default FINISHED SUCCEEDED
100%
http://varun-Inspiron-5558:19888/jobhistory/job/job_1454226137194_0003
{noformat}
> yarn list command to support filtering by tags
> --
>
> Key: YARN-4491
> URL: https://issues.apache.org/jira/browse/YARN-4491
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: YARN-4491.01.patch
>
>
> although you can filter the list of yarn applications using the --appTypes
> option; you can't use application tags. For finding applications on large
> processes, adding a --tag option would allow users to be more selective
> example:
> {code}
> yarn list --appTypes SPARK --tag production
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4491) yarn list command to support filtering by tags
[
https://issues.apache.org/jira/browse/YARN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125521#comment-15125521
]
Varun Saxena commented on YARN-4491:
Steve, would you want tags to be displayed against each application in the
output as well ?
> yarn list command to support filtering by tags
> --
>
> Key: YARN-4491
> URL: https://issues.apache.org/jira/browse/YARN-4491
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: YARN-4491.01.patch
>
>
> although you can filter the list of yarn applications using the --appTypes
> option; you can't use application tags. For finding applications on large
> processes, adding a --tag option would allow users to be more selective
> example:
> {code}
> yarn list --appTypes SPARK --tag production
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3902) Fair scheduler preempts ApplicationMaster
[ https://issues.apache.org/jira/browse/YARN-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125427#comment-15125427 ] Arun Suresh commented on YARN-3902: --- [~ka...@cloudera.com], can you do something about this ? > Fair scheduler preempts ApplicationMaster > - > > Key: YARN-3902 > URL: https://issues.apache.org/jira/browse/YARN-3902 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.3.0 > Environment: 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt2-1~bpo70+1 > (2014-12-08) x86_64 >Reporter: He Tianyi >Assignee: Arun Suresh > Original Estimate: 72h > Remaining Estimate: 72h > > YARN-2022 have fixed the similar issue related to CapacityScheduler. > However, FairScheduler still suffer, preempting AM while other normal > containers running out there. > I think we should take the same approach, avoid AM being preempted unless > there is no container running other than AM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4491) yarn list command to support filtering by tags
[
https://issues.apache.org/jira/browse/YARN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4491:
---
Attachment: YARN-4491.01.patch
> yarn list command to support filtering by tags
> --
>
> Key: YARN-4491
> URL: https://issues.apache.org/jira/browse/YARN-4491
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: YARN-4491.01.patch
>
>
> although you can filter the list of yarn applications using the --appTypes
> option; you can't use application tags. For finding applications on large
> processes, adding a --tag option would allow users to be more selective
> example:
> {code}
> yarn list --appTypes SPARK --tag production
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[
https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-3367:
Attachment: YARN-3367-YARN-2928.v1.007.patch
Thanks [~sjlee0], thanks for providing the patch having changes with
*FutureTask*, it was nice to know how to use it. As earlier you had provided
with a option to either use *CountdownLatch* or *FutureTask* , and as i was
aware of {{CountdownLatch}} went ahead with that. It was nice to know how to
use *FutureTask* and it was fitting aptly for this scenario.
bq. IMO I don't think it is an overkill. Sync or async,
Earlier had just skimmed through the *FutureTask* class and found a reference
to a *Thread* with name *runner*, hence i thought it was creating a thread to
publish it and mentioned it as *overhead*. Walked through the *FutureTask* code
thoroughly, to realize that it was not spawning a thread but just holds the
reference hence no problem.
Apart from other few comments which you had suggested and not included in your
trial patch have added a new Test class for V2.
Also would like to get further feedbacks on
* UGI or can it be handled later with Security
fixes ?
* {{TimelineClient}} class becoming bulky and little clumsy. So wanted to fork
a new Class {{TimelineClientV2Impl}} and have some sort of hierarchy with
{{TimelineClientImpl}}. For the EndUser there will be no change but just would
like to split the V2 part of {{TimelineClientImpl}} to {{TimelineClientV2Impl}}
, may be after this jira goes in can fork a new jira for this else can handle
in the same jira
> Replace starting a separate thread for post entity with event loop in
> TimelineClient
>
>
> Key: YARN-3367
> URL: https://issues.apache.org/jira/browse/YARN-3367
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-3367-YARN-2928.v1.005.patch,
> YARN-3367-YARN-2928.v1.006.patch, YARN-3367-YARN-2928.v1.007.patch,
> YARN-3367-feature-YARN-2928.003.patch,
> YARN-3367-feature-YARN-2928.v1.002.patch,
> YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch,
> sjlee-suggestion.patch
>
>
> Since YARN-3039, we add loop in TimelineClient to wait for
> collectorServiceAddress ready before posting any entity. In consumer of
> TimelineClient (like AM), we are starting a new thread for each call to get
> rid of potential deadlock in main thread. This way has at least 3 major
> defects:
> 1. The consumer need some additional code to wrap a thread before calling
> putEntities() in TimelineClient.
> 2. It cost many thread resources which is unnecessary.
> 3. The sequence of events could be out of order because each posting
> operation thread get out of waiting loop randomly.
> We should have something like event loop in TimelineClient side,
> putEntities() only put related entities into a queue of entities and a
> separated thread handle to deliver entities in queue to collector via REST
> call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4615) TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt fails occasionally
[
https://issues.apache.org/jira/browse/YARN-4615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125785#comment-15125785
]
Sunil G commented on YARN-4615:
---
Thanks [~rohithsharma] for the review and commit!
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> fails occasionally
>
>
> Key: YARN-4615
> URL: https://issues.apache.org/jira/browse/YARN-4615
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Jason Lowe
>Assignee: Sunil G
> Fix For: 2.9.0
>
> Attachments: 0001-YARN-4615.patch, 0002-YARN-4615.patch,
> 0003-YARN-4615.patch, 0004-YARN-4615.patch, 0005-YARN-4615.patch
>
>
> Sometimes
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> will fail like this:
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
> testResourceRequestRecoveryToTheRightAppAttempt[1](org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler)
> Time elapsed: 77.427 sec <<< FAILURE!
> java.lang.AssertionError: Attempt state is not correct (timedout): expected:
> SCHEDULED actual: ALLOCATED for the application attempt
> appattempt_1453254869107_0001_02
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:197)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForAttemptScheduled(MockRM.java:831)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler.testResourceRequestRecoveryToTheRightAppAttempt(TestAbstractYarnScheduler.java:572)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4100) Add Documentation for Distributed and Delegated-Centralized Node Labels feature
[
https://issues.apache.org/jira/browse/YARN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125854#comment-15125854
]
Hadoop QA commented on YARN-4100:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 4s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s
{color} | {color:green} hadoop-yarn-site in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 7s
{color} | {color:green} hadoop-yarn-site in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 3s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785212/YARN-4100.v1.005.patch
|
| JIRA Issue | YARN-4100 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
|
[jira] [Updated] (YARN-4330) MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages.
[
https://issues.apache.org/jira/browse/YARN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4330:
---
Summary: MiniYARNCluster is showing multiple Failed to instantiate default
resource calculator warning messages. (was: MiniYARNCluster prints multiple
Failed to instantiate default resource calculator warning messages)
> MiniYARNCluster is showing multiple Failed to instantiate default resource
> calculator warning messages.
>
>
> Key: YARN-4330
> URL: https://issues.apache.org/jira/browse/YARN-4330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn
>Affects Versions: 2.8.0
> Environment: OSX, JUnit
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-4330.01.patch
>
>
> Whenever I try to start a MiniYARNCluster on Branch-2 (commit #0b61cca), I
> see multiple stack traces warning me that a resource calculator plugin could
> not be created
> {code}
> (ResourceCalculatorPlugin.java:getResourceCalculatorPlugin(184)) -
> java.lang.UnsupportedOperationException: Could not determine OS: Failed to
> instantiate default resource calculator.
> java.lang.UnsupportedOperationException: Could not determine OS
> {code}
> This is a minicluster. It doesn't need resource calculation. It certainly
> doesn't need test logs being cluttered with even more stack traces which will
> only generate false alarms about tests failing.
> There needs to be a way to turn this off, and the minicluster should have it
> that way by default.
> Being ruthless and marking as a blocker, because its a fairly major
> regression for anyone testing with the minicluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4491) yarn list command to support filtering by tags
[
https://issues.apache.org/jira/browse/YARN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125821#comment-15125821
]
Varun Saxena commented on YARN-4491:
Jenkins slave went offline during build.
I have invoked the build manually from Jenkins again.
> yarn list command to support filtering by tags
> --
>
> Key: YARN-4491
> URL: https://issues.apache.org/jira/browse/YARN-4491
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: YARN-4491.01.patch
>
>
> although you can filter the list of yarn applications using the --appTypes
> option; you can't use application tags. For finding applications on large
> processes, adding a --tag option would allow users to be more selective
> example:
> {code}
> yarn list --appTypes SPARK --tag production
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4100) Add Documentation for Distributed and Delegated-Centralized Node Labels feature
[ https://issues.apache.org/jira/browse/YARN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125856#comment-15125856 ] Naganarasimha G R commented on YARN-4100: - Thanks [~devaraj.k] for triggering the build ! Now it seems be better, > Add Documentation for Distributed and Delegated-Centralized Node Labels > feature > --- > > Key: YARN-4100 > URL: https://issues.apache.org/jira/browse/YARN-4100 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: NodeLabel.html, YARN-4100.v1.001.patch, > YARN-4100.v1.002.patch, YARN-4100.v1.003.patch, YARN-4100.v1.004.patch, > YARN-4100.v1.005.patch > > > Add Documentation for Distributed Node Labels feature -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4330) MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages.
[
https://issues.apache.org/jira/browse/YARN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125874#comment-15125874
]
Hadoop QA commented on YARN-4330:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 38s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 1 new +
24 unchanged - 1 fixed = 25 total (was 25) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 17s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 52s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 38s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 12s {color}
|
[jira] [Commented] (YARN-3902) Fair scheduler preempts ApplicationMaster
[
https://issues.apache.org/jira/browse/YARN-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125609#comment-15125609
]
He Tianyi commented on YARN-3902:
-
Is it necessary that we make this 'preempting containers other than AMs' thing
optional?
I investigated preemption related codes in {{CapacityScheduler}} and there is
no option to control this behavior.
> Fair scheduler preempts ApplicationMaster
> -
>
> Key: YARN-3902
> URL: https://issues.apache.org/jira/browse/YARN-3902
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.3.0
> Environment: 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt2-1~bpo70+1
> (2014-12-08) x86_64
>Reporter: He Tianyi
>Assignee: Arun Suresh
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> YARN-2022 have fixed the similar issue related to CapacityScheduler.
> However, FairScheduler still suffer, preempting AM while other normal
> containers running out there.
> I think we should take the same approach, avoid AM being preempted unless
> there is no container running other than AM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4340) Add "list" API to reservation system
[ https://issues.apache.org/jira/browse/YARN-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125698#comment-15125698 ] Wangda Tan commented on YARN-4340: -- +1, will commit once [~seanpo03]/[~subru] confirm there's no issue with the Jenkins run. > Add "list" API to reservation system > > > Key: YARN-4340 > URL: https://issues.apache.org/jira/browse/YARN-4340 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Sean Po > Attachments: YARN-4340.v1.patch, YARN-4340.v10.patch, > YARN-4340.v11.patch, YARN-4340.v12.patch, YARN-4340.v2.patch, > YARN-4340.v3.patch, YARN-4340.v4.patch, YARN-4340.v5.patch, > YARN-4340.v6.patch, YARN-4340.v7.patch, YARN-4340.v8.patch, YARN-4340.v9.patch > > > This JIRA tracks changes to the APIs of the reservation system, and enables > querying the reservation system on which reservation exists by "time-range, > reservation-id". > YARN-4420 and YARN-2575 has a dependency on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4617) LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy
[
https://issues.apache.org/jira/browse/YARN-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125705#comment-15125705
]
Rohith Sharma K S commented on YARN-4617:
-
thanks [~jianhe] for committing the patch:-)
> LeafQueue#pendingOrderingPolicy should always use fixed ordering policy
> instead of using same as active applications ordering policy
>
>
> Key: YARN-4617
> URL: https://issues.apache.org/jira/browse/YARN-4617
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 2.8.0
>
> Attachments: 0001-YARN-4617.patch, 0001-YARN-4617.patch,
> 0002-YARN-4617.patch, 0003-YARN-4617.patch, 0004-YARN-4617.patch,
> 0005-YARN-4617.patch, 0006-YARN-4617.patch
>
>
> In discussion with [~leftnoteasy] in the JIRA
> [comment|https://issues.apache.org/jira/browse/YARN-4479?focusedCommentId=15108236=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15108236]
> pointed out that {{LeafQueue#pendingOrderingPolicy}} should NOT be assumed
> to be same as active applications ordering policy. It causes an issue when
> using fair ordering policy.
> Expectations of this JIRA should include
> # Create FifoOrderingPolicyForPendingApps which extends FifoOrderingPolicy.
> # Comparator of new ordering policy should use
> RecoveryComparator,PriorityComparator and Fifocomparator in order
> respectively.
> # Clean up {{LeafQueue#pendingOPForRecoveredApps}} which is no more required
> once new fixed ordering policy is created pending applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3998) Add retry-times to let NM re-launch container when it fails to run
[ https://issues.apache.org/jira/browse/YARN-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125597#comment-15125597 ] Jun Gong commented on YARN-3998: hi [~vvasudev], could you please help review the latest patch if you have time? Thanks! > Add retry-times to let NM re-launch container when it fails to run > -- > > Key: YARN-3998 > URL: https://issues.apache.org/jira/browse/YARN-3998 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-3998.01.patch, YARN-3998.02.patch, > YARN-3998.03.patch, YARN-3998.04.patch, YARN-3998.05.patch > > > I'd like to add a field(retry-times) in ContainerLaunchContext. When AM > launches containers, it could specify the value. Then NM will re-launch the > container 'retry-times' times when it fails to run(e.g.exit code is not 0). > It will save a lot of time. It avoids container localization. RM does not > need to re-schedule the container. And local files in container's working > directory will be left for re-use.(If container have downloaded some big > files, it does not need to re-download them when running again.) > We find it is useful in systems like Storm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4615) TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt fails occasionally
[
https://issues.apache.org/jira/browse/YARN-4615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125733#comment-15125733
]
Rohith Sharma K S commented on YARN-4615:
-
+1 LGTM
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> fails occasionally
>
>
> Key: YARN-4615
> URL: https://issues.apache.org/jira/browse/YARN-4615
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Jason Lowe
>Assignee: Sunil G
> Attachments: 0001-YARN-4615.patch, 0002-YARN-4615.patch,
> 0003-YARN-4615.patch, 0004-YARN-4615.patch, 0005-YARN-4615.patch
>
>
> Sometimes
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> will fail like this:
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
> testResourceRequestRecoveryToTheRightAppAttempt[1](org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler)
> Time elapsed: 77.427 sec <<< FAILURE!
> java.lang.AssertionError: Attempt state is not correct (timedout): expected:
> SCHEDULED actual: ALLOCATED for the application attempt
> appattempt_1453254869107_0001_02
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:197)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForAttemptScheduled(MockRM.java:831)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler.testResourceRequestRecoveryToTheRightAppAttempt(TestAbstractYarnScheduler.java:572)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[
https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125601#comment-15125601
]
Hadoop QA commented on YARN-3367:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 31s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
41s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 10s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 4s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 19s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
44s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
42s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 52s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 1s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 14s
{color} | {color:red} root: patch generated 13 new + 712 unchanged - 11 fixed =
725 total (was 723) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 32s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 31s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 37s {color}
| {color:red}
[jira] [Commented] (YARN-4615) TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt fails occasionally
[
https://issues.apache.org/jira/browse/YARN-4615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125782#comment-15125782
]
Hudson commented on YARN-4615:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9212 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9212/])
YARN-4615. Fix random test failure in (rohithsharmaks: rev
2673cbaf556eb4d0e44519cdbb8c6f0f02412a21)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> fails occasionally
>
>
> Key: YARN-4615
> URL: https://issues.apache.org/jira/browse/YARN-4615
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Jason Lowe
>Assignee: Sunil G
> Fix For: 2.9.0
>
> Attachments: 0001-YARN-4615.patch, 0002-YARN-4615.patch,
> 0003-YARN-4615.patch, 0004-YARN-4615.patch, 0005-YARN-4615.patch
>
>
> Sometimes
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> will fail like this:
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
> testResourceRequestRecoveryToTheRightAppAttempt[1](org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler)
> Time elapsed: 77.427 sec <<< FAILURE!
> java.lang.AssertionError: Attempt state is not correct (timedout): expected:
> SCHEDULED actual: ALLOCATED for the application attempt
> appattempt_1453254869107_0001_02
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:197)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForAttemptScheduled(MockRM.java:831)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler.testResourceRequestRecoveryToTheRightAppAttempt(TestAbstractYarnScheduler.java:572)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-3902) Fair scheduler preempts ApplicationMaster
[ https://issues.apache.org/jira/browse/YARN-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S reassigned YARN-3902: --- Assignee: He Tianyi (was: Arun Suresh) FYI.. I added [~He Tianyi] to contributors list. Welcome [~He Tianyi]! > Fair scheduler preempts ApplicationMaster > - > > Key: YARN-3902 > URL: https://issues.apache.org/jira/browse/YARN-3902 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.3.0 > Environment: 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt2-1~bpo70+1 > (2014-12-08) x86_64 >Reporter: He Tianyi >Assignee: He Tianyi > Original Estimate: 72h > Remaining Estimate: 72h > > YARN-2022 have fixed the similar issue related to CapacityScheduler. > However, FairScheduler still suffer, preempting AM while other normal > containers running out there. > I think we should take the same approach, avoid AM being preempted unless > there is no container running other than AM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4100) Add Documentation for Distributed and Delegated-Centralized Node Labels feature
[
https://issues.apache.org/jira/browse/YARN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125771#comment-15125771
]
Naganarasimha G R commented on YARN-4100:
-
Hi [~devaraj.k],
White space issues, tabs and ASF license issue are all related to
{{testptch/hadoop/hadoop-tools/hadoop-sls/hs_err_pid6818.log}} hence not
related to the modifications in the attached patch. Verified Locally by
applying using {{git apply --whitespace=fix}} and there were no fixes found !
> Add Documentation for Distributed and Delegated-Centralized Node Labels
> feature
> ---
>
> Key: YARN-4100
> URL: https://issues.apache.org/jira/browse/YARN-4100
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: NodeLabel.html, YARN-4100.v1.001.patch,
> YARN-4100.v1.002.patch, YARN-4100.v1.003.patch, YARN-4100.v1.004.patch,
> YARN-4100.v1.005.patch
>
>
> Add Documentation for Distributed Node Labels feature
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[ https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125256#comment-15125256 ] Varun Vasudev commented on YARN-4649: - The single log line thing was to make grep and log parsing easier. +1 for the latest patch. I'll commit this tomorrow if no one objects. > Add additional logging to some NM state store operations > > > Key: YARN-4649 > URL: https://issues.apache.org/jira/browse/YARN-4649 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Attachments: YARN-4649.001.patch, YARN-4649.002.patch > > > Adding additional logging to NM container recovery code (specifically > application/container status operations) makes it easier to debug container > recovery related issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
