[jira] [Commented] (YARN-3102) Decommisioned Nodes not listed in Web UI
[
https://issues.apache.org/jira/browse/YARN-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120276#comment-15120276
]
Hadoop QA commented on YARN-3102:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 47s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 1s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784702/YARN-3102-v7.patch |
| JIRA Issue | YARN-3102 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
|
[jira] [Created] (YARN-4649) Add additional logging to some NM state store operations
Sidharta Seethana created YARN-4649: --- Summary: Add additional logging to some NM state store operations Key: YARN-4649 URL: https://issues.apache.org/jira/browse/YARN-4649 Project: Hadoop YARN Issue Type: Improvement Reporter: Sidharta Seethana Assignee: Sidharta Seethana Priority: Minor Adding additional logging to NM container recovery code (specifically application/container status operations) makes it easier to debug container recovery related issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4340) Add "list" API to reservation system
[ https://issues.apache.org/jira/browse/YARN-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120549#comment-15120549 ] Sean Po commented on YARN-4340: --- The test failures above have all been addressed in a previous comment, except for TestYarnCLI. This test passes on my local machine. > Add "list" API to reservation system > > > Key: YARN-4340 > URL: https://issues.apache.org/jira/browse/YARN-4340 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Sean Po > Attachments: YARN-4340.v1.patch, YARN-4340.v10.patch, > YARN-4340.v11.patch, YARN-4340.v2.patch, YARN-4340.v3.patch, > YARN-4340.v4.patch, YARN-4340.v5.patch, YARN-4340.v6.patch, > YARN-4340.v7.patch, YARN-4340.v8.patch, YARN-4340.v9.patch > > > This JIRA tracks changes to the APIs of the reservation system, and enables > querying the reservation system on which reservation exists by "time-range, > reservation-id". > YARN-4420 has a dependency on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4340) Add "list" API to reservation system
[ https://issues.apache.org/jira/browse/YARN-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120592#comment-15120592 ] Wangda Tan commented on YARN-4340: -- Manually kicked Jenkins. > Add "list" API to reservation system > > > Key: YARN-4340 > URL: https://issues.apache.org/jira/browse/YARN-4340 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Sean Po > Attachments: YARN-4340.v1.patch, YARN-4340.v10.patch, > YARN-4340.v11.patch, YARN-4340.v2.patch, YARN-4340.v3.patch, > YARN-4340.v4.patch, YARN-4340.v5.patch, YARN-4340.v6.patch, > YARN-4340.v7.patch, YARN-4340.v8.patch, YARN-4340.v9.patch > > > This JIRA tracks changes to the APIs of the reservation system, and enables > querying the reservation system on which reservation exists by "time-range, > reservation-id". > YARN-4420 has a dependency on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4643) Container recovery is broken with delegating container runtime
[
https://issues.apache.org/jira/browse/YARN-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120322#comment-15120322
]
Hadoop QA commented on YARN-4643:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 31s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 0s {color} |
{color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784724/YARN-4643.002.patch |
| JIRA Issue | YARN-4643 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7b6db2f9b388 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fb238d7 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |
[jira] [Created] (YARN-4650) The AM should be launched with its own set of configs instead of using the NM's configs
Daniel Templeton created YARN-4650: -- Summary: The AM should be launched with its own set of configs instead of using the NM's configs Key: YARN-4650 URL: https://issues.apache.org/jira/browse/YARN-4650 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Daniel Templeton Assignee: Daniel Templeton There are cases, such as a secure LDAP configuration where the NM may need access to credentials that should not be exposed to the user. As long as the NM and AM share the same configuration files, anything exposed to the NM is also exposed to the AM and hence the users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4353) Provide short circuit user group mapping for NM/AM
[
https://issues.apache.org/jira/browse/YARN-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120518#comment-15120518
]
Daniel Templeton commented on YARN-4353:
The base issue here is that in a secure LDAP environment, the NM wants
credentials to resolve the groups even though it doesn't use them. The issue
there is that the AM shares the config with the NM, meaning that exposing the
credentials to the NM opens them up to the users.
[~vinodkv] said earlier:
bq. Regarding this JIRA, a lots of places in NM do depend on the user/group
information
I looked, but I couldn't find them. I even instrumented the
{{UserGroupInformation.getGroupNames()}} method and ran some different kinds of
jobs: nothing. Outside of code to facilitate testing,
{{UserGroupInformation.getGroupNames()}} is the only thing that uses the groups
data in the NM, MR AM, or dist shell AM. What am I missing?
Grabbing the user group data and localizing it for the AM is a really nice
idea, but given that no one is using the data, it seems like extra work for no
present reason. Given that the existence of the {{NullGroupMapping}}
(HADOOP-12566) provides a clean way to avoid sharing the credentials, however,
I'm happy to convert this JIRA over to storing the groups data for the AM that
may one day eventually need it.
> Provide short circuit user group mapping for NM/AM
> --
>
> Key: YARN-4353
> URL: https://issues.apache.org/jira/browse/YARN-4353
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4353.prelim.patch
>
>
> When the NM launches an AM, the {{ContainerLocalizer}} gets the current user
> from {{UserGroupInformation}}, which triggers user group mapping, even though
> the user groups are never accessed. If secure LDAP is configured for group
> mapping, then there are some additional complications created by the
> unnecessary group resolution. Additionally, it adds unnecessary latency to
> the container launch time.
> To address the issue, before getting the current user, the
> {{ContainerLocalizer}} should configure {{UserGroupInformation}} with a null
> group mapping service that quickly and quietly returns an empty group list
> for all users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[ https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120497#comment-15120497 ] Sangjin Lee commented on YARN-3367: --- Thanks for the latest patch [~Naganarasimha]! I'm going over it, and will post my review pretty soon. Regarding your point about UGI.doAs(), I think it is an important a bit challenging question. First of all, is it fair to assume that a TimelineClient belongs to a single user for the duration of the client? Or can it submit entities from multiple users for the duration of the client? If the former, then possibly we could pass in the UGI to the constructor so that the actual calls are made under doAs(). If the latter, then we might have to change the method signature of putEntities() to pass the UGI. Also, if the latter (a single timeline client needing to handle multiple users), combining entities into a single call becomes immediately complicated. In that case, we probably shouldn't combine entities from different users. Thoughts? > Replace starting a separate thread for post entity with event loop in > TimelineClient > > > Key: YARN-3367 > URL: https://issues.apache.org/jira/browse/YARN-3367 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: YARN-3367-YARN-2928.v1.005.patch, > YARN-3367-YARN-2928.v1.006.patch, YARN-3367-feature-YARN-2928.003.patch, > YARN-3367-feature-YARN-2928.v1.002.patch, > YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch > > > Since YARN-3039, we add loop in TimelineClient to wait for > collectorServiceAddress ready before posting any entity. In consumer of > TimelineClient (like AM), we are starting a new thread for each call to get > rid of potential deadlock in main thread. This way has at least 3 major > defects: > 1. The consumer need some additional code to wrap a thread before calling > putEntities() in TimelineClient. > 2. It cost many thread resources which is unnecessary. > 3. The sequence of events could be out of order because each posting > operation thread get out of waiting loop randomly. > We should have something like event loop in TimelineClient side, > putEntities() only put related entities into a queue of entities and a > separated thread handle to deliver entities in queue to collector via REST > call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4428) Redirect RM page to AHS page when AHS turned on and RM page is not avaialable
[
https://issues.apache.org/jira/browse/YARN-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120517#comment-15120517
]
Hadoop QA commented on YARN-4428:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 6s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 11s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 45s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784731/YARN-4428.8.patch |
| JIRA Issue | YARN-4428 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
|
[jira] [Commented] (YARN-4629) Distributed shell breaks under strong security
[
https://issues.apache.org/jira/browse/YARN-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120550#comment-15120550
]
Hadoop QA commented on YARN-4629:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 1 new +
103 unchanged - 0 fixed = 104 total (was 103) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 39s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 53s
{color} | {color:green} hadoop-yarn-applications-distributedshell in the patch
passed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 48s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 5s {color} |
{color:red} hadoop-yarn-applications-distributedshell in the patch failed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s
[jira] [Updated] (YARN-4649) Add additional logging to some NM state store operations
[ https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4649: Attachment: YARN-4649.001.patch Uploaded a patch - added additional debug logging to NM state store operations corresponding to key container/application lifecycle events. > Add additional logging to some NM state store operations > > > Key: YARN-4649 > URL: https://issues.apache.org/jira/browse/YARN-4649 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Attachments: YARN-4649.001.patch > > > Adding additional logging to NM container recovery code (specifically > application/container status operations) makes it easier to debug container > recovery related issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4617) LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy
[
https://issues.apache.org/jira/browse/YARN-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120703#comment-15120703
]
Rohith Sharma K S commented on YARN-4617:
-
Right, no more required to recreate the pendingApps list object. I will upload
patch with this change.
> LeafQueue#pendingOrderingPolicy should always use fixed ordering policy
> instead of using same as active applications ordering policy
>
>
> Key: YARN-4617
> URL: https://issues.apache.org/jira/browse/YARN-4617
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4617.patch, 0002-YARN-4617.patch
>
>
> In discussion with [~leftnoteasy] in the JIRA
> [comment|https://issues.apache.org/jira/browse/YARN-4479?focusedCommentId=15108236=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15108236]
> pointed out that {{LeafQueue#pendingOrderingPolicy}} should NOT be assumed
> to be same as active applications ordering policy. It causes an issue when
> using fair ordering policy.
> Expectations of this JIRA should include
> # Create FifoOrderingPolicyForPendingApps which extends FifoOrderingPolicy.
> # Comparator of new ordering policy should use
> RecoveryComparator,PriorityComparator and Fifocomparator in order
> respectively.
> # Clean up {{LeafQueue#pendingOPForRecoveredApps}} which is no more required
> once new fixed ordering policy is created pending applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4617) LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy
[
https://issues.apache.org/jira/browse/YARN-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120720#comment-15120720
]
Rohith Sharma K S commented on YARN-4617:
-
Missed to maintain order of patch number. Again uploaded the patch with new
sequence number {{0003-YARN-4617.patch}}, Kindly review {{0003-YARN-4617.patch}}
> LeafQueue#pendingOrderingPolicy should always use fixed ordering policy
> instead of using same as active applications ordering policy
>
>
> Key: YARN-4617
> URL: https://issues.apache.org/jira/browse/YARN-4617
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4617.patch, 0001-YARN-4617.patch,
> 0002-YARN-4617.patch, 0003-YARN-4617.patch
>
>
> In discussion with [~leftnoteasy] in the JIRA
> [comment|https://issues.apache.org/jira/browse/YARN-4479?focusedCommentId=15108236=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15108236]
> pointed out that {{LeafQueue#pendingOrderingPolicy}} should NOT be assumed
> to be same as active applications ordering policy. It causes an issue when
> using fair ordering policy.
> Expectations of this JIRA should include
> # Create FifoOrderingPolicyForPendingApps which extends FifoOrderingPolicy.
> # Comparator of new ordering policy should use
> RecoveryComparator,PriorityComparator and Fifocomparator in order
> respectively.
> # Clean up {{LeafQueue#pendingOPForRecoveredApps}} which is no more required
> once new fixed ordering policy is created pending applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[ https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120757#comment-15120757 ] Naganarasimha G R (Naga) commented on YARN-3367: Hi Sangjin, Skimmed throught the comments, Thanks for patiently reviewing, yes please share the prototype. Also just one doubt all the entityholders need not be FutureTask as its only required for sync calls, so would it be sometimes overkill (cases where only asyncs are used)? Regards, + Naga > Replace starting a separate thread for post entity with event loop in > TimelineClient > > > Key: YARN-3367 > URL: https://issues.apache.org/jira/browse/YARN-3367 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: YARN-3367-YARN-2928.v1.005.patch, > YARN-3367-YARN-2928.v1.006.patch, YARN-3367-feature-YARN-2928.003.patch, > YARN-3367-feature-YARN-2928.v1.002.patch, > YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch > > > Since YARN-3039, we add loop in TimelineClient to wait for > collectorServiceAddress ready before posting any entity. In consumer of > TimelineClient (like AM), we are starting a new thread for each call to get > rid of potential deadlock in main thread. This way has at least 3 major > defects: > 1. The consumer need some additional code to wrap a thread before calling > putEntities() in TimelineClient. > 2. It cost many thread resources which is unnecessary. > 3. The sequence of events could be out of order because each posting > operation thread get out of waiting loop randomly. > We should have something like event loop in TimelineClient side, > putEntities() only put related entities into a queue of entities and a > separated thread handle to deliver entities in queue to collector via REST > call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
[ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120605#comment-15120605 ] Wangda Tan commented on YARN-3215: -- [~Naganarasimha]/[~sunilg], Thanks for sharing your thoughts, my major concern of excluding NO_LABEL by default is. Applications could ask nothing in the first AllocateRequest, get headroom back and decide how many resources to ask, we cannot presume application's behavior. Telling an app: "your headroom is 0" but there're plenty of resources looks not good enough to me. The solution for longer term is to add by-partition headroom, it will be a clear protocol to return what application requested. I won't strongly against returning only requested headroom to app, but to make it behavior compatible (previously we returns headroom even if application doesn't ask) and code simpler, I would prefer to add NO_LABEL by default. Thoughts? > Respect labels in CapacityScheduler when computing headroom > --- > > Key: YARN-3215 > URL: https://issues.apache.org/jira/browse/YARN-3215 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Naganarasimha G R > Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch, > YARN-3215.v2.002.patch > > > In existing CapacityScheduler, when computing headroom of an application, it > will only consider "non-labeled" nodes of this application. > But it is possible the application is asking for labeled resources, so > headroom-by-label (like 5G resource available under node-label=red) is > required to get better resource allocation and avoid deadlocks such as > MAPREDUCE-5928. > This JIRA could involve both API changes (such as adding a > label-to-available-resource map in AllocateResponse) and also internal > changes in CapacityScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4633) TestRMRestart.testRMRestartAfterPreemption fails intermittently in trunk
[
https://issues.apache.org/jira/browse/YARN-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120700#comment-15120700
]
Rohith Sharma K S commented on YARN-4633:
-
[~bibinchundatt] thanks for the patch. I think your patch should solve the
issue.
Are you able to reproduce the test failure? For ensuring fix is proper, can you
give reproducible steps so that I can test the patch.
> TestRMRestart.testRMRestartAfterPreemption fails intermittently in trunk
> -
>
> Key: YARN-4633
> URL: https://issues.apache.org/jira/browse/YARN-4633
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.9.0
> Environment: Jenkin
>Reporter: Rohith Sharma K S
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4633.patch
>
>
> Jenkins
> [Build|https://builds.apache.org/job/PreCommit-YARN-Build/10366/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt]
> failed for below test case,
> {code}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
> support was removed in 8.0
> Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> Tests run: 54, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 455.808 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartAfterPreemption[0](org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 60.145 sec <<< FAILURE!
> java.lang.AssertionError: Attempt state is not correct (timedout): expected:
> SCHEDULED actual: FAILED for the application attempt
> appattempt_1453461355278_0001_04
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:197)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForAttemptScheduled(MockRM.java:831)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAM(MockRM.java:818)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartAfterPreemption(TestRMRestart.java:2352)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4617) LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy
[
https://issues.apache.org/jira/browse/YARN-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4617:
Attachment: 0001-YARN-4617.patch
updating the patch fixing Jian He comment. Kindly review the updated patch.
> LeafQueue#pendingOrderingPolicy should always use fixed ordering policy
> instead of using same as active applications ordering policy
>
>
> Key: YARN-4617
> URL: https://issues.apache.org/jira/browse/YARN-4617
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4617.patch, 0001-YARN-4617.patch,
> 0002-YARN-4617.patch
>
>
> In discussion with [~leftnoteasy] in the JIRA
> [comment|https://issues.apache.org/jira/browse/YARN-4479?focusedCommentId=15108236=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15108236]
> pointed out that {{LeafQueue#pendingOrderingPolicy}} should NOT be assumed
> to be same as active applications ordering policy. It causes an issue when
> using fair ordering policy.
> Expectations of this JIRA should include
> # Create FifoOrderingPolicyForPendingApps which extends FifoOrderingPolicy.
> # Comparator of new ordering policy should use
> RecoveryComparator,PriorityComparator and Fifocomparator in order
> respectively.
> # Clean up {{LeafQueue#pendingOPForRecoveredApps}} which is no more required
> once new fixed ordering policy is created pending applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4617) LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy
[
https://issues.apache.org/jira/browse/YARN-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4617:
Attachment: 0003-YARN-4617.patch
> LeafQueue#pendingOrderingPolicy should always use fixed ordering policy
> instead of using same as active applications ordering policy
>
>
> Key: YARN-4617
> URL: https://issues.apache.org/jira/browse/YARN-4617
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4617.patch, 0001-YARN-4617.patch,
> 0002-YARN-4617.patch, 0003-YARN-4617.patch
>
>
> In discussion with [~leftnoteasy] in the JIRA
> [comment|https://issues.apache.org/jira/browse/YARN-4479?focusedCommentId=15108236=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15108236]
> pointed out that {{LeafQueue#pendingOrderingPolicy}} should NOT be assumed
> to be same as active applications ordering policy. It causes an issue when
> using fair ordering policy.
> Expectations of this JIRA should include
> # Create FifoOrderingPolicyForPendingApps which extends FifoOrderingPolicy.
> # Comparator of new ordering policy should use
> RecoveryComparator,PriorityComparator and Fifocomparator in order
> respectively.
> # Clean up {{LeafQueue#pendingOPForRecoveredApps}} which is no more required
> once new fixed ordering policy is created pending applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3946) Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message
[ https://issues.apache.org/jira/browse/YARN-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120810#comment-15120810 ] Akira AJISAKA commented on YARN-3946: - Hi [~leftnoteasy], TestNetworkedJob is still failing. Would you please review the patch in MAPREDUCE-6579? > Update exact reason as to why a submitted app is in ACCEPTED state to app's > diagnostic message > -- > > Key: YARN-3946 > URL: https://issues.apache.org/jira/browse/YARN-3946 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Sumit Nigam >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: 3946WebImages.zip, YARN-3946.v1.001.patch, > YARN-3946.v1.002.patch, YARN-3946.v1.003.Images.zip, YARN-3946.v1.003.patch, > YARN-3946.v1.004.patch, YARN-3946.v1.005.patch, YARN-3946.v1.006.patch, > YARN-3946.v1.007.patch, YARN-3946.v1.008.patch > > > Currently there is no direct way to get the exact reason as to why a > submitted app is still in ACCEPTED state. It should be possible to know > through RM REST API as to what aspect is not being met - say, queue limits > being reached, or core/ memory requirement not being met, or AM limit being > reached, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[
https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-3367:
--
Attachment: sjlee-suggestion.patch
Diff against the v.6 patch.
Hopefully it illustrates how simpler things can become. This is in line with
the suggestion that we should use the higher level concurrency abstractions
whenever possible to reduce cruft and increase safety and performance.
IMO I don't think it is an overkill. Sync or async, {{FutureTask}} is pretty
smart about how to manage things. Especially since we're not really getting the
"result" from the work, the only thing it might add in the case of async is
setting the exception on the result when it is not going to be used. However,
if no one "gets" the exception, it will be subject to garbage collection
promptly.
> Replace starting a separate thread for post entity with event loop in
> TimelineClient
>
>
> Key: YARN-3367
> URL: https://issues.apache.org/jira/browse/YARN-3367
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-3367-YARN-2928.v1.005.patch,
> YARN-3367-YARN-2928.v1.006.patch, YARN-3367-feature-YARN-2928.003.patch,
> YARN-3367-feature-YARN-2928.v1.002.patch,
> YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch,
> sjlee-suggestion.patch
>
>
> Since YARN-3039, we add loop in TimelineClient to wait for
> collectorServiceAddress ready before posting any entity. In consumer of
> TimelineClient (like AM), we are starting a new thread for each call to get
> rid of potential deadlock in main thread. This way has at least 3 major
> defects:
> 1. The consumer need some additional code to wrap a thread before calling
> putEntities() in TimelineClient.
> 2. It cost many thread resources which is unnecessary.
> 3. The sequence of events could be out of order because each posting
> operation thread get out of waiting loop randomly.
> We should have something like event loop in TimelineClient side,
> putEntities() only put related entities into a queue of entities and a
> separated thread handle to deliver entities in queue to collector via REST
> call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4617) LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy
[
https://issues.apache.org/jira/browse/YARN-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120653#comment-15120653
]
Naganarasimha G R commented on YARN-4617:
-
[~rohithsharma],
Apart from the [~jianhe]'s comment, Patch LGTM...
> LeafQueue#pendingOrderingPolicy should always use fixed ordering policy
> instead of using same as active applications ordering policy
>
>
> Key: YARN-4617
> URL: https://issues.apache.org/jira/browse/YARN-4617
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4617.patch, 0002-YARN-4617.patch
>
>
> In discussion with [~leftnoteasy] in the JIRA
> [comment|https://issues.apache.org/jira/browse/YARN-4479?focusedCommentId=15108236=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15108236]
> pointed out that {{LeafQueue#pendingOrderingPolicy}} should NOT be assumed
> to be same as active applications ordering policy. It causes an issue when
> using fair ordering policy.
> Expectations of this JIRA should include
> # Create FifoOrderingPolicyForPendingApps which extends FifoOrderingPolicy.
> # Comparator of new ordering policy should use
> RecoveryComparator,PriorityComparator and Fifocomparator in order
> respectively.
> # Clean up {{LeafQueue#pendingOPForRecoveredApps}} which is no more required
> once new fixed ordering policy is created pending applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4411) ResourceManager IllegalArgumentException error
[
https://issues.apache.org/jira/browse/YARN-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120871#comment-15120871
]
Devaraj K commented on YARN-4411:
-
Thanks [~bibinchundatt] for the updated patch.
- I don't understand why do we need this, Do you see any problem if we invoke
attempt.createApplicationAttemptReport() when the state is other than
RMAppAttemptState.FINAL_SAVING? I think we can we can create
ApplicationAttemptReport irrespective of the state.
{code:xml}
+ if (!rmAppAttemptState.equals(RMAppAttemptState.FINAL_SAVING)) {
{code}
- Can you tell me when the application attempt state would be null? If it is
not really needed we can remove this assertion and if you have decided to keep
this statement then please add an assertion message.
{code:xml}
+ assertTrue(null != attemptreport.getYarnApplicationAttemptState());
{code}
> ResourceManager IllegalArgumentException error
> --
>
> Key: YARN-4411
> URL: https://issues.apache.org/jira/browse/YARN-4411
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: yarntime
>Assignee: Bibin A Chundatt
> Attachments: 0002-YARN-4411.patch, YARN-4411.001.patch
>
>
> in version 2.7.1, line 1914 may cause IllegalArgumentException in
> RMAppAttemptImpl:
> YarnApplicationAttemptState.valueOf(this.getState().toString())
> cause by this.getState() returns type RMAppAttemptState which may not be
> converted to YarnApplicationAttemptState.
> {noformat}
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState.LAUNCHED_UNMANAGED_SAVING
> at java.lang.Enum.valueOf(Enum.java:236)
> at
> org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState.valueOf(YarnApplicationAttemptState.java:27)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.createApplicationAttemptReport(RMAppAttemptImpl.java:1870)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationAttemptReport(ClientRMService.java:355)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationAttemptReport(ApplicationClientProtocolPBServiceImpl.java:355)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
[
https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120725#comment-15120725
]
Naganarasimha G R commented on YARN-3215:
-
Thanks [~sunilg] & [~wangda] for sharing your views.
bq. Applications could ask nothing in the first AllocateRequest, get headroom
back and decide how many resources to ask, we cannot presume application's
behavior. Telling an app: "your headroom is 0" but there're plenty of resources
looks not good enough to me.
IIUC i have taken this approach in {{CapacityHeadRoomProvider.getHeadroom}}
{code}
if (requestedPartitions.isEmpty() || (requestedPartitions.size() == 1
&& requestedPartitions.contains(RMNodeLabelsManager.NO_LABEL))) {
headroom = queue.getHeadroom(user, queueCurrentLimit,
clusterResource,
application);
}
{code}
So basically when there is no partitions requested i send the headroom of
default Partition hence IMO it doesnt break the compatability now.
Would this not be sufficient ? i can add a test case for the same and verify it
!
> Respect labels in CapacityScheduler when computing headroom
> ---
>
> Key: YARN-3215
> URL: https://issues.apache.org/jira/browse/YARN-3215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch,
> YARN-3215.v2.002.patch
>
>
> In existing CapacityScheduler, when computing headroom of an application, it
> will only consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so
> headroom-by-label (like 5G resource available under node-label=red) is
> required to get better resource allocation and avoid deadlocks such as
> MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a
> label-to-available-resource map in AllocateResponse) and also internal
> changes in CapacityScheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4647) RegisterNodeManagerRequestPBImpl needs better locking
[ https://issues.apache.org/jira/browse/YARN-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120595#comment-15120595 ] Wangda Tan commented on YARN-4647: -- [~kasha], +1 to the patch, just curious when race condition could happen. > RegisterNodeManagerRequestPBImpl needs better locking > - > > Key: YARN-4647 > URL: https://issues.apache.org/jira/browse/YARN-4647 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-4647.001.patch > > > While working on YARN-4512, I noticed there are potential race conditions in > RegisterNodeManagerRequestPBImpl. We need to add more locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4615) TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt fails occasionally
[
https://issues.apache.org/jira/browse/YARN-4615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120689#comment-15120689
]
Rohith Sharma K S commented on YARN-4615:
-
Analysis make sense to me, I am able to reproduce the issue keeping very small
delay like below
{code}
node.nodeHeartbeat(applicationAttemptOneID, 1, ContainerState.COMPLETE);
Thread.sleep(200);
rm.waitForState(node, am1ContainerID, RMContainerState.COMPLETED);
{code}
About the patch, would you use existing method
{{MockRM#waitForContainerState(ContainerId containerId,RMContainerState
state)}} and modify a little bit for incorporating your new method logic.
For testing purpose, you can use small delay like in the above.
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> fails occasionally
>
>
> Key: YARN-4615
> URL: https://issues.apache.org/jira/browse/YARN-4615
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Jason Lowe
>Assignee: Sunil G
> Attachments: 0001-YARN-4615.patch
>
>
> Sometimes
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> will fail like this:
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
> testResourceRequestRecoveryToTheRightAppAttempt[1](org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler)
> Time elapsed: 77.427 sec <<< FAILURE!
> java.lang.AssertionError: Attempt state is not correct (timedout): expected:
> SCHEDULED actual: ALLOCATED for the application attempt
> appattempt_1453254869107_0001_02
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:197)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForAttemptScheduled(MockRM.java:831)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler.testResourceRequestRecoveryToTheRightAppAttempt(TestAbstractYarnScheduler.java:572)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4465) SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled
[
https://issues.apache.org/jira/browse/YARN-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120726#comment-15120726
]
Wangda Tan commented on YARN-4465:
--
Hi [~bibinchundatt],
I felt I made a mistake for previous comment, when the label is not enabled, we
shouldn't simply ignore checking labels in request. Request with label will be
starving when labels are disabled. Reset such labels to empty is also
problematic, we should let application knows about the failure.
Here is my proposal, when label is disabled:
- Any request with label will be rejected (throws invalid label exception)
- Queues with label configured will be rejected, including
default-node-label-expression, accessible-node-labels (accessible-node-labels
is * by default, but we shouldn't allow explicitly set accessible-node-labels
to non-empty and non *), and label-related capacities (check
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils#loadCapacitiesByLabelsFromConf}})
Queue related changes could be done in a separated JIRA.
Thoughts?
> SchedulerUtils#validateRequest for Label check should happen only when
> nodelabel enabled
>
>
> Key: YARN-4465
> URL: https://issues.apache.org/jira/browse/YARN-4465
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4465.patch, 0002-YARN-4465.patch,
> 0003-YARN-4465.patch
>
>
> Disable label from rm side yarn.nodelabel.enable=false
> Capacity scheduler label configuration for queue is available as below
> default label for queue = b1 as 3 and accessible labels as 1,3
> Submit application to queue A .
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):
> Invalid resource request, queue=b1 doesn't have permission to access all
> labels in resource request. labelExpression of resource request=3. Queue
> labels=1,3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:216)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:401)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:340)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:602)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:247)
> {noformat}
> # Ignore default label expression when label is disabled *or*
> # NormalizeResourceRequest we can set label expression to
> when node label is not enabled *or*
> # Improve message
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
[
https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120741#comment-15120741
]
Wangda Tan commented on YARN-3215:
--
bq. Would this not be sufficient ? i can add a test case for the same and
verify it !
It sounds good to me, adding one more test will be better.
We have changed existing behavior of calculating headroom when labels enabled,
I would like to see if this will benefit/harm other projects.
*To summary,*
Old behavior:
- Return headroom of default partition only.
New behavior in the patch:
- Return headroom of partition that application requested:
{{Σ partition.headroom (partition ∈ application requested)}}
If application requested nothing, returns headroom of default partition.
Future behavior:
- Return headroom each partition to application.
I think this works for MR. [~ste...@apache.org], could you share your thoughts
of this? Does it work for Slider?
> Respect labels in CapacityScheduler when computing headroom
> ---
>
> Key: YARN-3215
> URL: https://issues.apache.org/jira/browse/YARN-3215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch,
> YARN-3215.v2.002.patch
>
>
> In existing CapacityScheduler, when computing headroom of an application, it
> will only consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so
> headroom-by-label (like 5G resource available under node-label=red) is
> required to get better resource allocation and avoid deadlocks such as
> MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a
> label-to-available-resource map in AllocateResponse) and also internal
> changes in CapacityScheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4643) Container recovery is broken with delegating container runtime
[
https://issues.apache.org/jira/browse/YARN-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120398#comment-15120398
]
Sidharta Seethana commented on YARN-4643:
-
Test failures are unrelated to this patch.
> Container recovery is broken with delegating container runtime
> --
>
> Key: YARN-4643
> URL: https://issues.apache.org/jira/browse/YARN-4643
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 2.8.0
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Critical
> Attachments: YARN-4643.001.patch, YARN-4643.002.patch
>
>
> Delegating container runtime uses the container's launch context to determine
> which runtime to use. However, during container recovery, a container object
> is not passed as input which leads to a {{NullPointerException}} when
> attempting to access the container's launch context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[
https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120399#comment-15120399
]
Hadoop QA commented on YARN-4630:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 0s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 52s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 2s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 28s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 55s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 30s {color}
| {color:red} hadoop-yarn-client in the patch
[jira] [Commented] (YARN-4644) TestRMRestart fails on YARN-2928 branch
[
https://issues.apache.org/jira/browse/YARN-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118832#comment-15118832
]
Naganarasimha G R commented on YARN-4644:
-
Thanks [~varun_saxena] for analyzing it, IMO we can handle FindBugs issue in
this jira itself, as they are small issues.
> TestRMRestart fails on YARN-2928 branch
> ---
>
> Key: YARN-4644
> URL: https://issues.apache.org/jira/browse/YARN-4644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4644-YARN-2928.01.patch
>
>
> This was reported by YARN-4238 QA report. Refer to
> https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
> Error reported is as under :
> {noformat}
> org.mockito.exceptions.verification.TooManyActualInvocations:
> noOpSystemMetricPublisher.appCreated(
> ,
>
> );
> Wanted 3 times:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> But was 6 times. Undesired invocation:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> {noformat}
> Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
> called twice.
> Has been introduced during rebase I guess.
> After removing the duplicate call, the test passes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
[ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119624#comment-15119624 ] Sunil G commented on YARN-3215: --- Hi [~Naganarasimha Garla] and [~leftnoteasy] I feel the point mentioned by Naga make sense to me. If we have non-exclusive labels in cluster, and those resources are majorly used by NO_LABEL applications (apps w/o any labels), then any headroom calculation from this NO_LABEL may give a big headroom for those apps with labels. > Respect labels in CapacityScheduler when computing headroom > --- > > Key: YARN-3215 > URL: https://issues.apache.org/jira/browse/YARN-3215 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Naganarasimha G R > Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch, > YARN-3215.v2.002.patch > > > In existing CapacityScheduler, when computing headroom of an application, it > will only consider "non-labeled" nodes of this application. > But it is possible the application is asking for labeled resources, so > headroom-by-label (like 5G resource available under node-label=red) is > required to get better resource allocation and avoid deadlocks such as > MAPREDUCE-5928. > This JIRA could involve both API changes (such as adding a > label-to-available-resource map in AllocateResponse) and also internal > changes in CapacityScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4238) correctly set createdTime and remove modifiedTime when publishing entities
[ https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-4238: -- Summary: correctly set createdTime and remove modifiedTime when publishing entities (was: createdTime and modifiedTime is not reported while publishing entities to ATSv2) > correctly set createdTime and remove modifiedTime when publishing entities > -- > > Key: YARN-4238 > URL: https://issues.apache.org/jira/browse/YARN-4238 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4238-YARN-2928.01.patch, > YARN-4238-YARN-2928.04.patch, YARN-4238-YARN-2928.05.patch, > YARN-4238-feature-YARN-2928.002.patch, YARN-4238-feature-YARN-2928.003.patch > > > While publishing entities from RM and elsewhere we are not sending created > time. For instance, created time in TimelineServiceV2Publisher class and for > other entities in other such similar classes is not updated. We can easily > update created time when sending application created event. Likewise for > modification time on every write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4512) Provide a knob to turn on over-allocation
[ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4512: --- Attachment: yarn-4512-yarn-1011.004.patch > Provide a knob to turn on over-allocation > - > > Key: YARN-4512 > URL: https://issues.apache.org/jira/browse/YARN-4512 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-4512-YARN-1011.001.patch, > yarn-4512-yarn-1011.002.patch, yarn-4512-yarn-1011.003.patch, > yarn-4512-yarn-1011.004.patch > > > We need two configs for overallocation - one to specify the threshold upto > which it is okay to over-allocate, another to specify the threshold after > which OPPORTUNISTIC containers should be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4512) Provide a knob to turn on over-allocation
[ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4512: --- Attachment: (was: yarn-4512-yarn-1011.004.patch) > Provide a knob to turn on over-allocation > - > > Key: YARN-4512 > URL: https://issues.apache.org/jira/browse/YARN-4512 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-4512-YARN-1011.001.patch, > yarn-4512-yarn-1011.002.patch, yarn-4512-yarn-1011.003.patch, > yarn-4512-yarn-1011.004.patch > > > We need two configs for overallocation - one to specify the threshold upto > which it is okay to over-allocate, another to specify the threshold after > which OPPORTUNISTIC containers should be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4643) Container recovery is broken with delegating container runtime
[
https://issues.apache.org/jira/browse/YARN-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119642#comment-15119642
]
Jason Lowe commented on YARN-4643:
--
I noticed that TestLinuxContainerExecutor also builds a
ContainerReacquisitionContext without setting a container. Don't we need to
address that as well?
> Container recovery is broken with delegating container runtime
> --
>
> Key: YARN-4643
> URL: https://issues.apache.org/jira/browse/YARN-4643
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 2.8.0
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Critical
> Attachments: YARN-4643.001.patch
>
>
> Delegating container runtime uses the container's launch context to determine
> which runtime to use. However, during container recovery, a container object
> is not passed as input which leads to a {{NullPointerException}} when
> attempting to access the container's launch context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4411) ResourceManager IllegalArgumentException error
[
https://issues.apache.org/jira/browse/YARN-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119710#comment-15119710
]
Bibin A Chundatt commented on YARN-4411:
[~rohithsharma]/[~devaraj.k]
Could you please review patch attached
> ResourceManager IllegalArgumentException error
> --
>
> Key: YARN-4411
> URL: https://issues.apache.org/jira/browse/YARN-4411
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: yarntime
>Assignee: Bibin A Chundatt
> Attachments: 0002-YARN-4411.patch, YARN-4411.001.patch
>
>
> in version 2.7.1, line 1914 may cause IllegalArgumentException in
> RMAppAttemptImpl:
> YarnApplicationAttemptState.valueOf(this.getState().toString())
> cause by this.getState() returns type RMAppAttemptState which may not be
> converted to YarnApplicationAttemptState.
> {noformat}
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState.LAUNCHED_UNMANAGED_SAVING
> at java.lang.Enum.valueOf(Enum.java:236)
> at
> org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState.valueOf(YarnApplicationAttemptState.java:27)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.createApplicationAttemptReport(RMAppAttemptImpl.java:1870)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationAttemptReport(ClientRMService.java:355)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationAttemptReport(ApplicationClientProtocolPBServiceImpl.java:355)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4512) Provide a knob to turn on over-allocation
[ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4512: --- Attachment: yarn-4512-yarn-1011.004.patch [~elgoiri] - good point. The v4 patch sends the over allocation info on register instead of node heartbeat. > Provide a knob to turn on over-allocation > - > > Key: YARN-4512 > URL: https://issues.apache.org/jira/browse/YARN-4512 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-4512-YARN-1011.001.patch, > yarn-4512-yarn-1011.002.patch, yarn-4512-yarn-1011.003.patch, > yarn-4512-yarn-1011.004.patch > > > We need two configs for overallocation - one to specify the threshold upto > which it is okay to over-allocate, another to specify the threshold after > which OPPORTUNISTIC containers should be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4644) TestRMRestart fails on YARN-2928 branch
[
https://issues.apache.org/jira/browse/YARN-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119694#comment-15119694
]
Hadoop QA commented on YARN-4644:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
6s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 14s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in YARN-2928 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 33s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 33s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 154m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
[jira] [Updated] (YARN-4238) correctly set createdTime and remove modifiedTime when publishing entities
[ https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-4238: -- Hadoop Flags: Reviewed > correctly set createdTime and remove modifiedTime when publishing entities > -- > > Key: YARN-4238 > URL: https://issues.apache.org/jira/browse/YARN-4238 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4238-YARN-2928.01.patch, > YARN-4238-YARN-2928.04.patch, YARN-4238-YARN-2928.05.patch, > YARN-4238-feature-YARN-2928.002.patch, YARN-4238-feature-YARN-2928.003.patch > > > While publishing entities from RM and elsewhere we are not sending created > time. For instance, created time in TimelineServiceV2Publisher class and for > other entities in other such similar classes is not updated. We can easily > update created time when sending application created event. Likewise for > modification time on every write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4172) Extend DominantResourceCalculator to account for all resources
[ https://issues.apache.org/jira/browse/YARN-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4172: Attachment: YARN-4172-YARN-3926.003.patch Uploaded a new patch which fixes javadoc error. > Extend DominantResourceCalculator to account for all resources > -- > > Key: YARN-4172 > URL: https://issues.apache.org/jira/browse/YARN-4172 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4172-YARN-3926.001.patch, > YARN-4172-YARN-3926.002.patch, YARN-4172-YARN-3926.003.patch > > > Now that support for multiple resources is present in the resource class, we > need to modify DominantResourceCalculator to account for the new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4633) TestRMRestart.testRMRestartAfterPreemption fails intermittently in trunk
[
https://issues.apache.org/jira/browse/YARN-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120909#comment-15120909
]
Bibin A Chundatt commented on YARN-4633:
[~rohithsharma]
Thanks for looking into patch
I tried increasing the number of attempt launch and was able to reproduce.
{noformat}
for (int i = 0; i < 100; i++) {
am0 = MockRM.launchAM(app0, rm1, nm1);
am0.registerAppAttempt();
}
{noformat}
> TestRMRestart.testRMRestartAfterPreemption fails intermittently in trunk
> -
>
> Key: YARN-4633
> URL: https://issues.apache.org/jira/browse/YARN-4633
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.9.0
> Environment: Jenkin
>Reporter: Rohith Sharma K S
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4633.patch
>
>
> Jenkins
> [Build|https://builds.apache.org/job/PreCommit-YARN-Build/10366/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt]
> failed for below test case,
> {code}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
> support was removed in 8.0
> Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> Tests run: 54, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 455.808 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartAfterPreemption[0](org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 60.145 sec <<< FAILURE!
> java.lang.AssertionError: Attempt state is not correct (timedout): expected:
> SCHEDULED actual: FAILED for the application attempt
> appattempt_1453461355278_0001_04
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:197)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForAttemptScheduled(MockRM.java:831)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAM(MockRM.java:818)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartAfterPreemption(TestRMRestart.java:2352)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4172) Extend DominantResourceCalculator to account for all resources
[
https://issues.apache.org/jira/browse/YARN-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120942#comment-15120942
]
Hadoop QA commented on YARN-4172:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} YARN-3926 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} YARN-3926 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} YARN-3926 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} YARN-3926 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
patch generated 0 new + 5 unchanged - 6 fixed = 5 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-yarn-common in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 15s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12779018/YARN-4172-YARN-3926.002.patch
|
| JIRA Issue | YARN-4172 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9ece939a7927 3.13.0-36-lowlatency #63-Ubuntu SMP
[jira] [Commented] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[
https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120951#comment-15120951
]
Hadoop QA commented on YARN-3367:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s {color}
| {color:red} YARN-3367 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784843/sjlee-suggestion.patch
|
| JIRA Issue | YARN-3367 |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/10414/console |
This message was automatically generated.
> Replace starting a separate thread for post entity with event loop in
> TimelineClient
>
>
> Key: YARN-3367
> URL: https://issues.apache.org/jira/browse/YARN-3367
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-3367-YARN-2928.v1.005.patch,
> YARN-3367-YARN-2928.v1.006.patch, YARN-3367-feature-YARN-2928.003.patch,
> YARN-3367-feature-YARN-2928.v1.002.patch,
> YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch,
> sjlee-suggestion.patch
>
>
> Since YARN-3039, we add loop in TimelineClient to wait for
> collectorServiceAddress ready before posting any entity. In consumer of
> TimelineClient (like AM), we are starting a new thread for each call to get
> rid of potential deadlock in main thread. This way has at least 3 major
> defects:
> 1. The consumer need some additional code to wrap a thread before calling
> putEntities() in TimelineClient.
> 2. It cost many thread resources which is unnecessary.
> 3. The sequence of events could be out of order because each posting
> operation thread get out of waiting loop randomly.
> We should have something like event loop in TimelineClient side,
> putEntities() only put related entities into a queue of entities and a
> separated thread handle to deliver entities in queue to collector via REST
> call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4651) movetoqueue option does not documented in 'YARN Commands'
Takashi Ohnishi created YARN-4651: - Summary: movetoqueue option does not documented in 'YARN Commands' Key: YARN-4651 URL: https://issues.apache.org/jira/browse/YARN-4651 Project: Hadoop YARN Issue Type: Bug Components: documentation Reporter: Takashi Ohnishi In 'YARN Commands', the movetoqueue and queue option are not listed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4411) ResourceManager IllegalArgumentException error
[
https://issues.apache.org/jira/browse/YARN-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120895#comment-15120895
]
Bibin A Chundatt commented on YARN-4411:
Thanks [~devaraj.k] for review
{quote}
I don't understand why do we need this, Do you see any problem if we invoke
attempt.createApplicationAttemptReport() when the state is other than
RMAppAttemptState.FINAL_SAVING? I think we can we can create
ApplicationAttemptReport irrespective of the state.
{quote}
Only for FINAL_SAVING the previous state will be send for create report.
{{RMAppAttemptImpl#createApplicationAttemptState}}
{noformat}
// If AppAttempt is in FINAL_SAVING state, return its previous state.
if (state.equals(RMAppAttemptState.FINAL_SAVING)) {
state = stateBeforeFinalSaving;
}
{noformat}
So for RMAppAttemptState=FINAL_SAVING the test is done as below.
{quote}
924 applicationAttempt.handle(new RMAppAttemptEvent(
925 applicationAttempt.getAppAttemptId(),
RMAppAttemptEventType.KILL));
926 assertEquals(RMAppAttemptState.FINAL_SAVING,
927 applicationAttempt.getAppAttemptState());
928 // ALLOCATED to FINAL_SAVING REPORT
929 ApplicationAttemptReport attemptreport =
930 applicationAttempt.createApplicationAttemptReport();
{quote}
For testing createApplicationReport with all other {{RMAppAttemptState}}
except FINAL_SAVING the check is done
{noformat}
+ if (!rmAppAttemptState.equals(RMAppAttemptState.FINAL_SAVING)) {
{noformat}
Second point will remove assert
> ResourceManager IllegalArgumentException error
> --
>
> Key: YARN-4411
> URL: https://issues.apache.org/jira/browse/YARN-4411
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: yarntime
>Assignee: Bibin A Chundatt
> Attachments: 0002-YARN-4411.patch, YARN-4411.001.patch
>
>
> in version 2.7.1, line 1914 may cause IllegalArgumentException in
> RMAppAttemptImpl:
> YarnApplicationAttemptState.valueOf(this.getState().toString())
> cause by this.getState() returns type RMAppAttemptState which may not be
> converted to YarnApplicationAttemptState.
> {noformat}
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState.LAUNCHED_UNMANAGED_SAVING
> at java.lang.Enum.valueOf(Enum.java:236)
> at
> org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState.valueOf(YarnApplicationAttemptState.java:27)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.createApplicationAttemptReport(RMAppAttemptImpl.java:1870)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationAttemptReport(ClientRMService.java:355)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationAttemptReport(ApplicationClientProtocolPBServiceImpl.java:355)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4108: - Attachment: YARN-4108.2.patch Attached ver.2 patch, fixed a javac warning, findbugs warning and test failures. Tested this patch in a local single node cluster, it works end-to-end. > CapacityScheduler: Improve preemption to preempt only those containers that > would satisfy the incoming request > -- > > Key: YARN-4108 > URL: https://issues.apache.org/jira/browse/YARN-4108 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4108-design-doc-V3.pdf, > YARN-4108-design-doc-v1.pdf, YARN-4108-design-doc-v2.pdf, YARN-4108.1.patch, > YARN-4108.2.patch, YARN-4108.poc.1.patch, YARN-4108.poc.2-WIP.patch, > YARN-4108.poc.3-WIP.patch, YARN-4108.poc.4-WIP.patch > > > This is sibling JIRA for YARN-2154. We should make sure container preemption > is more effective. > *Requirements:*: > 1) Can handle case of user-limit preemption > 2) Can handle case of resource placement requirements, such as: hard-locality > (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I > don't want to use rack1 and host\[1-3\]) > 3) Can handle preemption within a queue: cross user preemption (YARN-2113), > cross applicaiton preemption (such as priority-based (YARN-1963) / > fairness-based (YARN-3319)). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4172) Extend DominantResourceCalculator to account for all resources
[ https://issues.apache.org/jira/browse/YARN-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120971#comment-15120971 ] Wangda Tan commented on YARN-4172: -- +1, pending Jenkins. > Extend DominantResourceCalculator to account for all resources > -- > > Key: YARN-4172 > URL: https://issues.apache.org/jira/browse/YARN-4172 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4172-YARN-3926.001.patch, > YARN-4172-YARN-3926.002.patch, YARN-4172-YARN-3926.003.patch > > > Now that support for multiple resources is present in the resource class, we > need to modify DominantResourceCalculator to account for the new resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4617) LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy
[
https://issues.apache.org/jira/browse/YARN-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120905#comment-15120905
]
Wangda Tan commented on YARN-4617:
--
+1 to the patch, thanks [~rohithsharma]!
> LeafQueue#pendingOrderingPolicy should always use fixed ordering policy
> instead of using same as active applications ordering policy
>
>
> Key: YARN-4617
> URL: https://issues.apache.org/jira/browse/YARN-4617
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4617.patch, 0001-YARN-4617.patch,
> 0002-YARN-4617.patch, 0003-YARN-4617.patch
>
>
> In discussion with [~leftnoteasy] in the JIRA
> [comment|https://issues.apache.org/jira/browse/YARN-4479?focusedCommentId=15108236=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15108236]
> pointed out that {{LeafQueue#pendingOrderingPolicy}} should NOT be assumed
> to be same as active applications ordering policy. It causes an issue when
> using fair ordering policy.
> Expectations of this JIRA should include
> # Create FifoOrderingPolicyForPendingApps which extends FifoOrderingPolicy.
> # Comparator of new ordering policy should use
> RecoveryComparator,PriorityComparator and Fifocomparator in order
> respectively.
> # Clean up {{LeafQueue#pendingOPForRecoveredApps}} which is no more required
> once new fixed ordering policy is created pending applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4651) movetoqueue option does not documented in 'YARN Commands'
[ https://issues.apache.org/jira/browse/YARN-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takashi Ohnishi updated YARN-4651: -- Attachment: YARN-4651.1.patch Attached patch. > movetoqueue option does not documented in 'YARN Commands' > - > > Key: YARN-4651 > URL: https://issues.apache.org/jira/browse/YARN-4651 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Reporter: Takashi Ohnishi > Attachments: YARN-4651.1.patch > > > In 'YARN Commands', the movetoqueue and queue option are not listed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4428) Redirect RM page to AHS page when AHS turned on and RM page is not avaialable
[
https://issues.apache.org/jira/browse/YARN-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119802#comment-15119802
]
Hadoop QA commented on YARN-4428:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 45s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 24s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784646/YARN-4428.7.patch |
| JIRA Issue | YARN-4428 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
|
[jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
[
https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119825#comment-15119825
]
Karthik Kambatla commented on YARN-1011:
[~nroberts] - interesting ideas. The notion of two schedulers definitely frees
us from the confines we have today.
How about doing the following on node update (along the lines of two scheduler
suggestion):
{code}
process_newly_launched_containers();
process_completed_containers();
while (guaranteed_resources_on_node_are_still_available) {
app = pickApp();
if (app.hasOpportunisticContainersRunningOnTheNode()) {
promote_container_to_guaranteed();
} else {
allocate_guaranteed_container_as_we_do_today(); // includes reservation
}
}
while (opportunistic_resources_on_node_are_still_available) {
app = pickAppOkayWithOpportunisticContainers();
allocate_opportunistic_container(); // reservations are not allowed
}
{code}
This way:
# Apps that can't tolerate using opportunistic containers can state their
preference to have only guaranteed containers.
# For apps that are okay with opportunistic containers, a container is promoted
only when the app deserves guaranteed containers.
> [Umbrella] Schedule containers based on utilization of currently allocated
> containers
> -
>
> Key: YARN-1011
> URL: https://issues.apache.org/jira/browse/YARN-1011
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Arun C Murthy
> Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf,
> yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated
> containers and, if appropriate, allocate more (speculative?) containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[ https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated YARN-4630: - Attachment: (was: YARN-4630.patch.0) > Remove useless boxing/unboxing code (Hadoop YARN) > - > > Key: YARN-4630 > URL: https://issues.apache.org/jira/browse/YARN-4630 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Priority: Minor > Attachments: YARN-4630.0.patch > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4630) Remove useless boxing/unboxing code (Hadoop YARN)
[ https://issues.apache.org/jira/browse/YARN-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated YARN-4630: - Attachment: YARN-4630.0.patch Fixed style issue. > Remove useless boxing/unboxing code (Hadoop YARN) > - > > Key: YARN-4630 > URL: https://issues.apache.org/jira/browse/YARN-4630 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Priority: Minor > Attachments: YARN-4630.0.patch > > > There are lots of places where useless boxing/unboxing occur. > To avoid performance issue, let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119830#comment-15119830 ] Karthik Kambatla commented on YARN-1011: BTW, sorry for the delay in circling back here. Was out of the country last week. > [Umbrella] Schedule containers based on utilization of currently allocated > containers > - > > Key: YARN-1011 > URL: https://issues.apache.org/jira/browse/YARN-1011 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Arun C Murthy > Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf, > yarn-1011-design-v2.pdf > > > Currently RM allocates containers and assumes resources allocated are > utilized. > RM can, and should, get to a point where it measures utilization of allocated > containers and, if appropriate, allocate more (speculative?) containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4622) TestDistributedShell fails for v2 test cases after modifications for 1.5
[
https://issues.apache.org/jira/browse/YARN-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4622:
---
Hadoop Flags: Reviewed
> TestDistributedShell fails for v2 test cases after modifications for 1.5
>
>
> Key: YARN-4622
> URL: https://issues.apache.org/jira/browse/YARN-4622
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Labels: test
> Fix For: YARN-2928
>
> Attachments: YARN-4622-YARN-2928.v1.001.patch
>
>
> TestDistributedShell fails for v2 test cases :
> *testDSShellWithoutDomainV2DefaultFlow and
> testDSShellWithoutDomainV2CustomizedFlow* after trunk rebase with
> modifications for 1.5,
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at com.google.common.base.Joiner.toString(Joiner.java:532)
> at com.google.common.base.Joiner.appendTo(Joiner.java:124)
> at com.google.common.base.Joiner.appendTo(Joiner.java:181)
> at com.google.common.base.Joiner.join(Joiner.java:237)
> at com.google.common.base.Joiner.join(Joiner.java:226)
> at com.google.common.base.Joiner.join(Joiner.java:253)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.constructResURI(TimelineClientImpl.java:726)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:336)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.createAndStartTimelineClient(ApplicationImpl.java:149)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:113)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainerInternal(ContainerManagerImpl.java:971)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:830)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:65)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4644) TestRMRestart fails on YARN-2928 branch
[
https://issues.apache.org/jira/browse/YARN-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119822#comment-15119822
]
Varun Saxena commented on YARN-4644:
Seems to be some problem with the build. Findbugs warning is coming in QA
report despite fixing it.
Its showing a variable as unused despite removal of that variable from code.
> TestRMRestart fails on YARN-2928 branch
> ---
>
> Key: YARN-4644
> URL: https://issues.apache.org/jira/browse/YARN-4644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4644-YARN-2928.01.patch,
> YARN-4644-YARN-2928.02.patch
>
>
> This was reported by YARN-4238 QA report. Refer to
> https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
> Error reported is as under :
> {noformat}
> org.mockito.exceptions.verification.TooManyActualInvocations:
> noOpSystemMetricPublisher.appCreated(
> ,
>
> );
> Wanted 3 times:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> But was 6 times. Undesired invocation:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> {noformat}
> Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
> called twice.
> Has been introduced during rebase I guess.
> After removing the duplicate call, the test passes.
> There is a *findbugs warning* in resourcemanager in YARN-2928 branch as well.
> Fix it as part of this JIRA itself.
> {noformat}
> DLS Dead store to keepAliveApps in
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest)
> Bug type DLS_DEAD_LOCAL_STORE (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService
> In method
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest)
> Local variable named keepAliveApps
> At ResourceTrackerService.java:[line 486]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4629) Distributed shell breaks under strong security
[
https://issues.apache.org/jira/browse/YARN-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119829#comment-15119829
]
Daniel Templeton commented on YARN-4629:
Where would you recommend putting {{YarnClientUtils}}, [~steve_l]?
{{o.a.h.yarn.client}}, {{o.a.h.yarn.client.api}}, {{o.a.h.yarn.api}},
{{o.a.h.yarn.conf}}? They're all mostly mir Wurst. I would probably pick
{{o.a.h.yarn.conf}}.
> Distributed shell breaks under strong security
> --
>
> Key: YARN-4629
> URL: https://issues.apache.org/jira/browse/YARN-4629
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4629.001.patch
>
>
> If the auth_to_local is set to map requests from unknown hosts to nobody, the
> dist shell app fails. The reason is that the client doesn't translate the
> _HOST placeholder to the local hostname.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4512) Provide a knob to turn on over-allocation
[
https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119856#comment-15119856
]
Hadoop QA commented on YARN-4512:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 57s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 7 new +
312 unchanged - 0 fixed = 319 total (was 312) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 1s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
[jira] [Commented] (YARN-4224) Support fetching entities by UID and change the REST interface to conform to current REST APIs' in YARN
[ https://issues.apache.org/jira/browse/YARN-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119794#comment-15119794 ] Sangjin Lee commented on YARN-4224: --- +1 LGTM. Let me know if you have any more feedback. I'll commit it today unless there are more comments. > Support fetching entities by UID and change the REST interface to conform to > current REST APIs' in YARN > --- > > Key: YARN-4224 > URL: https://issues.apache.org/jira/browse/YARN-4224 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4224-YARN-2928.01.patch, > YARN-4224-YARN-2928.05.patch, YARN-4224-YARN-2928.06.patch, > YARN-4224-feature-YARN-2928.04.patch, > YARN-4224-feature-YARN-2928.wip.02.patch, > YARN-4224-feature-YARN-2928.wip.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2599) Standby RM should also expose some jmx and metrics
[ https://issues.apache.org/jira/browse/YARN-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-2599: - Target Version/s: 2.7.3, 2.6.5 (was: 2.7.3, 2.6.4) > Standby RM should also expose some jmx and metrics > -- > > Key: YARN-2599 > URL: https://issues.apache.org/jira/browse/YARN-2599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Karthik Kambatla >Assignee: Rohith Sharma K S > > YARN-1898 redirects jmx and metrics to the Active. As discussed there, we > need to separate out metrics displayed so the Standby RM can also be > monitored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4328) Findbugs warning in resourcemanager in branch-2.7 and branch-2.6
[
https://issues.apache.org/jira/browse/YARN-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119966#comment-15119966
]
Junping Du commented on YARN-4328:
--
Remove it from 2.6.4 which is mostly for critical bug fix.
> Findbugs warning in resourcemanager in branch-2.7 and branch-2.6
>
>
> Key: YARN-4328
> URL: https://issues.apache.org/jira/browse/YARN-4328
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1, 2.6.2
>Reporter: Varun Saxena
>Assignee: Akira AJISAKA
>Priority: Minor
> Attachments: YARN-4328.branch-2.6.00.patch,
> YARN-4328.branch-2.7.00.patch
>
>
> This issue exists in both branch-2.7 and branch-2.6
> {noformat}
> classname='org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKSyncOperationCallback'>
> category='PERFORMANCE' message='Should
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKSyncOperationCallback
> be a _static_ inner class?' lineNumber='118'/>
> {noformat}
> Below issue exists only in branch-2.6
> {noformat}
> classname='org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt'>
> category='MT_CORRECTNESS' message='Inconsistent synchronization of
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.queue;
> locked 57% of time' lineNumber='261'/>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4632) Replacing _HOST in RM_PRINCIPAL should not be the responsibility of the client code
[
https://issues.apache.org/jira/browse/YARN-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120025#comment-15120025
]
Daniel Templeton commented on YARN-4632:
Well, *that* was an entertaining rabbit hole. [~daryn], it would appear that
you are correct. To save others from tilting at this particular windmill in
the future, I will document my journey here.
When the client wants to run a YARN app against a secure HDFS cluster, it has
to be pass that application a delegation token (DT) that tells HDFS that the
YARN is allowed to access HDFS as the client. That token gets serialized to
bytes and passed over to the RM as part of the launch context. When the RM
launches the app, the first thing it does is try to renew all the tokens for
the app. The DT contains the principal of the service that is allowed to renew
it. If that service principal doesn't match the name of the RM, the token
renewal will fail, causing the app launch to fail.
The token's renewer string is something the client provides to HDFS when it
requests the DT. To find the RM's service name, the client typically looks up
the {{YarnConfiguration.RM_PRINCIPAL}} in the conf. That configuration will
typically list the service's hostname as {{_HOST}} to allow the hostname to be
replaced in the event of HA YARN. That {{_HOST}} placeholder has to be
replaced by the RM's hostname by the client. The point of this JIRA was to try
to move that hostname replacement out of the client's hands.
The problem is that the renewer string that the client sets is stashed away
deep inside the token, so once it's set, it's hard to unset. It is
theoretically possible to create a new token based on the original token,
except fixing the {{_HOST}} string in the renewer, but the bad news is that
creating a new HDFS token can only be done by a service that has access to
HDFS's delegation secret manager service, which is not exposed outside HDFS.
The net result is that the only code that can change the renewer principal is
HDFS itself. Since the token creation also requires the client's principal,
the client is the only one that can do it. Q.E.D.
I now realize that it doesn't actually make any sense to have the renewer set
anywhere other than the client code. The whole point of the operation is for
the client to explicitly grant access to the renewer to keep a token alive.
Anything other than the client being the one to name the renewer breaks the
security chain. The only viable approach to solving the problem this JIRA set
out to solve is to make the {{_HOST}} replacement as simple as possible. See
YARN-4629. It's not a complete solution, but it's all we've got.
> Replacing _HOST in RM_PRINCIPAL should not be the responsibility of the
> client code
> ---
>
> Key: YARN-4632
> URL: https://issues.apache.org/jira/browse/YARN-4632
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api, resourcemanager
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> It is currently the client's responsibility to call
> {{SecurityUtil.getServerPrincipal()}} to replace the _HOST placeholder in any
> principal name used for a delegation token. This is a non-optional operation
> and should not be pushed onto the client.
> All client apps that followed the distributed shell as the canonical example
> failed to do the replacement because distributed shell fails to do the
> replacement. (See YARN-4629.) Rather than fixing the whole world, since the
> whole world use distributed shell as a model, let's move the operation into
> YARN where it belongs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4328) Findbugs warning in resourcemanager in branch-2.7 and branch-2.6
[
https://issues.apache.org/jira/browse/YARN-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-4328:
-
Target Version/s: 2.7.3 (was: 2.7.3, 2.6.4)
> Findbugs warning in resourcemanager in branch-2.7 and branch-2.6
>
>
> Key: YARN-4328
> URL: https://issues.apache.org/jira/browse/YARN-4328
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1, 2.6.2
>Reporter: Varun Saxena
>Assignee: Akira AJISAKA
>Priority: Minor
> Attachments: YARN-4328.branch-2.6.00.patch,
> YARN-4328.branch-2.7.00.patch
>
>
> This issue exists in both branch-2.7 and branch-2.6
> {noformat}
> classname='org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKSyncOperationCallback'>
> category='PERFORMANCE' message='Should
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKSyncOperationCallback
> be a _static_ inner class?' lineNumber='118'/>
> {noformat}
> Below issue exists only in branch-2.6
> {noformat}
> classname='org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt'>
> category='MT_CORRECTNESS' message='Inconsistent synchronization of
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.queue;
> locked 57% of time' lineNumber='261'/>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3102) Decommisioned Nodes not listed in Web UI
[ https://issues.apache.org/jira/browse/YARN-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated YARN-3102: -- Attachment: YARN-3102-v7.patch Thanks a lot [~jlowe] for the very useful comments and help. I have updated the patch to have a default DrainDispatcher for MockRM. Also reduced the containsKey-get-remove to a remove and get-remove to remove in RMNodeImpl. I fixed 2 test classes where the drainDispatcher was following older code style along with the writetoHostFile change. Test cases' timeouts are now 30 seconds. > Decommisioned Nodes not listed in Web UI > > > Key: YARN-3102 > URL: https://issues.apache.org/jira/browse/YARN-3102 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 > Environment: 2 Node Manager and 1 Resource Manager >Reporter: Bibin A Chundatt >Assignee: Kuhu Shukla >Priority: Minor > Attachments: YARN-3102-v1.patch, YARN-3102-v2.patch, > YARN-3102-v3.patch, YARN-3102-v4.patch, YARN-3102-v5.patch, > YARN-3102-v6.patch, YARN-3102-v7.patch > > > Configure yarn.resourcemanager.nodes.exclude-path in yarn-site.xml to > yarn.exlude file In RM1 machine > Add Yarn.exclude with NM1 Host Name > Start the node as listed below NM1,NM2 Resource manager > Now check Nodes decommisioned in /cluster/nodes > Number of decommisioned node is listed as 1 but Table is empty in > /cluster/nodes/decommissioned (detail of Decommision node not shown) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4647) RegisterNodeManagerRequestPBImpl needs more synchronization
Karthik Kambatla created YARN-4647: -- Summary: RegisterNodeManagerRequestPBImpl needs more synchronization Key: YARN-4647 URL: https://issues.apache.org/jira/browse/YARN-4647 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.8.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla While working on YARN-4512, I noticed there are potential race conditions in RegisterNodeManagerRequestPBImpl. We need to add more locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4060) Revisit default retry config for connection with RM
[ https://issues.apache.org/jira/browse/YARN-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-4060: - Target Version/s: 2.7.3, 2.6.5 (was: 2.7.3, 2.6.4) > Revisit default retry config for connection with RM > > > Key: YARN-4060 > URL: https://issues.apache.org/jira/browse/YARN-4060 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > > 15 minutes timeout for AM/NM connection with RM in non-ha scenario turns out > to be short in production environment. The suggestion is to increase that > to 30 min. Also, the retry-interval is set to 30 seconds which appears too > long. We may reduce that to 10 seconds ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4060) Revisit default retry config for connection with RM
[ https://issues.apache.org/jira/browse/YARN-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119993#comment-15119993 ] Junping Du commented on YARN-4060: -- Move it out of 2.6.4. > Revisit default retry config for connection with RM > > > Key: YARN-4060 > URL: https://issues.apache.org/jira/browse/YARN-4060 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > > 15 minutes timeout for AM/NM connection with RM in non-ha scenario turns out > to be short in production environment. The suggestion is to increase that > to 30 min. Also, the retry-interval is set to 30 seconds which appears too > long. We may reduce that to 10 seconds ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4647) RegisterNodeManagerRequestPBImpl needs more synchronization
[ https://issues.apache.org/jira/browse/YARN-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119997#comment-15119997 ] Karthik Kambatla commented on YARN-4647: Trivial patch that adds synchronization similar to NodeStatusPBImpl and does a little more cleanup while at it. > RegisterNodeManagerRequestPBImpl needs more synchronization > --- > > Key: YARN-4647 > URL: https://issues.apache.org/jira/browse/YARN-4647 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-4647.001.patch > > > While working on YARN-4512, I noticed there are potential race conditions in > RegisterNodeManagerRequestPBImpl. We need to add more locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4647) RegisterNodeManagerRequestPBImpl needs more synchronization
[ https://issues.apache.org/jira/browse/YARN-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4647: --- Attachment: yarn-4647.001.patch > RegisterNodeManagerRequestPBImpl needs more synchronization > --- > > Key: YARN-4647 > URL: https://issues.apache.org/jira/browse/YARN-4647 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-4647.001.patch > > > While working on YARN-4512, I noticed there are potential race conditions in > RegisterNodeManagerRequestPBImpl. We need to add more locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4647) RegisterNodeManagerRequestPBImpl needs better locking
[ https://issues.apache.org/jira/browse/YARN-4647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4647: --- Summary: RegisterNodeManagerRequestPBImpl needs better locking (was: RegisterNodeManagerRequestPBImpl needs more synchronization) > RegisterNodeManagerRequestPBImpl needs better locking > - > > Key: YARN-4647 > URL: https://issues.apache.org/jira/browse/YARN-4647 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-4647.001.patch > > > While working on YARN-4512, I noticed there are potential race conditions in > RegisterNodeManagerRequestPBImpl. We need to add more locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2599) Standby RM should also expose some jmx and metrics
[ https://issues.apache.org/jira/browse/YARN-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119974#comment-15119974 ] Junping Du commented on YARN-2599: -- Move it to 2.6.5 as 2.6.4 is almost get kickoff for vote. > Standby RM should also expose some jmx and metrics > -- > > Key: YARN-2599 > URL: https://issues.apache.org/jira/browse/YARN-2599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Karthik Kambatla >Assignee: Rohith Sharma K S > > YARN-1898 redirects jmx and metrics to the Active. As discussed there, we > need to separate out metrics displayed so the Standby RM can also be > monitored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-4632) Replacing _HOST in RM_PRINCIPAL should not be the responsibility of the client code
[
https://issues.apache.org/jira/browse/YARN-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton resolved YARN-4632.
Resolution: Won't Fix
Because we can't, we won't, and we don't fix.
> Replacing _HOST in RM_PRINCIPAL should not be the responsibility of the
> client code
> ---
>
> Key: YARN-4632
> URL: https://issues.apache.org/jira/browse/YARN-4632
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api, resourcemanager
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> It is currently the client's responsibility to call
> {{SecurityUtil.getServerPrincipal()}} to replace the _HOST placeholder in any
> principal name used for a delegation token. This is a non-optional operation
> and should not be pushed onto the client.
> All client apps that followed the distributed shell as the canonical example
> failed to do the replacement because distributed shell fails to do the
> replacement. (See YARN-4629.) Rather than fixing the whole world, since the
> whole world use distributed shell as a model, let's move the operation into
> YARN where it belongs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4462) FairScheduler: Disallow preemption from a queue
[ https://issues.apache.org/jira/browse/YARN-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120058#comment-15120058 ] Karthik Kambatla commented on YARN-4462: bq. I don't think it is a reasonable scenario to set this default value to non-preemptable. Fair point. I agree it makes more sense to not allow setting a default value of non-preemptable. The patch itself looks good to me, but for the test being added to TestFairScheduler instead of TestFairSchedulerPreemption. Filed YARN-4648 to track that. +1 on this patch. Checking it in. > FairScheduler: Disallow preemption from a queue > --- > > Key: YARN-4462 > URL: https://issues.apache.org/jira/browse/YARN-4462 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie > Attachments: YARN-4462.001.patch, YARN-4462.002.patch, > YARN-4462.003.patch, YARN-4462.004.patch > > > When scheduler preemption is enabled, applications could be preempted if they > obtain resource over they should take. > When a mapreduce application is preempted some resource, it just runs slower. > However, when the preempted application is a long-run service, such as tomcat > running in slider, the service would fail. > So we should have a flag for application to indicate the scheduler that those > application should not be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4648) Move preemption related tests from TestFairScheduler to TestFairSchedulerPreemption
Karthik Kambatla created YARN-4648: -- Summary: Move preemption related tests from TestFairScheduler to TestFairSchedulerPreemption Key: YARN-4648 URL: https://issues.apache.org/jira/browse/YARN-4648 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.8.0 Reporter: Karthik Kambatla -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4512) Provide a knob to turn on over-allocation
[ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120065#comment-15120065 ] Inigo Goiri commented on YARN-4512: --- Findbugs error. > Provide a knob to turn on over-allocation > - > > Key: YARN-4512 > URL: https://issues.apache.org/jira/browse/YARN-4512 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-4512-YARN-1011.001.patch, > yarn-4512-yarn-1011.002.patch, yarn-4512-yarn-1011.003.patch, > yarn-4512-yarn-1011.004.patch > > > We need two configs for overallocation - one to specify the threshold upto > which it is okay to over-allocate, another to specify the threshold after > which OPPORTUNISTIC containers should be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4512) Provide a knob to turn on over-allocation
[ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120076#comment-15120076 ] Inigo Goiri commented on YARN-4512: --- +1 on v4 > Provide a knob to turn on over-allocation > - > > Key: YARN-4512 > URL: https://issues.apache.org/jira/browse/YARN-4512 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-4512-YARN-1011.001.patch, > yarn-4512-yarn-1011.002.patch, yarn-4512-yarn-1011.003.patch, > yarn-4512-yarn-1011.004.patch > > > We need two configs for overallocation - one to specify the threshold upto > which it is okay to over-allocate, another to specify the threshold after > which OPPORTUNISTIC containers should be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4428) Redirect RM page to AHS page when AHS turned on and RM page is not avaialable
[ https://issues.apache.org/jira/browse/YARN-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120028#comment-15120028 ] Chang Li commented on YARN-4428: [~jlowe] please help review the latest patch, thanks! > Redirect RM page to AHS page when AHS turned on and RM page is not avaialable > - > > Key: YARN-4428 > URL: https://issues.apache.org/jira/browse/YARN-4428 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: YARN-4428.1.2.patch, YARN-4428.1.patch, > YARN-4428.2.2.patch, YARN-4428.2.patch, YARN-4428.3.patch, YARN-4428.3.patch, > YARN-4428.4.patch, YARN-4428.5.patch, YARN-4428.6.patch, YARN-4428.7.patch > > > When AHS is turned on, if we can't view application in RM page, RM page > should redirect us to AHS page. For example, when you go to > cluster/app/application_1, if RM no longer remember the application, we will > simply get "Failed to read the application application_1", but it will be > good for RM ui to smartly try to redirect to AHS ui > /applicationhistory/app/application_1 to see if it's there. The redirect > usage already exist for logs in nodemanager UI. > Also, when AHS is enabled, WebAppProxyServlet should redirect to AHS page on > fall back of RM not remembering the app. YARN-3975 tried to do this only when > original tracking url is not set. But there are many cases, such as when app > failed at launch, original tracking url will be set to point to RM page, so > redirect to AHS page won't work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4462) FairScheduler: Disallow preemption from a queue
[ https://issues.apache.org/jira/browse/YARN-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120120#comment-15120120 ] Hudson commented on YARN-4462: -- FAILURE: Integrated in Hadoop-trunk-Commit #9194 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9194/]) YARN-4462. FairScheduler: Disallow preemption from a queue. (Tao Jie via (kasha: rev fb238d7e5dcd96466c8938b13ca7f13cedecb08a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java * hadoop-yarn-project/CHANGES.txt > FairScheduler: Disallow preemption from a queue > --- > > Key: YARN-4462 > URL: https://issues.apache.org/jira/browse/YARN-4462 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie > Attachments: YARN-4462.001.patch, YARN-4462.002.patch, > YARN-4462.003.patch, YARN-4462.004.patch > > > When scheduler preemption is enabled, applications could be preempted if they > obtain resource over they should take. > When a mapreduce application is preempted some resource, it just runs slower. > However, when the preempted application is a long-run service, such as tomcat > running in slider, the service would fail. > So we should have a flag for application to indicate the scheduler that those > application should not be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4512) Provide a knob to turn on over-allocation
[ https://issues.apache.org/jira/browse/YARN-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120067#comment-15120067 ] Inigo Goiri commented on YARN-4512: --- I guess we need [~leftnoteasy] or somebody else to push that one first. > Provide a knob to turn on over-allocation > - > > Key: YARN-4512 > URL: https://issues.apache.org/jira/browse/YARN-4512 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-4512-YARN-1011.001.patch, > yarn-4512-yarn-1011.002.patch, yarn-4512-yarn-1011.003.patch, > yarn-4512-yarn-1011.004.patch > > > We need two configs for overallocation - one to specify the threshold upto > which it is okay to over-allocate, another to specify the threshold after > which OPPORTUNISTIC containers should be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2457) FairScheduler: Handle preemption to help starved parent queues
[ https://issues.apache.org/jira/browse/YARN-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120135#comment-15120135 ] Junping Du commented on YARN-2457: -- Moving out all non-critical issues that didn't make it out of 2.6.4 into 2.6.5. > FairScheduler: Handle preemption to help starved parent queues > -- > > Key: YARN-2457 > URL: https://issues.apache.org/jira/browse/YARN-2457 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN-2395/YARN-2394 add preemption timeout and threshold per queue, but don't > check for parent queue starvation. > We need to check that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2457) FairScheduler: Handle preemption to help starved parent queues
[ https://issues.apache.org/jira/browse/YARN-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-2457: - Target Version/s: 2.7.3, 2.6.5 (was: 2.7.3, 2.6.4) > FairScheduler: Handle preemption to help starved parent queues > -- > > Key: YARN-2457 > URL: https://issues.apache.org/jira/browse/YARN-2457 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN-2395/YARN-2394 add preemption timeout and threshold per queue, but don't > check for parent queue starvation. > We need to check that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2037) Add restart support for Unmanaged AMs
[ https://issues.apache.org/jira/browse/YARN-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120143#comment-15120143 ] Junping Du commented on YARN-2037: -- Move all non-critical pending issues out of 2.6.4 into 2.6.5. > Add restart support for Unmanaged AMs > - > > Key: YARN-2037 > URL: https://issues.apache.org/jira/browse/YARN-2037 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla > > It would be nice to allow Unmanaged AMs also to restart in a work-preserving > way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2037) Add restart support for Unmanaged AMs
[ https://issues.apache.org/jira/browse/YARN-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-2037: - Target Version/s: 2.7.3, 2.6.5 (was: 2.7.3, 2.6.4) > Add restart support for Unmanaged AMs > - > > Key: YARN-2037 > URL: https://issues.apache.org/jira/browse/YARN-2037 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla > > It would be nice to allow Unmanaged AMs also to restart in a work-preserving > way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4570) Nodemanager leaking RawLocalFilesystem instances for user "testing"
[ https://issues.apache.org/jira/browse/YARN-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-4570: - Target Version/s: (was: 2.7.3, 2.6.4) > Nodemanager leaking RawLocalFilesystem instances for user "testing" > --- > > Key: YARN-4570 > URL: https://issues.apache.org/jira/browse/YARN-4570 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Chang Li > > I recently ran across a NodeManager that was running slowly due to excessive > GC. Digging into the heap I saw that most of the issue was leaked filesystem > statistics data objects which has been fixed in HADOOP-12107. However I also > noticed there were many thousands of RawLocalFilesystem objects on the heap, > far more than any other FileSystem type. Sampling a number of them showed > that they were for the "testing" user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4428) Redirect RM page to AHS page when AHS turned on and RM page is not avaialable
[
https://issues.apache.org/jira/browse/YARN-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120181#comment-15120181
]
Jason Lowe commented on YARN-4428:
--
Thanks for updating the patch. We're almost there!
- ToDo should be TODO to be consistent with other TODO's in the source
- The point of using slf4j instead of the commons logging is to _not_ have to
check for debug enabled before logging. The log statement should use the slf4j
string formatting directives and pass the parts as arguments rather than
construct it explicitly, which is why we don't have to check first. For
example, this:
{code}
if (LOG.isDebugEnabled()) {
LOG.debug("String " + parts[3] +
" fails to be parsed as ApplicationAttemptId.", e);
}
{code}
should become something simpler like this:
{code}
LOG.debug("Error parsing '{}' as an ApplicationAttemptId",
parts[3], e);
{code}
- ahsRedirectPath should just early-out with null if the ahs is not enabled
rather than having each switch case check for it. There's no need to waste
time trying to parse the URL looking for apps, attempts, or containers if the
AHS isn't enabled, and that's an extremely fast check to do up front now that
it's cached.
> Redirect RM page to AHS page when AHS turned on and RM page is not avaialable
> -
>
> Key: YARN-4428
> URL: https://issues.apache.org/jira/browse/YARN-4428
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: YARN-4428.1.2.patch, YARN-4428.1.patch,
> YARN-4428.2.2.patch, YARN-4428.2.patch, YARN-4428.3.patch, YARN-4428.3.patch,
> YARN-4428.4.patch, YARN-4428.5.patch, YARN-4428.6.patch, YARN-4428.7.patch
>
>
> When AHS is turned on, if we can't view application in RM page, RM page
> should redirect us to AHS page. For example, when you go to
> cluster/app/application_1, if RM no longer remember the application, we will
> simply get "Failed to read the application application_1", but it will be
> good for RM ui to smartly try to redirect to AHS ui
> /applicationhistory/app/application_1 to see if it's there. The redirect
> usage already exist for logs in nodemanager UI.
> Also, when AHS is enabled, WebAppProxyServlet should redirect to AHS page on
> fall back of RM not remembering the app. YARN-3975 tried to do this only when
> original tracking url is not set. But there are many cases, such as when app
> failed at launch, original tracking url will be set to point to RM page, so
> redirect to AHS page won't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4643) Container recovery is broken with delegating container runtime
[
https://issues.apache.org/jira/browse/YARN-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sidharta Seethana updated YARN-4643:
Attachment: YARN-4643.002.patch
hi [~jlowe], thanks for catching that. The tests in that file don't run
automatically and it also looks like the instructions there are out of date. In
any case, I have uploaded a new patch that fixes the test as well.
> Container recovery is broken with delegating container runtime
> --
>
> Key: YARN-4643
> URL: https://issues.apache.org/jira/browse/YARN-4643
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 2.8.0
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Critical
> Attachments: YARN-4643.001.patch, YARN-4643.002.patch
>
>
> Delegating container runtime uses the container's launch context to determine
> which runtime to use. However, during container recovery, a container object
> is not passed as input which leads to a {{NullPointerException}} when
> attempting to access the container's launch context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4428) Redirect RM page to AHS page when AHS turned on and RM page is not avaialable
[ https://issues.apache.org/jira/browse/YARN-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated YARN-4428: --- Attachment: YARN-4428.8.patch [~jlowe] thanks a lot for patient and careful review! updated.8 patch accordingly > Redirect RM page to AHS page when AHS turned on and RM page is not avaialable > - > > Key: YARN-4428 > URL: https://issues.apache.org/jira/browse/YARN-4428 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: YARN-4428.1.2.patch, YARN-4428.1.patch, > YARN-4428.2.2.patch, YARN-4428.2.patch, YARN-4428.3.patch, YARN-4428.3.patch, > YARN-4428.4.patch, YARN-4428.5.patch, YARN-4428.6.patch, YARN-4428.7.patch, > YARN-4428.8.patch > > > When AHS is turned on, if we can't view application in RM page, RM page > should redirect us to AHS page. For example, when you go to > cluster/app/application_1, if RM no longer remember the application, we will > simply get "Failed to read the application application_1", but it will be > good for RM ui to smartly try to redirect to AHS ui > /applicationhistory/app/application_1 to see if it's there. The redirect > usage already exist for logs in nodemanager UI. > Also, when AHS is enabled, WebAppProxyServlet should redirect to AHS page on > fall back of RM not remembering the app. YARN-3975 tried to do this only when > original tracking url is not set. But there are many cases, such as when app > failed at launch, original tracking url will be set to point to RM page, so > redirect to AHS page won't work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4629) Distributed shell breaks under strong security
[ https://issues.apache.org/jira/browse/YARN-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4629: --- Attachment: YARN-4629.002.patch How about something like this? If this looks alright, I'll go add some tests. > Distributed shell breaks under strong security > -- > > Key: YARN-4629 > URL: https://issues.apache.org/jira/browse/YARN-4629 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-4629.001.patch, YARN-4629.002.patch > > > If the auth_to_local is set to map requests from unknown hosts to nobody, the > dist shell app fails. The reason is that the client doesn't translate the > _HOST placeholder to the local hostname. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4224) Support fetching entities by UID and change the REST interface to conform to current REST APIs' in YARN
[ https://issues.apache.org/jira/browse/YARN-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120232#comment-15120232 ] Li Lu commented on YARN-4224: - +1. I'll commit shortly. > Support fetching entities by UID and change the REST interface to conform to > current REST APIs' in YARN > --- > > Key: YARN-4224 > URL: https://issues.apache.org/jira/browse/YARN-4224 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4224-YARN-2928.01.patch, > YARN-4224-YARN-2928.05.patch, YARN-4224-YARN-2928.06.patch, > YARN-4224-feature-YARN-2928.04.patch, > YARN-4224-feature-YARN-2928.wip.02.patch, > YARN-4224-feature-YARN-2928.wip.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[
https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120627#comment-15120627
]
Sangjin Lee commented on YARN-3367:
---
I went over the patch in some more detail, and it's definitely much closer. I
have one major (\?) suggestion and a few minor ones.
What you have implemented in {{TimelineClientImpl.EntitiesHolder}} is basically
what's already available in the JDK in {{FutureTask}}. What's needed here is a
result-bearing synchronizer along with the ability to run the task. That's
exactly what {{FutureTask}} is. You can define {{EntitiesHolder}} this way to
take advantage of it:
{code}
private final class EntitiesHolder extends FutureTask {
private final
org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities;
private final boolean isSync;
EntitiesHolder(
final
org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities,
final boolean isSync) {
super(new Callable() {
// publishEntities()
public Void call() throws Exception {
MultivaluedMap params = new MultivaluedMapImpl();
params.add("appid", contextAppId.toString());
params.add("async", Boolean.toString(!isSync));
putObjects("entities", params, entities);
return null;
}
});
this.entities = entities;
this.isSync = isSync;
}
public boolean isSync() {
return isSync;
}
public org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities
getEntities() {
return entities;
}
}
{code}
If you have a {{FutureTask}}, when you need to run {{publishEntities()}} you
can simply call {{run()}}. For example,
{code}
if (entitiesHolder != null) {
if (entitiesHolder.isSync()) {
entitiesHolder.run();
} else {
{code}
When you need to join with the result of that run on another thread (either its
normal completion or an exception), you can simply call {{get()}}. You can
catch {{ExecutionException}} and look at its cause to get the real exception:
{code}
// In sync call we need to wait till its published and if any error then
// throw it back
try {
entitiesHolder.get();
} catch (ExecutionException e) {
throw new YarnException(
"Failed while adding entity to the queue for publishing",
e.getCause());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new YarnException(
"Failed while adding entity to the queue for publishing", e);
}
{code}
Doing it this way removes a significant amount of code and results in much
simpler and more robust code. I have saved the prototype, and I'd be happy to
share the diff. Let me know.
Onto other feedback,
(TimelineClientImpl.java)
- l.877: I would get rid of this constructor, and simply pass in the boolean in
both cases
- l.921: Why only 2 calls? Also, should this be configurable?
- l.923: This doesn't need to be {{ScheduledExecutorService}}.
{{ExecutorService}} is all we need. Also, let's rename it to {{executor}}
instead of {{scheduler}}.
- l.925: {{stopped}} is superfluous because the underlying {{ExecutorService}}
manages the stoppage via interrupt. Let's remove it.
- l.931: name {{createThread()}} is not quite accurate; how about
{{createRunnable()}}?
- l.1007: if you want to check the status, you can replace {{stopped}} with
{{scheduler.isShutdown()}}
- l.1025: restore the interrupt status (via
{{Thread.currentThread().interrupt()}})
- l.1040: same
- l.1047: {{Executors.newSingleThreadedExecutor()}}
- l.1048: {{executor.execute(createRunnable());}}
> Replace starting a separate thread for post entity with event loop in
> TimelineClient
>
>
> Key: YARN-3367
> URL: https://issues.apache.org/jira/browse/YARN-3367
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-3367-YARN-2928.v1.005.patch,
> YARN-3367-YARN-2928.v1.006.patch, YARN-3367-feature-YARN-2928.003.patch,
> YARN-3367-feature-YARN-2928.v1.002.patch,
> YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch
>
>
> Since YARN-3039, we add loop in TimelineClient to wait for
> collectorServiceAddress ready before posting any entity. In consumer of
> TimelineClient (like AM), we are starting a new thread for each call to get
> rid of potential deadlock in main thread. This way has at least 3 major
> defects:
> 1. The consumer need some additional code to
[jira] [Commented] (YARN-4646) AMRMClient crashed when RM transition from active to standby
[
https://issues.apache.org/jira/browse/YARN-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120626#comment-15120626
]
sandflee commented on YARN-4646:
MR AM catches most remote exceptions and retry, I don't know if it's the proper
action if using in AMRMClientAsync, as in my mind, most remote exceptions are
caused by bad request, retry has no effect such as a very big resource request.
> AMRMClient crashed when RM transition from active to standby
>
>
> Key: YARN-4646
> URL: https://issues.apache.org/jira/browse/YARN-4646
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: sandflee
>
> when RM transition to standby, ApplicationMasterService#allocate() is
> interrupted and the exception is passed to AM.
> the following is the exception msg:
> {quote}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:266)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:448)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:258)
> ... 11 more
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:107)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy35.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:274)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:237)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException):
> java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:266)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:448)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
>
[jira] [Commented] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[ https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120646#comment-15120646 ] Naganarasimha G R commented on YARN-3367: - Thanks [~sjlee0], for sharing these points: bq. First of all, is it fair to assume that a TimelineClient belongs to a single user for the duration of the client? Even i was little skeptical about it, hence did not take a call there. But IMO if its used by *AM* or a *Container* then i think it would be safe to assume that its only used by a single UGI, as security Tokens needs to be got by the client from ATS server and then send it as part of launch context. So ideally single UGI is used for a timeline client but currently there is no restriction, but may be we can document the same. bq. If the former, then possibly we could pass in the UGI to the constructor so that the actual calls are made under doAs(). Based on above thoughts i think most propable solution is this. bq. combining entities into a single call becomes immediately complicated. In that case, we probably shouldn't combine entities from different users. Good point, did not think about this ! If we conclude as multiple UGI can use the same Timeline Client then we need to merge Entities only if the UGI is same (because this would be the most common use case). Further to avoid this disadvantage i would prefer to use single UGI by each timeline client. Thoughts? > Replace starting a separate thread for post entity with event loop in > TimelineClient > > > Key: YARN-3367 > URL: https://issues.apache.org/jira/browse/YARN-3367 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: YARN-3367-YARN-2928.v1.005.patch, > YARN-3367-YARN-2928.v1.006.patch, YARN-3367-feature-YARN-2928.003.patch, > YARN-3367-feature-YARN-2928.v1.002.patch, > YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch > > > Since YARN-3039, we add loop in TimelineClient to wait for > collectorServiceAddress ready before posting any entity. In consumer of > TimelineClient (like AM), we are starting a new thread for each call to get > rid of potential deadlock in main thread. This way has at least 3 major > defects: > 1. The consumer need some additional code to wrap a thread before calling > putEntities() in TimelineClient. > 2. It cost many thread resources which is unnecessary. > 3. The sequence of events could be out of order because each posting > operation thread get out of waiting loop randomly. > We should have something like event loop in TimelineClient side, > putEntities() only put related entities into a queue of entities and a > separated thread handle to deliver entities in queue to collector via REST > call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
[ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120685#comment-15120685 ] Sunil G commented on YARN-3215: --- Yes [~leftnoteasy], Thanks for pointing out. Current approach will return 0 in this initial case . I think a general safe approach (more headroom can be assumed safe for now) is what looks suits here considering some corner scenarios like this. With clear long term approach "by-partition headroom" ahead, as you mentioned all this can be handled. Till then, i think its safer to assume NO_LABEL in default case. > Respect labels in CapacityScheduler when computing headroom > --- > > Key: YARN-3215 > URL: https://issues.apache.org/jira/browse/YARN-3215 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Naganarasimha G R > Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch, > YARN-3215.v2.002.patch > > > In existing CapacityScheduler, when computing headroom of an application, it > will only consider "non-labeled" nodes of this application. > But it is possible the application is asking for labeled resources, so > headroom-by-label (like 5G resource available under node-label=red) is > required to get better resource allocation and avoid deadlocks such as > MAPREDUCE-5928. > This JIRA could involve both API changes (such as adding a > label-to-available-resource map in AllocateResponse) and also internal > changes in CapacityScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[
https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120731#comment-15120731
]
Wangda Tan commented on YARN-3940:
--
I think you can take a look at AppSchedulingInfo#getRequestedPartitions (will
be added with YARN-3215) to check if an app can run in another queue. It could
be better than simply use labels in ASC.
[~Naganarasimha].
> Application moveToQueue should check NodeLabel permission
> --
>
> Key: YARN-3940
> URL: https://issues.apache.org/jira/browse/YARN-3940
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch,
> 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch,
> 0006-YARN-3940.patch
>
>
> Configure capacity scheduler
> Configure node label an submit application {{queue=A Label=X}}
> Move application to queue {{B}} and x is not having access
> {code}
> 2015-07-20 19:46:19,626 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Application attempt appattempt_1437385548409_0005_01 released container
> container_e08_1437385548409_0005_01_02 on node: host:
> host-10-19-92-117:64318 #containers=1 available=
> used= with event: KILL
> 2015-07-20 19:46:20,970 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1437385548409_0005_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, queue=b1 doesn't have permission to access all labels in
> resource request. labelExpression of resource request=x. Queue labels=y
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168)
> {code}
> Same exception will be thrown till *heartbeat timeout*
> Then application state will be updated to *FAILED*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4224) Support fetching entities by UID and change the REST interface to conform to current REST APIs' in YARN
[
https://issues.apache.org/jira/browse/YARN-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119085#comment-15119085
]
Hadoop QA commented on YARN-4224:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 46s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
53s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 46s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 9 new +
46 unchanged - 14 fixed = 55 total (was 60) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} |
[jira] [Commented] (YARN-4633) TestRMRestart.testRMRestartAfterPreemption fails intermittently in trunk
[
https://issues.apache.org/jira/browse/YARN-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119108#comment-15119108
]
Hadoop QA commented on YARN-4633:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 14s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 57s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784595/0001-YARN-4633.patch |
| JIRA Issue | YARN-4633 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
|
[jira] [Commented] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[
https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118877#comment-15118877
]
Hadoop QA commented on YARN-4108:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 14s {color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66
with JDK v1.8.0_66 generated 1 new + 1 unchanged - 1 fixed = 2 total (was 2)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
patch generated 60 new + 496 unchanged - 8 fixed = 556 total (was 504) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 26s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 3s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 24s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 17s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
[jira] [Updated] (YARN-4644) TestRMRestart fails on YARN-2928 branch
[
https://issues.apache.org/jira/browse/YARN-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4644:
---
Description:
This was reported by YARN-4238 QA report. Refer to
https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
Error reported is as under :
{noformat}
org.mockito.exceptions.verification.TooManyActualInvocations:
noOpSystemMetricPublisher.appCreated(
,
);
Wanted 3 times:
-> at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
But was 6 times. Undesired invocation:
-> at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
{noformat}
Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
called twice.
Has been introduced during rebase I guess.
After removing the duplicate call, the test passes.
There is a *findbugs warning* in resourcemanager in YARN-2928 branch as well.
Fix it as part of this JIRA itself.
{noformat}
DLS Dead store to keepAliveApps in
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest)
Bug type DLS_DEAD_LOCAL_STORE (click for details)
In class org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService
In method
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest)
Local variable named keepAliveApps
At ResourceTrackerService.java:[line 486]
{noformat}
was:
This was reported by YARN-4238 QA report. Refer to
https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
Error reported is as under :
{noformat}
org.mockito.exceptions.verification.TooManyActualInvocations:
noOpSystemMetricPublisher.appCreated(
,
);
Wanted 3 times:
-> at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
But was 6 times. Undesired invocation:
-> at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
{noformat}
Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
called twice.
Has been introduced during rebase I guess.
After removing the duplicate call, the test passes.
There is a *findbugs
> TestRMRestart fails on YARN-2928 branch
> ---
>
> Key: YARN-4644
> URL: https://issues.apache.org/jira/browse/YARN-4644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4644-YARN-2928.01.patch,
> YARN-4644-YARN-2928.02.patch
>
>
> This was reported by YARN-4238 QA report. Refer to
> https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
> Error reported is as under :
> {noformat}
> org.mockito.exceptions.verification.TooManyActualInvocations:
> noOpSystemMetricPublisher.appCreated(
> ,
>
> );
> Wanted 3 times:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> But was 6 times. Undesired invocation:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> {noformat}
> Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
> called twice.
> Has been introduced during rebase I guess.
> After removing the duplicate call, the test passes.
> There is a *findbugs warning* in resourcemanager in YARN-2928 branch as well.
> Fix it as part of this JIRA itself.
> {noformat}
> DLS Dead store to keepAliveApps in
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest)
> Bug type DLS_DEAD_LOCAL_STORE (click for details)
> In class org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService
> In method
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest)
> Local variable named keepAliveApps
> At ResourceTrackerService.java:[line 486]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4644) TestRMRestart fails on YARN-2928 branch
[
https://issues.apache.org/jira/browse/YARN-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4644:
---
Description:
This was reported by YARN-4238 QA report. Refer to
https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
Error reported is as under :
{noformat}
org.mockito.exceptions.verification.TooManyActualInvocations:
noOpSystemMetricPublisher.appCreated(
,
);
Wanted 3 times:
-> at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
But was 6 times. Undesired invocation:
-> at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
{noformat}
Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
called twice.
Has been introduced during rebase I guess.
After removing the duplicate call, the test passes.
There is a *findbugs
was:
This was reported by YARN-4238 QA report. Refer to
https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
Error reported is as under :
{noformat}
org.mockito.exceptions.verification.TooManyActualInvocations:
noOpSystemMetricPublisher.appCreated(
,
);
Wanted 3 times:
-> at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
But was 6 times. Undesired invocation:
-> at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
{noformat}
Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
called twice.
Has been introduced during rebase I guess.
After removing the duplicate call, the test passes.
> TestRMRestart fails on YARN-2928 branch
> ---
>
> Key: YARN-4644
> URL: https://issues.apache.org/jira/browse/YARN-4644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4644-YARN-2928.01.patch,
> YARN-4644-YARN-2928.02.patch
>
>
> This was reported by YARN-4238 QA report. Refer to
> https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/
> Error reported is as under :
> {noformat}
> org.mockito.exceptions.verification.TooManyActualInvocations:
> noOpSystemMetricPublisher.appCreated(
> ,
>
> );
> Wanted 3 times:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> But was 6 times. Undesired invocation:
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955)
> {noformat}
> Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been
> called twice.
> Has been introduced during rebase I guess.
> After removing the duplicate call, the test passes.
> There is a *findbugs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4615) TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt fails occasionally
[
https://issues.apache.org/jira/browse/YARN-4615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118890#comment-15118890
]
Rohith Sharma K S commented on YARN-4615:
-
Great!!
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> fails occasionally
>
>
> Key: YARN-4615
> URL: https://issues.apache.org/jira/browse/YARN-4615
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Jason Lowe
>
> Sometimes
> TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt
> will fail like this:
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
> support was removed in 8.0
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 116.776 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
> testApplicationPriorityAllocationWithChangeInPriority(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority)
> Time elapsed: 50.687 sec <<< FAILURE!
> java.lang.AssertionError: Attempt state is not correct (timedout): expected:
> SCHEDULED actual: ALLOCATED for the application attempt
> appattempt_1453255879005_0002_01
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:197)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForAttemptScheduled(MockRM.java:831)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAM(MockRM.java:818)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority.testApplicationPriorityAllocationWithChangeInPriority(TestApplicationPriority.java:494)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
