[jira] [Commented] (YARN-2370) Made comment more accurate in AppSchedulingInfo.java
[
https://issues.apache.org/jira/browse/YARN-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078870#comment-14078870
]
Hadoop QA commented on YARN-2370:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658574/YARN-2370.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4478//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4478//console
This message is automatically generated.
Made comment more accurate in AppSchedulingInfo.java
Key: YARN-2370
URL: https://issues.apache.org/jira/browse/YARN-2370
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 3.0.0
Reporter: Wenwu Peng
Assignee: Wenwu Peng
Priority: Trivial
Attachments: YARN-2370.0.patch
in method allocateOffSwitch of AppSchedulingInfo.java, only invoke update
OffRack request, the comment should be Update cloned OffRack requests for
recovery not Update cloned RackLocal and OffRack requests for recover
{code}
// Update cloned RackLocal and OffRack requests for recovery
resourceRequests.add(cloneResourceRequest(offSwitchRequest));
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-18) Configurable Hierarchical Topology for YARN
[
https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079016#comment-14079016
]
Hadoop QA commented on YARN-18:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658610/YARN-18-v8-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4479//console
This message is automatically generated.
Configurable Hierarchical Topology for YARN
---
Key: YARN-18
URL: https://issues.apache.org/jira/browse/YARN-18
Project: Hadoop YARN
Issue Type: New Feature
Affects Versions: 2.0.3-alpha
Reporter: Junping Du
Assignee: Junping Du
Labels: features
Attachments:
HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch,
HierachicalTopologyForYARNr1.pdf, MAPREDUCE-4309-v2.patch,
MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch,
MAPREDUCE-4309-v6.patch, MAPREDUCE-4309-v7.patch, MAPREDUCE-4309.patch,
Pluggable topologies with NodeGroup for YARN.pdf, YARN-18-v2.patch,
YARN-18-v3.1.patch, YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch,
YARN-18-v4.2.patch, YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch,
YARN-18-v5.patch, YARN-18-v6.1.patch, YARN-18-v6.2.patch, YARN-18-v6.3.patch,
YARN-18-v6.4.patch, YARN-18-v6.patch, YARN-18-v7.1.patch, YARN-18-v7.2.patch,
YARN-18-v7.3.patch, YARN-18-v7.patch, YARN-18-v8-1.patch, YARN-18.patch
Per discussion in the design lounge of Hadoop Summit 2013, we agreed to
change the design of “Pluggable topologies with NodeGroup for YARN” to
support a configurable hierarchical topology that makes adding additional
locality layers simple. Please refer attached doc
HierachicalTopologyForYARNr1.pdf for details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-18) Configurable Hierarchical Topology for YARN
[
https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079064#comment-14079064
]
Hadoop QA commented on YARN-18:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658611/YARN-18-v8.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4480//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4480//console
This message is automatically generated.
Configurable Hierarchical Topology for YARN
---
Key: YARN-18
URL: https://issues.apache.org/jira/browse/YARN-18
Project: Hadoop YARN
Issue Type: New Feature
Affects Versions: 2.0.3-alpha
Reporter: Junping Du
Assignee: Junping Du
Labels: features
Attachments:
HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch,
HierachicalTopologyForYARNr1.pdf, MAPREDUCE-4309-v2.patch,
MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch,
MAPREDUCE-4309-v6.patch, MAPREDUCE-4309-v7.patch, MAPREDUCE-4309.patch,
Pluggable topologies with NodeGroup for YARN.pdf, YARN-18-v2.patch,
YARN-18-v3.1.patch, YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch,
YARN-18-v4.2.patch, YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch,
YARN-18-v5.patch, YARN-18-v6.1.patch, YARN-18-v6.2.patch, YARN-18-v6.3.patch,
YARN-18-v6.4.patch, YARN-18-v6.patch, YARN-18-v7.1.patch, YARN-18-v7.2.patch,
YARN-18-v7.3.patch, YARN-18-v7.patch, YARN-18-v8.0.patch, YARN-18.patch
Per discussion in the design lounge of Hadoop Summit 2013, we agreed to
change the design of “Pluggable topologies with NodeGroup for YARN” to
support a configurable hierarchical topology that makes adding additional
locality layers simple. Please refer attached doc
HierachicalTopologyForYARNr1.pdf for details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2372) There are Chinese Characters in the FairScheduler's document
[
https://issues.apache.org/jira/browse/YARN-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079162#comment-14079162
]
Hadoop QA commented on YARN-2372:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658625/YARN-2372.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4481//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4481//console
This message is automatically generated.
There are Chinese Characters in the FairScheduler's document
Key: YARN-2372
URL: https://issues.apache.org/jira/browse/YARN-2372
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.1
Reporter: Fengdong Yu
Assignee: Fengdong Yu
Priority: Minor
Attachments: YARN-2372.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2348) ResourceManager web UI should display server-side time instead of UTC time
[
https://issues.apache.org/jira/browse/YARN-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079341#comment-14079341
]
Hadoop QA commented on YARN-2348:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658650/4.after-patch.JPG
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4482//console
This message is automatically generated.
ResourceManager web UI should display server-side time instead of UTC time
--
Key: YARN-2348
URL: https://issues.apache.org/jira/browse/YARN-2348
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.4.1
Reporter: Leitao Guo
Attachments: 3.before-patch.JPG, 4.after-patch.JPG,
YARN-2348.2.patch, YARN-2348.patch
ResourceManager web UI, including application list and scheduler, displays
UTC time in default, this will confuse users who do not use UTC time. This
web UI should display server-side time in default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2348) ResourceManager web UI should display server-side time instead of UTC time
[
https://issues.apache.org/jira/browse/YARN-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079534#comment-14079534
]
Hadoop QA commented on YARN-2348:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658662/YARN-2348.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4483//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4483//console
This message is automatically generated.
ResourceManager web UI should display server-side time instead of UTC time
--
Key: YARN-2348
URL: https://issues.apache.org/jira/browse/YARN-2348
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.4.1
Reporter: Leitao Guo
Attachments: 3.before-patch.JPG, 4.after-patch.JPG,
YARN-2348.2.patch, YARN-2348.2.patch, YARN-2348.patch
ResourceManager web UI, including application list and scheduler, displays
UTC time in default, this will confuse users who do not use UTC time. This
web UI should display server-side time in default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2008) CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
[
https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079785#comment-14079785
]
Hadoop QA commented on YARN-2008:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658692/YARN-2008.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4484//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4484//console
This message is automatically generated.
CapacityScheduler may report incorrect queueMaxCap if there is hierarchy
queue structure
-
Key: YARN-2008
URL: https://issues.apache.org/jira/browse/YARN-2008
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Chen He
Assignee: Craig Welch
Attachments: YARN-2008.1.patch, YARN-2008.2.patch, YARN-2008.3.patch
If there are two queues, both allowed to use 100% of the actual resources in
the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and
there is not actual space available. If we use current method to get
headroom, CapacityScheduler thinks there are still available resources for
users in Q1 but they have been used by Q2.
If the CapacityScheduelr has a hierarchy queue structure, it may report
incorrect queueMaxCap. Here is a example
||||rootQueue|| ||
| | / |
\ |
| L1ParentQueue1 | |
L1ParentQueue2|
| (allowed to use up 80% of its parent)| | (allowed to use 20%
in minimum of its parent)|
|/ | \ ||
| L2LeafQueue1 |L2LeafQueue2 | |
|(50% of its parent) | (50% of its parent in minimum) | |
When we calculate headroom of a user in L2LeafQueue2, current method will
think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources.
However, without checking L1ParentQueue1, we are not sure. It is possible
that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually,
L2LeafQueue2 can only use 30% (60%*50%).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1994) Expose YARN/MR endpoints on multiple interfaces
[
https://issues.apache.org/jira/browse/YARN-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079871#comment-14079871
]
Hadoop QA commented on YARN-1994:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658703/YARN-1994.13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4485//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4485//console
This message is automatically generated.
Expose YARN/MR endpoints on multiple interfaces
---
Key: YARN-1994
URL: https://issues.apache.org/jira/browse/YARN-1994
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager, resourcemanager, webapp
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Craig Welch
Attachments: YARN-1994.0.patch, YARN-1994.1.patch,
YARN-1994.11.patch, YARN-1994.11.patch, YARN-1994.12.patch,
YARN-1994.13.patch, YARN-1994.2.patch, YARN-1994.3.patch, YARN-1994.4.patch,
YARN-1994.5.patch, YARN-1994.6.patch, YARN-1994.7.patch
YARN and MapReduce daemons currently do not support specifying a wildcard
address for the server endpoints. This prevents the endpoints from being
accessible from all interfaces on a multihomed machine.
Note that if we do specify INADDR_ANY for any of the options, it will break
clients as they will attempt to connect to 0.0.0.0. We need a solution that
allows specifying a hostname or IP-address for clients while requesting
wildcard bind for the servers.
(List of endpoints is in a comment below)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1994) Expose YARN/MR endpoints on multiple interfaces
[
https://issues.apache.org/jira/browse/YARN-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080402#comment-14080402
]
Hadoop QA commented on YARN-1994:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658793/YARN-1994.14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/4486//artifact/trunk/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4486//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4486//console
This message is automatically generated.
Expose YARN/MR endpoints on multiple interfaces
---
Key: YARN-1994
URL: https://issues.apache.org/jira/browse/YARN-1994
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager, resourcemanager, webapp
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Craig Welch
Attachments: YARN-1994.0.patch, YARN-1994.1.patch,
YARN-1994.11.patch, YARN-1994.11.patch, YARN-1994.12.patch,
YARN-1994.13.patch, YARN-1994.14.patch, YARN-1994.2.patch, YARN-1994.3.patch,
YARN-1994.4.patch, YARN-1994.5.patch, YARN-1994.6.patch, YARN-1994.7.patch
YARN and MapReduce daemons currently do not support specifying a wildcard
address for the server endpoints. This prevents the endpoints from being
accessible from all interfaces on a multihomed machine.
Note that if we do specify INADDR_ANY for any of the options, it will break
clients as they will attempt to connect to 0.0.0.0. We need a solution that
allows specifying a hostname or IP-address for clients while requesting
wildcard bind for the servers.
(List of endpoints is in a comment below)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2372) There are Chinese Characters in the FairScheduler's document
[
https://issues.apache.org/jira/browse/YARN-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080431#comment-14080431
]
Hadoop QA commented on YARN-2372:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658825/YARN-2372.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4488//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4488//console
This message is automatically generated.
There are Chinese Characters in the FairScheduler's document
Key: YARN-2372
URL: https://issues.apache.org/jira/browse/YARN-2372
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.1
Reporter: Fengdong Yu
Assignee: Fengdong Yu
Priority: Minor
Attachments: YARN-2372.patch, YARN-2372.patch, YARN-2372.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2372) There are Chinese Characters in the FairScheduler's document
[
https://issues.apache.org/jira/browse/YARN-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080444#comment-14080444
]
Hadoop QA commented on YARN-2372:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658828/YARN-2372.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4489//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4489//console
This message is automatically generated.
There are Chinese Characters in the FairScheduler's document
Key: YARN-2372
URL: https://issues.apache.org/jira/browse/YARN-2372
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.1
Reporter: Fengdong Yu
Assignee: Fengdong Yu
Priority: Minor
Attachments: YARN-2372.patch, YARN-2372.patch, YARN-2372.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1994) Expose YARN/MR endpoints on multiple interfaces
[
https://issues.apache.org/jira/browse/YARN-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080469#comment-14080469
]
Hadoop QA commented on YARN-1994:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658822/YARN-1994.15.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4487//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4487//console
This message is automatically generated.
Expose YARN/MR endpoints on multiple interfaces
---
Key: YARN-1994
URL: https://issues.apache.org/jira/browse/YARN-1994
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager, resourcemanager, webapp
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Craig Welch
Attachments: YARN-1994.0.patch, YARN-1994.1.patch,
YARN-1994.11.patch, YARN-1994.11.patch, YARN-1994.12.patch,
YARN-1994.13.patch, YARN-1994.14.patch, YARN-1994.15.patch,
YARN-1994.2.patch, YARN-1994.3.patch, YARN-1994.4.patch, YARN-1994.5.patch,
YARN-1994.6.patch, YARN-1994.7.patch
YARN and MapReduce daemons currently do not support specifying a wildcard
address for the server endpoints. This prevents the endpoints from being
accessible from all interfaces on a multihomed machine.
Note that if we do specify INADDR_ANY for any of the options, it will break
clients as they will attempt to connect to 0.0.0.0. We need a solution that
allows specifying a hostname or IP-address for clients while requesting
wildcard bind for the servers.
(List of endpoints is in a comment below)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2372) There are Chinese Characters in the FairScheduler's document
[
https://issues.apache.org/jira/browse/YARN-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080582#comment-14080582
]
Hadoop QA commented on YARN-2372:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658851/YARN-2372.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4490//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4490//console
This message is automatically generated.
There are Chinese Characters in the FairScheduler's document
Key: YARN-2372
URL: https://issues.apache.org/jira/browse/YARN-2372
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.1
Reporter: Fengdong Yu
Assignee: Fengdong Yu
Priority: Minor
Attachments: YARN-2372.patch, YARN-2372.patch, YARN-2372.patch,
YARN-2372.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2374) YARN trunk build failing TestDistributedShell.testDSShell
[
https://issues.apache.org/jira/browse/YARN-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080655#comment-14080655
]
Hadoop QA commented on YARN-2374:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658863/apache-yarn-2374.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4493//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4493//console
This message is automatically generated.
YARN trunk build failing TestDistributedShell.testDSShell
-
Key: YARN-2374
URL: https://issues.apache.org/jira/browse/YARN-2374
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2374.0.patch
The YARN trunk build has been failing for the last few days in the
distributed shell module.
{noformat}
testDSShell(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
Time elapsed: 27.269 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:188)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2212) ApplicationMaster needs to find a way to update the AMRMToken periodically
[
https://issues.apache.org/jira/browse/YARN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080665#comment-14080665
]
Hadoop QA commented on YARN-2212:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658859/YARN-2212.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnAMRMTokenRollOver
org.apache.hadoop.yarn.client.TestApplicationMasterServiceOnHA
org.apache.hadoop.yarn.client.TestRMFailover
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.api.impl.TestNMClient
org.apache.hadoop.yarn.client.TestGetGroups
org.apache.hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
org.apache.hadoop.yarn.client.api.impl.TestYarnClient
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4491//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4491//console
This message is automatically generated.
ApplicationMaster needs to find a way to update the AMRMToken periodically
--
Key: YARN-2212
URL: https://issues.apache.org/jira/browse/YARN-2212
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2212.1.patch, YARN-2212.2.patch,
YARN-2212.3.1.patch, YARN-2212.3.patch, YARN-2212.4.patch, YARN-2212.5.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2347) Consolidate RMStateVersion and NMDBSchemaVersion into StateVersion in yarn-server-common
[
https://issues.apache.org/jira/browse/YARN-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080669#comment-14080669
]
Hadoop QA commented on YARN-2347:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658862/YARN-2347-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4492//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4492//console
This message is automatically generated.
Consolidate RMStateVersion and NMDBSchemaVersion into StateVersion in
yarn-server-common
Key: YARN-2347
URL: https://issues.apache.org/jira/browse/YARN-2347
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2347-v2.patch, YARN-2347-v3.patch,
YARN-2347-v4.patch, YARN-2347-v5.patch, YARN-2347-v6.patch, YARN-2347.patch
We have similar things for version state for RM, NM, TS (TimelineServer),
etc. I think we should consolidate them into a common object.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2372) There are Chinese Characters in the FairScheduler's document
[
https://issues.apache.org/jira/browse/YARN-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080689#comment-14080689
]
Hadoop QA commented on YARN-2372:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658877/YARN-2372.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4494//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4494//console
This message is automatically generated.
There are Chinese Characters in the FairScheduler's document
Key: YARN-2372
URL: https://issues.apache.org/jira/browse/YARN-2372
Project: Hadoop YARN
Issue Type: Improvement
Components: documentation
Affects Versions: 2.4.1
Reporter: Fengdong Yu
Assignee: Fengdong Yu
Priority: Minor
Attachments: YARN-2372.patch, YARN-2372.patch, YARN-2372.patch,
YARN-2372.patch, YARN-2372.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2051) Fix code bug and add more unit tests for PBImpls
[
https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080750#comment-14080750
]
Hadoop QA commented on YARN-2051:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655677/YARN-2051.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4495//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4495//console
This message is automatically generated.
Fix code bug and add more unit tests for PBImpls
Key: YARN-2051
URL: https://issues.apache.org/jira/browse/YARN-2051
Project: Hadoop YARN
Issue Type: Test
Reporter: Junping Du
Assignee: Binglin Chang
Priority: Critical
Attachments: YARN-2051.v1.patch
From YARN-2016, we can see some bug could exist in PB implementation of
protocol. The bad news is most of these PBImpl don't have any unit test to
verify the info is not lost or changed after serialization/deserialization.
We should add more tests for it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2051) Fix code bug and add more unit tests for PBImpls
[
https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080829#comment-14080829
]
Hadoop QA commented on YARN-2051:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658915/YARN-2051.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4496//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4496//console
This message is automatically generated.
Fix code bug and add more unit tests for PBImpls
Key: YARN-2051
URL: https://issues.apache.org/jira/browse/YARN-2051
Project: Hadoop YARN
Issue Type: Test
Reporter: Junping Du
Assignee: Binglin Chang
Priority: Critical
Attachments: YARN-2051.v1.patch, YARN-2051.v2.patch
From YARN-2016, we can see some bug could exist in PB implementation of
protocol. The bad news is most of these PBImpl don't have any unit test to
verify the info is not lost or changed after serialization/deserialization.
We should add more tests for it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2212) ApplicationMaster needs to find a way to update the AMRMToken periodically
[
https://issues.apache.org/jira/browse/YARN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081090#comment-14081090
]
Hadoop QA commented on YARN-2212:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658944/YARN-2212.5.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4497//console
This message is automatically generated.
ApplicationMaster needs to find a way to update the AMRMToken periodically
--
Key: YARN-2212
URL: https://issues.apache.org/jira/browse/YARN-2212
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2212.1.patch, YARN-2212.2.patch,
YARN-2212.3.1.patch, YARN-2212.3.patch, YARN-2212.4.patch, YARN-2212.5.patch,
YARN-2212.5.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
[
https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081227#comment-14081227
]
Hadoop QA commented on YARN-2033:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658949/YARN-2033_ALL.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 20 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4498//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4498//console
This message is automatically generated.
Investigate merging generic-history into the Timeline Store
---
Key: YARN-2033
URL: https://issues.apache.org/jira/browse/YARN-2033
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf,
YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch,
YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch,
YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
Having two different stores isn't amicable to generic insights on what's
happening with applications. This is to investigate porting generic-history
into the Timeline Store.
One goal is to try and retain most of the client side interfaces as close to
what we have today.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2069) CS queue level preemption should respect user-limits
[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081331#comment-14081331
]
Hadoop QA commented on YARN-2069:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658971/YARN-2069-trunk-8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4499//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4499//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4499//console
This message is automatically generated.
CS queue level preemption should respect user-limits
Key: YARN-2069
URL: https://issues.apache.org/jira/browse/YARN-2069
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch,
YARN-2069-trunk-3.patch, YARN-2069-trunk-4.patch, YARN-2069-trunk-5.patch,
YARN-2069-trunk-6.patch, YARN-2069-trunk-7.patch, YARN-2069-trunk-8.patch
This is different from (even if related to, and likely share code with)
YARN-2113.
YARN-2113 focuses on making sure that even if queue has its guaranteed
capacity, it's individual users are treated in-line with their limits
irrespective of when they join in.
This JIRA is about respecting user-limits while preempting containers to
balance queue capacities.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2008) CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
[
https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081382#comment-14081382
]
Hadoop QA commented on YARN-2008:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658970/YARN-2008.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4500//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4500//console
This message is automatically generated.
CapacityScheduler may report incorrect queueMaxCap if there is hierarchy
queue structure
-
Key: YARN-2008
URL: https://issues.apache.org/jira/browse/YARN-2008
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Chen He
Assignee: Craig Welch
Attachments: YARN-2008.1.patch, YARN-2008.2.patch, YARN-2008.3.patch,
YARN-2008.4.patch
If there are two queues, both allowed to use 100% of the actual resources in
the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and
there is not actual space available. If we use current method to get
headroom, CapacityScheduler thinks there are still available resources for
users in Q1 but they have been used by Q2.
If the CapacityScheduelr has a hierarchy queue structure, it may report
incorrect queueMaxCap. Here is a example
||||rootQueue|| ||
| | / |
\ |
| L1ParentQueue1 | |
L1ParentQueue2|
| (allowed to use up 80% of its parent)| | (allowed to use 20%
in minimum of its parent)|
|/ | \ ||
| L2LeafQueue1 |L2LeafQueue2 | |
|(50% of its parent) | (50% of its parent in minimum) | |
When we calculate headroom of a user in L2LeafQueue2, current method will
think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources.
However, without checking L1ParentQueue1, we are not sure. It is possible
that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually,
L2LeafQueue2 can only use 30% (60%*50%).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2212) ApplicationMaster needs to find a way to update the AMRMToken periodically
[
https://issues.apache.org/jira/browse/YARN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081826#comment-14081826
]
Hadoop QA commented on YARN-2212:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658995/YARN-2212.5.rebase.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4501//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4501//console
This message is automatically generated.
ApplicationMaster needs to find a way to update the AMRMToken periodically
--
Key: YARN-2212
URL: https://issues.apache.org/jira/browse/YARN-2212
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2212.1.patch, YARN-2212.2.patch,
YARN-2212.3.1.patch, YARN-2212.3.patch, YARN-2212.4.patch, YARN-2212.5.patch,
YARN-2212.5.patch, YARN-2212.5.rebase.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2069) CS queue level preemption should respect user-limits
[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081985#comment-14081985
]
Hadoop QA commented on YARN-2069:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659051/YARN-2069-trunk-9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4502//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4502//console
This message is automatically generated.
CS queue level preemption should respect user-limits
Key: YARN-2069
URL: https://issues.apache.org/jira/browse/YARN-2069
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch,
YARN-2069-trunk-3.patch, YARN-2069-trunk-4.patch, YARN-2069-trunk-5.patch,
YARN-2069-trunk-6.patch, YARN-2069-trunk-7.patch, YARN-2069-trunk-8.patch,
YARN-2069-trunk-9.patch
This is different from (even if related to, and likely share code with)
YARN-2113.
YARN-2113 focuses on making sure that even if queue has its guaranteed
capacity, it's individual users are treated in-line with their limits
irrespective of when they join in.
This JIRA is about respecting user-limits while preempting containers to
balance queue capacities.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2008) CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
[
https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081988#comment-14081988
]
Hadoop QA commented on YARN-2008:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659068/YARN-2008.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4503//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4503//console
This message is automatically generated.
CapacityScheduler may report incorrect queueMaxCap if there is hierarchy
queue structure
-
Key: YARN-2008
URL: https://issues.apache.org/jira/browse/YARN-2008
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Chen He
Assignee: Craig Welch
Attachments: YARN-2008.1.patch, YARN-2008.2.patch, YARN-2008.3.patch,
YARN-2008.4.patch, YARN-2008.5.patch
If there are two queues, both allowed to use 100% of the actual resources in
the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and
there is not actual space available. If we use current method to get
headroom, CapacityScheduler thinks there are still available resources for
users in Q1 but they have been used by Q2.
If the CapacityScheduelr has a hierarchy queue structure, it may report
incorrect queueMaxCap. Here is a example
||||rootQueue|| ||
| | / |
\ |
| L1ParentQueue1 | |
L1ParentQueue2|
| (allowed to use up 80% of its parent)| | (allowed to use 20%
in minimum of its parent)|
|/ | \ ||
| L2LeafQueue1 |L2LeafQueue2 | |
|(50% of its parent) | (50% of its parent in minimum) | |
When we calculate headroom of a user in L2LeafQueue2, current method will
think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources.
However, without checking L1ParentQueue1, we are not sure. It is possible
that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually,
L2LeafQueue2 can only use 30% (60%*50%).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2377) Localization exception stack traces are not passed as diagnostic info
[
https://issues.apache.org/jira/browse/YARN-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082064#comment-14082064
]
Hadoop QA commented on YARN-2377:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659061/YARN-2377.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4504//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4504//console
This message is automatically generated.
Localization exception stack traces are not passed as diagnostic info
-
Key: YARN-2377
URL: https://issues.apache.org/jira/browse/YARN-2377
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: YARN-2377.v01.patch
In the Localizer log one can only see this kind of message
{code}
14/07/31 10:29:00 INFO localizer.ResourceLocalizationService: DEBUG: FAILED {
hdfs://ha-nn-uri-0:8020/tmp/hadoop-yarn/staging/gshegalov/.staging/job_1406825443306_0004/job.jar,
1406827248944, PATTERN, (?:classes/|lib/).* }, java.net.UnknownHos
tException: ha-nn-uri-0
{code}
And then only {{ java.net.UnknownHostException: ha-nn-uri-0}} message is
propagated as diagnostics.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082076#comment-14082076
]
Hadoop QA commented on YARN-2288:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659076/YARN-2288.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4505//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4505//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2374) YARN trunk build failing TestDistributedShell.testDSShell
[
https://issues.apache.org/jira/browse/YARN-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082292#comment-14082292
]
Hadoop QA commented on YARN-2374:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659134/apache-yarn-2374.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4506//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4506//console
This message is automatically generated.
YARN trunk build failing TestDistributedShell.testDSShell
-
Key: YARN-2374
URL: https://issues.apache.org/jira/browse/YARN-2374
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2374.0.patch, apache-yarn-2374.1.patch
The YARN trunk build has been failing for the last few days in the
distributed shell module.
{noformat}
testDSShell(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
Time elapsed: 27.269 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:188)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-535) TestUnmanagedAMLauncher can corrupt target/test-classes/yarn-site.xml during write phase, breaks later test runs
[
https://issues.apache.org/jira/browse/YARN-535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082297#comment-14082297
]
Hadoop QA commented on YARN-535:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583010/YARN-535-02.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4507//console
This message is automatically generated.
TestUnmanagedAMLauncher can corrupt target/test-classes/yarn-site.xml during
write phase, breaks later test runs
Key: YARN-535
URL: https://issues.apache.org/jira/browse/YARN-535
Project: Hadoop YARN
Issue Type: Bug
Components: applications
Affects Versions: 3.0.0
Environment: OS/X laptop, HFS+ filesystem
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Attachments: YARN-535-02.patch, YARN-535.patch
the setup phase of {{TestUnmanagedAMLauncher}} overwrites {{yarn-site.xml}}.
As {{Configuration.writeXml()}} does a reread of all resources, this will
break if the (open-for-writing) resource is already visible as an empty file.
This leaves a corrupted {{target/test-classes/yarn-site.xml}}, which breaks
later test runs -because it is not overwritten by later incremental builds,
due to timestamps.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2374) YARN trunk build failing TestDistributedShell.testDSShell
[
https://issues.apache.org/jira/browse/YARN-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082701#comment-14082701
]
Hadoop QA commented on YARN-2374:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659190/apache-yarn-2374.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4509//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4509//console
This message is automatically generated.
YARN trunk build failing TestDistributedShell.testDSShell
-
Key: YARN-2374
URL: https://issues.apache.org/jira/browse/YARN-2374
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2374.0.patch, apache-yarn-2374.1.patch,
apache-yarn-2374.2.patch, apache-yarn-2374.3.patch
The YARN trunk build has been failing for the last few days in the
distributed shell module.
{noformat}
testDSShell(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
Time elapsed: 27.269 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:188)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2008) CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
[
https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082815#comment-14082815
]
Hadoop QA commented on YARN-2008:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659207/YARN-2008.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4510//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4510//console
This message is automatically generated.
CapacityScheduler may report incorrect queueMaxCap if there is hierarchy
queue structure
-
Key: YARN-2008
URL: https://issues.apache.org/jira/browse/YARN-2008
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Chen He
Assignee: Craig Welch
Attachments: YARN-2008.1.patch, YARN-2008.2.patch, YARN-2008.3.patch,
YARN-2008.4.patch, YARN-2008.5.patch, YARN-2008.6.patch, YARN-2008.7.patch
If there are two queues, both allowed to use 100% of the actual resources in
the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and
there is not actual space available. If we use current method to get
headroom, CapacityScheduler thinks there are still available resources for
users in Q1 but they have been used by Q2.
If the CapacityScheduelr has a hierarchy queue structure, it may report
incorrect queueMaxCap. Here is a example
||||rootQueue|| ||
| | / |
\ |
| L1ParentQueue1 | |
L1ParentQueue2|
| (allowed to use up 80% of its parent)| | (allowed to use 20%
in minimum of its parent)|
|/ | \ ||
| L2LeafQueue1 |L2LeafQueue2 | |
|(50% of its parent) | (50% of its parent in minimum) | |
When we calculate headroom of a user in L2LeafQueue2, current method will
think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources.
However, without checking L1ParentQueue1, we are not sure. It is possible
that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually,
L2LeafQueue2 can only use 30% (60%*50%).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2379) Scheduler UI can't sort by field 9(total memory) in both 2.0.5 and 2.4
[
https://issues.apache.org/jira/browse/YARN-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082992#comment-14082992
]
Hadoop QA commented on YARN-2379:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659237/YARN-2379.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4511//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4511//console
This message is automatically generated.
Scheduler UI can't sort by field 9(total memory) in both 2.0.5 and 2.4
--
Key: YARN-2379
URL: https://issues.apache.org/jira/browse/YARN-2379
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.5-alpha, 2.4.0
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-2379.v1.patch
sorting field 9 total memory in Scheduler UI doesn't work correctly
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[
https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083028#comment-14083028
]
Hadoop QA commented on YARN-2277:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659244/YARN-2277-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4512//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4512//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4512//console
This message is automatically generated.
Add Cross-Origin support to the ATS REST API
Key: YARN-2277
URL: https://issues.apache.org/jira/browse/YARN-2277
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch,
YARN-2277-v2.patch
As the Application Timeline Server is not provided with built-in UI, it may
make sense to enable JSONP or CORS Rest API capabilities to allow for remote
UI to access the data directly via javascript without cross side server
browser blocks coming into play.
Example client may be like
http://api.jquery.com/jQuery.getJSON/
This can alleviate the need to create a local proxy cache.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2343) Improve error message on token expire exception
[
https://issues.apache.org/jira/browse/YARN-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083190#comment-14083190
]
Hadoop QA commented on YARN-2343:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659248/YARN-2343-080114.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4513//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4513//console
This message is automatically generated.
Improve error message on token expire exception
---
Key: YARN-2343
URL: https://issues.apache.org/jira/browse/YARN-2343
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Li Lu
Assignee: Li Lu
Priority: Trivial
Labels: usability
Attachments: YARN-2343-072314.patch, YARN-2343-080114.patch
Some of token expire exception is triggered by wrong time settings on cluster
nodes, but the current exception message does not explicitly address that. It
would be helpful to add some message explicitly pointing out that this
exception could be caused by machines out of sync in time, or even wrong time
zone settings.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[
https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083266#comment-14083266
]
Hadoop QA commented on YARN-2277:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659270/YARN-2277-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4514//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4514//console
This message is automatically generated.
Add Cross-Origin support to the ATS REST API
Key: YARN-2277
URL: https://issues.apache.org/jira/browse/YARN-2277
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch,
YARN-2277-v2.patch, YARN-2277-v3.patch
As the Application Timeline Server is not provided with built-in UI, it may
make sense to enable JSONP or CORS Rest API capabilities to allow for remote
UI to access the data directly via javascript without cross side server
browser blocks coming into play.
Example client may be like
http://api.jquery.com/jQuery.getJSON/
This can alleviate the need to create a local proxy cache.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[
https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083632#comment-14083632
]
Hadoop QA commented on YARN-2373:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659480/YARN-2373.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4515//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4515//console
This message is automatically generated.
WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
Key: YARN-2373
URL: https://issues.apache.org/jira/browse/YARN-2373
Project: Hadoop YARN
Issue Type: Bug
Reporter: Larry McCay
Attachments: YARN-2373.patch
As part of HADOOP-10904, this jira represents a change to WebAppUtils to
uptake the use of the credential provider API through the new method on
Configuration called getPassword.
This provides an alternative to storing the passwords in clear text within
the ssl-server.xml file while maintaining backward compatibility with that
behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[
https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083779#comment-14083779
]
Hadoop QA commented on YARN-2373:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659508/YARN-2373.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4516//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4516//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4516//console
This message is automatically generated.
WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
Key: YARN-2373
URL: https://issues.apache.org/jira/browse/YARN-2373
Project: Hadoop YARN
Issue Type: Bug
Reporter: Larry McCay
Attachments: YARN-2373.patch, YARN-2373.patch
As part of HADOOP-10904, this jira represents a change to WebAppUtils to
uptake the use of the credential provider API through the new method on
Configuration called getPassword.
This provides an alternative to storing the passwords in clear text within
the ssl-server.xml file while maintaining backward compatibility with that
behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[
https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083807#comment-14083807
]
Hadoop QA commented on YARN-2373:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659512/YARN-2373.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4517//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4517//console
This message is automatically generated.
WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
Key: YARN-2373
URL: https://issues.apache.org/jira/browse/YARN-2373
Project: Hadoop YARN
Issue Type: Bug
Reporter: Larry McCay
Attachments: YARN-2373.patch, YARN-2373.patch, YARN-2373.patch
As part of HADOOP-10904, this jira represents a change to WebAppUtils to
uptake the use of the credential provider API through the new method on
Configuration called getPassword.
This provides an alternative to storing the passwords in clear text within
the ssl-server.xml file while maintaining backward compatibility with that
behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1354) Recover applications upon nodemanager restart
[
https://issues.apache.org/jira/browse/YARN-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083820#comment-14083820
]
Hadoop QA commented on YARN-1354:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659513/YARN-1354-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4518//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4518//console
This message is automatically generated.
Recover applications upon nodemanager restart
-
Key: YARN-1354
URL: https://issues.apache.org/jira/browse/YARN-1354
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.3.0
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-1354-v1.patch,
YARN-1354-v2-and-YARN-1987-and-YARN-1362.patch, YARN-1354-v3.patch,
YARN-1354-v4.patch, YARN-1354-v5.patch, YARN-1354-v6.patch
The set of active applications in the nodemanager context need to be
recovered for work-preserving nodemanager restart
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2212) ApplicationMaster needs to find a way to update the AMRMToken periodically
[
https://issues.apache.org/jira/browse/YARN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085722#comment-14085722
]
Hadoop QA commented on YARN-2212:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659777/YARN-2212.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4519//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4519//console
This message is automatically generated.
ApplicationMaster needs to find a way to update the AMRMToken periodically
--
Key: YARN-2212
URL: https://issues.apache.org/jira/browse/YARN-2212
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2212.1.patch, YARN-2212.2.patch,
YARN-2212.3.1.patch, YARN-2212.3.patch, YARN-2212.4.patch, YARN-2212.5.patch,
YARN-2212.5.patch, YARN-2212.5.rebase.patch, YARN-2212.6.patch,
YARN-2212.6.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance
[
https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085725#comment-14085725
]
Hadoop QA commented on YARN-2352:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659794/yarn-2352-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.ha.TestZKFailoverControllerStress
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4520//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4520//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4520//console
This message is automatically generated.
FairScheduler: Collect metrics on duration of critical methods that affect
performance
--
Key: YARN-2352
URL: https://issues.apache.org/jira/browse/YARN-2352
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.4.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: fs-perf-metrics.png, yarn-2352-1.patch
We need more metrics for better visibility into FairScheduler performance. At
the least, we need to do this for (1) handle node events, (2) update, (3)
compute fairshares, (4) preemption.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2229) ContainerId can overflow with RM restart
[
https://issues.apache.org/jira/browse/YARN-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085855#comment-14085855
]
Hadoop QA commented on YARN-2229:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659801/YARN-2229.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4521//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4521//console
This message is automatically generated.
ContainerId can overflow with RM restart
Key: YARN-2229
URL: https://issues.apache.org/jira/browse/YARN-2229
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2229.1.patch, YARN-2229.10.patch,
YARN-2229.10.patch, YARN-2229.11.patch, YARN-2229.2.patch, YARN-2229.2.patch,
YARN-2229.3.patch, YARN-2229.4.patch, YARN-2229.5.patch, YARN-2229.6.patch,
YARN-2229.7.patch, YARN-2229.8.patch, YARN-2229.9.patch
On YARN-2052, we changed containerId format: upper 10 bits are for epoch,
lower 22 bits are for sequence number of Ids. This is for preserving
semantics of {{ContainerId#getId()}}, {{ContainerId#toString()}},
{{ContainerId#compareTo()}}, {{ContainerId#equals}}, and
{{ConverterUtils#toContainerId}}. One concern is epoch can overflow after RM
restarts 1024 times.
To avoid the problem, its better to make containerId long. We need to define
the new format of container Id with preserving backward compatibility on this
JIRA.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance
[
https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085977#comment-14085977
]
Hadoop QA commented on YARN-2352:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659837/yarn-2352-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.ipc.TestRPCCallBenchmark
org.apache.hadoop.net.TestNetUtils
org.apache.hadoop.security.TestSecurityUtil
org.apache.hadoop.crypto.key.TestKeyProviderFactory
org.apache.hadoop.ipc.TestRPC
org.apache.hadoop.security.TestDoAsEffectiveUser
org.apache.hadoop.conf.TestConfiguration
org.apache.hadoop.ipc.TestCallQueueManager
org.apache.hadoop.ipc.TestIPC
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4522//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4522//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4522//console
This message is automatically generated.
FairScheduler: Collect metrics on duration of critical methods that affect
performance
--
Key: YARN-2352
URL: https://issues.apache.org/jira/browse/YARN-2352
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.4.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: fs-perf-metrics.png, yarn-2352-1.patch, yarn-2352-2.patch
We need more metrics for better visibility into FairScheduler performance. At
the least, we need to do this for (1) handle node events, (2) update, (3)
compute fairshares, (4) preemption.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2374) YARN trunk build failing TestDistributedShell.testDSShell
[
https://issues.apache.org/jira/browse/YARN-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086074#comment-14086074
]
Hadoop QA commented on YARN-2374:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659868/apache-yarn-2374.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4523//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4523//console
This message is automatically generated.
YARN trunk build failing TestDistributedShell.testDSShell
-
Key: YARN-2374
URL: https://issues.apache.org/jira/browse/YARN-2374
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2374.0.patch, apache-yarn-2374.1.patch,
apache-yarn-2374.2.patch, apache-yarn-2374.3.patch, apache-yarn-2374.4.patch
The YARN trunk build has been failing for the last few days in the
distributed shell module.
{noformat}
testDSShell(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
Time elapsed: 27.269 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:188)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1337) Recover containers upon nodemanager restart
[
https://issues.apache.org/jira/browse/YARN-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086909#comment-14086909
]
Hadoop QA commented on YARN-1337:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659958/YARN-1337-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.TestPBLocalizerRPC
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4524//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4524//console
This message is automatically generated.
Recover containers upon nodemanager restart
---
Key: YARN-1337
URL: https://issues.apache.org/jira/browse/YARN-1337
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.3.0
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-1337-v1.patch
To support work-preserving NM restart we need to recover the state of the
containers when the nodemanager went down. This includes informing the RM of
containers that have exited in the interim and a strategy for dealing with
the exit codes from those containers along with how to reacquire the active
containers and determine their exit codes when they terminate. The state of
finished containers also needs to be recovered.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2212) ApplicationMaster needs to find a way to update the AMRMToken periodically
[
https://issues.apache.org/jira/browse/YARN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087111#comment-14087111
]
Hadoop QA commented on YARN-2212:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1265/YARN-2212.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4525//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4525//console
This message is automatically generated.
ApplicationMaster needs to find a way to update the AMRMToken periodically
--
Key: YARN-2212
URL: https://issues.apache.org/jira/browse/YARN-2212
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2212.1.patch, YARN-2212.2.patch,
YARN-2212.3.1.patch, YARN-2212.3.patch, YARN-2212.4.patch, YARN-2212.5.patch,
YARN-2212.5.patch, YARN-2212.5.rebase.patch, YARN-2212.6.patch,
YARN-2212.6.patch, YARN-2212.7.patch, YARN-2212.7.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2359) Application is hung without timeout and retry after DNS/network is down.
[
https://issues.apache.org/jira/browse/YARN-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087117#comment-14087117
]
Hadoop QA commented on YARN-2359:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1266/YARN-2359.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4526//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4526//console
This message is automatically generated.
Application is hung without timeout and retry after DNS/network is down.
-
Key: YARN-2359
URL: https://issues.apache.org/jira/browse/YARN-2359
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Critical
Attachments: YARN-2359.000.patch, YARN-2359.001.patch,
YARN-2359.002.patch
Application is hung without timeout and retry after DNS/network is down.
It is because right after the container is allocated for the AM, the
DNS/network is down for the node which has the AM container.
The application attempt is at state RMAppAttemptState.SCHEDULED, it receive
RMAppAttemptEventType.CONTAINER_ALLOCATED event, because the
IllegalArgumentException(due to DNS error) happened, it stay at state
RMAppAttemptState.SCHEDULED. In the state machine, only two events will be
processed at this state:
RMAppAttemptEventType.CONTAINER_ALLOCATED and RMAppAttemptEventType.KILL.
The code didn't handle the event(RMAppAttemptEventType.CONTAINER_FINISHED)
which will be generated when the node and container timeout. So even the node
is removed, the Application is still hung in this state
RMAppAttemptState.SCHEDULED.
The only way to make the application exit this state is to send
RMAppAttemptEventType.KILL event which will only be generated when you
manually kill the application from Job Client by forceKillApplication.
To fix the issue, we should add an entry in the state machine table to handle
RMAppAttemptEventType.CONTAINER_FINISHED event at state
RMAppAttemptState.SCHEDULED
add the following code in StateMachineFactory:
{code}.addTransition(RMAppAttemptState.SCHEDULED,
RMAppAttemptState.FINAL_SAVING,
RMAppAttemptEventType.CONTAINER_FINISHED,
new FinalSavingTransition(
new AMContainerCrashedBeforeRunningTransition(),
RMAppAttemptState.FAILED)){code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2378) Adding support for moving apps between queues in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087192#comment-14087192
]
Hadoop QA commented on YARN-2378:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660005/YARN-2378.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/4527//artifact/trunk/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4527//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4527//console
This message is automatically generated.
Adding support for moving apps between queues in Capacity Scheduler
---
Key: YARN-2378
URL: https://issues.apache.org/jira/browse/YARN-2378
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Subramaniam Venkatraman Krishnan
Assignee: Subramaniam Venkatraman Krishnan
Labels: capacity-scheduler
Attachments: YARN-2378.patch
As discussed with [~leftnoteasy] and [~jianhe], we are breaking up YARN-1707
to smaller patches for manageability. This JIRA will address adding support
for moving apps between queues in Capacity Scheduler.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087222#comment-14087222
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660025/YARN-2288-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4528//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4528//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2374) YARN trunk build failing TestDistributedShell.testDSShell
[
https://issues.apache.org/jira/browse/YARN-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087241#comment-14087241
]
Hadoop QA commented on YARN-2374:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659868/apache-yarn-2374.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4529//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4529//console
This message is automatically generated.
YARN trunk build failing TestDistributedShell.testDSShell
-
Key: YARN-2374
URL: https://issues.apache.org/jira/browse/YARN-2374
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2374.0.patch, apache-yarn-2374.1.patch,
apache-yarn-2374.2.patch, apache-yarn-2374.3.patch, apache-yarn-2374.4.patch
The YARN trunk build has been failing for the last few days in the
distributed shell module.
{noformat}
testDSShell(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
Time elapsed: 27.269 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:188)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087380#comment-14087380
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660025/YARN-2288-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4530//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4530//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1954) Add waitFor to AMRMClient(Async)
[
https://issues.apache.org/jira/browse/YARN-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087501#comment-14087501
]
Hadoop QA commented on YARN-1954:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660096/YARN-1954.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.api.async.impl.TestAMRMClientAsync
org.apache.hadoop.yarn.client.TestApplicationMasterServiceOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4531//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4531//console
This message is automatically generated.
Add waitFor to AMRMClient(Async)
Key: YARN-1954
URL: https://issues.apache.org/jira/browse/YARN-1954
Project: Hadoop YARN
Issue Type: New Feature
Components: client
Affects Versions: 3.0.0, 2.4.0
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1954.1.patch, YARN-1954.2.patch, YARN-1954.3.patch,
YARN-1954.4.patch, YARN-1954.4.patch, YARN-1954.5.patch
Recently, I saw some use cases of AMRMClient(Async). The painful thing is
that the main non-daemon thread has to sit in a dummy loop to prevent AM
process exiting before all the tasks are done, while unregistration is
triggered on a separate another daemon thread by callback methods (in
particular when using AMRMClientAsync). IMHO, it should be beneficial to add
a waitFor method to AMRMClient(Async) to block the AM until unregistration or
user supplied check point, such that users don't need to write the loop
themselves.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1954) Add waitFor to AMRMClient(Async)
[
https://issues.apache.org/jira/browse/YARN-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087523#comment-14087523
]
Hadoop QA commented on YARN-1954:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660096/YARN-1954.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.api.async.impl.TestAMRMClientAsync
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4532//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4532//console
This message is automatically generated.
Add waitFor to AMRMClient(Async)
Key: YARN-1954
URL: https://issues.apache.org/jira/browse/YARN-1954
Project: Hadoop YARN
Issue Type: New Feature
Components: client
Affects Versions: 3.0.0, 2.4.0
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1954.1.patch, YARN-1954.2.patch, YARN-1954.3.patch,
YARN-1954.4.patch, YARN-1954.4.patch, YARN-1954.5.patch
Recently, I saw some use cases of AMRMClient(Async). The painful thing is
that the main non-daemon thread has to sit in a dummy loop to prevent AM
process exiting before all the tasks are done, while unregistration is
triggered on a separate another daemon thread by callback methods (in
particular when using AMRMClientAsync). IMHO, it should be beneficial to add
a waitFor method to AMRMClient(Async) to block the AM until unregistration or
user supplied check point, such that users don't need to write the loop
themselves.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2138) Cleanup notifyDone* methods in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087557#comment-14087557
]
Hadoop QA commented on YARN-2138:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660104/YARN-2138.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4533//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4533//console
This message is automatically generated.
Cleanup notifyDone* methods in RMStateStore
---
Key: YARN-2138
URL: https://issues.apache.org/jira/browse/YARN-2138
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Varun Saxena
Attachments: YARN-2138.patch
The storedException passed into notifyDoneStoringApplication is always null.
Similarly for other notifyDone* methods. We can clean up these methods as
this control flow path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2130) Cleanup: Adding getRMAppManager, getQueueACLsManager, getApplicationACLsManager to RMContext
[
https://issues.apache.org/jira/browse/YARN-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087583#comment-14087583
]
Hadoop QA commented on YARN-2130:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660110/YARN-2130.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationMasterServiceOnHA
org.apache.hadoop.yarn.client.TestRMFailover
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.api.impl.TestNMClient
org.apache.hadoop.yarn.client.TestGetGroups
org.apache.hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
org.apache.hadoop.yarn.client.api.impl.TestYarnClient
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4534//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4534//console
This message is automatically generated.
Cleanup: Adding getRMAppManager, getQueueACLsManager,
getApplicationACLsManager to RMContext
Key: YARN-2130
URL: https://issues.apache.org/jira/browse/YARN-2130
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2130.1.patch, YARN-2130.2.patch, YARN-2130.3.patch,
YARN-2130.4.patch, YARN-2130.5.patch, YARN-2130.6.patch, YARN-2130.7-2.patch,
YARN-2130.7.patch, YARN-2130.8.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1954) Add waitFor to AMRMClient(Async)
[
https://issues.apache.org/jira/browse/YARN-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087690#comment-14087690
]
Hadoop QA commented on YARN-1954:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660129/YARN-1954.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4535//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4535//console
This message is automatically generated.
Add waitFor to AMRMClient(Async)
Key: YARN-1954
URL: https://issues.apache.org/jira/browse/YARN-1954
Project: Hadoop YARN
Issue Type: New Feature
Components: client
Affects Versions: 3.0.0, 2.4.0
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1954.1.patch, YARN-1954.2.patch, YARN-1954.3.patch,
YARN-1954.4.patch, YARN-1954.4.patch, YARN-1954.5.patch, YARN-1954.6.patch
Recently, I saw some use cases of AMRMClient(Async). The painful thing is
that the main non-daemon thread has to sit in a dummy loop to prevent AM
process exiting before all the tasks are done, while unregistration is
triggered on a separate another daemon thread by callback methods (in
particular when using AMRMClientAsync). IMHO, it should be beneficial to add
a waitFor method to AMRMClient(Async) to block the AM until unregistration or
user supplied check point, such that users don't need to write the loop
themselves.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2130) Cleanup: Adding getRMAppManager, getQueueACLsManager, getApplicationACLsManager to RMContext
[
https://issues.apache.org/jira/browse/YARN-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087843#comment-14087843
]
Hadoop QA commented on YARN-2130:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660150/YARN-2130.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4536//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4536//console
This message is automatically generated.
Cleanup: Adding getRMAppManager, getQueueACLsManager,
getApplicationACLsManager to RMContext
Key: YARN-2130
URL: https://issues.apache.org/jira/browse/YARN-2130
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2130.1.patch, YARN-2130.2.patch, YARN-2130.3.patch,
YARN-2130.4.patch, YARN-2130.5.patch, YARN-2130.6.patch, YARN-2130.7-2.patch,
YARN-2130.7.patch, YARN-2130.8.patch, YARN-2130.8.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance
[
https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087860#comment-14087860
]
Hadoop QA commented on YARN-2352:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660151/yarn-2352-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4537//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4537//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4537//console
This message is automatically generated.
FairScheduler: Collect metrics on duration of critical methods that affect
performance
--
Key: YARN-2352
URL: https://issues.apache.org/jira/browse/YARN-2352
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.4.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: fs-perf-metrics.png, yarn-2352-1.patch,
yarn-2352-2.patch, yarn-2352-2.patch
We need more metrics for better visibility into FairScheduler performance. At
the least, we need to do this for (1) handle node events, (2) update, (3)
compute fairshares, (4) preemption.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1337) Recover containers upon nodemanager restart
[
https://issues.apache.org/jira/browse/YARN-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088204#comment-14088204
]
Hadoop QA commented on YARN-1337:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659958/YARN-1337-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4538//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4538//console
This message is automatically generated.
Recover containers upon nodemanager restart
---
Key: YARN-1337
URL: https://issues.apache.org/jira/browse/YARN-1337
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.3.0
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-1337-v1.patch
To support work-preserving NM restart we need to recover the state of the
containers when the nodemanager went down. This includes informing the RM of
containers that have exited in the interim and a strategy for dealing with
the exit codes from those containers along with how to reacquire the active
containers and determine their exit codes when they terminate. The state of
finished containers also needs to be recovered.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2388) TestTimelineWebServices fails on trunk after HADOOP-10791
[
https://issues.apache.org/jira/browse/YARN-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088238#comment-14088238
]
Hadoop QA commented on YARN-2388:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660201/YARN-2388.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4539//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4539//console
This message is automatically generated.
TestTimelineWebServices fails on trunk after HADOOP-10791
-
Key: YARN-2388
URL: https://issues.apache.org/jira/browse/YARN-2388
Project: Hadoop YARN
Issue Type: Test
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: YARN-2388.1.patch
https://builds.apache.org/job/PreCommit-YARN-Build/4530//testReport/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2008) CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
[
https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088470#comment-14088470
]
Hadoop QA commented on YARN-2008:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660231/YARN-2008.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4540//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4540//console
This message is automatically generated.
CapacityScheduler may report incorrect queueMaxCap if there is hierarchy
queue structure
-
Key: YARN-2008
URL: https://issues.apache.org/jira/browse/YARN-2008
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Chen He
Assignee: Craig Welch
Attachments: YARN-2008.1.patch, YARN-2008.2.patch, YARN-2008.3.patch,
YARN-2008.4.patch, YARN-2008.5.patch, YARN-2008.6.patch, YARN-2008.7.patch,
YARN-2008.8.patch
If there are two queues, both allowed to use 100% of the actual resources in
the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and
there is not actual space available. If we use current method to get
headroom, CapacityScheduler thinks there are still available resources for
users in Q1 but they have been used by Q2.
If the CapacityScheduelr has a hierarchy queue structure, it may report
incorrect queueMaxCap. Here is a example
||||rootQueue|| ||
| | / |
\ |
| L1ParentQueue1 | |
L1ParentQueue2|
| (allowed to use up 80% of its parent)| | (allowed to use 20%
in minimum of its parent)|
|/ | \ ||
| L2LeafQueue1 |L2LeafQueue2 | |
|(50% of its parent) | (50% of its parent in minimum) | |
When we calculate headroom of a user in L2LeafQueue2, current method will
think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources.
However, without checking L1ParentQueue1, we are not sure. It is possible
that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually,
L2LeafQueue2 can only use 30% (60%*50%).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback
[
https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088646#comment-14088646
]
Hadoop QA commented on YARN-415:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660287/YARN-415.201408062232.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4542//console
This message is automatically generated.
Capture memory utilization at the app-level for chargeback
--
Key: YARN-415
URL: https://issues.apache.org/jira/browse/YARN-415
Project: Hadoop YARN
Issue Type: New Feature
Components: resourcemanager
Affects Versions: 0.23.6
Reporter: Kendall Thrapp
Assignee: Andrey Klochkov
Attachments: YARN-415--n10.patch, YARN-415--n2.patch,
YARN-415--n3.patch, YARN-415--n4.patch, YARN-415--n5.patch,
YARN-415--n6.patch, YARN-415--n7.patch, YARN-415--n8.patch,
YARN-415--n9.patch, YARN-415.201405311749.txt, YARN-415.201406031616.txt,
YARN-415.201406262136.txt, YARN-415.201407042037.txt,
YARN-415.201407071542.txt, YARN-415.201407171553.txt,
YARN-415.201407172144.txt, YARN-415.201407232237.txt,
YARN-415.201407242148.txt, YARN-415.201407281816.txt,
YARN-415.201408062232.txt, YARN-415.patch
For the purpose of chargeback, I'd like to be able to compute the cost of an
application in terms of cluster resource usage. To start out, I'd like to
get the memory utilization of an application. The unit should be MB-seconds
or something similar and, from a chargeback perspective, the memory amount
should be the memory reserved for the application, as even if the app didn't
use all that memory, no one else was able to use it.
(reserved ram for container 1 * lifetime of container 1) + (reserved ram for
container 2 * lifetime of container 2) + ... + (reserved ram for container n
* lifetime of container n)
It'd be nice to have this at the app level instead of the job level because:
1. We'd still be able to get memory usage for jobs that crashed (and wouldn't
appear on the job history server).
2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).
This new metric should be available both through the RM UI and RM Web
Services REST API.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2212) ApplicationMaster needs to find a way to update the AMRMToken periodically
[
https://issues.apache.org/jira/browse/YARN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088653#comment-14088653
]
Hadoop QA commented on YARN-2212:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660260/YARN-2212.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4541//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4541//console
This message is automatically generated.
ApplicationMaster needs to find a way to update the AMRMToken periodically
--
Key: YARN-2212
URL: https://issues.apache.org/jira/browse/YARN-2212
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2212.1.patch, YARN-2212.2.patch,
YARN-2212.3.1.patch, YARN-2212.3.patch, YARN-2212.4.patch, YARN-2212.5.patch,
YARN-2212.5.patch, YARN-2212.5.rebase.patch, YARN-2212.6.patch,
YARN-2212.6.patch, YARN-2212.7.patch, YARN-2212.7.patch, YARN-2212.8.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2249) RM may receive container release request on AM resync before container is actually recovered
[
https://issues.apache.org/jira/browse/YARN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088663#comment-14088663
]
Hadoop QA commented on YARN-2249:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660275/YARN-2249.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4543//console
This message is automatically generated.
RM may receive container release request on AM resync before container is
actually recovered
Key: YARN-2249
URL: https://issues.apache.org/jira/browse/YARN-2249
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-2249.1.patch
AM resync on RM restart will send outstanding container release requests back
to the new RM. In the meantime, NMs report the container statuses back to RM
to recover the containers. If RM receives the container release request
before the container is actually recovered in scheduler, the container won't
be released and the release request will be lost.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance
[
https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088745#comment-14088745
]
Hadoop QA commented on YARN-2352:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660272/yarn-2352-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4544//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4544//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4544//console
This message is automatically generated.
FairScheduler: Collect metrics on duration of critical methods that affect
performance
--
Key: YARN-2352
URL: https://issues.apache.org/jira/browse/YARN-2352
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.4.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: fs-perf-metrics.png, yarn-2352-1.patch,
yarn-2352-2.patch, yarn-2352-2.patch, yarn-2352-3.patch
We need more metrics for better visibility into FairScheduler performance. At
the least, we need to do this for (1) handle node events, (2) update, (3)
compute fairshares, (4) preemption.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088841#comment-14088841
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660309/YARN-2288-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
org.apache.hadoop.yarn.server.timeline.TestLeveldbTimelineStore
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4545//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4545//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch,
YARN-2288-v4.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2249) RM may receive container release request on AM resync before container is actually recovered
[
https://issues.apache.org/jira/browse/YARN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1404#comment-1404
]
Hadoop QA commented on YARN-2249:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660316/YARN-2249.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestMaxRunningAppsEnforcer
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSSchedulerApp
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerApplicationAttempt
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4546//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4546//console
This message is automatically generated.
RM may receive container release request on AM resync before container is
actually recovered
Key: YARN-2249
URL: https://issues.apache.org/jira/browse/YARN-2249
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-2249.1.patch, YARN-2249.1.patch
AM resync on RM restart will send outstanding container release requests back
to the new RM. In the meantime, NMs report the container statuses back to RM
to recover the containers. If RM receives the container release request
before the container is actually recovered in scheduler, the container won't
be released and the release request will be lost.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1408#comment-1408
]
Hadoop QA commented on YARN-2288:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660309/YARN-2288-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.TestLeveldbTimelineStore
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4547//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4547//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch,
YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2288) Data persistent in timelinestore should be versioned
[
https://issues.apache.org/jira/browse/YARN-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088980#comment-14088980
]
Hadoop QA commented on YARN-2288:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660327/YARN-2288-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4548//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4548//console
This message is automatically generated.
Data persistent in timelinestore should be versioned
Key: YARN-2288
URL: https://issues.apache.org/jira/browse/YARN-2288
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: 2.4.1
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-2288-v2.patch, YARN-2288-v3.patch,
YARN-2288-v4.patch, YARN-2288-v5.patch, YARN-2288.patch
We have LevelDB-backed TimelineStore, it should have schema version for
changes in schema in future.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2008) CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure
[
https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089606#comment-14089606
]
Hadoop QA commented on YARN-2008:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660415/YARN-2008.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4549//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4549//console
This message is automatically generated.
CapacityScheduler may report incorrect queueMaxCap if there is hierarchy
queue structure
-
Key: YARN-2008
URL: https://issues.apache.org/jira/browse/YARN-2008
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Chen He
Assignee: Craig Welch
Attachments: YARN-2008.1.patch, YARN-2008.2.patch, YARN-2008.3.patch,
YARN-2008.4.patch, YARN-2008.5.patch, YARN-2008.6.patch, YARN-2008.7.patch,
YARN-2008.8.patch, YARN-2008.9.patch
If there are two queues, both allowed to use 100% of the actual resources in
the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and
there is not actual space available. If we use current method to get
headroom, CapacityScheduler thinks there are still available resources for
users in Q1 but they have been used by Q2.
If the CapacityScheduelr has a hierarchy queue structure, it may report
incorrect queueMaxCap. Here is a example
||||rootQueue|| ||
| | / |
\ |
| L1ParentQueue1 | |
L1ParentQueue2|
| (allowed to use up 80% of its parent)| | (allowed to use 20%
in minimum of its parent)|
|/ | \ ||
| L2LeafQueue1 |L2LeafQueue2 | |
|(50% of its parent) | (50% of its parent in minimum) | |
When we calculate headroom of a user in L2LeafQueue2, current method will
think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources.
However, without checking L1ParentQueue1, we are not sure. It is possible
that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually,
L2LeafQueue2 can only use 30% (60%*50%).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance
[
https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089665#comment-14089665
]
Hadoop QA commented on YARN-2352:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660423/yarn-2352-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4550//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4550//console
This message is automatically generated.
FairScheduler: Collect metrics on duration of critical methods that affect
performance
--
Key: YARN-2352
URL: https://issues.apache.org/jira/browse/YARN-2352
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.4.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: fs-perf-metrics.png, yarn-2352-1.patch,
yarn-2352-2.patch, yarn-2352-2.patch, yarn-2352-3.patch, yarn-2352-4.patch
We need more metrics for better visibility into FairScheduler performance. At
the least, we need to do this for (1) handle node events, (2) update, (3)
compute fairshares, (4) preemption.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2212) ApplicationMaster needs to find a way to update the AMRMToken periodically
[
https://issues.apache.org/jira/browse/YARN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089970#comment-14089970
]
Hadoop QA commented on YARN-2212:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660467/YARN-2212.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4552//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4552//console
This message is automatically generated.
ApplicationMaster needs to find a way to update the AMRMToken periodically
--
Key: YARN-2212
URL: https://issues.apache.org/jira/browse/YARN-2212
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-2212.1.patch, YARN-2212.2.patch,
YARN-2212.3.1.patch, YARN-2212.3.patch, YARN-2212.4.patch, YARN-2212.5.patch,
YARN-2212.5.patch, YARN-2212.5.rebase.patch, YARN-2212.6.patch,
YARN-2212.6.patch, YARN-2212.7.patch, YARN-2212.7.patch, YARN-2212.8.patch,
YARN-2212.9.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2026) Fair scheduler : Fair share for inactive queues causes unfair allocation in some scenarios
[
https://issues.apache.org/jira/browse/YARN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090026#comment-14090026
]
Hadoop QA commented on YARN-2026:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660479/YARN-2026-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4553//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4553//console
This message is automatically generated.
Fair scheduler : Fair share for inactive queues causes unfair allocation in
some scenarios
--
Key: YARN-2026
URL: https://issues.apache.org/jira/browse/YARN-2026
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Labels: scheduler
Attachments: YARN-2026-v1.txt, YARN-2026-v2.txt, YARN-2026-v3.txt,
YARN-2026-v4.txt
Problem1- While using hierarchical queues in fair scheduler,there are few
scenarios where we have seen a leaf queue with least fair share can take
majority of the cluster and starve a sibling parent queue which has greater
weight/fair share and preemption doesn’t kick in to reclaim resources.
The root cause seems to be that fair share of a parent queue is distributed
to all its children irrespective of whether its an active or an inactive(no
apps running) queue. Preemption based on fair share kicks in only if the
usage of a queue is less than 50% of its fair share and if it has demands
greater than that. When there are many queues under a parent queue(with high
fair share),the child queue’s fair share becomes really low. As a result when
only few of these child queues have apps running,they reach their *tiny* fair
share quickly and preemption doesn’t happen even if other leaf
queues(non-sibling) are hogging the cluster.
This can be solved by dividing fair share of parent queue only to active
child queues.
Here is an example describing the problem and proposed solution:
root.lowPriorityQueue is a leaf queue with weight 2
root.HighPriorityQueue is parent queue with weight 8
root.HighPriorityQueue has 10 child leaf queues :
root.HighPriorityQueue.childQ(1..10)
Above config,results in root.HighPriorityQueue having 80% fair share
and each of its ten child queue would have 8% fair share. Preemption would
happen only if the child queue is 4% (0.5*8=4).
Lets say at the moment no apps are running in any of the
root.HighPriorityQueue.childQ(1..10) and few apps are running in
root.lowPriorityQueue which is taking up 95% of the cluster.
Up till this point,the behavior of FS is correct.
Now,lets say root.HighPriorityQueue.childQ1 got a big job which requires 30%
of the cluster. It would get only the available 5% in the cluster and
preemption wouldn't kick in since its above 4%(half fair share).This is bad
considering childQ1 is under a highPriority parent queue which has *80% fair
share*.
Until root.lowPriorityQueue starts relinquishing containers,we would see the
following allocation on the scheduler page:
*root.lowPriorityQueue = 95%*
*root.HighPriorityQueue.childQ1=5%*
This can be solved by distributing a parent’s fair share only to active
queues.
So in the example above,since childQ1 is the only active queue
under root.HighPriorityQueue, it would get all its parent’s fair share i.e.
80%.
This would cause preemption to reclaim the 30% needed by childQ1 from
root.lowPriorityQueue after fairSharePreemptionTimeout seconds.
Problem2 - Also note that similar situation can happen between
root.HighPriorityQueue.childQ1 and root.HighPriorityQueue.childQ2,if childQ2
hogs the cluster. childQ2 can take up 95% cluster
[jira] [Commented] (YARN-2378) Adding support for moving apps between queues in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090028#comment-14090028
]
Hadoop QA commented on YARN-2378:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660481/YARN-2378.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4554//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4554//console
This message is automatically generated.
Adding support for moving apps between queues in Capacity Scheduler
---
Key: YARN-2378
URL: https://issues.apache.org/jira/browse/YARN-2378
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Subramaniam Venkatraman Krishnan
Assignee: Subramaniam Venkatraman Krishnan
Labels: capacity-scheduler
Attachments: YARN-2378.patch, YARN-2378.patch
As discussed with [~leftnoteasy] and [~jianhe], we are breaking up YARN-1707
to smaller patches for manageability. This JIRA will address adding support
for moving apps between queues in Capacity Scheduler.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2352) FairScheduler: Collect metrics on duration of critical methods that affect performance
[
https://issues.apache.org/jira/browse/YARN-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090126#comment-14090126
]
Hadoop QA commented on YARN-2352:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660496/yarn-2352-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4555//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4555//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4555//console
This message is automatically generated.
FairScheduler: Collect metrics on duration of critical methods that affect
performance
--
Key: YARN-2352
URL: https://issues.apache.org/jira/browse/YARN-2352
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.4.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: fs-perf-metrics.png, yarn-2352-1.patch,
yarn-2352-2.patch, yarn-2352-2.patch, yarn-2352-3.patch, yarn-2352-4.patch,
yarn-2352-5.patch
We need more metrics for better visibility into FairScheduler performance. At
the least, we need to do this for (1) handle node events, (2) update, (3)
compute fairshares, (4) preemption.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2249) RM may receive container release request on AM resync before container is actually recovered
[
https://issues.apache.org/jira/browse/YARN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090217#comment-14090217
]
Hadoop QA commented on YARN-2249:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660510/YARN-2249.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4556//console
This message is automatically generated.
RM may receive container release request on AM resync before container is
actually recovered
Key: YARN-2249
URL: https://issues.apache.org/jira/browse/YARN-2249
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-2249.1.patch, YARN-2249.1.patch, YARN-2249.2.patch,
YARN-2249.2.patch
AM resync on RM restart will send outstanding container release requests back
to the new RM. In the meantime, NMs report the container statuses back to RM
to recover the containers. If RM receives the container release request
before the container is actually recovered in scheduler, the container won't
be released and the release request will be lost.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback
[
https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090302#comment-14090302
]
Hadoop QA commented on YARN-415:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660535/YARN-415.201408080204.txt
against trunk revision .
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4557//console
This message is automatically generated.
Capture memory utilization at the app-level for chargeback
--
Key: YARN-415
URL: https://issues.apache.org/jira/browse/YARN-415
Project: Hadoop YARN
Issue Type: New Feature
Components: resourcemanager
Affects Versions: 0.23.6
Reporter: Kendall Thrapp
Assignee: Andrey Klochkov
Attachments: YARN-415--n10.patch, YARN-415--n2.patch,
YARN-415--n3.patch, YARN-415--n4.patch, YARN-415--n5.patch,
YARN-415--n6.patch, YARN-415--n7.patch, YARN-415--n8.patch,
YARN-415--n9.patch, YARN-415.201405311749.txt, YARN-415.201406031616.txt,
YARN-415.201406262136.txt, YARN-415.201407042037.txt,
YARN-415.201407071542.txt, YARN-415.201407171553.txt,
YARN-415.201407172144.txt, YARN-415.201407232237.txt,
YARN-415.201407242148.txt, YARN-415.201407281816.txt,
YARN-415.201408062232.txt, YARN-415.201408080204.txt, YARN-415.patch
For the purpose of chargeback, I'd like to be able to compute the cost of an
application in terms of cluster resource usage. To start out, I'd like to
get the memory utilization of an application. The unit should be MB-seconds
or something similar and, from a chargeback perspective, the memory amount
should be the memory reserved for the application, as even if the app didn't
use all that memory, no one else was able to use it.
(reserved ram for container 1 * lifetime of container 1) + (reserved ram for
container 2 * lifetime of container 2) + ... + (reserved ram for container n
* lifetime of container n)
It'd be nice to have this at the app level instead of the job level because:
1. We'd still be able to get memory usage for jobs that crashed (and wouldn't
appear on the job history server).
2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).
This new metric should be available both through the RM UI and RM Web
Services REST API.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2397) RM web interface sometimes returns request is a replay error in secure mode
[
https://issues.apache.org/jira/browse/YARN-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090368#comment-14090368
]
Hadoop QA commented on YARN-2397:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660551/apache-yarn-2397.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4559//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4559//console
This message is automatically generated.
RM web interface sometimes returns request is a replay error in secure mode
---
Key: YARN-2397
URL: https://issues.apache.org/jira/browse/YARN-2397
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2397.0.patch
The RM web interface sometimes returns a request is a replay error if the
default kerberos http filter is enabled. This is because it uses the new
RMAuthenticationFilter in addition to the AuthenticationFilter. There is a
workaround to set
yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled to false.
This bug is to fix the code to use only the RMAuthenticationFilter and not
both.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2249) RM may receive container release request on AM resync before container is actually recovered
[
https://issues.apache.org/jira/browse/YARN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090406#comment-14090406
]
Hadoop QA commented on YARN-2249:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660542/YARN-2249.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4560//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4560//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4560//console
This message is automatically generated.
RM may receive container release request on AM resync before container is
actually recovered
Key: YARN-2249
URL: https://issues.apache.org/jira/browse/YARN-2249
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-2249.1.patch, YARN-2249.1.patch, YARN-2249.2.patch,
YARN-2249.2.patch, YARN-2249.3.patch
AM resync on RM restart will send outstanding container release requests back
to the new RM. In the meantime, NMs report the container statuses back to RM
to recover the containers. If RM receives the container release request
before the container is actually recovered in scheduler, the container won't
be released and the release request will be lost.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2397) RM web interface sometimes returns request is a replay error in secure mode
[
https://issues.apache.org/jira/browse/YARN-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090566#comment-14090566
]
Hadoop QA commented on YARN-2397:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660578/apache-yarn-2397.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4561//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4561//console
This message is automatically generated.
RM web interface sometimes returns request is a replay error in secure mode
---
Key: YARN-2397
URL: https://issues.apache.org/jira/browse/YARN-2397
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2397.0.patch, apache-yarn-2397.1.patch
The RM web interface sometimes returns a request is a replay error if the
default kerberos http filter is enabled. This is because it uses the new
RMAuthenticationFilter in addition to the AuthenticationFilter. There is a
workaround to set
yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled to false.
This bug is to fix the code to use only the RMAuthenticationFilter and not
both.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2138) Cleanup notifyDone* methods in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090876#comment-14090876
]
Hadoop QA commented on YARN-2138:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660614/YARN-2138.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4562//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4562//console
This message is automatically generated.
Cleanup notifyDone* methods in RMStateStore
---
Key: YARN-2138
URL: https://issues.apache.org/jira/browse/YARN-2138
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Varun Saxena
Attachments: YARN-2138.002.patch, YARN-2138.patch
The storedException passed into notifyDoneStoringApplication is always null.
Similarly for other notifyDone* methods. We can clean up these methods as
this control flow path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2396) RpcClientFactoryPBImpl.stopClient always throws due to missing close method
[
https://issues.apache.org/jira/browse/YARN-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091250#comment-14091250
]
Hadoop QA commented on YARN-2396:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660649/yarn2396.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4563//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4563//console
This message is automatically generated.
RpcClientFactoryPBImpl.stopClient always throws due to missing close method
---
Key: YARN-2396
URL: https://issues.apache.org/jira/browse/YARN-2396
Project: Hadoop YARN
Issue Type: Bug
Components: api
Affects Versions: 2.4.1
Reporter: Jason Lowe
Assignee: chang li
Attachments: yarn2396.patch
RpcClientFactoryPBImpl.stopClient will throw a YarnRuntimeException if the
protocol does not have a close method, despite the log message indicating it
is ignoring errors. It's interesting to note that none of the YARN protocol
classes currently have a close method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2026) Fair scheduler : Fair share for inactive queues causes unfair allocation in some scenarios
[
https://issues.apache.org/jira/browse/YARN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091516#comment-14091516
]
Hadoop QA commented on YARN-2026:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660716/YARN-2026-v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4564//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4564//console
This message is automatically generated.
Fair scheduler : Fair share for inactive queues causes unfair allocation in
some scenarios
--
Key: YARN-2026
URL: https://issues.apache.org/jira/browse/YARN-2026
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Labels: scheduler
Attachments: YARN-2026-v1.txt, YARN-2026-v2.txt, YARN-2026-v3.txt,
YARN-2026-v4.txt, YARN-2026-v5.txt
Problem1- While using hierarchical queues in fair scheduler,there are few
scenarios where we have seen a leaf queue with least fair share can take
majority of the cluster and starve a sibling parent queue which has greater
weight/fair share and preemption doesn’t kick in to reclaim resources.
The root cause seems to be that fair share of a parent queue is distributed
to all its children irrespective of whether its an active or an inactive(no
apps running) queue. Preemption based on fair share kicks in only if the
usage of a queue is less than 50% of its fair share and if it has demands
greater than that. When there are many queues under a parent queue(with high
fair share),the child queue’s fair share becomes really low. As a result when
only few of these child queues have apps running,they reach their *tiny* fair
share quickly and preemption doesn’t happen even if other leaf
queues(non-sibling) are hogging the cluster.
This can be solved by dividing fair share of parent queue only to active
child queues.
Here is an example describing the problem and proposed solution:
root.lowPriorityQueue is a leaf queue with weight 2
root.HighPriorityQueue is parent queue with weight 8
root.HighPriorityQueue has 10 child leaf queues :
root.HighPriorityQueue.childQ(1..10)
Above config,results in root.HighPriorityQueue having 80% fair share
and each of its ten child queue would have 8% fair share. Preemption would
happen only if the child queue is 4% (0.5*8=4).
Lets say at the moment no apps are running in any of the
root.HighPriorityQueue.childQ(1..10) and few apps are running in
root.lowPriorityQueue which is taking up 95% of the cluster.
Up till this point,the behavior of FS is correct.
Now,lets say root.HighPriorityQueue.childQ1 got a big job which requires 30%
of the cluster. It would get only the available 5% in the cluster and
preemption wouldn't kick in since its above 4%(half fair share).This is bad
considering childQ1 is under a highPriority parent queue which has *80% fair
share*.
Until root.lowPriorityQueue starts relinquishing containers,we would see the
following allocation on the scheduler page:
*root.lowPriorityQueue = 95%*
*root.HighPriorityQueue.childQ1=5%*
This can be solved by distributing a parent’s fair share only to active
queues.
So in the example above,since childQ1 is the only active queue
under root.HighPriorityQueue, it would get all its parent’s fair share i.e.
80%.
This would cause preemption to reclaim the 30% needed by childQ1 from
root.lowPriorityQueue after fairSharePreemptionTimeout seconds.
Problem2 - Also note that similar situation can happen between
root.HighPriorityQueue.childQ1 and root.HighPriorityQueue.childQ2,if childQ2
hogs the cluster. childQ2 can
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[
https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091527#comment-14091527
]
Hadoop QA commented on YARN-2277:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660715/YARN-2277-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4565//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4565//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4565//console
This message is automatically generated.
Add Cross-Origin support to the ATS REST API
Key: YARN-2277
URL: https://issues.apache.org/jira/browse/YARN-2277
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch,
YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch
As the Application Timeline Server is not provided with built-in UI, it may
make sense to enable JSONP or CORS Rest API capabilities to allow for remote
UI to access the data directly via javascript without cross side server
browser blocks coming into play.
Example client may be like
http://api.jquery.com/jQuery.getJSON/
This can alleviate the need to create a local proxy cache.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2302) Refactor TimelineWebServices
[
https://issues.apache.org/jira/browse/YARN-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091539#comment-14091539
]
Hadoop QA commented on YARN-2302:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660727/YARN-2302.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4566//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4566//console
This message is automatically generated.
Refactor TimelineWebServices
Key: YARN-2302
URL: https://issues.apache.org/jira/browse/YARN-2302
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: YARN-2302.1.patch, YARN-2302.2.patch
Now TimelineWebServices contains non-trivial logic to process the HTTP
requests, manipulate the data, check the access, and interact with the
timeline store.
I propose the move the data-oriented logic to a middle layer (so called
TimelineDataManager), and TimelineWebServices only processes the requests,
and call TimelineDataManager to complete the remaining tasks.
By doing this, we make the generic history module reuse TimelineDataManager
internally (YARN-2033), invoking the putting/getting methods directly.
Otherwise, we have to send the HTTP requests to TimelineWebServices to query
the generic history data, which is not an efficient way.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2249) RM may receive container release request on AM resync before container is actually recovered
[
https://issues.apache.org/jira/browse/YARN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091583#comment-14091583
]
Hadoop QA commented on YARN-2249:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660744/YARN-2249.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4567//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4567//console
This message is automatically generated.
RM may receive container release request on AM resync before container is
actually recovered
Key: YARN-2249
URL: https://issues.apache.org/jira/browse/YARN-2249
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-2249.1.patch, YARN-2249.1.patch, YARN-2249.2.patch,
YARN-2249.2.patch, YARN-2249.3.patch, YARN-2249.4.patch
AM resync on RM restart will send outstanding container release requests back
to the new RM. In the meantime, NMs report the container statuses back to RM
to recover the containers. If RM receives the container release request
before the container is actually recovered in scheduler, the container won't
be released and the release request will be lost.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2138) Cleanup notifyDone* methods in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091608#comment-14091608
]
Hadoop QA commented on YARN-2138:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660749/YARN-2138.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4568//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4568//console
This message is automatically generated.
Cleanup notifyDone* methods in RMStateStore
---
Key: YARN-2138
URL: https://issues.apache.org/jira/browse/YARN-2138
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Varun Saxena
Attachments: YARN-2138.002.patch, YARN-2138.003.patch, YARN-2138.patch
The storedException passed into notifyDoneStoringApplication is always null.
Similarly for other notifyDone* methods. We can clean up these methods as
this control flow path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2399) FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt
[
https://issues.apache.org/jira/browse/YARN-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091612#comment-14091612
]
Hadoop QA commented on YARN-2399:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660780/yarn-2399-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4569//console
This message is automatically generated.
FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt
Key: YARN-2399
URL: https://issues.apache.org/jira/browse/YARN-2399
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.5.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: yarn-2399-1.patch
FairScheduler has two data structures for an application, making the code
hard to track. We should merge these for better maintainability in the
long-term.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1954) Add waitFor to AMRMClient(Async)
[
https://issues.apache.org/jira/browse/YARN-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091621#comment-14091621
]
Hadoop QA commented on YARN-1954:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660786/YARN-1954.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4570//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4570//console
This message is automatically generated.
Add waitFor to AMRMClient(Async)
Key: YARN-1954
URL: https://issues.apache.org/jira/browse/YARN-1954
Project: Hadoop YARN
Issue Type: New Feature
Components: client
Affects Versions: 3.0.0, 2.4.0
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1954.1.patch, YARN-1954.2.patch, YARN-1954.3.patch,
YARN-1954.4.patch, YARN-1954.4.patch, YARN-1954.5.patch, YARN-1954.6.patch,
YARN-1954.7.patch
Recently, I saw some use cases of AMRMClient(Async). The painful thing is
that the main non-daemon thread has to sit in a dummy loop to prevent AM
process exiting before all the tasks are done, while unregistration is
triggered on a separate another daemon thread by callback methods (in
particular when using AMRMClientAsync). IMHO, it should be beneficial to add
a waitFor method to AMRMClient(Async) to block the AM until unregistration or
user supplied check point, such that users don't need to write the loop
themselves.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1954) Add waitFor to AMRMClient(Async)
[
https://issues.apache.org/jira/browse/YARN-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091839#comment-14091839
]
Hadoop QA commented on YARN-1954:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660811/YARN-1954.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4571//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4571//console
This message is automatically generated.
Add waitFor to AMRMClient(Async)
Key: YARN-1954
URL: https://issues.apache.org/jira/browse/YARN-1954
Project: Hadoop YARN
Issue Type: New Feature
Components: client
Affects Versions: 3.0.0, 2.4.0
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1954.1.patch, YARN-1954.2.patch, YARN-1954.3.patch,
YARN-1954.4.patch, YARN-1954.4.patch, YARN-1954.5.patch, YARN-1954.6.patch,
YARN-1954.7.patch, YARN-1954.8.patch
Recently, I saw some use cases of AMRMClient(Async). The painful thing is
that the main non-daemon thread has to sit in a dummy loop to prevent AM
process exiting before all the tasks are done, while unregistration is
triggered on a separate another daemon thread by callback methods (in
particular when using AMRMClientAsync). IMHO, it should be beneficial to add
a waitFor method to AMRMClient(Async) to block the AM until unregistration or
user supplied check point, such that users don't need to write the loop
themselves.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2400) TestAMRestart fails intermittently
[
https://issues.apache.org/jira/browse/YARN-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091874#comment-14091874
]
Hadoop QA commented on YARN-2400:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660816/YARN-2400.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4573//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4573//console
This message is automatically generated.
TestAMRestart fails intermittently
--
Key: YARN-2400
URL: https://issues.apache.org/jira/browse/YARN-2400
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-2400.1.patch
java.lang.AssertionError: AppAttempt state is not correct (timedout)
expected:ALLOCATED but was:SCHEDULED
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at
org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:417)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAM(MockRM.java:579)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAndRegisterAM(MockRM.java:586)
at
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testShouldNotCountFailureToMaxAttemptRetry(TestAMRestart.java:389)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-415) Capture memory utilization at the app-level for chargeback
[
https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091929#comment-14091929
]
Hadoop QA commented on YARN-415:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660822/YARN-415.201408092006.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.cli.TestYarnCLI
org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4574//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4574//console
This message is automatically generated.
Capture memory utilization at the app-level for chargeback
--
Key: YARN-415
URL: https://issues.apache.org/jira/browse/YARN-415
Project: Hadoop YARN
Issue Type: New Feature
Components: resourcemanager
Affects Versions: 0.23.6
Reporter: Kendall Thrapp
Assignee: Andrey Klochkov
Attachments: YARN-415--n10.patch, YARN-415--n2.patch,
YARN-415--n3.patch, YARN-415--n4.patch, YARN-415--n5.patch,
YARN-415--n6.patch, YARN-415--n7.patch, YARN-415--n8.patch,
YARN-415--n9.patch, YARN-415.201405311749.txt, YARN-415.201406031616.txt,
YARN-415.201406262136.txt, YARN-415.201407042037.txt,
YARN-415.201407071542.txt, YARN-415.201407171553.txt,
YARN-415.201407172144.txt, YARN-415.201407232237.txt,
YARN-415.201407242148.txt, YARN-415.201407281816.txt,
YARN-415.201408062232.txt, YARN-415.201408080204.txt,
YARN-415.201408092006.txt, YARN-415.patch
For the purpose of chargeback, I'd like to be able to compute the cost of an
application in terms of cluster resource usage. To start out, I'd like to
get the memory utilization of an application. The unit should be MB-seconds
or something similar and, from a chargeback perspective, the memory amount
should be the memory reserved for the application, as even if the app didn't
use all that memory, no one else was able to use it.
(reserved ram for container 1 * lifetime of container 1) + (reserved ram for
container 2 * lifetime of container 2) + ... + (reserved ram for container n
* lifetime of container n)
It'd be nice to have this at the app level instead of the job level because:
1. We'd still be able to get memory usage for jobs that crashed (and wouldn't
appear on the job history server).
2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).
This new metric should be available both through the RM UI and RM Web
Services REST API.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2400) TestAMRestart fails intermittently
[
https://issues.apache.org/jira/browse/YARN-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092005#comment-14092005
]
Hadoop QA commented on YARN-2400:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660834/YARN-2240.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4575//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4575//console
This message is automatically generated.
TestAMRestart fails intermittently
--
Key: YARN-2400
URL: https://issues.apache.org/jira/browse/YARN-2400
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Fix For: 2.6.0
Attachments: YARN-2240.2.patch, YARN-2400.1.patch
java.lang.AssertionError: AppAttempt state is not correct (timedout)
expected:ALLOCATED but was:SCHEDULED
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at
org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:417)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAM(MockRM.java:579)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAndRegisterAM(MockRM.java:586)
at
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testShouldNotCountFailureToMaxAttemptRetry(TestAMRestart.java:389)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2302) Refactor TimelineWebServices
[
https://issues.apache.org/jira/browse/YARN-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092010#comment-14092010
]
Hadoop QA commented on YARN-2302:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660836/YARN-2302.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4576//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4576//console
This message is automatically generated.
Refactor TimelineWebServices
Key: YARN-2302
URL: https://issues.apache.org/jira/browse/YARN-2302
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: YARN-2302.1.patch, YARN-2302.2.patch, YARN-2302.3.patch,
YARN-2302.4.patch
Now TimelineWebServices contains non-trivial logic to process the HTTP
requests, manipulate the data, check the access, and interact with the
timeline store.
I propose the move the data-oriented logic to a middle layer (so called
TimelineDataManager), and TimelineWebServices only processes the requests,
and call TimelineDataManager to complete the remaining tasks.
By doing this, we make the generic history module reuse TimelineDataManager
internally (YARN-2033), invoking the putting/getting methods directly.
Otherwise, we have to send the HTTP requests to TimelineWebServices to query
the generic history data, which is not an efficient way.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2277) Add Cross-Origin support to the ATS REST API
[
https://issues.apache.org/jira/browse/YARN-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092017#comment-14092017
]
Hadoop QA commented on YARN-2277:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660837/YARN-2277-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4577//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4577//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4577//console
This message is automatically generated.
Add Cross-Origin support to the ATS REST API
Key: YARN-2277
URL: https://issues.apache.org/jira/browse/YARN-2277
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Attachments: YARN-2277-CORS.patch, YARN-2277-JSONP.patch,
YARN-2277-v2.patch, YARN-2277-v3.patch, YARN-2277-v3.patch, YARN-2277-v4.patch
As the Application Timeline Server is not provided with built-in UI, it may
make sense to enable JSONP or CORS Rest API capabilities to allow for remote
UI to access the data directly via javascript without cross side server
browser blocks coming into play.
Example client may be like
http://api.jquery.com/jQuery.getJSON/
This can alleviate the need to create a local proxy cache.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1370) Fair scheduler to re-populate container allocation state
[
https://issues.apache.org/jira/browse/YARN-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092029#comment-14092029
]
Hadoop QA commented on YARN-1370:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660838/YARN-1370.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4578//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4578//console
This message is automatically generated.
Fair scheduler to re-populate container allocation state
Key: YARN-1370
URL: https://issues.apache.org/jira/browse/YARN-1370
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Anubhav Dhoot
Attachments: YARN-1370.001.patch
YARN-1367 and YARN-1368 enable the NM to tell the RM about currently running
containers and the RM will pass this information to the schedulers along with
the node information. The schedulers are currently already informed about
previously running apps when the app data is recovered from the store. The
scheduler is expected to be able to repopulate its allocation state from the
above 2 sources of information.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1915) ClientToAMTokenMasterKey should be provided to AM at launch time
[
https://issues.apache.org/jira/browse/YARN-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092196#comment-14092196
]
Hadoop QA commented on YARN-1915:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660870/YARN-1915.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4580//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4580//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4580//console
This message is automatically generated.
ClientToAMTokenMasterKey should be provided to AM at launch time
Key: YARN-1915
URL: https://issues.apache.org/jira/browse/YARN-1915
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.2.0
Reporter: Hitesh Shah
Assignee: Jason Lowe
Priority: Critical
Attachments: YARN-1915.patch
Currently, the AM receives the key as part of registration. This introduces a
race where a client can connect to the AM when the AM has not received the
key.
Current Flow:
1) AM needs to start the client listening service in order to get host:port
and send it to the RM as part of registration
2) RM gets the port info in register() and transitions the app to RUNNING.
Responds back with client secret to AM.
3) User asks RM for client token. Gets it and pings the AM. AM hasn't
received client secret from RM and so RPC itself rejects the request.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2308) NPE happened when RM restart after CapacityScheduler queue configuration changed
[
https://issues.apache.org/jira/browse/YARN-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092245#comment-14092245
]
Hadoop QA commented on YARN-2308:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660878/jira2308.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4581//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4581//console
This message is automatically generated.
NPE happened when RM restart after CapacityScheduler queue configuration
changed
-
Key: YARN-2308
URL: https://issues.apache.org/jira/browse/YARN-2308
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, scheduler
Affects Versions: 2.6.0
Reporter: Wangda Tan
Assignee: chang li
Priority: Critical
Attachments: jira2308.patch
I encountered a NPE when RM restart
{code}
2014-07-16 07:22:46,957 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
handling event type APP_ATTEMPT_ADDED to the scheduler
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:566)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:922)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:98)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:594)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:654)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:698)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:682)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:744)
{code}
And RM will be failed to restart.
This is caused by queue configuration changed, I removed some queues and
added new queues. So when RM restarts, it tries to recover history
applications, and when any of queues of these applications removed, NPE will
be raised.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2138) Cleanup notifyDone* methods in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092538#comment-14092538
]
Hadoop QA commented on YARN-2138:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660936/YARN-2138.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4584//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4584//console
This message is automatically generated.
Cleanup notifyDone* methods in RMStateStore
---
Key: YARN-2138
URL: https://issues.apache.org/jira/browse/YARN-2138
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Varun Saxena
Attachments: YARN-2138.002.patch, YARN-2138.003.patch,
YARN-2138.004.patch, YARN-2138.patch
The storedException passed into notifyDoneStoringApplication is always null.
Similarly for other notifyDone* methods. We can clean up these methods as
this control flow path is not used anymore.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
