[jira] [Commented] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321812#comment-14321812 ] Yongjun Zhang commented on YARN-3200: - This is a subtask that I will be working on, but it seems I can't assign the jira to myself. Anyone who has the privilege, appreciate your help to assign it to me. Thanks. Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned YARN-3200: -- Assignee: Brahma Reddy Battula Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Assignee: Brahma Reddy Battula Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2799) cleanup TestLogAggregationService based on the change in YARN-90
[
https://issues.apache.org/jira/browse/YARN-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321754#comment-14321754
]
zhihai xu commented on YARN-2799:
-
Hi [~djp],
Yes, you are right. that is a good catch. I uploaded a new patch
YARN-2799.002.patch which addressed your comment.
Please review it.
thanks
zhihai
cleanup TestLogAggregationService based on the change in YARN-90
Key: YARN-2799
URL: https://issues.apache.org/jira/browse/YARN-2799
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Attachments: YARN-2799.000.patch, YARN-2799.001.patch,
YARN-2799.002.patch
cleanup TestLogAggregationService based on the change in YARN-90.
The following code is added to setup in YARN-90,
{code}
dispatcher = createDispatcher();
appEventHandler = mock(EventHandler.class);
dispatcher.register(ApplicationEventType.class, appEventHandler);
{code}
In this case, we should remove all these code from each test function to
avoid duplicate code.
Same for dispatcher.stop() which is in tearDown,
we can remove dispatcher.stop() from from each test function also because it
will always be called from tearDown for each test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2799) cleanup TestLogAggregationService based on the change in YARN-90
[
https://issues.apache.org/jira/browse/YARN-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321762#comment-14321762
]
Hadoop QA commented on YARN-2799:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12698939/YARN-2799.002.patch
against trunk revision ef950ea.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6639//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6639//console
This message is automatically generated.
cleanup TestLogAggregationService based on the change in YARN-90
Key: YARN-2799
URL: https://issues.apache.org/jira/browse/YARN-2799
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Attachments: YARN-2799.000.patch, YARN-2799.001.patch,
YARN-2799.002.patch
cleanup TestLogAggregationService based on the change in YARN-90.
The following code is added to setup in YARN-90,
{code}
dispatcher = createDispatcher();
appEventHandler = mock(EventHandler.class);
dispatcher.register(ApplicationEventType.class, appEventHandler);
{code}
In this case, we should remove all these code from each test function to
avoid duplicate code.
Same for dispatcher.stop() which is in tearDown,
we can remove dispatcher.stop() from from each test function also because it
will always be called from tearDown for each test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3199) Fair Scheduler documentation improvements
[
https://issues.apache.org/jira/browse/YARN-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321798#comment-14321798
]
Rohit Agarwal commented on YARN-3199:
-
[~aw], [~gururaj] Please also pick this up when converting the docs from apt to
markdown as part of YARN-3168.
Fair Scheduler documentation improvements
-
Key: YARN-3199
URL: https://issues.apache.org/jira/browse/YARN-3199
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.6.0
Reporter: Rohit Agarwal
Priority: Minor
Labels: documentation
Attachments: YARN-3199.patch
{{yarn.scheduler.increment-allocation-mb}} and
{{yarn.scheduler.increment-allocation-vcores}} are not documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3199) Fair Scheduler documentation improvements
[
https://issues.apache.org/jira/browse/YARN-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321805#comment-14321805
]
Hadoop QA commented on YARN-3199:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12698946/YARN-3199.patch
against trunk revision 3338f6d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6640//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6640//console
This message is automatically generated.
Fair Scheduler documentation improvements
-
Key: YARN-3199
URL: https://issues.apache.org/jira/browse/YARN-3199
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.6.0
Reporter: Rohit Agarwal
Priority: Minor
Labels: documentation
Attachments: YARN-3199.patch
{{yarn.scheduler.increment-allocation-mb}} and
{{yarn.scheduler.increment-allocation-vcores}} are not documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3199) Fair Scheduler documentation improvements
Rohit Agarwal created YARN-3199:
---
Summary: Fair Scheduler documentation improvements
Key: YARN-3199
URL: https://issues.apache.org/jira/browse/YARN-3199
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.6.0
Reporter: Rohit Agarwal
Priority: Minor
{{yarn.scheduler.increment-allocation-mb}} and
{{yarn.scheduler.increment-allocation-vcores}} are not documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3199) Fair Scheduler documentation improvements
[
https://issues.apache.org/jira/browse/YARN-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Agarwal updated YARN-3199:
Attachment: YARN-3199.patch
Fair Scheduler documentation improvements
-
Key: YARN-3199
URL: https://issues.apache.org/jira/browse/YARN-3199
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.6.0
Reporter: Rohit Agarwal
Priority: Minor
Labels: documentation
Attachments: YARN-3199.patch
{{yarn.scheduler.increment-allocation-mb}} and
{{yarn.scheduler.increment-allocation-vcores}} are not documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2749) Some testcases from TestLogAggregationService fails in trunk
[ https://issues.apache.org/jira/browse/YARN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322007#comment-14322007 ] Hudson commented on YARN-2749: -- FAILURE: Integrated in Hadoop-trunk-Commit #7117 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7117/]) YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. (Contributed by Xuan Gong) (junping_du: rev ab0b958a522d502426b91b6e4ab6dd29caccc372) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java Some testcases from TestLogAggregationService fails in trunk Key: YARN-2749 URL: https://issues.apache.org/jira/browse/YARN-2749 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Fix For: 2.7.0 Attachments: YARN-2749.1.patch, YARN-2749.2.patch, YARN-2749.2.patch Some testcases from TestLogAggregationService fails in trunk. Those can be reproduced in centos Stack Trace: java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1362) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention(TestLogAggregationService.java:1290) Stack Trace: java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationService(TestLogAggregationService.java:1362) at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLogAggregationServiceWithRetention(TestLogAggregationService.java:1290) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322064#comment-14322064 ] Tsuyoshi OZAWA commented on YARN-3200: -- s/head/ahead/ Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Assignee: Yongjun Zhang Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322063#comment-14322063 ] Tsuyoshi OZAWA commented on YARN-3200: -- [~yzhangal] done. Please go head. Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Assignee: Yongjun Zhang Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322070#comment-14322070 ] Yongjun Zhang commented on YARN-3200: - Thanks a lot [~brahmareddy] and [~ozawa]! I will submit patch after HADOOP-11597 is resolved because of the dependency. Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Assignee: Yongjun Zhang Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-3200: - Assignee: Brahma Reddy Battula (was: Tsuyoshi OZAWA) Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Assignee: Brahma Reddy Battula Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-3200: - Assignee: (was: Brahma Reddy Battula) Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-3200: - Assignee: Yongjun Zhang Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Assignee: Yongjun Zhang Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3200) Factor OSType out from Shell: changes in YARN
Yongjun Zhang created YARN-3200: --- Summary: Factor OSType out from Shell: changes in YARN Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2799) cleanup TestLogAggregationService based on the change in YARN-90
[
https://issues.apache.org/jira/browse/YARN-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-2799:
Attachment: YARN-2799.002.patch
cleanup TestLogAggregationService based on the change in YARN-90
Key: YARN-2799
URL: https://issues.apache.org/jira/browse/YARN-2799
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Attachments: YARN-2799.000.patch, YARN-2799.001.patch,
YARN-2799.002.patch
cleanup TestLogAggregationService based on the change in YARN-90.
The following code is added to setup in YARN-90,
{code}
dispatcher = createDispatcher();
appEventHandler = mock(EventHandler.class);
dispatcher.register(ApplicationEventType.class, appEventHandler);
{code}
In this case, we should remove all these code from each test function to
avoid duplicate code.
Same for dispatcher.stop() which is in tearDown,
we can remove dispatcher.stop() from from each test function also because it
will always be called from tearDown for each test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3194) After NM restart,completed containers are not released which are sent during NM registration
[
https://issues.apache.org/jira/browse/YARN-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322277#comment-14322277
]
Rohith commented on YARN-3194:
--
bq. it's removing the old node and adding the newly connected node. RM is also
not restarted.
{{RMNodeImpl#ReconnectNodeTransition#.transition}} does not remove old node if
any applications are running. In the below code, if noRunningApps is false then
Node is not removed. Instead just handling running applications.
{code}
public void transition(RMNodeImpl rmNode, RMNodeEvent event) {
RMNodeReconnectEvent reconnectEvent = (RMNodeReconnectEvent) event;
RMNode newNode = reconnectEvent.getReconnectedNode();
rmNode.nodeManagerVersion = newNode.getNodeManagerVersion();
ListApplicationId runningApps = reconnectEvent.getRunningApplications();
boolean noRunningApps =
(runningApps == null) || (runningApps.size() == 0);
// No application running on the node, so send node-removal event with
// cleaning up old container info.
if (noRunningApps) {
// Remove the node from scheduler
// Add node to the scheduler
} else {
rmNode.httpPort = newNode.getHttpPort();
rmNode.httpAddress = newNode.getHttpAddress();
rmNode.totalCapability = newNode.getTotalCapability();
// Reset heartbeat ID since node just restarted.
rmNode.getLastNodeHeartBeatResponse().setResponseId(0);
}
// Handles running app on this node
// resource update to schedule code
}
}
{code}
After NM restart,completed containers are not released which are sent during
NM registration
Key: YARN-3194
URL: https://issues.apache.org/jira/browse/YARN-3194
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Environment: NM restart is enabled
Reporter: Rohith
Assignee: Rohith
On NM restart ,NM sends all the outstanding NMContainerStatus to RM. But RM
process only ContainerState.RUNNING. If container is completed when NM was
down then those containers resources wont be release which result in
applications to hang.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3194) After NM restart,completed containers are not released which are sent during NM registration
[ https://issues.apache.org/jira/browse/YARN-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322154#comment-14322154 ] Jian He commented on YARN-3194: --- bq. I have one doubt on this method ResourceTrackerService#handleNMContainerStatus This is legacy code for non-work-preserving restart. we could remove that. Just disregard this method. bq. NM RESTART is Enabled – Problem is here For node_reconnect event, it's removing the old node and adding the newly connected node. RM is also not restarted. I don't think we need to handle the RMNodeReconnectEvent After NM restart,completed containers are not released which are sent during NM registration Key: YARN-3194 URL: https://issues.apache.org/jira/browse/YARN-3194 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Environment: NM restart is enabled Reporter: Rohith Assignee: Rohith On NM restart ,NM sends all the outstanding NMContainerStatus to RM. But RM process only ContainerState.RUNNING. If container is completed when NM was down then those containers resources wont be release which result in applications to hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3194) After NM restart,completed containers are not released which are sent during NM registration
[
https://issues.apache.org/jira/browse/YARN-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322144#comment-14322144
]
Rohith commented on YARN-3194:
--
And Not related specific to this jira, I have one doubt on this method
{{ResourceTrackerService#handleNMContainerStatus}} is sending
container_finished event only to master container. Why other containers are not
considered? I think it is made intentionally for optimization so if
container_finished event would release other containers resources. Is this is
the reason?
After NM restart,completed containers are not released which are sent during
NM registration
Key: YARN-3194
URL: https://issues.apache.org/jira/browse/YARN-3194
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Environment: NM restart is enabled
Reporter: Rohith
Assignee: Rohith
On NM restart ,NM sends all the outstanding NMContainerStatus to RM. But RM
process only ContainerState.RUNNING. If container is completed when NM was
down then those containers resources wont be release which result in
applications to hang.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3194) After NM restart,completed containers are not released which are sent during NM registration
[
https://issues.apache.org/jira/browse/YARN-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322143#comment-14322143
]
Rohith commented on YARN-3194:
--
Thanks [~jianhe] for pointing me out container recovery flow!! Issue priority
can decided later,not a problem.
I had deeper look about NM registration flow. There are 2 scenario's can occur
# Node added event : Again here 2 scenario's can occur
## New node is registering with different ip:port -- NOT A PROBLEM
## Old node is re-registering because of RESYNC command from RM when RM restart
-- NOT A PROBLEM
# Node reconnected event :
## Existing node is re-registering i.e RM treat it as reconnecting node when RM
is not restarted
### NM RESTART NOT Enabled -- NOT A PROBLEM
### NM RESTART is Enabled -- {color:red}Problem is here{color}
When Node is reconnected and applications are running in that node,
NMContainerStatus are ignored. I think RMNodeReconnectEvent should consider
NMContainerStatus and process it.
After NM restart,completed containers are not released which are sent during
NM registration
Key: YARN-3194
URL: https://issues.apache.org/jira/browse/YARN-3194
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Environment: NM restart is enabled
Reporter: Rohith
Assignee: Rohith
On NM restart ,NM sends all the outstanding NMContainerStatus to RM. But RM
process only ContainerState.RUNNING. If container is completed when NM was
down then those containers resources wont be release which result in
applications to hang.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3110) Faulty link and state in ApplicationHistory when aplication is in unassigned state
[
https://issues.apache.org/jira/browse/YARN-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321775#comment-14321775
]
Naganarasimha G R commented on YARN-3110:
-
Hi [~ozawa], Can you please review this patch too ?
Faulty link and state in ApplicationHistory when aplication is in unassigned
state
--
Key: YARN-3110
URL: https://issues.apache.org/jira/browse/YARN-3110
Project: Hadoop YARN
Issue Type: Bug
Components: applications, timelineserver
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Naganarasimha G R
Priority: Minor
Attachments: YARN-3110.20150209-1.patch
Application state and History link wrong when Application is in unassigned
state
1.Configure capacity schedular with queue size as 1 also max Absolute Max
Capacity: 10.0%
(Current application state is Accepted and Unassigned from resource manager
side)
2.Submit application to queue and check the state and link in Application
history
State= null and History link shown as N/A in applicationhistory page
Kill the same application . In timeline server logs the below is show when
selecting application link.
{quote}
2015-01-29 15:39:50,956 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
read the AM container of the application attempt
appattempt_1422467063659_0007_01.
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainer(ApplicationHistoryManagerOnTimelineStore.java:162)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAMContainer(ApplicationHistoryManagerOnTimelineStore.java:184)
at
org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:160)
at
org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:156)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSController.app(AHSController.java:38)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
[jira] [Assigned] (YARN-3200) Factor OSType out from Shell: changes in YARN
[ https://issues.apache.org/jira/browse/YARN-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned YARN-3200: -- Assignee: Tsuyoshi OZAWA (was: Brahma Reddy Battula) Hi [~ozawa] can you please assign to [~yzhangal].. Factor OSType out from Shell: changes in YARN - Key: YARN-3200 URL: https://issues.apache.org/jira/browse/YARN-3200 Project: Hadoop YARN Issue Type: Improvement Reporter: Yongjun Zhang Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3201) add args for DistributedShell to specify a image for tasks that will run on docker
zhangwei created YARN-3201: -- Summary: add args for DistributedShell to specify a image for tasks that will run on docker Key: YARN-3201 URL: https://issues.apache.org/jira/browse/YARN-3201 Project: Hadoop YARN Issue Type: Wish Components: applications/distributed-shell Reporter: zhangwei It's very useful to execute a script on docker to do some test, but the distributedshell has no args to set the image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-3195) [YARN]Missing uniformity In Yarn Queue CLI command
[
https://issues.apache.org/jira/browse/YARN-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K reassigned YARN-3195:
---
Assignee: Jagadesh Kiran N
[YARN]Missing uniformity In Yarn Queue CLI command
---
Key: YARN-3195
URL: https://issues.apache.org/jira/browse/YARN-3195
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.6.0
Environment: SUSE Linux SP3
Reporter: Jagadesh Kiran N
Assignee: Jagadesh Kiran N
Priority: Minor
Fix For: 2.7.0
Attachments: Helptobe removed in Queue.png, YARN-3195.patch,
YARN-3195.patch
Help is generic command should not be placed here because of this uniformity
is missing compared to other commands.Remove -help command inside ./yarn
queue as uniformity with respect to other commands
{code}
SO486LDPag65:/home/OpenSource/HA/install/hadoop/resourcemanager/bin # ./yarn
queue -help
15/02/13 19:30:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
usage: queue
* -help Displays help for all commands.*
-status Queue Name List queue information about given queue.
SO486LDPag65:/home/OpenSource/HA/install/hadoop/resourcemanager/bin # ./yarn
queue
15/02/13 19:33:14 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Invalid Command Usage :
usage: queue
* -help Displays help for all commands.*
-status Queue Name List queue information about given queue.
{code}
* -help Displays help for all commands.*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3025) Provide API for retrieving blacklisted nodes
[
https://issues.apache.org/jira/browse/YARN-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated YARN-3025:
-
Attachment: yarn-3025-v2.txt
Patch v2 does what was proposed above.
Next step is to add getter for black listed nodes in ApplicationMasterProtocol
Provide API for retrieving blacklisted nodes
Key: YARN-3025
URL: https://issues.apache.org/jira/browse/YARN-3025
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: yarn-3025-v1.txt, yarn-3025-v2.txt
We have the following method which updates blacklist:
{code}
public synchronized void updateBlacklist(ListString blacklistAdditions,
ListString blacklistRemovals) {
{code}
Upon AM failover, there should be an API which returns the blacklisted nodes
so that the new AM can make consistent decisions.
The new API can be:
{code}
public synchronized ListString getBlacklistedNodes()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3076) YarnClient implementation to retrieve label to node mapping
[ https://issues.apache.org/jira/browse/YARN-3076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322395#comment-14322395 ] Varun Saxena commented on YARN-3076: [~leftnoteasy], ok will update the cases and upload a new patch today. YarnClient implementation to retrieve label to node mapping --- Key: YARN-3076 URL: https://issues.apache.org/jira/browse/YARN-3076 Project: Hadoop YARN Issue Type: Sub-task Components: client Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Attachments: YARN-3076.001.patch, YARN-3076.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3168) Convert site documentation from apt to markdown
[ https://issues.apache.org/jira/browse/YARN-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322414#comment-14322414 ] Gururaj Shetty commented on YARN-3168: -- Thanks Allen Wittenauer for the patch. Do you convert the .apt to markdown files manually or do you use any tool to do the same? Convert site documentation from apt to markdown --- Key: YARN-3168 URL: https://issues.apache.org/jira/browse/YARN-3168 Project: Hadoop YARN Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Gururaj Shetty Attachments: YARN-3168-00.patch YARN analog to HADOOP-11495 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-44) NM webapp returns 500 error when container log directory is missing
[ https://issues.apache.org/jira/browse/YARN-44?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322453#comment-14322453 ] Devaraj K commented on YARN-44: --- It doesn't seem to be a problem anymore with the changes happened. [~jlowe], can you confirm to close this issue? NM webapp returns 500 error when container log directory is missing --- Key: YARN-44 URL: https://issues.apache.org/jira/browse/YARN-44 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3 Reporter: Jason Lowe When log aggregation fails to create the application log directory it kills the application. This apparently can leave the container with no log directory, as the user encounters a 500 error when trying to access the AM logs via the webapp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
