[jira] [Updated] (YARN-2844) WebAppProxyServlet cannot handle urls which contain encoded characters
[
https://issues.apache.org/jira/browse/YARN-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu updated YARN-2844:
---
Attachment: YARN-2844.patch
> WebAppProxyServlet cannot handle urls which contain encoded characters
> --
>
> Key: YARN-2844
> URL: https://issues.apache.org/jira/browse/YARN-2844
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: Shixiong Zhu
>Priority: Minor
> Attachments: YARN-2844.patch
>
>
> WebAppProxyServlet has a bug about the URL encode/decode. This was found when
> running Spark on Yarn.
> When a user accesses
> "http://example.com:8088/proxy/application_1415344371838_0006/executors/threadDump/?executorId=%3Cdriver%3E";,
> WebAppProxyServlet will require
> "http://example.com:36429/executors/threadDump/?executorId=%25253Cdriver%25253E";.
> But Spark Web Server expects
> "http://example.com:36429/executors/threadDump/?executorId=%3Cdriver%3E";.
> Here are problems I found in WebAppProxyServlet.
> 1. java.net.URI.toString returns an encoded url string. So the following code
> in WebAppProxyServlet should use `true` instead of `false`.
> {code:java}
> org.apache.commons.httpclient.URI uri =
> new org.apache.commons.httpclient.URI(link.toString(), false);
> {code}
> 2.
> [HttpServletRequest.getPathInfo()|https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html#getPathInfo()]
> will returns a decoded string. Therefore, if the link is
> http://example.com:8088/proxy/application_1415344371838_0006/John%2FHunter,
> pathInfo will be "/application_1415344371838_0006/John/Hunter". Then the URI
> created in WebAppProxyServlet will be something like ".../John/Hunter", but
> the correct link should be ".../John%2FHunber". We can use
> [HttpServletRequest.getRequestURI()|https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html#getRequestURI()]
> to get the raw path.
> {code:java}
> final String pathInfo = req.getPathInfo();
> {code}
> 3. Use wrong URI constructor. [URI(String scheme, String authority, String
> path, String query, String
> fragment)|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#URI(java.lang.String,%20java.lang.String,%20java.lang.String,%20java.lang.String,%20java.lang.String)]
> will encode the path and query which have already been encoded. Should use
> [URI(String
> str)|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#URI(java.lang.String)]
> directly since the url has already been encoded.
> {code:java}
> URI toFetch = new URI(trackingUri.getScheme(),
> trackingUri.getAuthority(),
> StringHelper.ujoin(trackingUri.getPath(), rest),
> req.getQueryString(),
> null);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2844) WebAppProxyServlet cannot handle urls which contain encoded characters
Shixiong Zhu created YARN-2844:
--
Summary: WebAppProxyServlet cannot handle urls which contain
encoded characters
Key: YARN-2844
URL: https://issues.apache.org/jira/browse/YARN-2844
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Reporter: Shixiong Zhu
Priority: Minor
WebAppProxyServlet has a bug about the URL encode/decode. This was found when
running Spark on Yarn.
When a user accesses
"http://example.com:8088/proxy/application_1415344371838_0006/executors/threadDump/?executorId=%3Cdriver%3E";,
WebAppProxyServlet will require
"http://example.com:36429/executors/threadDump/?executorId=%25253Cdriver%25253E";.
But Spark Web Server expects
"http://example.com:36429/executors/threadDump/?executorId=%3Cdriver%3E";.
Here are problems I found in WebAppProxyServlet.
1. java.net.URI.toString returns an encoded url string. So the following code
in WebAppProxyServlet should use `true` instead of `false`.
{code:java}
org.apache.commons.httpclient.URI uri =
new org.apache.commons.httpclient.URI(link.toString(), false);
{code}
2.
[HttpServletRequest.getPathInfo()|https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html#getPathInfo()]
will returns a decoded string. Therefore, if the link is
http://example.com:8088/proxy/application_1415344371838_0006/John%2FHunter,
pathInfo will be "/application_1415344371838_0006/John/Hunter". Then the URI
created in WebAppProxyServlet will be something like ".../John/Hunter", but the
correct link should be ".../John%2FHunber". We can use
[HttpServletRequest.getRequestURI()|https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html#getRequestURI()]
to get the raw path.
{code:java}
final String pathInfo = req.getPathInfo();
{code}
3. Use wrong URI constructor. [URI(String scheme, String authority, String
path, String query, String
fragment)|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#URI(java.lang.String,%20java.lang.String,%20java.lang.String,%20java.lang.String,%20java.lang.String)]
will encode the path and query which have already been encoded. Should use
[URI(String
str)|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#URI(java.lang.String)]
directly since the url has already been encoded.
{code:java}
URI toFetch = new URI(trackingUri.getScheme(),
trackingUri.getAuthority(),
StringHelper.ujoin(trackingUri.getPath(), rest), req.getQueryString(),
null);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[
https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205994#comment-14205994
]
Hadoop QA commented on YARN-2843:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680740/YARN-2843-2.patch
against trunk revision 58e9bf4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5812//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5812//console
This message is automatically generated.
> NodeLabels manager should trim all inputs for hosts and labels
> --
>
> Key: YARN-2843
> URL: https://issues.apache.org/jira/browse/YARN-2843
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sushmitha Sreenivasan
>Assignee: Wangda Tan
> Attachments: YARN-2843-1.patch, YARN-2843-2.patch
>
>
> NodeLabels manager should trim all inputs for hosts and labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2843: -- Priority: Major (was: Blocker) Target Version/s: 2.7.0 (was: 2.6.0) Looking at the patch for review - not a blocker. > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2843: - Attachment: YARN-2843-2.patch Updated patch to fix test failures > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-2843-1.patch, YARN-2843-2.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[
https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205953#comment-14205953
]
Hadoop QA commented on YARN-2843:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680728/YARN-2843-1.patch
against trunk revision 58e9bf4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.yarn.client.cli.TestRMAdminCLI
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5811//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5811//console
This message is automatically generated.
> NodeLabels manager should trim all inputs for hosts and labels
> --
>
> Key: YARN-2843
> URL: https://issues.apache.org/jira/browse/YARN-2843
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sushmitha Sreenivasan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-2843-1.patch
>
>
> NodeLabels manager should trim all inputs for hosts and labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2843: - Attachment: YARN-2843-1.patch Attached fix this issue. > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha Sreenivasan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-2843-1.patch > > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205850#comment-14205850 ] Hudson commented on YARN-2841: -- FAILURE: Integrated in Hadoop-trunk-Commit #6509 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6509/]) YARN-2841: Correct fix version from branch-2.6 to branch-2.7 in the (xgong: rev 58e9bf4b908e0b21309006eba49899b092f38071) * hadoop-yarn-project/CHANGES.txt > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.7.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2842) TestApplicationClientProtocolOnHA fails against Java 8
[
https://issues.apache.org/jira/browse/YARN-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved YARN-2842.
--
Resolution: Duplicate
Should have searched :-)
> TestApplicationClientProtocolOnHA fails against Java 8
> --
>
> Key: YARN-2842
> URL: https://issues.apache.org/jira/browse/YARN-2842
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/1/consoleFull :
> {code}
> testGetNewApplicationOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
> Time elapsed: 8.959 sec <<< ERROR!
> java.net.ConnectException: Call From asf908.gq1.ygridcore.net/67.195.81.152
> to asf908.gq1.ygridcore.net:28032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy17.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:217)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy18.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:206)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:214)
> at
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetNewApplicationOnHA(TestApplicationClientProtocolOnHA.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205842#comment-14205842 ] Hudson commented on YARN-2841: -- FAILURE: Integrated in Hadoop-trunk-Commit #6508 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6508/]) YARN-2841. RMProxy should retry EOFException. Contributed by Jian He (xgong: rev 5c9a51f140ba76ddb25580aeb288db25e3f9653f) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.7.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205840#comment-14205840 ] Xuan Gong commented on YARN-2841: - Committed into trunk/branch-2. Thanks, Jian > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.7.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2841: Priority: Critical (was: Blocker) > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.7.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2841: Fix Version/s: 2.7.0 > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Critical > Fix For: 2.7.0 > > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[
https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205832#comment-14205832
]
Hadoop QA commented on YARN-2841:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680710/YARN-2841.1.patch
against trunk revision 86bf8c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5810//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5810//console
This message is automatically generated.
> RMProxy should retry EOFException
> --
>
> Key: YARN-2841
> URL: https://issues.apache.org/jira/browse/YARN-2841
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Jian He
>Assignee: Jian He
>Priority: Blocker
> Attachments: YARN-2841.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned YARN-2843: Assignee: Wangda Tan > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha >Assignee: Wangda Tan >Priority: Blocker > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
[ https://issues.apache.org/jira/browse/YARN-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205824#comment-14205824 ] Wangda Tan commented on YARN-2843: -- Thanks [~ssreenivasan] reporting this issue. Taking over, working on it. > NodeLabels manager should trim all inputs for hosts and labels > -- > > Key: YARN-2843 > URL: https://issues.apache.org/jira/browse/YARN-2843 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sushmitha >Assignee: Wangda Tan >Priority: Blocker > > NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2843) NodeLabels manager should trim all inputs for hosts and labels
Sushmitha created YARN-2843: --- Summary: NodeLabels manager should trim all inputs for hosts and labels Key: YARN-2843 URL: https://issues.apache.org/jira/browse/YARN-2843 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sushmitha Priority: Blocker NodeLabels manager should trim all inputs for hosts and labels -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205819#comment-14205819 ] Xuan Gong commented on YARN-2841: - +1. Looks good. Will commit when Jenkins give +1 > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Blocker > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2842) TestApplicationClientProtocolOnHA fails against Java 8
[
https://issues.apache.org/jira/browse/YARN-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205800#comment-14205800
]
Wangda Tan commented on YARN-2842:
--
This is not Java 8 specific issue, it is tracked by YARN-2710
> TestApplicationClientProtocolOnHA fails against Java 8
> --
>
> Key: YARN-2842
> URL: https://issues.apache.org/jira/browse/YARN-2842
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/1/consoleFull :
> {code}
> testGetNewApplicationOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
> Time elapsed: 8.959 sec <<< ERROR!
> java.net.ConnectException: Call From asf908.gq1.ygridcore.net/67.195.81.152
> to asf908.gq1.ygridcore.net:28032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy17.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:217)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy18.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:206)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:214)
> at
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetNewApplicationOnHA(TestApplicationClientProtocolOnHA.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2842) TestApplicationClientProtocolOnHA fails against Java 8
Ted Yu created YARN-2842:
Summary: TestApplicationClientProtocolOnHA fails against Java 8
Key: YARN-2842
URL: https://issues.apache.org/jira/browse/YARN-2842
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
>From https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/1/consoleFull :
{code}
testGetNewApplicationOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
Time elapsed: 8.959 sec <<< ERROR!
java.net.ConnectException: Call From asf908.gq1.ygridcore.net/67.195.81.152 to
asf908.gq1.ygridcore.net:28032 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy17.getNewApplication(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy18.getNewApplication(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:206)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:214)
at
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetNewApplicationOnHA(TestApplicationClientProtocolOnHA.java:76)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2791) Add Disk as a resource for scheduling
[ https://issues.apache.org/jira/browse/YARN-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Daingade updated YARN-2791: --- Attachment: DiskDriveAsResourceInYARN.pdf Uploading design doc for Disk as a resource > Add Disk as a resource for scheduling > - > > Key: YARN-2791 > URL: https://issues.apache.org/jira/browse/YARN-2791 > Project: Hadoop YARN > Issue Type: New Feature > Components: scheduler >Affects Versions: 2.5.1 >Reporter: Swapnil Daingade >Assignee: Yuliya Feldman > Attachments: DiskDriveAsResourceInYARN.pdf > > > Currently, the number of disks present on a node is not considered a factor > while scheduling containers on that node. Having large amount of memory on a > node can lead to high number of containers being launched on that node, all > of which compete for I/O bandwidth. This multiplexing of I/O across > containers can lead to slower overall progress and sub-optimal resource > utilization as containers starved for I/O bandwidth hold on to other > resources like cpu and memory. This problem can be solved by considering disk > as a resource and including it in deciding how many containers can be > concurrently run on a node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205768#comment-14205768 ] Zhijie Shen commented on YARN-2837: --- Test the patch on a single node secure cluster: 1. Start and restart the timeline server, and the DT information is recovered properly. 2. The DT generated in before the timeline server can be renewed properly afterwards. Some other issues I've observed while doing test: At the very early seconds the http server is started, the MR job, which tries to emit the timeline data, gets a number of 404 error. I guess the server is not fully ready before it taking the incoming requests. > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-2837.1.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2607) TestDistributedShell fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205704#comment-14205704
]
Wangda Tan commented on YARN-2607:
--
Hmm.. it seems lots of distributed related tests are timeout in your machine.
I've tried several times locally but cannot reproduce. I cannot get any idea
from the log you uploaded.
I think it should be a environment-specific problem, could you please file a
separated ticket to track the timeout issue? Let's see if anybody else suffered
this.
Thanks,
> TestDistributedShell fails in trunk
> ---
>
> Key: YARN-2607
> URL: https://issues.apache.org/jira/browse/YARN-2607
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Wangda Tan
> Fix For: 2.6.0
>
> Attachments: YARN-2607-1.patch, YARN-2607-2.patch, YARN-2607-3.patch,
> test.log
>
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk/691/console :
> {code}
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.641 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
> On Linux, I got the following locally:
> {code}
> testDSAttemptFailuresValidityIntervalFailed(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 64.715 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalFailed(TestDistributedShell.java:384)
> testDSAttemptFailuresValidityIntervalSucess(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 115.842 sec <<< ERROR!
> java.lang.Exception: test timed out after 9 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:680)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:661)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalSucess(TestDistributedShell.java:342)
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.633 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205689#comment-14205689
]
Wangda Tan commented on YARN-2495:
--
bq. I have already added the below code in the NodeHeartbeatRequest interface
for this(or i dint get your comment correctly, please elaborate):
I think I've misreading your patch, your patch should have done what we
discusse.
bq. This if check will be there in the overloaded case to identify which
overloaded method to choose right ? i am not able to see any benefit from this.
The benefit are
1) You don't have to update test cases for that
2) The semanic are clear, create a register request with label or not.
bq. NM_NODE_LABELS_FETCH_INTERVAL_MS & DEFAULT_NM_NODE_LABELS_FETCH_INTERVAL_MS
is used in both the script based and also config based hence had not moved it
under config-based ?
I suggest to have different option for script-based/config-based, even if we
can combine them together.
bq. if reuse then can delete NM_NODE_LABELS_FROM_CONFIG, as its not used.
IIUC, NM_NODE_LABELS_FROM_CONFIG is a list of labels, even if we want to
separate the two properties, we cannot remove NM_NODE_LABELS_FROM_CONFIG,
correct?
(But I still suggest you to change it to: NM_NODE_LABELS_PREFIX +
"config-based" + ".node-labels".)
bq. These i wanted to discuss with you , based on your patch changes for labels
had figured out this class but as i was modifying the existing PB class awas
wondering why these existing PB's are not added here.
I think it's better to leverage existing utility class instead of implement
your own. For example, you have set values but not check them, which is
incorrect, but using utility class can avoid such problem. Even if you added
new fields, tests will cover them without any changes:
{code}
-ApplicationId.newInstance(1234L, 2)));
+ApplicationId.newInstance(1234L, 2)),new HashSet());
{code}
Will include review of unit tests after you uploaded new patch.
> Allow admin specify labels from each NM (Distributed configuration)
> ---
>
> Key: YARN-2495
> URL: https://issues.apache.org/jira/browse/YARN-2495
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
> YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
> YARN-2495_20141022.1.patch
>
>
> Target of this JIRA is to allow admin specify labels in each NM, this covers
> - User can set labels in each NM (by setting yarn-site.xml or using script
> suggested by [~aw])
> - NM will send labels to RM via ResourceTracker API
> - RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-2841: -- Attachment: YARN-2841.1.patch > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Blocker > Attachments: YARN-2841.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2607) TestDistributedShell fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated YARN-2607:
-
Attachment: test.log
[~leftnoteasy] yeah, it failed on trunk. Attaching a log.
> TestDistributedShell fails in trunk
> ---
>
> Key: YARN-2607
> URL: https://issues.apache.org/jira/browse/YARN-2607
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Wangda Tan
> Fix For: 2.6.0
>
> Attachments: YARN-2607-1.patch, YARN-2607-2.patch, YARN-2607-3.patch,
> test.log
>
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk/691/console :
> {code}
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.641 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
> On Linux, I got the following locally:
> {code}
> testDSAttemptFailuresValidityIntervalFailed(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 64.715 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalFailed(TestDistributedShell.java:384)
> testDSAttemptFailuresValidityIntervalSucess(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 115.842 sec <<< ERROR!
> java.lang.Exception: test timed out after 9 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:680)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:661)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalSucess(TestDistributedShell.java:342)
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.633 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2423) TimelineClient should wrap all GET APIs to facilitate Java users
[
https://issues.apache.org/jira/browse/YARN-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205545#comment-14205545
]
Hadoop QA commented on YARN-2423:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680675/YARN-2423.patch
against trunk revision eace218.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.client.api.impl.TestTimelineClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5809//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5809//console
This message is automatically generated.
> TimelineClient should wrap all GET APIs to facilitate Java users
>
>
> Key: YARN-2423
> URL: https://issues.apache.org/jira/browse/YARN-2423
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: YARN-2423.patch, YARN-2423.patch, YARN-2423.patch
>
>
> TimelineClient provides the Java method to put timeline entities. It's also
> good to wrap over all GET APIs (both entity and domain), and deserialize the
> json response into Java POJO objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2423) TimelineClient should wrap all GET APIs to facilitate Java users
[ https://issues.apache.org/jira/browse/YARN-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205508#comment-14205508 ] Hitesh Shah commented on YARN-2423: --- I think it is a bit too early to mark these APIs as Public ( as it enforces a suggestion that the APIs are stable ) . A proper underlying storage such as HBase is yet to be completed and that may have repercussions on what is/is not supported in a scalable manner. > TimelineClient should wrap all GET APIs to facilitate Java users > > > Key: YARN-2423 > URL: https://issues.apache.org/jira/browse/YARN-2423 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: YARN-2423.patch, YARN-2423.patch, YARN-2423.patch > > > TimelineClient provides the Java method to put timeline entities. It's also > good to wrap over all GET APIs (both entity and domain), and deserialize the > json response into Java POJO objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2607) TestDistributedShell fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205494#comment-14205494
]
Wangda Tan commented on YARN-2607:
--
Hi [~ozawa],
I just tried it can pass locally, does it still failing?
Thanks,
> TestDistributedShell fails in trunk
> ---
>
> Key: YARN-2607
> URL: https://issues.apache.org/jira/browse/YARN-2607
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Wangda Tan
> Fix For: 2.6.0
>
> Attachments: YARN-2607-1.patch, YARN-2607-2.patch, YARN-2607-3.patch
>
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk/691/console :
> {code}
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.641 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
> On Linux, I got the following locally:
> {code}
> testDSAttemptFailuresValidityIntervalFailed(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 64.715 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalFailed(TestDistributedShell.java:384)
> testDSAttemptFailuresValidityIntervalSucess(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 115.842 sec <<< ERROR!
> java.lang.Exception: test timed out after 9 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:680)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:661)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalSucess(TestDistributedShell.java:342)
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.633 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2423) TimelineClient should wrap all GET APIs to facilitate Java users
[ https://issues.apache.org/jira/browse/YARN-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-2423: Attachment: YARN-2423.patch I rebased the new patch and fixed the test failures. For the bug I mentioned, I talked to Zhijie offline and that is the correct behavior, at least for now; so I commented out the line that checks the related entities in the test. > TimelineClient should wrap all GET APIs to facilitate Java users > > > Key: YARN-2423 > URL: https://issues.apache.org/jira/browse/YARN-2423 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: YARN-2423.patch, YARN-2423.patch, YARN-2423.patch > > > TimelineClient provides the Java method to put timeline entities. It's also > good to wrap over all GET APIs (both entity and domain), and deserialize the > json response into Java POJO objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205445#comment-14205445 ] Eric Payne commented on YARN-1963: -- Thanks a lot [~sunilg] for taking the lead on this and putting together the design document. I have a question about per-priority ACLs. Can per-priority ACLs within a queue be inherited from queue-level ACLs if the per-priority ACLs aren't there? In a cluster that already has queues divided to be specific to business units, they will want to only specify the queue-level ACL list. In other words, in this use case, the queue-level users are already trusted enough to modify apps on that queue, regardless of priority, so they won't want the extra overhead of specifying additional priority-level ACLs. Is that part of the design? > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: YARN Application Priorities Design.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2009) Priority support for preemption in ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205419#comment-14205419 ] Eric Payne commented on YARN-2009: -- Hi [~sunilg]. If I understand correctly, this Jira is not intended to implement preemption within a single queue. Rather, the intent is to change ProportionalCapacityPreemptionPolicy so that when selecting containers to preempt from a queue, choose the lower priority ones first. Is that correct? > Priority support for preemption in ProportionalCapacityPreemptionPolicy > --- > > Key: YARN-2009 > URL: https://issues.apache.org/jira/browse/YARN-2009 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Devaraj K >Assignee: Sunil G > > While preempting containers based on the queue ideal assignment, we may need > to consider preempting the low priority application containers first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-2841: -- Affects Version/s: 2.6.0 > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jian He >Assignee: Jian He >Priority: Blocker > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2841) RMProxy should retry EOFException
[ https://issues.apache.org/jira/browse/YARN-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-2841: -- Summary: RMProxy should retry EOFException (was: RMProxy should retry EOFException after HADOOP-9576) > RMProxy should retry EOFException > -- > > Key: YARN-2841 > URL: https://issues.apache.org/jira/browse/YARN-2841 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He >Priority: Blocker > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2841) RMProxy should retry EOFException after HADOOP-9576
Jian He created YARN-2841: - Summary: RMProxy should retry EOFException after HADOOP-9576 Key: YARN-2841 URL: https://issues.apache.org/jira/browse/YARN-2841 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Priority: Blocker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2840) Timeline should support creation of Domains where domainId is not provided by the user
Hitesh Shah created YARN-2840: - Summary: Timeline should support creation of Domains where domainId is not provided by the user Key: YARN-2840 URL: https://issues.apache.org/jira/browse/YARN-2840 Project: Hadoop YARN Issue Type: Improvement Components: timelineserver Reporter: Hitesh Shah Current expectation is that the user has to come up with a unique domain id. When using this with applications such as Pig/Hive/Oozie, these applications will need to come up with a cluster-wide unique id to be able to create a domain as domainIds need to be unique. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2495) Allow admin specify labels from each NM (Distributed configuration)
[
https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204974#comment-14204974
]
Naganarasimha G R commented on YARN-2495:
-
Hi [~wangda], Thanks for reviewing and sorry for the delayed reply ...
{quote}
Major comments:
1.
Makes sense to me, I suggest to add a field like node-labels-updated in
NodeHeartbeatRequest
{quote}
I have already added the below code in the NodeHeartbeatRequest interface for
this(or i dint get your comment correctly, please elaborate):
{quote}
@@ -26,7 +28,8 @@
public static NodeHeartbeatRequest newInstance(NodeStatus nodeStatus,
MasterKey lastKnownContainerTokenMasterKey,
- MasterKey lastKnownNMTokenMasterKey) {
+ MasterKey lastKnownNMTokenMasterKey, Set nodeLabels,
+ boolean isNodeLabelsUpdated
@@ -45,4 +50,10 @@ public static NodeHeartbeatRequest newInstance(NodeStatus
nodeStatus,
+ public abstract boolean isNodeLabelsUpdated();
+ public abstract void setNodeLabelsUpdated(boolean isNodeLabelsUpdated);
{quote}
{quote}
3.Suggest to create a overload newInstance function without labels for
RegisterNodeManagerRequest to avoid check like:
(nodeLabelsProvider == null) ? null : nodeLabels);
{quote}
This if check will be there in the overloaded case to identify which overloaded
method to choose right ? i am not able to see any benefit from this.
{quote}
4.YarnConfiguration:
NM_NODE_LABELS_FROM_CONFIG sounds like a boolean value to me, how about call it
NM_NODE_LABELS_PREFIX + "config-based" + ".node-labels"?
NM_NODE_LABELS_FETCH_INTERVAL_MS should also be a part of config-based.
{quote}
NM_NODE_LABELS_FETCH_INTERVAL_MS & DEFAULT_NM_NODE_LABELS_FETCH_INTERVAL_MS is
used in both the script based and also config based hence had not moved it
under config-based ?
whats your opinion having two properties separately for script and config based
is better? if reuse then can delete NM_NODE_LABELS_FROM_CONFIG, as its not used.
{quote}
5. PB tests
I think you can leverage TestPBImplRecords do all PB related tests, does it
enough?
{quote}
These i wanted to discuss with you , based on your patch changes for labels had
figured out this class but as i was modifying the existing PB class awas
wondering why these existing PB's are not added here.
Others comments have reworked, after clarifications of the above points will
upload the patch tomorrow...
> Allow admin specify labels from each NM (Distributed configuration)
> ---
>
> Key: YARN-2495
> URL: https://issues.apache.org/jira/browse/YARN-2495
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Naganarasimha G R
> Attachments: YARN-2495.20141023-1.patch, YARN-2495.20141024-1.patch,
> YARN-2495.20141030-1.patch, YARN-2495.20141031-1.patch,
> YARN-2495_20141022.1.patch
>
>
> Target of this JIRA is to allow admin specify labels in each NM, this covers
> - User can set labels in each NM (by setting yarn-site.xml or using script
> suggested by [~aw])
> - NM will send labels to RM via ResourceTracker API
> - RM will set labels in NodeLabelManager when NM register/update labels
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2839) YARN minicluster doesn't bail out if all the NM disks are dead
[
https://issues.apache.org/jira/browse/YARN-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204901#comment-14204901
]
Steve Loughran commented on YARN-2839:
--
We don't see a stack trace; we see ERRORS in the logs
{code}
2014-11-10 03:02:18,431 [Thread-2] INFO nodemanager.LocalDirsHandlerService
(LocalDirsHandlerService.java:logDiskStatus(339)) - Disk(s) failed: 1/1
local-dirs are bad:
/tmp/jenkins/workspace/slider-core/target/testexistsfailswithunknowncluster/testexistsfailswithunknowncluster-localDir-nm-0_0;
1/1 log-dirs are bad:
/tmp/jenkins/workspace/slider-core/target/testexistsfailswithunknowncluster/testexistsfailswithunknowncluster-logDir-nm-0_0
2014-11-10 03:02:18,432 [Thread-2] ERROR nodemanager.LocalDirsHandlerService
(LocalDirsHandlerService.java:updateDirsAfterTest(332)) - Most of the disks
failed. 1/1 local-dirs are bad:
/tmp/jenkins/workspace/slider-core/target/testexistsfailswithunknowncluster/testexistsfailswithunknowncluster-localDir-nm-0_0;
1/1 log-dirs are bad:
/tmp/jenkins/workspace/slider-core/target/testexistsfailswithunknowncluster/testexistsfailswithunknowncluster-logDir-nm-0_0
2014-11-10 03:02:18,433 [Thread-2] INFO localizer.ResourceLocalizationService
(ResourceLocalizationService.java:validateConf(216)) - per directory file limit
= 8192
{code}
> YARN minicluster doesn't bail out if all the NM disks are dead
> --
>
> Key: YARN-2839
> URL: https://issues.apache.org/jira/browse/YARN-2839
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>
> Some jenkins tests of mine have been failing deep in the resource
> localization process. Iif all the disks of the NMs are considered bad they
> refuse to work, but the Yarn Minicluster doesn't fail itself.
> YARN-90 assumes that the NM disks will come back. This isn't likely to hold
> in a short-lived mini cluster —better to have it probe the NMs and fail if
> they aren't healthy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2839) YARN minicluster doesn't bail out if all the NM disks are dead
Steve Loughran created YARN-2839: Summary: YARN minicluster doesn't bail out if all the NM disks are dead Key: YARN-2839 URL: https://issues.apache.org/jira/browse/YARN-2839 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Steve Loughran Some jenkins tests of mine have been failing deep in the resource localization process. Iif all the disks of the NMs are considered bad they refuse to work, but the Yarn Minicluster doesn't fail itself. YARN-90 assumes that the NM disks will come back. This isn't likely to hold in a short-lived mini cluster —better to have it probe the NMs and fail if they aren't healthy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204869#comment-14204869
]
Hudson commented on YARN-2834:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1953 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1953/])
YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery
consistent with the (somewhat incorrect) behaviour in the non-recovery case.
Contributed by Jian He. (vinodkv: rev e76faebc9589654e83c8244ef9aff88391e56b80)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Blocker
> Fix For: 2.6.0
>
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTran
[jira] [Commented] (YARN-2830) Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
[ https://issues.apache.org/jira/browse/YARN-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204872#comment-14204872 ] Hudson commented on YARN-2830: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1953 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1953/]) YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. (acmurthy: rev 43cd07b408c6613d2c9aa89203cfa3110d830538) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithHistoryService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceDecrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebApp.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryStoreTestUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ahs/TestRMApplicationHistoryWriter.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204811#comment-14204811
]
Hudson commented on YARN-2834:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1929 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1929/])
YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery
consistent with the (somewhat incorrect) behaviour in the non-recovery case.
Contributed by Jian He. (vinodkv: rev e76faebc9589654e83c8244ef9aff88391e56b80)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Blocker
> Fix For: 2.6.0
>
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(Sta
[jira] [Commented] (YARN-2830) Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
[ https://issues.apache.org/jira/browse/YARN-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204814#comment-14204814 ] Hudson commented on YARN-2830: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1929 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1929/]) YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. (acmurthy: rev 43cd07b408c6613d2c9aa89203cfa3110d830538) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestNMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryStoreTestUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/
[jira] [Commented] (YARN-2838) Issues with TimeLineServer (Application History)
[ https://issues.apache.org/jira/browse/YARN-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204789#comment-14204789 ] Naganarasimha G R commented on YARN-2838: - Missed to add point 4 : In YARNClientIMPL;history data can be either got from HistoryServer (old manager) or from TimeLineServer (new) So historyServiceEnabled flag needs to check for both Timeline server configurations and ApplicationHistoryServer configurations, as data can be got from either of them. > Issues with TimeLineServer (Application History) > > > Key: YARN-2838 > URL: https://issues.apache.org/jira/browse/YARN-2838 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: IssuesInTimelineServer.pdf > > > Few issues in usage of Timeline server for generic application history access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204747#comment-14204747
]
Naganarasimha G R commented on YARN-2808:
-
"historyClient" in YarnClientImpl will behave differently for getContainers
call for a running application. When deprecated AHS is configured it throws
exception and timeline history server is configured all containers result is
got of which completed containers can be utilized. So based on which server is
configured, YarnClientImpl need to take decision whether containers for running
application needs to be considered from history client. hence this issue
depends on yarn-2838
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204737#comment-14204737
]
Naganarasimha G R commented on YARN-2808:
-
While modifying for this issue faced some issues in timelineserver for generic
application history data, as some changes are related to that, will start
working on this issue once clarifications are got for yarn-2838
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2838) Issues with TimeLineServer (Application History)
[ https://issues.apache.org/jira/browse/YARN-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-2838: Attachment: IssuesInTimelineServer.pdf While working on YARN-2808, faced few issues with Timelineserver. Gathering them in this attached file. Correct me in case of any issue reported is incorrect. > Issues with TimeLineServer (Application History) > > > Key: YARN-2838 > URL: https://issues.apache.org/jira/browse/YARN-2838 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: IssuesInTimelineServer.pdf > > > Few issues in usage of Timeline server for generic application history access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2838) Issues with TimeLineServer (Application History)
Naganarasimha G R created YARN-2838: --- Summary: Issues with TimeLineServer (Application History) Key: YARN-2838 URL: https://issues.apache.org/jira/browse/YARN-2838 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.5.1 Reporter: Naganarasimha G R Assignee: Naganarasimha G R Few issues in usage of Timeline server for generic application history access -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2830) Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
[ https://issues.apache.org/jira/browse/YARN-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204686#comment-14204686 ] Hudson commented on YARN-2830: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #739 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/739/]) YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. (acmurthy: rev 43cd07b408c6613d2c9aa89203cfa3110d830538) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/api/protocolrecords/TestProtocolRecords.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKillAMPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceDecrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestNMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/api/protocolrecords/TestRegisterNodeManagerRequest.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-r
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204683#comment-14204683
]
Hudson commented on YARN-2834:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #739 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/739/])
YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery
consistent with the (somewhat incorrect) behaviour in the non-recovery case.
Contributed by Jian He. (vinodkv: rev e76faebc9589654e83c8244ef9aff88391e56b80)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Blocker
> Fix For: 2.6.0
>
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(State
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204525#comment-14204525
]
Hadoop QA commented on YARN-2837:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680544/YARN-2837.1.patch
against trunk revision ab30d51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5808//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5808//console
This message is automatically generated.
> Timeline server needs to recover the timeline DT when restarting
>
>
> Key: YARN-2837
> URL: https://issues.apache.org/jira/browse/YARN-2837
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Blocker
> Attachments: YARN-2837.1.patch
>
>
> Timeline server needs to recover the stateful information when restarting as
> RM/NM/JHS does now. So far the stateful information only includes the
> timeline DT. Without recovery, the timeline DT of the existing YARN apps is
> not long valid, and cannot be renewed any more after the timeline server is
> restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2664) Improve RM webapp to expose info about reservations.
[ https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-2664: Assignee: Matteo Mazzucchelli > Improve RM webapp to expose info about reservations. > > > Key: YARN-2664 > URL: https://issues.apache.org/jira/browse/YARN-2664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Matteo Mazzucchelli > Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch, > YARN-2664.patch > > > YARN-1051 provides a new functionality in the RM to ask for reservation on > resources. Exposing this through the webapp GUI is important. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2837: -- Attachment: (was: YARN-2834.1.patch) > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-2837.1.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2837: -- Attachment: YARN-2837.1.patch > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-2837.1.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2837: -- Attachment: YARN-2834.1.patch Create a patch to make the timeline state store, I choose to use Leveldb impl because: 1. Timeline server already uses leveldb. 2. It provides atomic operations, and isolate the system dependent FS. 3. Less heavy and complex than using HDFS (in particular in secure mode) 4. Easy to implement the operations. > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-2834.1.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
