[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303431#comment-14303431
]
Hudson commented on YARN-2808:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2044 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2044/])
YARN-2808. Made YARN CLI list attemptâs finished containers of a running
application. Contributed by Naganarasimha G R. (zjshen: rev
52575ff2240ea0b7cea818ef34503576c0ecf9f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
* hadoop-yarn-project/CHANGES.txt
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303389#comment-14303389
]
Hudson commented on YARN-2808:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #94 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/94/])
YARN-2808. Made YARN CLI list attempt’s finished containers of a running
application. Contributed by Naganarasimha G R. (zjshen: rev
52575ff2240ea0b7cea818ef34503576c0ecf9f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
* hadoop-yarn-project/CHANGES.txt
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303323#comment-14303323
]
Hudson commented on YARN-2808:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #90 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/90/])
YARN-2808. Made YARN CLI list attemptâs finished containers of a running
application. Contributed by Naganarasimha G R. (zjshen: rev
52575ff2240ea0b7cea818ef34503576c0ecf9f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
* hadoop-yarn-project/CHANGES.txt
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303300#comment-14303300
]
Hudson commented on YARN-2808:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2025 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2025/])
YARN-2808. Made YARN CLI list attemptâs finished containers of a running
application. Contributed by Naganarasimha G R. (zjshen: rev
52575ff2240ea0b7cea818ef34503576c0ecf9f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
* hadoop-yarn-project/CHANGES.txt
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303139#comment-14303139
]
Hudson commented on YARN-2808:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #827 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/827/])
YARN-2808. Made YARN CLI list attemptâs finished containers of a running
application. Contributed by Naganarasimha G R. (zjshen: rev
52575ff2240ea0b7cea818ef34503576c0ecf9f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
* hadoop-yarn-project/CHANGES.txt
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303112#comment-14303112
]
Hudson commented on YARN-2808:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #93 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/93/])
YARN-2808. Made YARN CLI list attemptâs finished containers of a running
application. Contributed by Naganarasimha G R. (zjshen: rev
52575ff2240ea0b7cea818ef34503576c0ecf9f7)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302537#comment-14302537
]
Naganarasimha G R commented on YARN-2808:
-
Thanks [~zjshen] for reviewing and committing the patch, i will look into
YARN-3123 too.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301842#comment-14301842
]
Hudson commented on YARN-2808:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #6981 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6981/])
YARN-2808. Made YARN CLI list attemptâs finished containers of a running
application. Contributed by Naganarasimha G R. (zjshen: rev
52575ff2240ea0b7cea818ef34503576c0ecf9f7)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Fix For: 2.7.0
>
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301795#comment-14301795
]
Zhijie Shen commented on YARN-2808:
---
+1 LGTM, will commit the patch.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299782#comment-14299782
]
Hadoop QA commented on YARN-2808:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695749/YARN-2808.20150131-1.patch
against trunk revision 26c2de3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6477//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6477//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6477//console
This message is automatically generated.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch,
> YARN-2808.20150131-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/cont
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299101#comment-14299101
]
Zhijie Shen commented on YARN-2808:
---
I think the patch should work, though it's not guarantee all the containers
will be returned for a running attempt due to some race condition that
container is finished, its info is pushed to timeline server, but is still not
persisted. Anyway, it will be a good improvement in terms of user experience.
Some minor comments:
1. Is it possible to improve the performance? The application could be big to
have hundreds of containers. It's not efficient to loop through them many
times. Maybe run through them once, and put the ids in a hashset for check?
{code}
for (int i = 0; i < containersFromHistoryServer.size(); i++) {
if (containersFromHistoryServer.get(i).getContainerId()
.equals(tmp.getContainerId())) {
containersFromHistoryServer.remove(i);
//Remove containers from AHS as container from RM will
have latest
//information
break;
}
}
{code}
2. In the test can we add a case that the running container is in RM, and it's
also in the timeline server as part of its information is written there, the
container info cached in RM is sourced instead of the partial info in the
timeline server.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:4702
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299033#comment-14299033
]
Hadoop QA commented on YARN-2808:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695573/YARN-2808.20150130-1.patch
against trunk revision f2c9109.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6466//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6466//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6466//console
This message is automatically generated.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Attachments: YARN-2808.20150126-1.patch, YARN-2808.20150130-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_000
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292141#comment-14292141
]
Hadoop QA commented on YARN-2808:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694541/YARN-2808.20150126-1.patch
against trunk revision 7574df1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
org.apache.hadoop.yarn.client.cli.TestRMAdminCLI
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6417//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6417//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6417//console
This message is automatically generated.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
> Attachments: YARN-2808.20150126-1.patch
>
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
>
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204747#comment-14204747
]
Naganarasimha G R commented on YARN-2808:
-
"historyClient" in YarnClientImpl will behave differently for getContainers
call for a running application. When deprecated AHS is configured it throws
exception and timeline history server is configured all containers result is
got of which completed containers can be utilized. So based on which server is
configured, YarnClientImpl need to take decision whether containers for running
application needs to be considered from history client. hence this issue
depends on yarn-2838
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204737#comment-14204737
]
Naganarasimha G R commented on YARN-2808:
-
While modifying for this issue faced some issues in timelineserver for generic
application history data, as some changes are related to that, will start
working on this issue once clarifications are got for yarn-2838
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201409#comment-14201409
]
George Wong commented on YARN-2808:
---
That's great. Looking forward to the new container command.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200081#comment-14200081
]
Naganarasimha G R commented on YARN-2808:
-
Hi [~GWong]
Earlier idea was the same, but i feel there might be lot of differences for
supporting "yarn container" command for both applicationID and application
attemptID with -list option itself, so as suggested by JianHe in
[YARN-2301|https://issues.apache.org/jira/browse/YARN-2301?focusedCommentId=14070512&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14070512],
i thought of splitting as follows.
# YARN-2301 : for first 3 small issues
# new jira for supporting "yarn container" command for both applicationID and
application attemptID
# listing of all containers even for running and completed apps as part of
yarn-1794 (similar to the current issue will confirm with Mayank and finalize
it )
Have already been working on this but was waiting for level DB based Timeline
server to be committed to get all the containers from timeline server itself
which will resolve most of the issues of "yarn container" command.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
>
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1429#comment-1429
]
George Wong commented on YARN-2808:
---
[~Naganarasimha], could you link this jira to yarn-2301? yarn-2808 could be one
piece of improvement for yarn container command.
As you are working on yarn-2301, can you fix this in yarn-2301?
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199963#comment-14199963
]
Naganarasimha G R commented on YARN-2808:
-
Hi [~wangg23] & [~GWong],
+1 for the approach mentioned by you Gordon wang.
Actually i had mentioned similar to these lines in YARN-2301 in one of my
comments
[https://issues.apache.org/jira/browse/YARN-2301?focusedCommentId=14085628&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085628
].
User should not be concerned from where the containers information is provided
they just need list of containers for a given appAttempt.
Earlier when i mention the above comment, Timelineserver store was file based
so only when the application is completed the file is completely written so
timelineserver will not be able to give the data when containers for running
applciations are requested.
[~GWong],
As i am working on the similar jira 2301 and already started working on it i
would like to continue and finish it . Sorry for the inconvenience
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199951#comment-14199951
]
George Wong commented on YARN-2808:
---
I hope I could take up this jira. Could any committer assign it to me?
Thanks.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>Assignee: Naganarasimha G R
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199862#comment-14199862
]
George Wong commented on YARN-2808:
---
+1 to [~wangg23]'s comment.
the container's info may exist in RM and TimeLineServer. Fetching all the info
from both servers is a solution.
I think I can provide a patch for the fix.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199559#comment-14199559
]
Gordon Wang commented on YARN-2808:
---
setting yarn.resourcemanager.max-completed-applications to 0 is not a solution
from my view.
RM could cache completed app generic info, so that it is easy for user to check
the job status. And more, user could find the job info in RM's web UI, then,
redirect to history-server/timeline-server for more app detail info.
I think YARN-1794 is similar, but not the same.
{quote}
And was intending to fix by "modifing YarnClientImpl.getContainers(). If the
size of containers is 0. We can get it from timeline."
{quote}
I am afraid this fix is only for YARN-2808 case. But generally speaking, since
RM does not cache completed container info, if a app is running, both the size
of running containers and the size of completed container could be larger than
0. So, one possible fix is that pulling the container info both from RM and
timeline-server regardless of the state of app attempt.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198123#comment-14198123
]
Naganarasimha G R commented on YARN-2808:
-
Hi [~beckham007]
Yes YARN-2301 is not solving this issue and YARN-2301 is for different purpose
altogether but the issue mentioned in this jira is almost same as in YARN-1794.
And was intending to fix by ??"modifing YarnClientImpl.getContainers(). If the
size of containers is 0. We can get it from timeline."??
But do you want this to be fixed by modifying the default value of
"yarn.resourcemanager.max-completed-applications" to 0 ?, AFAIK if modified to
0 then in the webui it will not show any finished apps which would not be a
good default configuration. what is your opinion ?
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14197797#comment-14197797
]
Beckham007 commented on YARN-2808:
--
[~Naganarasimha] YARN-2301 not sloved this.
The default "yarn.resourcemanager.max-completed-applications" is 1. So we
can get appAtempt from RM, not timeline. But RM couldn't get container infos
for finished app.
We can fix this by modify YarnClientImpl.getContainers(). If the size of
containers is 0. We can get it from timeline.
For now, we can set "yarn.resourcemanager.max-completed-applications" to 0.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14197684#comment-14197684
]
Naganarasimha G R commented on YARN-2808:
-
I think its similar to YARN-1794 , which was raised by Mayank Bansal.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14197672#comment-14197672
]
Gordon Wang commented on YARN-2808:
---
we need to fetch the container info from timeline server when the app is
finished. Since RM does not cache any containers info after the app is finished.
> yarn client tool can not list app_attempt's container info correctly
>
>
> Key: YARN-2808
> URL: https://issues.apache.org/jira/browse/YARN-2808
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Reporter: Gordon Wang
>
> When enabling timeline server, yarn client can not list the container info
> for a application attempt correctly.
> Here is the reproduce step.
> # enabling yarn timeline server
> # submit a MR job
> # after the job is finished. use yarn client to list the container info of
> the app attempt.
> Then, since the RM has cached the application's attempt info, the output show
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :0
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> {noformat}
> But if the rm is restarted, client can fetch the container info from timeline
> server correctly.
> {noformat}
> [hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
> appattempt_1415168250217_0001_01
> 14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
> http://0.0.0.0:8188/ws/v1/timeline/
> 14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
> server at /0.0.0.0:10200
> Total number of containers :4
> Container-Id Start Time Finish
> Time StateHost
> LOG-URL
> container_1415168250217_0001_01_01 1415168318376
> 1415168349896COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
> container_1415168250217_0001_01_02 1415168326399
> 1415168334858COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
> container_1415168250217_0001_01_03 1415168326400
> 1415168335277COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
> container_1415168250217_0001_01_04 1415168335825
> 1415168343873COMPLETElocalhost.localdomain:47024
> http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
