[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13919440#comment-13919440 ] Jason Lowe commented on YARN-1445: -- bq. Then, it is possible that AM is unregistered, and RM tells the client that the application is still running. When the client moves on to contact AM, AM has proceeded and exited before being able to respond the client request. This race will always exist and is inherent with asynchronous processes. The client could check the RM and the app could really be RUNNING, but by the time the client gets around to contacting the app the AM has rushed through the FINISHING and FINISHED state and could be gone by the time the client gets there. That's why ClientServiceDelegate retries on errors and re-evaluates whether to go to the AM or history server on each retry. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch, YARN-1445.6.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918984#comment-13918984 ] Zhijie Shen commented on YARN-1445: --- I've thought about postponing the unregistration success notification until the application is at FINISHED. It seems impossible, because it will result in a dead lock bellow 1. AM container is waiting for finishing unregistration to move on and exit; 2. Unregistration is waiting for RM notifying success; 3. RM is waiting for RMApp moving from FINISHING to FINISHED to return success; 4. RMApp is waiting for RMAppAttempt moving from FINISHING to FINISHED; 5. RMAppAttempt is waiting for AM container being finished. Then, if we return a prior state to the client given the internal FINISHING, and still return unregistration success when RMApp reaches FINISHING, client will see, for example, RUNNING, while the registration is already successful. The inconsistency here may result in some race condition for the process relying on checking the final state. For example, MR client will direct user to AM if the application is said not to be in a final state. Then, it is possible that AM is unregistered, and RM tells the client that the application is still running. When the client moves on to contact AM, AM has proceeded and exited before being able to respond the client request. It seems that we cannot avoid splitting the user-faced state, and FINISHING can map to a period of an application's life cycle, which is from unregistration to process exit. [~jlowe] and [~jianhe], how do you think about it? Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch, YARN-1445.6.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[
https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13915569#comment-13915569
]
Hadoop QA commented on YARN-1445:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631682/YARN-1445.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3214//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3214//console
This message is automatically generated.
Separate FINISHING and FINISHED state in YarnApplicationState
-
Key: YARN-1445
URL: https://issues.apache.org/jira/browse/YARN-1445
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch,
YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch, YARN-1445.6.patch
Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to
YarnApplicationState.FINISHED.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13916057#comment-13916057 ] Zhijie Shen commented on YARN-1445: --- Sorry for raising the new thought so late, but i think it may be more acceptable by multiple stakeholders. Referring to the recently added new internal state KILLING, which is mapped to a prior state before it, we may do the same thing for FINISHING. So, instead of merging FINISHING to FINISHED, we can merging FINISHING to a prior state as well. Then, FINISHED will always be a real final state with diagnostics information ready. One problem with this approach is that after unregistration succeeds, the client will still see the application in a non-final state for a while, because internally when an application is at FINISHING, client can already be notified of unregistration success. What we can do is to postpone the unregistration success notification until the application is at FINISHED. Then, when client receives the notification, the application will always in a final state. Any idea? Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch, YARN-1445.6.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850300#comment-13850300 ] Xuan Gong commented on YARN-1445: - Address all the latest comments Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch, YARN-1445.5.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[
https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850392#comment-13850392
]
Hadoop QA commented on YARN-1445:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619070/YARN-1445.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapred.TestYARNRunner
org.apache.hadoop.mapreduce.security.TestJHSSecurity
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl
org.apache.hadoop.mapred.TestClientRedirect
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2676//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2676//console
This message is automatically generated.
Separate FINISHING and FINISHED state in YarnApplicationState
-
Key: YARN-1445
URL: https://issues.apache.org/jira/browse/YARN-1445
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch,
YARN-1445.4.patch, YARN-1445.5.patch
Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to
YarnApplicationState.FINISHED.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[
https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851008#comment-13851008
]
Hadoop QA commented on YARN-1445:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619159/YARN-1445.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapred.TestYARNRunner
org.apache.hadoop.mapreduce.security.TestJHSSecurity
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapred.TestClientRedirect
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2681//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2681//console
This message is automatically generated.
Separate FINISHING and FINISHED state in YarnApplicationState
-
Key: YARN-1445
URL: https://issues.apache.org/jira/browse/YARN-1445
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch,
YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch
Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to
YarnApplicationState.FINISHED.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851299#comment-13851299 ] Zhijie Shen commented on YARN-1445: --- The three test failures should be not related (two for MAPREDUCE-5687 and one for YARN-1463). +1. The patch looks good to me. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849233#comment-13849233 ] Jason Lowe commented on YARN-1445: -- No, I think I'm good. If the client really needs to distinguish between the app being successful and the app completely finishing (e.g.: to obtain final metrics from RM) then we have to expose it. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849442#comment-13849442 ] Sandy Ryza commented on YARN-1445: -- In yarn_protos, we shouldn't change the numbers on the existing states. There's nothing wrong with adding FINISHING as 9. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848216#comment-13848216 ] Xuan Gong commented on YARN-1445: - [~jlowe] any further comments ? Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13839108#comment-13839108 ] Xuan Gong commented on YARN-1445: - bq. Sorry for raising the issue again, but YARN-1446 may ignore KILL request when an RMApp has entered FINISHING. If this is the case, we may want not to allow ApplicationCLI to kill an application when we have confirmed it is at FINISHING. Checked YARN-1446, added it back Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13839116#comment-13839116 ] Zhijie Shen commented on YARN-1445: --- bq. If an AM is really doing things after it unregisters in such a way that the client needs to distinguish FINISHING vs. FINISHED, that itself seems to be a problem. The AM must not do anything critical after it unregisters, because there are no retries at that point. Anything critical needs to be done before unregistering so if the AM crashes (or its node crashes) those critical steps will be re-attempted. I agree AM should be done (or almost done) when unregistering on RM. Unfortunately, after AM unregistration, RM still has to take additional management work before marking the application done from the point of view of RM. And only after the application is done from the point of view of RM, we are able to get its complete report from RM. FINISHING somehow becomes a transit state, which indicate the application has completed its work, but RM hasn't completed the post-processing work. bq. There still is the issue of final app time/diagnostics, and in that sense maybe FINISHING should map to the client state of RUNNING instead of FINISHED. I'm afraid it may be more confusing than merging FINISHING and FINISHED. If I remember it correctly, MR jobclient will check the application state from RM first. If the job is still running, the client will contact MR AM for detailed information. If we map FINISHING to RUNNING, the client will be redirected to MR AM when the application is actually in FINISHING, where RM AM is already unregistered and probably the process is ended. Hence the client will find RM AM cannot be reached. Personally, I think unregistration should be the boundary between RUNNING and the post states. FINISHING is proposed to indicate AM is unregistered, but RM hasn't finished the post unregistration work. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, YARN-1445.4.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[
https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13839262#comment-13839262
]
Hadoop QA commented on YARN-1445:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617027/YARN-1445.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.security.TestJHSSecurity
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2599//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2599//console
This message is automatically generated.
Separate FINISHING and FINISHED state in YarnApplicationState
-
Key: YARN-1445
URL: https://issues.apache.org/jira/browse/YARN-1445
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch,
YARN-1445.4.patch
Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to
YarnApplicationState.FINISHED.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838067#comment-13838067 ] Zhijie Shen commented on YARN-1445: --- bq. I've the different opinion here. RMAppImpl actually allows killing the app when it is in FINISHING. Moreover, if we handle FINISHING in the same way as FINISHED, we will see that printApplicationReport tells the app is still in FINISHING, while killApplication says the app is finished. Thoughts? bq. Make sense Sorry for raising the issue again, but YARN-1446 may ignore KILL request when an RMApp has entered FINISHING. If this is the case, we may want not to allow ApplicationCLI to kill an application when we have confirmed it is at FINISHING. [~jlowe], how do you think about the plan of splitting FINISHING and FINISHED? Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838297#comment-13838297 ] Vinod Kumar Vavilapalli commented on YARN-1445: --- +1 for the general idea of splitting the FINISHING and FINISHED states. FINISHING state is a first class state, in that AMs are given the guarentee that they can do stuff after unregister call. It's useful to let the clients know. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838338#comment-13838338 ] Jason Lowe commented on YARN-1445: -- Personally, I'm still not convinced this is a good idea overall. If an AM is really doing things *after* it unregisters in such a way that the client needs to distinguish FINISHING vs. FINISHED, that itself seems to be a problem. The AM _must not_ do anything critical after it unregisters, because there are no retries at that point. Anything critical needs to be done _before_ unregistering so if the AM crashes (or its node crashes) those critical steps will be re-attempted. In addition there's a limited time in which the app can complete any of these after-unregister tasks, as the RM will kill it if it takes too long and there's no way to ask for more time. There still is the issue of final app time/diagnostics, and in that sense maybe FINISHING should map to the client state of RUNNING instead of FINISHED. By exposing this new, unrecognized state to existing clients, that's essentially how some of them will interpret it if they're looking for explicit, terminal states (like KILLED/FINISHED/FAILED). Others will interpret it as a terminal state since it's not RUNNING. That being said, if we're still bent on exposing this distinction then here's my take: - For killing FINISHING apps, I think it's OK to either go ahead and kill the container or ignore the request. For the latter case, the RM is going to try to kill it in a few minutes anyway and forget about it. - Be sure to check all the existing occurrences of YarnApplication.FINISHED and make sure we don't need to change that code to handle the new YarnApplication.FINISHING state. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836701#comment-13836701 ] Zhijie Shen commented on YARN-1445: --- 1. I agree on the most plans about dealing with FINISHING. bq. 3.WebAppProxyServlet::doGet(). I think we might need to handle FINISHING as well as FINISHED. After RMApp enters FINISHING, the AM has already been unregistered, and tracking url has already been updated. Therefore, we're able to redirect the request in this state. bq. 5.ClientServiceDelegate::getProxy(), we might need to handle YarnApplicationState.FINISHING, too. When the application is in FINISHING, the AM has already been unregistered, and MR job need to be redirected to JHS for the details. bq. 6.ApplicationCLI::killApplication(). Here is a question mark. We can kill the AM when the RMApp is at Finishing state, since the AM did not really exist. Just for DS and MR, when the AM did the unregisterApplicationMaster, that means the application is finished, so at this time, if we send kill event, it is meaningless. Here, I just handle the YarnApplicationState.Finishing and Finished with the same way. I've the different opinion here. RMAppImpl actually allows killing the app when it is in FINISHING. Moreover, if we handle FINISHING in the same way as FINISHED, we will see that printApplicationReport tells the app is still in FINISHING, while killApplication says the app is finished. Thoughts? 2. TestYarnClient#testSubmitApplication need to be changed accordingly as well. Would you please double check other test classes where FINISHED is referred, and check whether the test cases will be broken or not? 3. Is it good to document the difference in detail between FINISHING and FINISHED? Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836841#comment-13836841 ] Xuan Gong commented on YARN-1445: - bq. I've the different opinion here. RMAppImpl actually allows killing the app when it is in FINISHING. Moreover, if we handle FINISHING in the same way as FINISHED, we will see that printApplicationReport tells the app is still in FINISHING, while killApplication says the app is finished. Thoughts? Make sense bq. 2. TestYarnClient#testSubmitApplication need to be changed accordingly as well. Would you please double check other test classes where FINISHED is referred, and check whether the test cases will be broken or not? Good catch. Fixed. For other places, I think they are fine. I did the full run on all the test for hadoop-yarn project. bq. 3. Is it good to document the difference in detail between FINISHING and FINISHED Added Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[
https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836960#comment-13836960
]
Hadoop QA commented on YARN-1445:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616603/YARN-1445.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.security.TestJHSSecurity
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2569//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2569//console
This message is automatically generated.
Separate FINISHING and FINISHED state in YarnApplicationState
-
Key: YARN-1445
URL: https://issues.apache.org/jira/browse/YARN-1445
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch
Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to
YarnApplicationState.FINISHED.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833830#comment-13833830 ] Jason Lowe commented on YARN-1445: -- Curious, would having the UNREGISTERED event pass a diagnostic be a better fix? I'd rather not expose a new YarnApplicationState to clients unless we really need to, and exposing the diagnostic when the app unregistered seems like a better fix overall (client is notified sooner, no new state needs to be exposed). Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834049#comment-13834049 ] Xuan Gong commented on YARN-1445: - bq. A nit: ResourceManagerRest.apt.vm needs to be updated as well Updated bq. Do we need to take care of FINISHING case in ClientServiceDelegate, WebAppProxyServlet and etc? IMOH, it's good to double check the places where FINISHED is used, to see whether we need to handle FINISHING given the change. Let us walk thru the list: 1. At LogsCLI::verifyApplicationState(), the function is used to figure out the RMApp state, and see if it is good time to check Logs. If the YarnApplicationState is FINISHING, it will return -1. It means it is not good time to check log. I think that makes sense. RMApp will notify NM to start log aggregation when it is at FINISHED/KILLED/FAILED state 2. At Client::monitorApplication() and UnmanagedAMLauncher::run(). Both of them are used to monitor the application whether it is finished. So, handle YarnApplicationState.FINISHED alone should be enough. 3.WebAppProxyServlet::doGet(). I think we might need to handle FINISHING as well as FINISHED. 4.At TypeConvert::fromYarn(), We can handle FINISHING as well as FINISHED. It will check the FinalApplicationStatus, this FinalApplicationStatus is based on the currentAppAttempt's finalStatus, or is generated by currentState of RMApp (In this case, the RMApp's FINISHING and FINISHED will be counted as the same status) 5.ClientServiceDelegate::getProxy(), we might need to handle YarnApplicationState.FINISHING, too. 6.ApplicationCLI::killApplication(). Here is a question mark. We can kill the AM when the RMApp is at Finishing state, since the AM did not really exist. Just for DS and MR, when the AM did the unregisterApplicationMaster, that means the application is finished, so at this time, if we send kill event, it is meaningless. Here, I just handle the YarnApplicationState.Finishing and Finished with the same way. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834089#comment-13834089 ] Xuan Gong commented on YARN-1445: - [~jlowe] The diagnostic issue may be not a good example. Sorry for the confusing. As I mentioned in my previous comment, AM sent out unregisterApplicationMaster, it does not mean that AM is finished. (In MR case, or DS case, after AM sent out the unregisterApplicationMaster, it does mean the application is finished, because after that, AM will do nothing but exist). But it may not be the case for the other applications, (AM might do other things, such as monitoring, writing log, etc, after that). So, merging the finishing state and finished state might be not a good idea. Because the client will not known when the application is really finished. Hopefully, this can explain why we want to separate the FINISHING and FINISHED state. [~zjshen] Please add the comment if you have better explanation. Thanks Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[
https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834158#comment-13834158
]
Hadoop QA commented on YARN-1445:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616100/YARN-1445.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.security.TestJHSSecurity
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2548//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2548//console
This message is automatically generated.
Separate FINISHING and FINISHED state in YarnApplicationState
-
Key: YARN-1445
URL: https://issues.apache.org/jira/browse/YARN-1445
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1445.1.patch, YARN-1445.2.patch
Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to
YarnApplicationState.FINISHED.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834242#comment-13834242 ] Xuan Gong commented on YARN-1445: - This test case failure is un-related Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834244#comment-13834244 ] Zhijie Shen commented on YARN-1445: --- I did some more investigation. Here's some finding. Correct me if I'm wrong: 1. From FINISHING, RMApp can only go to FINISHED. Therefore, unless RM crashes or hangs, the client will finally get the final state of RMApp 2. At the point of ApplicationReport, two pieces of information are not available when RMApp is in FINISHING, but are available when RMApp is in FINISHED: diagnostics and finishTime. Note that diagnostics may come from RMAppAttempt as well as from the KILL signal (it is possible when RMApp is in FINISHING). This diagnostics cannot be passed to RMApp when it is still in FINISHING. On the other side, if the users/clients doesn't care about the diagnostics info, and the finishTime, they can deal with FINISHING/FINISHED in the same way. However, if they do, they may expect the complete information only when the state becomes FINISHED. Maybe this is also the guide of fixing the current code where FINISHED is used? Mover, projecting the future, we may have more information that are available in FINISHED, but not in FINISHING (or even transitions from FINISHING to other states?), as we are still actively fixing transition issues. Therefore I incline to splitting the state in YarnApplicationState. Of course, for new user oriented state, we need document clearly the difference between it and FINISHED. Any idea? Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch, YARN-1445.2.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[
https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832907#comment-13832907
]
Hadoop QA commented on YARN-1445:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615874/YARN-1445.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2534//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2534//console
This message is automatically generated.
Separate FINISHING and FINISHED state in YarnApplicationState
-
Key: YARN-1445
URL: https://issues.apache.org/jira/browse/YARN-1445
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1445.1.patch
Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to
YarnApplicationState.FINISHED.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833522#comment-13833522 ] Xuan Gong commented on YARN-1445: - Let me give more details: AM sends unregisterApplicationMaster event with the diagnostic message. When the RMAppAttempt receives this event, and send the RMAppAttemptEventType.UNREGISTERE to RMApp, and RMApp will go to finishing state. At this time, the yarnapplicationstate will become finished. But at this time, the diagnostic message which is from AM is still at RMAppAttempt until RMAppAttempt send the RMAppFinishedAttemptEvent to RMApp. So, before this, at the client side, when the client sees the YarnApplicationState become finished, the client will consider this application is finished, but at this time, he can not get any diagnostic message until this application is really finished (RMApp is at Finished state). My use case is Distributed Shell. We can not find any diagnostic message at the console (At this time, RMApp is at Finishing state, and YarnApplicationState is finished state). Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833537#comment-13833537 ] Zhijie Shen commented on YARN-1445: --- Looked at the patch: 1. Do we need to take care of FINISHING case in ClientServiceDelegate, WebAppProxyServlet and etc? IMOH, it's good to double check the places where FINISHED is used, to see whether we need to handle FINISHING given the change. 2. A nit: ResourceManagerRest.apt.vm needs to be updated as well Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1445.1.patch Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13831909#comment-13831909 ] Sandy Ryza commented on YARN-1445: -- Is there a rationale for why they are currently a single YarnApplicationState? What are the meanings of the FINISHING and FINISHED states? Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13831927#comment-13831927 ] Jason Lowe commented on YARN-1445: -- FINISHING means the app has unregistered and the RM is waiting for the application to exit on its own. FINISHED means the app is totally done and has exited. I added the distinction a while back because the RM was aggressively killing any application as soon as it unregistered, leading to unclean shutdowns (e..g: failure to cleanup the staging directory). See MAPREDUCE-4099 for details. In hindsight I should have simply called the new state UNREGISTERED to help avoid confusion, and my apologies for that. I mapped the two RMApp states to a single YarnApplicationState since I didn't think clients needed to know the distinction at the time and doing so meant breaking existing YARN clients since they would now see a new, unrecognized state. I'm assuming Xuan has a case where it's necessary to distinguish these on the client side, curious to hear what that is. And I hope clients are tolerant of the new state. ;-) Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832273#comment-13832273 ] Xuan Gong commented on YARN-1445: - [~jlowe] Thanks for the explanation. When the AM did the unregister, the RMApp and RMAppAttempt will go to the Finishing state, and when the AMContainer is finished, the RMAppAttempt and RMApp will go to the finished state. Looks like that if AM finished the unregister, we may consider this Application as complete, as least for the client point of view. But this is for the MR case. Right ? I am afraid that in the future, when clients write their own applicationMasters, they will not consider that the application is complete even if the application did the unregister. So, i think it may be better to separate these two states. Separate FINISHING and FINISHED state in YarnApplicationState - Key: YARN-1445 URL: https://issues.apache.org/jira/browse/YARN-1445 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1#6144)
