[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947813#comment-13947813 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #521 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/521/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947927#comment-13947927 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1713 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1713/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947972#comment-13947972 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1738 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1738/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13948184#comment-13948184
]
Bikas Saha commented on YARN-1521:
--
How are the following idempotent? Looks like submitApplication was made
idempotent in this patch itself? What about moveApplication? If the app has
been moved from A-B then asking another move from A would be an error. So a
retry after the operation should also return an error unless there is specific
handling for the retry.
{code}
+ @Idempotent
public SubmitApplicationResponse submitApplication(
SubmitApplicationRequest request)
+ @Idempotent
public MoveApplicationAcrossQueuesResponse moveApplicationAcrossQueues(
{code}
IMO, how can we simply create an empty response and return it? Currently this
is safe because that response object has no fields. But if a field is added
tomorrow then this code would be wrong since users would get a response object
from which they cannot get response information. We should be using RetryCache
to populate the response from the actual response that was returned for the
original request that got retried.
{code}- throw RPCUtil.getRemoteException(message);
+ LOG.info(This is an earlier submitted application: + applicationId);
+ return SubmitApplicationResponse.newInstance();{code}
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Priority: Blocker
Fix For: 2.4.0
Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch,
YARN-1521.3.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13948457#comment-13948457 ] Bikas Saha commented on YARN-1521: -- OK. So we are requiring the moveAppAcrossQueues API to be idempotent? How can that be done without ignoring an illegal move of an app from a queue in which the app does not exist? Is that correct behavior? Any response to the second comment above about returning a new SubmitApplicationResponse instead of the original response for the original request. What if there are member values in SubmitApplicationResponse (there aren't any right now). Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946246#comment-13946246
]
Zhijie Shen commented on YARN-1521:
---
Browsed through the newest, and have the following comments:
1. If the application is already in the RM cache, we shouldn't log success.
Otherwise, there may be multiple logs for one submission.
{code}
+ RMAuditLogger.logSuccess(user, AuditConstants.SUBMIT_APP_REQUEST,
+ ClientRMService, applicationId);
{code}
2. Should it sleep a while before retry?
{code}
+ try {
+client.getApplications();
+return;
+ } catch (Exception e) {
+LOG.error(e.getMessage());
+ } finally {
+client.stop();
+ }
{code}
3. Again, sleep 1ms before next try, yielding to the thread of API methods'
invoking. And, have max retry when exception?
{code}
while (keepRunning) {
+ if (cluster.getStartFailoverFlag()) {
+try {
+ explicitFailover();
+ keepRunning = false;
+} catch (Exception e) {
+ // Do Nothing
+} finally {
+ keepRunning = false;
+}
+ }
+}
{code}
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch, YARN-1521.1.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947059#comment-13947059 ] Xuan Gong commented on YARN-1521: - bq. 1. If the application is already in the RM cache, we shouldn't log success. Otherwise, there may be multiple logs for one submission. Removed bq. Should it sleep a while before retry? DONE bq. Again, sleep 1ms before next try, yielding to the thread of API methods' invoking. And, have max retry when exception? DONE. Did not add max retry. The failover thread will be killed after every testcases Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947125#comment-13947125
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636766/YARN-1521.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3453//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3453//console
This message is automatically generated.
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947139#comment-13947139 ] Xuan Gong commented on YARN-1521: - [~kkambatl] Could you take a look ? Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947295#comment-13947295
]
Jian He commented on YARN-1521:
---
- we should at least log this event saying this is an earlier submitted
application, which is good for debugging.
- TestProtocolHA - TestProtocolHABase.java
- I ran TestApplicationClientProtocolOnHA locally without core code changes,
the whole test eventually takes 15 mins to crash.
I observed that the test keeps doing failover even if the test is done. can you
investigate ?
{code}
14/03/25 15:02:08 WARN resourcemanager.RMAuditLogger: USER=jhe
OPERATION=transitionToStandby TARGET=RMHAProtocolService RESULT=FAILURE
DESCRIPTION=Exception transitioning to standby PERMISSIONS=All users are
allowed
14/03/25 15:02:09 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
14/03/25 15:02:11 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
14/03/25 15:02:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
14/03/25 15:02:23 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
14/03/25 15:02:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
14/03/25 15:02:57 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
14/03/25 15:03:30 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
{code}
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947302#comment-13947302 ] Vinod Kumar Vavilapalli commented on YARN-1521: --- bq. - TestProtocolHA - TestProtocolHABase.java Or ProtocolHATestBase. BTW, why is it ProtocolHA? Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947311#comment-13947311 ] Jian He commented on YARN-1521: --- In TestApplicationClientProtocolOnHA , each client API call involves a new miniYarnCluster start and shutdown, how long does it take for the whole test to finish ? If that's too long, we can just reuse one miniYarnCluster. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947324#comment-13947324 ] Zhijie Shen commented on YARN-1521: --- bq. If that's too long, we can just reuse one miniYarnCluster. Some input: It may or may not run into the case I saw in YARN-1873. Before reusing a single yarn cluster, it's good to make sure the change on miniYarnCluster in the current test case will not disturb the following test cases. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947392#comment-13947392
]
Karthik Kambatla commented on YARN-1521:
Sorry for the delay in getting to this. Haven't look at the patch itself yet.
{quote}
ApplicationMasterProtocol
1. registerApplicationMaster
2. finishApplicationMaster
3. allocate
{quote}
Don't remember the source corresponding to these methods off the top of my
head, but would think we *should* mark all of them as well. I am okay with
doing this in a separate JIRA to unblock this.
# Register resets the sequence number - Idempotent. YARN-556 allows a running
AM to re-register, albeit on resync.
# Finish should be similar to kill - Idempotent. Should be similar to
killApplication?
# Allocate - Idempotent or AtMostOnce - as the AM heartbeats periodically.
Preferably Idempotent.
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947399#comment-13947399
]
Karthik Kambatla commented on YARN-1521:
Looked at the non-test source only. Looks good. One nit - it should be okay to
fix at commit time.
# Reword to it checks whether the application already exists?
{code}
* pDuring the submission process, it checks whether the application
* has already exist. If the application exists, it will simply return
* SubmitApplicationResponse/p
{code}
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947414#comment-13947414 ] Xuan Gong commented on YARN-1521: - bq. In TestApplicationClientProtocolOnHA , each client API call involves a new miniYarnCluster start and shutdown, how long does it take for the whole test to finish ? If that's too long, we can just reuse one miniYarnCluster. It takes about 120s to finish all the test cases. bq. we should at least log this event saying this is an earlier submitted application, which is good for debugging. ADDED bq. TestProtocolHA - TestProtocolHABase.java changed Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947466#comment-13947466
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636836/YARN-1521.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3459//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3459//console
This message is automatically generated.
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch,
YARN-1521.3.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947511#comment-13947511 ] Jian He commented on YARN-1521: --- LGTM + 1. Regarding ApplicationMasterProtocol, since applications today are anyways killed after RM restarts, no point adding it now. We can add those in YARN-556. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Priority: Blocker Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947512#comment-13947512 ] Jian He commented on YARN-1521: --- checking this in. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Priority: Blocker Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947525#comment-13947525 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5404 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5404/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Priority: Blocker Fix For: 2.4.0 Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, YARN-1521.3.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945736#comment-13945736
]
Jian He commented on YARN-1521:
---
post initial review：
- this doesn’t seem correct, user is possible to get multiple delegation
tokens. Given this change, user can only get one token
{code}
// check whether the token exists or not.
// If this token existed, recover the token with
// DelegationTokenInformation
{code}
- Explicitly assert the App does not exist in RMStateStore or not.
{code}
// After submission, the applicationState will
// not be saved in RMStateStore
{code}
- Explicitly assert the app exists in RMContext after the 2nd RM comes Active.
{code}
// Submit Application
// After submission, the applicationState will be saved in RMStateStore.
{code}
- The bulk of test-specific hack in MiniYarnCluster can be moved to
TestHAProtocol, as MiniYarnCluster is commonly used by others.
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945905#comment-13945905 ] Xuan Gong commented on YARN-1521: - Thanks for the review. bq. The bulk of test-specific hack in MiniYarnCluster can be moved to TestHAProtocol, as MiniYarnCluster is commonly used by others. Yes, you are right. Remove those changes from MiniYarnCluster to TestHAProtocol bq. Explicitly assert the app exists in RMContext after the 2nd RM comes Active./ Explicitly assert the App does not exist in RMStateStore or not. DONE bq. this doesn’t seem correct, user is possible to get multiple delegation tokens. Given this change, user can only get one token Remove all changes from ClientRMService#getDelegationToken(). Just let the method re-entry if failover happens. [~jianhe] Please correct me if I am wrong. The getDelegationToken() is used to get a new Token, but is not saved in zookeeper yet. So, even if failover happens during getDelegationToken call, we can not get previous generated Token back, so just let the method re-generate a new token, and it should be fine. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch, YARN-1521.1.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945964#comment-13945964
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636467/YARN-1521.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3448//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3448//console
This message is automatically generated.
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch, YARN-1521.1.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944632#comment-13944632 ] Xuan Gong commented on YARN-1521: - updated: * mark renewDelegationToken, cancelDelegationToken, updateNodeResource and moveApplicationAcrossQueues as Idempotent * change submitApplication from AtMostOnce to Idempotetn * change nodeHeartbeat from Idempotent to AtMostOnce Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944633#comment-13944633 ] Xuan Gong commented on YARN-1521: - The patch includes: * Mark appropriate protocol methods with idempotent and atmostonce annotation based on the proposal. * Create testcases to test the annotation marked ** Limited scope: For all the testcases, we only test whether the method will be re-entry when failover happens. Does not cover the entire logic test. ** Test strategy: create a separate failover thread with a trigger flag, override all APIs that added trigger flag. When the apis are called, we will set trigger flag as true to kick off the failover. So We can make sure the failover happens during process of the method. If this API is marked as idempotent or atmostonce, the testcases will pass; otherwise, they will throw the exception. ** Did not add testcases for ResourceManagerAdministrationProtocol. All refresh* will be called during the process of transitionToActive that will break the test strategy I used here. But I did the manually testing. Simply add sleep thread into the refresh*, and verified that all refresh* apes can be re-entry when failover happens after we marked all refresh* as idempotent. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944675#comment-13944675
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636277/YARN-1521.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3439//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3439//console
This message is automatically generated.
Mark appropriate protocol methods with the idempotent annotation or
AtMostOnce annotation
-
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
Attachments: YARN-1521.0.patch
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944723#comment-13944723 ] Karthik Kambatla commented on YARN-1521: I would like to take a closer look at the annotations before this gets committed. If not urgent, please wait for me until Tuesday. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1521.0.patch After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943793#comment-13943793 ] Jian He commented on YARN-1521: --- - getNewApplicationId , getDelegationToken. since each call returns a new ID/Token, not sure this matches with idempotency. - For the registers protocols. For example, registerNodeManager : if previous call succeeds, RM didn't crash, registerNodeManager retry because of some network problem, the next call comes in, this node is deemed as a reconnected node instead of a new node. Probably AtMostOnce? Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942409#comment-13942409 ] Xuan Gong commented on YARN-1521: - We have three options for all the protocol APIs: 1. marked as Idempotent 2. marked as AtMostOnce. 3. do nothing The apis can be marked idempotent: * ResourceManagerAdministrationProtocol: 1. refreshQueues 2. refreshNodes 3. refreshSuperUserGroupsConfiguration 4. refreshUserToGroupsMappings 5. refreshAdminAcls 6. refreshServiceAcls * ResourceTracker: 1. registerNodeManager 2. nodeHeartbeat * ApplicationClientProtocol 1. getNewApplication 2. forceKillApplication 3. getApplicationReport 4. getClusterMetrics 5. getApplications 6. getClusterNodes 7. getQueueInfo 8. getQueueUserAcls 9. getApplicationAttemptReport 10. getApplicationAttempts 11. getContainerReport 12. getContainers 13. getDelegationToken These APIs can be marked as AtMostOnce: * ApplicationClientProtocol: 1. submitApplication These apis do not need to add any annotations * ResourceManagerAdministrationProtocol 1. updateNodeResource: Do not find any places using it. Maybe some other applications will use this API. Possibly can be marked idempotent. * ApplicationMasterProtocol 1. registerApplicationMaster 2. finishApplicationMaster 3. allocate * ApplicationClientProtocol: 1. moveApplicationAcrossQueues: Do not support yet There are two APIs which are not in this list: 1. renewDelegationToken 2. cancelDelegationToken We can add the proper annotations for them in https://issues.apache.org/jira/browse/YARN-1776 Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942555#comment-13942555 ] Zhijie Shen commented on YARN-1521: --- bq. 2. cancelDelegationToken The situation of cancelDelegationToken is still different from that of renewDelegationToken. It is summarized in YARN-1860. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935274#comment-13935274 ] Zhijie Shen commented on YARN-1521: --- Just a reminder, ApplicationClientProtocol has four more methods, whose idempotency needs to be verified as well: 1. getApplicationAttemptReport 2. getApplicationAttempts 3. getContainerReport 4. getContainers Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935336#comment-13935336 ] Tsuyoshi OZAWA commented on YARN-1521: -- We should introduce RetryCache to avoid returning incorrect errors to client if non-idempotent RPC(ie. submitApplication) are executed. Opened YARN-1836 for this. Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935821#comment-13935821 ] Xuan Gong commented on YARN-1521: - But first of all, we need to find which apis can be marked as Idempotent Here is the list of APIs that I think we can mark as Idempotent: * ResourceTracker ** registerNodeManager ** nodeHeartbeat * ResourceManagerAdministrationProtocol ** refreshQueues ** refreshNodes ** refreshSuperUserGroupsConfiguration ** refreshUserToGroupsMappings ** refreshAdminAcls ** refreshServiceAcls * ApplicationClientProtocol ** forceKillApplication ** getApplicationReport (already marked) ** getClusterMetrics ** getApplications ** getClusterNodes ** getQueueInfo ** getQueueUserAcls ** getApplicationAttemptReport ** getApplicationAttempts ** getContainerReport ** getContainers Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935873#comment-13935873 ] Xuan Gong commented on YARN-1521: - APIs are not in previous list: * ApplicationMasterProtocol ** registerApplicationMaster ** finishApplicationMaster ** allocate * ResourceManagerAdministrationProtocol ** updateNodeResource * ApplicationClientProtocol ** getNewApplication ** submitApplication ** getDelegationToken ** renewDelegationToken ** cancelDelegationToken ** moveApplicationAcrossQueues Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation - Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859206#comment-13859206 ] Karthik Kambatla commented on YARN-1521: I don't know the contracts on each of these APIs well enough (yet) to say whether they are idempotent - it would be nice for someone like [~vinodkv] to comment on this. I think allocate(), finishApplicationMaster(), getQueueInfo() are idempotent. If not already, get/renewDelegationToken(), getCluster*() should also be (made) idempotent. Other items in the list sound okay to me. Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13858604#comment-13858604 ] Xuan Gong commented on YARN-1521: - Thanks for the explanation, [~bikassaha] Here is the new list : The ops can be idempotent: * ApplicationClientProtocol: ** forceKillApplication() ** getApplicationReport() ** getApplications() ** getQueueUserAclInfo() ** cancelDelegationToken() * ResourceManagerAdministrationProtocol: ** refreshQueues() ** refreshNodes() ** refreshSuperUserGroupsConfiguration() ** RefreshUserToGroupsMappingsRequest() ** refreshAdminAcls() ** refreshServiceAcls() * ResourceTracker ** registerNodeManager() ** nodeHeartbeat() The ops do not need to be idempotent: * ApplicationClientProtocol: ** getQueueInfo() ** getNewApplication() ** submitApplication() ** renewDelegationToken() ** getDelegationToken() * ApplicationMasterProtocol: ** registerApplicationMaster() ** finishApplicationMaster() ** allocate() The ops I am not sure about: * ApplicationClientProtocol: ** getClusterMetrics() --- probably idempotent ** getClusterNodes() --- probably idempotent * ResourceManagerAdministrationProtocol ** updateNodeResource() --- probably idempotent All those ops that I am not sure is related to NM. So, when failover happens, a new RM will become active. This RM may not get all NM immediately (We need to wait for NM to do the heartbeat to reach a new RM) until all the NMs re-connect to a new RM. Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857315#comment-13857315
]
Xuan Gong commented on YARN-1521:
-
an idempotent operation is one that has no additional effect if it is called
more than once with the same input parameters. That means no matter how many
times you call the operation with the same parameter, the result will be the
same. In our HA case, i think that it means no matter which RM is active, by
calling the operations, we will get the same result.
Based on this, here is my proposal:
{code}
In ApplicationClientProtocol :
{code}
a. getNewApplication(), submitApplication(), forceKillApplication(),
getApplicationReport(),
getApplications(),getQueueUserAclInfo(),getDelegationToken(),renewDelegationToken(),
cancelDelegationToken() need to add idempotent annotation.
When RM becomes Active, it will read back all the Applications, and all related
DelegationTokens back from the zk, in that case, no matter which RM is active,
it will get all the information for all the applications, either running or
finished. That is the reason why we can always get the same results if we call
getApplicationReport(),
getApplications(),getDelegationToken(),renewDelegationToken(),
cancelDelegationToken(). So, the idempotent annotation need to be added for
those operations.
For forceKillApplication(), getNewApplication(), submitApplication(), those
operation need to add idempotent annotation too. Because, when the user calls
them, no matter which RM is active, the application will be submitted or killed.
For getQueueUserAclInfo(), YARN-1459 will guarantee that other RMs will get the
latest configuration if one of RMs are changed its configuration. In this case,
when getQueueUserAclInfo is called, we will always get the same results no
matter which RM is active.
b. getClusterMetrics(), getClusterNodes(),getQueueInfo() do not need to add.
I do not want to add idempotent annotation. Because those are based on the
Nodes information or queue information. The current active RM has the resource
information about all the NMs, and queue info for all the applications. But
when failover happens, the next active RMs will not have all the NMs
information immediately, and it does not have any queue info (such as which
application in which queue).
{code}
In ResourceTracker:
{code}
a.registerNodeManager needs to add idempotent annotation.
Same reason why I want to add idempotent annotation for submitApplication()
b. nodeHeartbeat does not need to add
The new active RM does not know all the NMs immediately.
{code}
In ApplicationMasterProtocol
{code}
Does not need to add for all the methods in ApplicationMasterProtocol.
There are three operations: registerApplicationMaster, finishApplicationMaster,
allocate(). When the failover happens, a new RM will start a new Attempt for
the running AM (just like the RM restart).
{code}
In ResourceManagerAdministrationProtocol
{code}
a.refreshQueues(), refreshNodes(),
refreshSuperUserGroupsConfiguration(),RefreshUserToGroupsMappingsRequest(),
refreshAdminAcls(), refreshServiceAcls() need to add idempotent annotation.
YARN-1459 will guarantee that other RMs will get the latest configuration if
one of RMs are changed its configuration.
b.updateNodeResource() do not need to add
The new active RM does not know all the NMs immediately.
Mark appropriate protocol methods with the idempotent annotation
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857318#comment-13857318 ] Xuan Gong commented on YARN-1521: - [~bikassaha], [~vinodkv], [~kkambatl] Comments, please Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857328#comment-13857328 ] Karthik Kambatla commented on YARN-1521: bq. an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters. That means no matter how many times you call the operation with the same parameter, the result will be the same. In our HA case, i think that it means no matter which RM is active, by calling the operations, we will get the same result. I am not sure if this is completely accurate. Rephrasing the definition from wiki - an idempotent operation can be applied multiple times without changing the result beyond the initial application. The relevance to that in the context of HA and client failover, I believe, has to do with whether an operation can be retried. For instance, submitting an application is not idempotent. Suppose a client is in the process of submitting an application to the Active (say, RM1) and RM1 goes down. RM2 takes over. Now, we don't know the status of the app submission and would be wrong to retry submitting to RM2. In this case, the client shouldn't failover to RM2. Unfortunately, there might be retries in MR and YARN without checking for idempotence or status of the previous attempt. I haven't run into any issues yet, but I think it would be nice to handle all of them cleanly. Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857358#comment-13857358 ] Bikas Saha commented on YARN-1521: -- After the first application of an idempotent operation, the subsequent application of the operation should not change the state of the system in any substantial manner. So its not clear to me how submitApplication() is idempotent. If the first operation succeeds then the second must fail. getNewApplication() will create a new application id on the RM for every invocation. It may be harmless to retry but its probably not idempotent. renewDelegationToken() - again harmless to retry but probably not idempotent. killApplication() is idempotent. getApplicationReport() is idempotent. getDelegationToken() - probably not since it results in creation of a new delegation token. Then there is the point of which operations are transferable via retries across RM failover. Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
