[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057354#comment-14057354 ] Hudson commented on YARN-1366: -- FAILURE: Integrated in Hadoop-Yarn-trunk #609 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/609/]) YARN-1366. Changed AMRMClient to re-register with RM and send outstanding requests back to RM on work-preserving RM restart. Contributed by Rohith (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1609254) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/resources/core-site.xml AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Fix For: 2.5.0 Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.13.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057472#comment-14057472 ] Hudson commented on YARN-1366: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1800 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1800/]) YARN-1366. Changed AMRMClient to re-register with RM and send outstanding requests back to RM on work-preserving RM restart. Contributed by Rohith (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1609254) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/resources/core-site.xml AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Fix For: 2.5.0 Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.13.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057539#comment-14057539 ] Hudson commented on YARN-1366: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1827 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1827/]) YARN-1366. Changed AMRMClient to re-register with RM and send outstanding requests back to RM on work-preserving RM restart. Contributed by Rohith (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1609254) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/resources/core-site.xml AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Fix For: 2.5.0 Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.13.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056570#comment-14056570 ] Hudson commented on YARN-1366: -- FAILURE: Integrated in Hadoop-trunk-Commit #5850 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5850/]) YARN-1366. Changed AMRMClient to re-register with RM and send outstanding requests back to RM on work-preserving RM restart. Contributed by Rohith (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1609254) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/resources/core-site.xml AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Fix For: 2.5.0 Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.13.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054839#comment-14054839
]
Hadoop QA commented on YARN-1366:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654564/YARN-1366.13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4218//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4218//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.10.patch,
YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.13.patch,
YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch,
YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch,
YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055200#comment-14055200 ] Jian He commented on YARN-1366: --- lgtm AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.13.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053200#comment-14053200
]
Jian He commented on YARN-1366:
---
Can you also clarify why AMRMClient automatically does re-register ?
ask list is an internal object which may not be obvious for outside users.
how about container requests?
{code}
then the re-register may end up sending the entire ask list
to the RM (including matched requests).
{code}
Looks good to me overall, [~bikassaha], any more comments?
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.10.patch,
YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053201#comment-14053201 ] Bikas Saha commented on YARN-1366: -- bq. I meant an empty response. After this does AMRMClientASync need to see AMCommand.RE_SYNC? That should be dead code right? AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053210#comment-14053210 ] Jian He commented on YARN-1366: --- right, we can remove that too AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051141#comment-14051141
]
Hadoop QA commented on YARN-1366:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653772/YARN-1366.12.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4189//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4189//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.10.patch,
YARN-1366.11.patch, YARN-1366.12.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050913#comment-14050913 ] Jian He commented on YARN-1366: --- looks good, +1 AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050965#comment-14050965
]
Bikas Saha commented on YARN-1366:
--
Why are we returning the old allocateResponse to the user? What is the user
expected to do with this allocateResponse that has a RESYNC command in it?
Should we make a second call to allocate (after re-registering) and then send
that response back up to the user?
{code}+// re register with RM
+registerApplicationMaster();
+return allocateResponse;
+ }{code}
There needs to be some clear documentation that if the user has not removed
container requests that have already been satisfied, then the re-register may
end up sending the entire ask list to the RM (including matched requests).
Which would mean the RM could end up giving it a lot of new allocated
containers.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.10.patch,
YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch,
YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch,
YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch,
YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050975#comment-14050975 ] Jian He commented on YARN-1366: --- bq. Should we make a second call to allocate (after re-registering) and then send that response back up to the user? Followup jira YARN-2209 will replace the resync command with a proper exception, in which case the returned allocate response should be null. bq. There needs to be some clear documentation make sense, Rohith, can you add some documentation about this , thx! AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050977#comment-14050977 ] Jian He commented on YARN-1366: --- bq. in which case the returned allocate response should be null I meant an empty response.. AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050978#comment-14050978 ] Bikas Saha commented on YARN-1366: -- Does a null response make sense for the user? AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051013#comment-14051013 ] Rohith commented on YARN-1366: -- bq. can you add some documentation about this Shall I add in java doc for AMRMClient#allocate()..? If it is in information guide, Could you please guide me how to add documentation.? AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051049#comment-14051049 ] Jian He commented on YARN-1366: --- add java doc for AMRMClient#allocate(). should be enough. bq. Does a null response make sense for the user? doing one more allocate makes the response more consistent. AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.10.patch, YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048567#comment-14048567
]
Hadoop QA commented on YARN-1366:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653324/YARN-1366.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4155//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4155//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048574#comment-14048574
]
Hadoop QA commented on YARN-1366:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653326/YARN-1366.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4156//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4156//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049167#comment-14049167
]
Jian He commented on YARN-1366:
---
- SecurityUtil.java loads configurations during class loading.
I see.
Patch looks good to me, just two more minor comments:
- These two synchronized block can be merged into one ?
{code}
synchronized (this) {
// reset lastResponseId to 0
lastResponseId = 0;
release.addAll(this.pendingRelease);
blacklistAdditions.addAll(this.blacklistedNodes);
}
// re register with RM
registerApplicationMaster();
synchronized (this) {
for (MapString, TreeMapResource, ResourceRequestInfo rr :
remoteRequestsTable
.values()) {
for (MapResource, ResourceRequestInfo capabalities : rr.values())
{
for (ResourceRequestInfo request : capabalities.values()) {
addResourceRequestToAsk(request.remoteRequest);
}
}
}
}
{code}
- The following reset of responseId in unregisterApplicationMaster is not
needed?
{code}
synchronized (this) {
// reset lastResponseId to 0
lastResponseId = 0;
}
{code}
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049563#comment-14049563 ] Rohith commented on YARN-1366: -- bq. These two synchronized block can be merged into one ? This I separated intentionally for the handling the very corner scenario i.e after AM gets resync it go for re registering the AM. By worst case, with this period of time, if again RM goes down, then registerapplicationmaster start retry both RM's. Thought not to block AMRMClient oprations such as updateblacklist,addContainerRequest and others so on... Would you think time taken to retry is not more and it can be blocked? AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049572#comment-14049572
]
Jian He commented on YARN-1366:
---
I meant, can we do this ?
{code}
synchronized (this) {
// reset lastResponseId to 0
lastResponseId = 0;
release.addAll(this.pendingRelease);
blacklistAdditions.addAll(this.blacklistedNodes);
for (MapString, TreeMapResource, ResourceRequestInfo rr :
remoteRequestsTable
.values()) {
for (MapResource, ResourceRequestInfo capabalities : rr.values())
{
for (ResourceRequestInfo request : capabalities.values()) {
addResourceRequestToAsk(request.remoteRequest);
}
}
}
}
// re register with RM
registerApplicationMaster();
{code}
and lastResponseId = 0; may be put in registerApplicationMaster call also ?
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.9.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049610#comment-14049610
]
Hadoop QA commented on YARN-1366:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653530/YARN-1366.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4171//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4171//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4171//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.10.patch,
YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch,
YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch, YARN-1366.9.patch,
YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049646#comment-14049646
]
Hadoop QA commented on YARN-1366:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653538/YARN-1366.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4173//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4173//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.10.patch,
YARN-1366.11.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch,
YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.8.patch,
YARN-1366.9.patch, YARN-1366.patch, YARN-1366.prototype.patch,
YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047412#comment-14047412
]
Rohith commented on YARN-1366:
--
Thank you for reviewing patch. I will update patch soon.
One update on the comment,
bq.testAMRMClientResendsRequestsOnRMRestart seems not testing re-sending
pendingReleases across RM restart, because the pending releases seems already
decremented to zero before restart happens
The test does verification of pending release too. Before restart , 1st
container is released. Below code releases only 1 container. One approach is
to make test stronger, number of container can be asserted before/after this
code.
{noformat}
int pendingRelease = 0;
IteratorContainer it = allocatedContainers.iterator();
while (it.hasNext()) {
amClient.releaseAssignedContainer(it.next().getId());
pendingRelease++;
it.remove();
break;// remove one container
}
{noformat}
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047432#comment-14047432 ] Jian He commented on YARN-1366: --- bq. The test does verification of pending release too I see. - We can move release.addAll(this.pendingRelease); to the first isResyncCommand check? similarly, blacklistAdditions.addAll(this.blacklistedNodes); - please add some comments to pendingRelease and release variables AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047718#comment-14047718
]
Hadoop QA commented on YARN-1366:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653162/YARN-1366.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4146//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4146//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.patch, YARN-1366.prototype.patch,
YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047962#comment-14047962
]
Jian He commented on YARN-1366:
---
Thanks for updating, some more comments:
- “blacklistRemovals.addAll(blacklistToRemove);”, we don't need to add this in
isResyncCommand check? as RM after restart will just forget all previously
blacklisted nodes.
- below code needs synchronize ?
{code}
for (MapString, TreeMapResource, ResourceRequestInfo rr :
remoteRequestsTable
.values()) {
for (MapResource, ResourceRequestInfo capabalities : rr.values()) {
for (ResourceRequestInfo request : capabalities.values()) {
addResourceRequestToAsk(request.remoteRequest);
}
}
}
{code}
- “isApplicationMasterRegistered = false;” not needed in allocate and
unregisterApplicationMaster.
- Instead of adding a new core-site.xml file, we can just set the config in the
test code conf object.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.patch, YARN-1366.prototype.patch,
YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048079#comment-14048079
]
Jian He commented on YARN-1366:
---
- we may check pendingRelease isEmpty as well to avoid unnecessary loops.
{code}
if (!allocateResponse.getCompletedContainersStatuses().isEmpty()) {
removePendingReleaseRequests(allocateResponse
.getCompletedContainersStatuses());
}
{code}
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.8.patch, YARN-1366.patch, YARN-1366.prototype.patch,
YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047072#comment-14047072
]
Hadoop QA commented on YARN-1366:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653045/YARN-1366.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4132//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4132//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047366#comment-14047366
]
Jian He commented on YARN-1366:
---
Thanks for working on the patch ! some comments:
- isApplicationMasterRegistered is actually not an argument, may be throw
ApplicationMasterNotRegsiteredException in this case ?
{code}
Preconditions.checkArgument(isApplicationMasterRegistered,
Application Master is trying to unregister before registering.);
{code}
- pom.xml format: use spaces instead of tabs
{code}
+dependency
+ groupIdorg.apache.hadoop/groupId
+ artifactIdhadoop-yarn-common/artifactId
+ typetest-jar/type
+ scopetest/scope
+ /dependency
{code}
- testAMRMClientResendsRequestsOnRMRestart seems not testing re-sending
pendingReleases across RM restart, because the pending releases seems already
decremented to zero before restart happens.
- Not related to this jira. Current ApplicationMasterService does not allow
multiple registers. Application may want to update its tracking url etc.
Should we make AMS accept multiple registers ?
{code} Preconditions.checkArgument(!isApplicationMasterRegistered,
ApplicationMaster is already registered);
{code}
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.7.patch,
YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046826#comment-14046826 ] Rohith commented on YARN-1366: -- Looking into fix findbug warning and test case. Will update patch once it is done. AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046900#comment-14046900
]
Hadoop QA commented on YARN-1366:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653008/YARN-1366.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4130//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4130//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4130//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.6.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[
https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046176#comment-14046176
]
Hadoop QA commented on YARN-1366:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652857/YARN-1366.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4121//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4121//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4121//console
This message is automatically generated.
AM should implement Resync with the ApplicationMasterService instead of
shutting down
-
Key: YARN-1366
URL: https://issues.apache.org/jira/browse/YARN-1366
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Bikas Saha
Assignee: Rohith
Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch,
YARN-1366.4.patch, YARN-1366.5.patch, YARN-1366.patch,
YARN-1366.prototype.patch, YARN-1366.prototype.patch
The ApplicationMasterService currently sends a resync response to which the
AM responds by shutting down. The AM behavior is expected to change to
calling resyncing with the RM. Resync means resetting the allocate RPC
sequence number to 0 and the AM should send its entire outstanding request to
the RM. Note that if the AM is making its first allocate call to the RM then
things should proceed like normal without needing a resync. The RM will
return all containers that have completed since the RM last synced with the
AM. Some container completions may be reported more than once.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043085#comment-14043085 ] Jian He commented on YARN-1366: --- btw, can you add necessary comments in the test body also, given it's a complicated test case. thx! AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043044#comment-14043044 ] Jian He commented on YARN-1366: --- Hi [~rohithsharma], start looking at the patch. it''s been a while, Mind updating the patch please ? thanks! some comment in the meanwhile: Skim through testAMRMClientResendsRequestsOnRMRestart, can you add comment above the test method to explain the steps involved? Ideally, the test should be 1.AMRMClient allocate some containers, 2. RM restarted before containers are allocated to AM. 3. On RM restart, previous container requests are automatically re-sent by AMRMClient on re-register. 4. assert the containers are allocated by the new RM. similarly for releaseList, blacklist. AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043059#comment-14043059 ] Rohith commented on YARN-1366: -- Thank you [~jianhe] for looking into patch :-) In current patch, test does above mentioned comments verification in an order. I will upload patch with current changes with yarn-1365 and mentioning test verification. AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.4.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013356#comment-14013356 ] Jian He commented on YARN-1366: --- The bulk of the patch here is MR changes. I think we should have a MR jira to track the MR changes? Both patches are very related and patch size seems reasonable to be consolidated. It's fine to leave as-is, but just easier for reviewer to have more context. AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013607#comment-14013607 ] Rohith commented on YARN-1366: -- Let this jira keep only for Yarn Client. I created MAPREDUCE-5910 for handling at MR. AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1366) AM should implement Resync with the ApplicationMasterService instead of shutting down
[ https://issues.apache.org/jira/browse/YARN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013281#comment-14013281 ] Anubhav Dhoot commented on YARN-1366: - Updated the title to more accurately reflect the changes covered in this jira AM should implement Resync with the ApplicationMasterService instead of shutting down - Key: YARN-1366 URL: https://issues.apache.org/jira/browse/YARN-1366 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Rohith Attachments: YARN-1366.1.patch, YARN-1366.2.patch, YARN-1366.3.patch, YARN-1366.patch, YARN-1366.prototype.patch, YARN-1366.prototype.patch The ApplicationMasterService currently sends a resync response to which the AM responds by shutting down. The AM behavior is expected to change to calling resyncing with the RM. Resync means resetting the allocate RPC sequence number to 0 and the AM should send its entire outstanding request to the RM. Note that if the AM is making its first allocate call to the RM then things should proceed like normal without needing a resync. The RM will return all containers that have completed since the RM last synced with the AM. Some container completions may be reported more than once. -- This message was sent by Atlassian JIRA (v6.2#6252)
