[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[ https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645589#comment-13645589 ] Hudson commented on YARN-599: - Integrated in Hadoop-Mapreduce-trunk #1415 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1415/]) YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager to separate out various validation checks depending on whether they rely on RM configuration or not. Contributed by Zhijie Shen. (Revision 1477478) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477478 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > Refactoring submitApplication in ClientRMService and RMAppManager > - > > Key: YARN-599 > URL: https://issues.apache.org/jira/browse/YARN-599 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: YARN-599.1.patch, YARN-599.2.patch > > > Currently, ClientRMService#submitApplication call RMAppManager#handle, and > consequently call RMAppMangager#submitApplication directly, though the code > looks like scheduling an APP_SUBMIT event. > In addition, the validation code before creating an RMApp instance is not > well organized. Ideally, the dynamic validation, which depends on the RM's > configuration, should be put in RMAppMangager#submitApplication. > RMAppMangager#submitApplication is called by > ClientRMService#submitApplication and RMAppMangager#recover. Since the > configuration may be changed after RM restarts, the validation needs to be > done again even in recovery mode. Therefore, resource request validation, > which based on min/max resource limits, should be moved from > ClientRMService#submitApplication to RMAppMangager#submitApplication. On the > other hand, the static validation, which is independent of the RM's > configuration should be put in ClientRMService#submitApplication, because it > is only need to be done once during the first submission. > Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. > RMAppMangager#submitApplication has a flaw is not synchronized. If two > application submissions with the same application ID enter the function, and > one progresses to the completion of RMApp instantiation, and the other > progresses the completion of putting the RMApp instance into rmContext, the > slower submission will cause an exception due to the duplicate application > ID. However, the exception will cause the RMApp instance already in rmContext > (belongs to the faster submission) being rejected with the current code flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[ https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645550#comment-13645550 ] Hudson commented on YARN-599: - Integrated in Hadoop-Hdfs-trunk #1388 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1388/]) YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager to separate out various validation checks depending on whether they rely on RM configuration or not. Contributed by Zhijie Shen. (Revision 1477478) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477478 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > Refactoring submitApplication in ClientRMService and RMAppManager > - > > Key: YARN-599 > URL: https://issues.apache.org/jira/browse/YARN-599 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: YARN-599.1.patch, YARN-599.2.patch > > > Currently, ClientRMService#submitApplication call RMAppManager#handle, and > consequently call RMAppMangager#submitApplication directly, though the code > looks like scheduling an APP_SUBMIT event. > In addition, the validation code before creating an RMApp instance is not > well organized. Ideally, the dynamic validation, which depends on the RM's > configuration, should be put in RMAppMangager#submitApplication. > RMAppMangager#submitApplication is called by > ClientRMService#submitApplication and RMAppMangager#recover. Since the > configuration may be changed after RM restarts, the validation needs to be > done again even in recovery mode. Therefore, resource request validation, > which based on min/max resource limits, should be moved from > ClientRMService#submitApplication to RMAppMangager#submitApplication. On the > other hand, the static validation, which is independent of the RM's > configuration should be put in ClientRMService#submitApplication, because it > is only need to be done once during the first submission. > Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. > RMAppMangager#submitApplication has a flaw is not synchronized. If two > application submissions with the same application ID enter the function, and > one progresses to the completion of RMApp instantiation, and the other > progresses the completion of putting the RMApp instance into rmContext, the > slower submission will cause an exception due to the duplicate application > ID. However, the exception will cause the RMApp instance already in rmContext > (belongs to the faster submission) being rejected with the current code flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[ https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645451#comment-13645451 ] Hudson commented on YARN-599: - Integrated in Hadoop-Yarn-trunk #199 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/199/]) YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager to separate out various validation checks depending on whether they rely on RM configuration or not. Contributed by Zhijie Shen. (Revision 1477478) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477478 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > Refactoring submitApplication in ClientRMService and RMAppManager > - > > Key: YARN-599 > URL: https://issues.apache.org/jira/browse/YARN-599 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: YARN-599.1.patch, YARN-599.2.patch > > > Currently, ClientRMService#submitApplication call RMAppManager#handle, and > consequently call RMAppMangager#submitApplication directly, though the code > looks like scheduling an APP_SUBMIT event. > In addition, the validation code before creating an RMApp instance is not > well organized. Ideally, the dynamic validation, which depends on the RM's > configuration, should be put in RMAppMangager#submitApplication. > RMAppMangager#submitApplication is called by > ClientRMService#submitApplication and RMAppMangager#recover. Since the > configuration may be changed after RM restarts, the validation needs to be > done again even in recovery mode. Therefore, resource request validation, > which based on min/max resource limits, should be moved from > ClientRMService#submitApplication to RMAppMangager#submitApplication. On the > other hand, the static validation, which is independent of the RM's > configuration should be put in ClientRMService#submitApplication, because it > is only need to be done once during the first submission. > Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. > RMAppMangager#submitApplication has a flaw is not synchronized. If two > application submissions with the same application ID enter the function, and > one progresses to the completion of RMApp instantiation, and the other > progresses the completion of putting the RMApp instance into rmContext, the > slower submission will cause an exception due to the duplicate application > ID. However, the exception will cause the RMApp instance already in rmContext > (belongs to the faster submission) being rejected with the current code flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[ https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645300#comment-13645300 ] Hudson commented on YARN-599: - Integrated in Hadoop-trunk-Commit #3698 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3698/]) YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager to separate out various validation checks depending on whether they rely on RM configuration or not. Contributed by Zhijie Shen. (Revision 1477478) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477478 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > Refactoring submitApplication in ClientRMService and RMAppManager > - > > Key: YARN-599 > URL: https://issues.apache.org/jira/browse/YARN-599 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: YARN-599.1.patch, YARN-599.2.patch > > > Currently, ClientRMService#submitApplication call RMAppManager#handle, and > consequently call RMAppMangager#submitApplication directly, though the code > looks like scheduling an APP_SUBMIT event. > In addition, the validation code before creating an RMApp instance is not > well organized. Ideally, the dynamic validation, which depends on the RM's > configuration, should be put in RMAppMangager#submitApplication. > RMAppMangager#submitApplication is called by > ClientRMService#submitApplication and RMAppMangager#recover. Since the > configuration may be changed after RM restarts, the validation needs to be > done again even in recovery mode. Therefore, resource request validation, > which based on min/max resource limits, should be moved from > ClientRMService#submitApplication to RMAppMangager#submitApplication. On the > other hand, the static validation, which is independent of the RM's > configuration should be put in ClientRMService#submitApplication, because it > is only need to be done once during the first submission. > Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. > RMAppMangager#submitApplication has a flaw is not synchronized. If two > application submissions with the same application ID enter the function, and > one progresses to the completion of RMApp instantiation, and the other > progresses the completion of putting the RMApp instance into rmContext, the > slower submission will cause an exception due to the duplicate application > ID. However, the exception will cause the RMApp instance already in rmContext > (belongs to the faster submission) being rejected with the current code flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[ https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645288#comment-13645288 ] Vinod Kumar Vavilapalli commented on YARN-599: -- Hm, it isn't straight-forward to figure that failures during RMAppManager.submitApplication() are properly put in Audit logs. But they are, I just verified. The latest patch looks good to me. +1, checking it in.. > Refactoring submitApplication in ClientRMService and RMAppManager > - > > Key: YARN-599 > URL: https://issues.apache.org/jira/browse/YARN-599 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-599.1.patch, YARN-599.2.patch > > > Currently, ClientRMService#submitApplication call RMAppManager#handle, and > consequently call RMAppMangager#submitApplication directly, though the code > looks like scheduling an APP_SUBMIT event. > In addition, the validation code before creating an RMApp instance is not > well organized. Ideally, the dynamic validation, which depends on the RM's > configuration, should be put in RMAppMangager#submitApplication. > RMAppMangager#submitApplication is called by > ClientRMService#submitApplication and RMAppMangager#recover. Since the > configuration may be changed after RM restarts, the validation needs to be > done again even in recovery mode. Therefore, resource request validation, > which based on min/max resource limits, should be moved from > ClientRMService#submitApplication to RMAppMangager#submitApplication. On the > other hand, the static validation, which is independent of the RM's > configuration should be put in ClientRMService#submitApplication, because it > is only need to be done once during the first submission. > Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. > RMAppMangager#submitApplication has a flaw is not synchronized. If two > application submissions with the same application ID enter the function, and > one progresses to the completion of RMApp instantiation, and the other > progresses the completion of putting the RMApp instance into rmContext, the > slower submission will cause an exception due to the duplicate application > ID. However, the exception will cause the RMApp instance already in rmContext > (belongs to the faster submission) being rejected with the current code flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[
https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645248#comment-13645248
]
Hadoop QA commented on YARN-599:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581118/YARN-599.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/844//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/844//console
This message is automatically generated.
> Refactoring submitApplication in ClientRMService and RMAppManager
> -
>
> Key: YARN-599
> URL: https://issues.apache.org/jira/browse/YARN-599
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-599.1.patch, YARN-599.2.patch
>
>
> Currently, ClientRMService#submitApplication call RMAppManager#handle, and
> consequently call RMAppMangager#submitApplication directly, though the code
> looks like scheduling an APP_SUBMIT event.
> In addition, the validation code before creating an RMApp instance is not
> well organized. Ideally, the dynamic validation, which depends on the RM's
> configuration, should be put in RMAppMangager#submitApplication.
> RMAppMangager#submitApplication is called by
> ClientRMService#submitApplication and RMAppMangager#recover. Since the
> configuration may be changed after RM restarts, the validation needs to be
> done again even in recovery mode. Therefore, resource request validation,
> which based on min/max resource limits, should be moved from
> ClientRMService#submitApplication to RMAppMangager#submitApplication. On the
> other hand, the static validation, which is independent of the RM's
> configuration should be put in ClientRMService#submitApplication, because it
> is only need to be done once during the first submission.
> Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw.
> RMAppMangager#submitApplication has a flaw is not synchronized. If two
> application submissions with the same application ID enter the function, and
> one progresses to the completion of RMApp instantiation, and the other
> progresses the completion of putting the RMApp instance into rmContext, the
> slower submission will cause an exception due to the duplicate application
> ID. However, the exception will cause the RMApp instance already in rmContext
> (belongs to the faster submission) being rejected with the current code flow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[ https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639848#comment-13639848 ] Zhijie Shen commented on YARN-599: -- In YARN-599.1.patch, there're the following changes: 1. ClientRMService#submitApplication calls RMAppManager#submitApplication directly. APP_SUBMIT event is removed at all. RMAppManager#submitApplication throws YarnRemoteException. 2. Move getCurrentUser and validateResourceRequest from ClientRMService#submitApplication to RMAppManager#submitApplication. Move getQueue and getApplicationName from RMAppManager#submitApplication to ClientRMService#submitApplication. Adjust the test cases in TestClientRMService and TestAppManger accordingly. 3. Refactor try-catch block in RMAppManager#submitApplication to avoid sending APP_REJECTED event to existing app in rmContext given duplicate applicateId. 4. Refactor TestAppManger to extract common part of the tests and push them to setup(). > Refactoring submitApplication in ClientRMService and RMAppManager > - > > Key: YARN-599 > URL: https://issues.apache.org/jira/browse/YARN-599 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-599.1.patch > > > Currently, ClientRMService#submitApplication call RMAppManager#handle, and > consequently call RMAppMangager#submitApplication directly, though the code > looks like scheduling an APP_SUBMIT event. > In addition, the validation code before creating an RMApp instance is not > well organized. Ideally, the dynamic validation, which depends on the RM's > configuration, should be put in RMAppMangager#submitApplication. > RMAppMangager#submitApplication is called by > ClientRMService#submitApplication and RMAppMangager#recover. Since the > configuration may be changed after RM restarts, the validation needs to be > done again even in recovery mode. Therefore, resource request validation, > which based on min/max resource limits, should be moved from > ClientRMService#submitApplication to RMAppMangager#submitApplication. On the > other hand, the static validation, which is independent of the RM's > configuration should be put in ClientRMService#submitApplication, because it > is only need to be done once during the first submission. > Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. > RMAppMangager#submitApplication has a flaw is not synchronized. If two > application submissions with the same application ID enter the function, and > one progresses to the completion of RMApp instantiation, and the other > progresses the completion of putting the RMApp instance into rmContext, the > slower submission will cause an exception due to the duplicate application > ID. However, the exception will cause the RMApp instance already in rmContext > (belongs to the faster submission) being rejected with the current code flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager
[
https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638596#comment-13638596
]
Hadoop QA commented on YARN-599:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579931/YARN-599.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/799//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/799//console
This message is automatically generated.
> Refactoring submitApplication in ClientRMService and RMAppManager
> -
>
> Key: YARN-599
> URL: https://issues.apache.org/jira/browse/YARN-599
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-599.1.patch
>
>
> Currently, ClientRMService#submitApplication call RMAppManager#handle, and
> consequently call RMAppMangager#submitApplication directly, though the code
> looks like scheduling an APP_SUBMIT event.
> In addition, the validation code before creating an RMApp instance is not
> well organized. Ideally, the dynamic validation, which depends on the RM's
> configuration, should be put in RMAppMangager#submitApplication.
> RMAppMangager#submitApplication is called by
> ClientRMService#submitApplication and RMAppMangager#recover. Since the
> configuration may be changed after RM restarts, the validation needs to be
> done again even in recovery mode. Therefore, resource request validation,
> which based on min/max resource limits, should be moved from
> ClientRMService#submitApplication to RMAppMangager#submitApplication. On the
> other hand, the static validation, which is independent of the RM's
> configuration should be put in ClientRMService#submitApplication, because it
> is only need to be done once during the first submission.
> Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw.
> RMAppMangager#submitApplication has a flaw is not synchronized. If two
> application submissions with the same application ID enter the function, and
> one progresses to the completion of RMApp instantiation, and the other
> progresses the completion of putting the RMApp instance into rmContext, the
> slower submission will cause an exception due to the duplicate application
> ID. However, the exception will cause the RMApp instance already in rmContext
> (belongs to the faster submission) being rejected with the current code flow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
