[jira] [Commented] (YARN-2988) Graph#save() may leak file descriptors
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258641#comment-14258641
]
Tsuyoshi OZAWA commented on YARN-2988:
--
[~ted_yu] Oops, I changed assignee wrongly. Sorry for that and thanks for your
fix.
> Graph#save() may leak file descriptors
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2988) Graph#save() may leak file descriptors
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned YARN-2988:
Assignee: Ted Yu (was: Tsuyoshi OZAWA)
> Graph#save() may leak file descriptors
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2988) Graph#save() may leak resource
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA reassigned YARN-2988:
Assignee: Tsuyoshi OZAWA (was: Ted Yu)
> Graph#save() may leak resource
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Tsuyoshi OZAWA
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2988) Graph#save() may leak resource
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258638#comment-14258638
]
Tsuyoshi OZAWA commented on YARN-2988:
--
Committing this to trunk and branch-2. Thanks [~ted_yu] for your reporting and
contribution!
> Graph#save() may leak resource
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2988) Graph#save() may leak file descriptors
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated YARN-2988:
-
Summary: Graph#save() may leak file descriptors (was: Graph#save() may
leak resource)
> Graph#save() may leak file descriptors
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Tsuyoshi OZAWA
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2988) Graph#save() may leak resource
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258635#comment-14258635
]
Hudson commented on YARN-2988:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6788 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6788/])
YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) (ozawa:
rev b256dd76006efbd4bcde3146a642fe0902d83dd2)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> Graph#save() may leak resource
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2988) Graph#save() may leak resource
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated YARN-2988:
-
Target Version/s: 2.7.0
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
> Graph#save() may leak resource
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2988) Graph#save() may leak resource
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258628#comment-14258628
]
Tsuyoshi OZAWA commented on YARN-2988:
--
+1
> Graph#save() may leak resource
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2988) Graph#save() may leak resource
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258580#comment-14258580
]
Hadoop QA commented on YARN-2988:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689073/YARN-2988-002.patch
against trunk revision 4f18018.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6189//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6189//console
This message is automatically generated.
> Graph#save() may leak resource
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2988) Graph#save() may leak resource
[
https://issues.apache.org/jira/browse/YARN-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated YARN-2988:
-
Attachment: YARN-2988-002.patch
How about this patch ?
> Graph#save() may leak resource
> --
>
> Key: YARN-2988
> URL: https://issues.apache.org/jira/browse/YARN-2988
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: YARN-2988-001.patch, YARN-2988-002.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/state/Graph.java
> :
> {code}
> public void save(String filepath) throws IOException {
> OutputStreamWriter fout = new OutputStreamWriter(
> new FileOutputStream(filepath), Charset.forName("UTF-8"));
> fout.write(generateGraphViz());
> fout.close();
> {code}
> The close of fout should be enclosed in finally clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[
https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258506#comment-14258506
]
Hadoop QA commented on YARN-2938:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689062/YARN-2938.003.patch
against trunk revision 4f18018.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6188//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6188//console
This message is automatically generated.
> Fix new findbugs warnings in hadoop-yarn-resourcemanager and
> hadoop-yarn-applicationhistoryservice
> --
>
> Key: YARN-2938
> URL: https://issues.apache.org/jira/browse/YARN-2938
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Fix For: 2.7.0
>
> Attachments: FindBugs Report.html, YARN-2938.001.patch,
> YARN-2938.002.patch, YARN-2938.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[ https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-2938: --- Attachment: YARN-2938.003.patch > Fix new findbugs warnings in hadoop-yarn-resourcemanager and > hadoop-yarn-applicationhistoryservice > -- > > Key: YARN-2938 > URL: https://issues.apache.org/jira/browse/YARN-2938 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.7.0 > > Attachments: FindBugs Report.html, YARN-2938.001.patch, > YARN-2938.002.patch, YARN-2938.003.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258337#comment-14258337 ] Hudson commented on YARN-2937: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2002 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2002/]) YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. (zjshen: rev 41a548a916d4248164cb9495320f123ec215d70e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java > Fix new findbugs warnings in hadoop-yarn-nodemanager > > > Key: YARN-2937 > URL: https://issues.apache.org/jira/browse/YARN-2937 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.7.0 > > Attachments: YARN-2937.001.patch, YARN-2937.002.patch, > YARN-2937.003.patch, YARN-2937.004.patch, YARN-2937.005.patch, > YARN-2937.006.patch, YARN-2937.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2940) Fix new findbugs warnings in rest of the hadoop-yarn components
[ https://issues.apache.org/jira/browse/YARN-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258335#comment-14258335 ] Hudson commented on YARN-2940: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2002 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2002/]) CHANGES.txt: add YARN-2940 (junping_du: rev 20674255ad891bb693ecb2faa5bdb37477073dac) * hadoop-yarn-project/CHANGES.txt YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) (junping_du: rev d468c9aaf15e977f12e6214b47996412c907e069) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/RegistryAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/QueueCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java > Fix new findbugs warnings in rest of the hadoop-yarn components > --- > > Key: YARN-2940 > URL: https://issues.apache.org/jira/browse/YARN-2940 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Li Lu > Fix For: 2.7.0 > > Attachments: YARN-2940-121014-1.patch, YARN-2940-121014.patch, > YARN-2940-122314.patch, findbugs_report_yarn.zip > > > Fix findbugs warnings in the following YARN components: > hadoop-yarn-applications-distributedshell > hadoop-yarn-applications-unmanaged-am-launcher > hadoop-yarn-server-web-proxy > hadoop-yarn-registry > hadoop-yarn-server-common > hadoop-yarn-client -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2970) NodeLabel operations in RMAdmin CLI get missing in help command.
[
https://issues.apache.org/jira/browse/YARN-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258334#comment-14258334
]
Hudson commented on YARN-2970:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2002 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2002/])
YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command.
(Contributed by Varun Saxena) (junping_du: rev
e1ee0d45ea3315e7fbe1d28a7a35a0ce0c54483b)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
> NodeLabel operations in RMAdmin CLI get missing in help command.
>
>
> Key: YARN-2970
> URL: https://issues.apache.org/jira/browse/YARN-2970
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Junping Du
>Assignee: Varun Saxena
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2970.001.patch, YARN-2970.patch
>
>
> NodeLabel operations in RMAdmin CLI get missing in help command when I am
> debugging YARN-313, we should add them on as other cmds:
> {noformat}
> yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshResources]
> [-refreshSuperUserGroupsConfiguration] [-refreshUserToGroupsMappings]
> [-refreshAdminAcls] [-refreshServiceAcl] [-getGroup [username]]
> [-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]) [-help
> [cmd]]
>-refreshQueues: Reload the queues' acls, states and scheduler specific
> properties.
> ResourceManager will reload the mapred-queues configuration
> file.
>-refreshNodes: Refresh the hosts information at the ResourceManager.
>-refreshResources: Refresh resources of NodeManagers at the
> ResourceManager.
>-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>-refreshUserToGroupsMappings: Refresh user-to-groups mappings
>-refreshAdminAcls: Refresh acls for administration of ResourceManager
>-refreshServiceAcl: Reload the service-level authorization policy file.
> ResoureceManager will reload the authorization policy file.
>-getGroups [username]: Get the groups which given user belongs to.
>-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]):
> Update resource on specific node.
>-help [cmd]: Displays help for the given command or all commands if none
> is specified.
>-addToClusterNodeLabels [label1,label2,label3] (label splitted by ","):
> add to cluster node labels
>-removeFromClusterNodeLabels [label1,label2,label3] (label splitted by
> ","): remove from cluster node labels
>-replaceLabelsOnNode [node1:port,label1,label2 node2:port,label1,label2]:
> replace labels on nodes
>-directlyAccessNodeLabelStore: Directly access node label store, with this
> option, all node label related operations will not connect RM. Instead, they
> will access/modify stored node labels directly. By default, it is false
> (access via RM). AND PLEASE NOTE: if you configured
> yarn.node-labels.fs-store.root-dir to a local directory (instead of NFS or
> HDFS), this option will only work when the command run on the machine where
> RM is running.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258332#comment-14258332 ] Hudson commented on YARN-2837: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2002 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2002/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, > YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLocks in RMStateStore<->ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258336#comment-14258336
]
Hudson commented on YARN-2946:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2002 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2002/])
YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith
Sharmaks (jianhe: rev 4f18018b7abb405f38d269da096ec9fd80c81d9a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreAMRMTokenEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTMasterKeyEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreEventType.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
> DeadLocks in RMStateStore<->ZKRMStateStore
> --
>
> Key: YARN-2946
> URL: https://issues.apache.org/jira/browse/YARN-2946
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Blocker
> Attachments: 0001-YARN-2946.patch, 0001-YARN-2946.patch,
> 0002-YARN-2946.patch, 0003-YARN-2946.patch, 0003-YARN-2946.patch,
> 0004-YARN-2946.patch, RM_BeforeFix_Deadlock_cycle_1.png,
> RM_BeforeFix_Deadlock_cycle_2.png, TestYARN2946.java
>
>
> Found one deadlock in ZKRMStateStore.
> # Initial stage zkClient is null because of zk disconnected event.
> # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
> re establish zookeeper connection either via synconnected or expired event,
> it is highly possible that any other thred can obtain lock on
> {{ZKRMStateStore.this}} from state machine transition events. This cause
> Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258320#comment-14258320 ] Hudson commented on YARN-2937: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #52 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/52/]) YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. (zjshen: rev 41a548a916d4248164cb9495320f123ec215d70e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java > Fix new findbugs warnings in hadoop-yarn-nodemanager > > > Key: YARN-2937 > URL: https://issues.apache.org/jira/browse/YARN-2937 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.7.0 > > Attachments: YARN-2937.001.patch, YARN-2937.002.patch, > YARN-2937.003.patch, YARN-2937.004.patch, YARN-2937.005.patch, > YARN-2937.006.patch, YARN-2937.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2940) Fix new findbugs warnings in rest of the hadoop-yarn components
[ https://issues.apache.org/jira/browse/YARN-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258317#comment-14258317 ] Hudson commented on YARN-2940: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #52 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/52/]) CHANGES.txt: add YARN-2940 (junping_du: rev 20674255ad891bb693ecb2faa5bdb37477073dac) * hadoop-yarn-project/CHANGES.txt YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) (junping_du: rev d468c9aaf15e977f12e6214b47996412c907e069) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/RegistryAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/QueueCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/CHANGES.txt > Fix new findbugs warnings in rest of the hadoop-yarn components > --- > > Key: YARN-2940 > URL: https://issues.apache.org/jira/browse/YARN-2940 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Li Lu > Fix For: 2.7.0 > > Attachments: YARN-2940-121014-1.patch, YARN-2940-121014.patch, > YARN-2940-122314.patch, findbugs_report_yarn.zip > > > Fix findbugs warnings in the following YARN components: > hadoop-yarn-applications-distributedshell > hadoop-yarn-applications-unmanaged-am-launcher > hadoop-yarn-server-web-proxy > hadoop-yarn-registry > hadoop-yarn-server-common > hadoop-yarn-client -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLocks in RMStateStore<->ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258319#comment-14258319
]
Hudson commented on YARN-2946:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #52 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/52/])
YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith
Sharmaks (jianhe: rev 4f18018b7abb405f38d269da096ec9fd80c81d9a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreAMRMTokenEvent.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreEventType.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTMasterKeyEvent.java
> DeadLocks in RMStateStore<->ZKRMStateStore
> --
>
> Key: YARN-2946
> URL: https://issues.apache.org/jira/browse/YARN-2946
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Blocker
> Attachments: 0001-YARN-2946.patch, 0001-YARN-2946.patch,
> 0002-YARN-2946.patch, 0003-YARN-2946.patch, 0003-YARN-2946.patch,
> 0004-YARN-2946.patch, RM_BeforeFix_Deadlock_cycle_1.png,
> RM_BeforeFix_Deadlock_cycle_2.png, TestYARN2946.java
>
>
> Found one deadlock in ZKRMStateStore.
> # Initial stage zkClient is null because of zk disconnected event.
> # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
> re establish zookeeper connection either via synconnected or expired event,
> it is highly possible that any other thred can obtain lock on
> {{ZKRMStateStore.this}} from state machine transition events. This cause
> Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2970) NodeLabel operations in RMAdmin CLI get missing in help command.
[
https://issues.apache.org/jira/browse/YARN-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258316#comment-14258316
]
Hudson commented on YARN-2970:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #52 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/52/])
YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command.
(Contributed by Varun Saxena) (junping_du: rev
e1ee0d45ea3315e7fbe1d28a7a35a0ce0c54483b)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
> NodeLabel operations in RMAdmin CLI get missing in help command.
>
>
> Key: YARN-2970
> URL: https://issues.apache.org/jira/browse/YARN-2970
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Junping Du
>Assignee: Varun Saxena
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2970.001.patch, YARN-2970.patch
>
>
> NodeLabel operations in RMAdmin CLI get missing in help command when I am
> debugging YARN-313, we should add them on as other cmds:
> {noformat}
> yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshResources]
> [-refreshSuperUserGroupsConfiguration] [-refreshUserToGroupsMappings]
> [-refreshAdminAcls] [-refreshServiceAcl] [-getGroup [username]]
> [-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]) [-help
> [cmd]]
>-refreshQueues: Reload the queues' acls, states and scheduler specific
> properties.
> ResourceManager will reload the mapred-queues configuration
> file.
>-refreshNodes: Refresh the hosts information at the ResourceManager.
>-refreshResources: Refresh resources of NodeManagers at the
> ResourceManager.
>-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>-refreshUserToGroupsMappings: Refresh user-to-groups mappings
>-refreshAdminAcls: Refresh acls for administration of ResourceManager
>-refreshServiceAcl: Reload the service-level authorization policy file.
> ResoureceManager will reload the authorization policy file.
>-getGroups [username]: Get the groups which given user belongs to.
>-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]):
> Update resource on specific node.
>-help [cmd]: Displays help for the given command or all commands if none
> is specified.
>-addToClusterNodeLabels [label1,label2,label3] (label splitted by ","):
> add to cluster node labels
>-removeFromClusterNodeLabels [label1,label2,label3] (label splitted by
> ","): remove from cluster node labels
>-replaceLabelsOnNode [node1:port,label1,label2 node2:port,label1,label2]:
> replace labels on nodes
>-directlyAccessNodeLabelStore: Directly access node label store, with this
> option, all node label related operations will not connect RM. Instead, they
> will access/modify stored node labels directly. By default, it is false
> (access via RM). AND PLEASE NOTE: if you configured
> yarn.node-labels.fs-store.root-dir to a local directory (instead of NFS or
> HDFS), this option will only work when the command run on the machine where
> RM is running.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258314#comment-14258314 ] Hudson commented on YARN-2837: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #52 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/52/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, > YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258285#comment-14258285 ] Hudson commented on YARN-2837: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #48 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/48/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, > YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258291#comment-14258291 ] Hudson commented on YARN-2937: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #48 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/48/]) YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. (zjshen: rev 41a548a916d4248164cb9495320f123ec215d70e) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java > Fix new findbugs warnings in hadoop-yarn-nodemanager > > > Key: YARN-2937 > URL: https://issues.apache.org/jira/browse/YARN-2937 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.7.0 > > Attachments: YARN-2937.001.patch, YARN-2937.002.patch, > YARN-2937.003.patch, YARN-2937.004.patch, YARN-2937.005.patch, > YARN-2937.006.patch, YARN-2937.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2940) Fix new findbugs warnings in rest of the hadoop-yarn components
[ https://issues.apache.org/jira/browse/YARN-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258288#comment-14258288 ] Hudson commented on YARN-2940: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #48 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/48/]) CHANGES.txt: add YARN-2940 (junping_du: rev 20674255ad891bb693ecb2faa5bdb37477073dac) * hadoop-yarn-project/CHANGES.txt YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) (junping_du: rev d468c9aaf15e977f12e6214b47996412c907e069) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/QueueCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/RegistryAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java > Fix new findbugs warnings in rest of the hadoop-yarn components > --- > > Key: YARN-2940 > URL: https://issues.apache.org/jira/browse/YARN-2940 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Li Lu > Fix For: 2.7.0 > > Attachments: YARN-2940-121014-1.patch, YARN-2940-121014.patch, > YARN-2940-122314.patch, findbugs_report_yarn.zip > > > Fix findbugs warnings in the following YARN components: > hadoop-yarn-applications-distributedshell > hadoop-yarn-applications-unmanaged-am-launcher > hadoop-yarn-server-web-proxy > hadoop-yarn-registry > hadoop-yarn-server-common > hadoop-yarn-client -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLocks in RMStateStore<->ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258289#comment-14258289
]
Hudson commented on YARN-2946:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #48 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/48/])
YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith
Sharmaks (jianhe: rev 4f18018b7abb405f38d269da096ec9fd80c81d9a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreAMRMTokenEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreEventType.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTMasterKeyEvent.java
> DeadLocks in RMStateStore<->ZKRMStateStore
> --
>
> Key: YARN-2946
> URL: https://issues.apache.org/jira/browse/YARN-2946
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Blocker
> Attachments: 0001-YARN-2946.patch, 0001-YARN-2946.patch,
> 0002-YARN-2946.patch, 0003-YARN-2946.patch, 0003-YARN-2946.patch,
> 0004-YARN-2946.patch, RM_BeforeFix_Deadlock_cycle_1.png,
> RM_BeforeFix_Deadlock_cycle_2.png, TestYARN2946.java
>
>
> Found one deadlock in ZKRMStateStore.
> # Initial stage zkClient is null because of zk disconnected event.
> # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
> re establish zookeeper connection either via synconnected or expired event,
> it is highly possible that any other thred can obtain lock on
> {{ZKRMStateStore.this}} from state machine transition events. This cause
> Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2970) NodeLabel operations in RMAdmin CLI get missing in help command.
[
https://issues.apache.org/jira/browse/YARN-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258287#comment-14258287
]
Hudson commented on YARN-2970:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #48 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/48/])
YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command.
(Contributed by Varun Saxena) (junping_du: rev
e1ee0d45ea3315e7fbe1d28a7a35a0ce0c54483b)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
> NodeLabel operations in RMAdmin CLI get missing in help command.
>
>
> Key: YARN-2970
> URL: https://issues.apache.org/jira/browse/YARN-2970
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Junping Du
>Assignee: Varun Saxena
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2970.001.patch, YARN-2970.patch
>
>
> NodeLabel operations in RMAdmin CLI get missing in help command when I am
> debugging YARN-313, we should add them on as other cmds:
> {noformat}
> yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshResources]
> [-refreshSuperUserGroupsConfiguration] [-refreshUserToGroupsMappings]
> [-refreshAdminAcls] [-refreshServiceAcl] [-getGroup [username]]
> [-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]) [-help
> [cmd]]
>-refreshQueues: Reload the queues' acls, states and scheduler specific
> properties.
> ResourceManager will reload the mapred-queues configuration
> file.
>-refreshNodes: Refresh the hosts information at the ResourceManager.
>-refreshResources: Refresh resources of NodeManagers at the
> ResourceManager.
>-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>-refreshUserToGroupsMappings: Refresh user-to-groups mappings
>-refreshAdminAcls: Refresh acls for administration of ResourceManager
>-refreshServiceAcl: Reload the service-level authorization policy file.
> ResoureceManager will reload the authorization policy file.
>-getGroups [username]: Get the groups which given user belongs to.
>-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]):
> Update resource on specific node.
>-help [cmd]: Displays help for the given command or all commands if none
> is specified.
>-addToClusterNodeLabels [label1,label2,label3] (label splitted by ","):
> add to cluster node labels
>-removeFromClusterNodeLabels [label1,label2,label3] (label splitted by
> ","): remove from cluster node labels
>-replaceLabelsOnNode [node1:port,label1,label2 node2:port,label1,label2]:
> replace labels on nodes
>-directlyAccessNodeLabelStore: Directly access node label store, with this
> option, all node label related operations will not connect RM. Instead, they
> will access/modify stored node labels directly. By default, it is false
> (access via RM). AND PLEASE NOTE: if you configured
> yarn.node-labels.fs-store.root-dir to a local directory (instead of NFS or
> HDFS), this option will only work when the command run on the machine where
> RM is running.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258276#comment-14258276 ] Hudson commented on YARN-2937: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1983 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1983/]) YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. (zjshen: rev 41a548a916d4248164cb9495320f123ec215d70e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java > Fix new findbugs warnings in hadoop-yarn-nodemanager > > > Key: YARN-2937 > URL: https://issues.apache.org/jira/browse/YARN-2937 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.7.0 > > Attachments: YARN-2937.001.patch, YARN-2937.002.patch, > YARN-2937.003.patch, YARN-2937.004.patch, YARN-2937.005.patch, > YARN-2937.006.patch, YARN-2937.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258271#comment-14258271 ] Hudson commented on YARN-2837: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1983 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1983/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, > YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2940) Fix new findbugs warnings in rest of the hadoop-yarn components
[ https://issues.apache.org/jira/browse/YARN-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258274#comment-14258274 ] Hudson commented on YARN-2940: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1983 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1983/]) CHANGES.txt: add YARN-2940 (junping_du: rev 20674255ad891bb693ecb2faa5bdb37477073dac) * hadoop-yarn-project/CHANGES.txt YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) (junping_du: rev d468c9aaf15e977f12e6214b47996412c907e069) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/RegistryAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/QueueCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java > Fix new findbugs warnings in rest of the hadoop-yarn components > --- > > Key: YARN-2940 > URL: https://issues.apache.org/jira/browse/YARN-2940 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Li Lu > Fix For: 2.7.0 > > Attachments: YARN-2940-121014-1.patch, YARN-2940-121014.patch, > YARN-2940-122314.patch, findbugs_report_yarn.zip > > > Fix findbugs warnings in the following YARN components: > hadoop-yarn-applications-distributedshell > hadoop-yarn-applications-unmanaged-am-launcher > hadoop-yarn-server-web-proxy > hadoop-yarn-registry > hadoop-yarn-server-common > hadoop-yarn-client -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLocks in RMStateStore<->ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258275#comment-14258275
]
Hudson commented on YARN-2946:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1983 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1983/])
YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith
Sharmaks (jianhe: rev 4f18018b7abb405f38d269da096ec9fd80c81d9a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreEventType.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTMasterKeyEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreAMRMTokenEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
* hadoop-yarn-project/CHANGES.txt
> DeadLocks in RMStateStore<->ZKRMStateStore
> --
>
> Key: YARN-2946
> URL: https://issues.apache.org/jira/browse/YARN-2946
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Blocker
> Attachments: 0001-YARN-2946.patch, 0001-YARN-2946.patch,
> 0002-YARN-2946.patch, 0003-YARN-2946.patch, 0003-YARN-2946.patch,
> 0004-YARN-2946.patch, RM_BeforeFix_Deadlock_cycle_1.png,
> RM_BeforeFix_Deadlock_cycle_2.png, TestYARN2946.java
>
>
> Found one deadlock in ZKRMStateStore.
> # Initial stage zkClient is null because of zk disconnected event.
> # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
> re establish zookeeper connection either via synconnected or expired event,
> it is highly possible that any other thred can obtain lock on
> {{ZKRMStateStore.this}} from state machine transition events. This cause
> Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2970) NodeLabel operations in RMAdmin CLI get missing in help command.
[
https://issues.apache.org/jira/browse/YARN-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258273#comment-14258273
]
Hudson commented on YARN-2970:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1983 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1983/])
YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command.
(Contributed by Varun Saxena) (junping_du: rev
e1ee0d45ea3315e7fbe1d28a7a35a0ce0c54483b)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
> NodeLabel operations in RMAdmin CLI get missing in help command.
>
>
> Key: YARN-2970
> URL: https://issues.apache.org/jira/browse/YARN-2970
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Junping Du
>Assignee: Varun Saxena
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2970.001.patch, YARN-2970.patch
>
>
> NodeLabel operations in RMAdmin CLI get missing in help command when I am
> debugging YARN-313, we should add them on as other cmds:
> {noformat}
> yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshResources]
> [-refreshSuperUserGroupsConfiguration] [-refreshUserToGroupsMappings]
> [-refreshAdminAcls] [-refreshServiceAcl] [-getGroup [username]]
> [-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]) [-help
> [cmd]]
>-refreshQueues: Reload the queues' acls, states and scheduler specific
> properties.
> ResourceManager will reload the mapred-queues configuration
> file.
>-refreshNodes: Refresh the hosts information at the ResourceManager.
>-refreshResources: Refresh resources of NodeManagers at the
> ResourceManager.
>-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>-refreshUserToGroupsMappings: Refresh user-to-groups mappings
>-refreshAdminAcls: Refresh acls for administration of ResourceManager
>-refreshServiceAcl: Reload the service-level authorization policy file.
> ResoureceManager will reload the authorization policy file.
>-getGroups [username]: Get the groups which given user belongs to.
>-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]):
> Update resource on specific node.
>-help [cmd]: Displays help for the given command or all commands if none
> is specified.
>-addToClusterNodeLabels [label1,label2,label3] (label splitted by ","):
> add to cluster node labels
>-removeFromClusterNodeLabels [label1,label2,label3] (label splitted by
> ","): remove from cluster node labels
>-replaceLabelsOnNode [node1:port,label1,label2 node2:port,label1,label2]:
> replace labels on nodes
>-directlyAccessNodeLabelStore: Directly access node label store, with this
> option, all node label related operations will not connect RM. Instead, they
> will access/modify stored node labels directly. By default, it is false
> (access via RM). AND PLEASE NOTE: if you configured
> yarn.node-labels.fs-store.root-dir to a local directory (instead of NFS or
> HDFS), this option will only work when the command run on the machine where
> RM is running.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2919) Potential race between renew and cancel in DelegationTokenRenwer
[
https://issues.apache.org/jira/browse/YARN-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258208#comment-14258208
]
Naganarasimha G R commented on YARN-2919:
-
Hi [~kasha]
Thanks for the review,
AFAIU you just want to add a atomic "canceling" state to
{{org.apache.hadoop.security.token.Token}} itself, and once
{{org.apache.hadoop.security.token.Token.cancel(Configuration)}} is invoked
"canceling" state of token will be set to true and when
{{org.apache.hadoop.security.token.Token.renew(Configuration)}} try to renew
the token we should check this state and stop the renew and return expiration
date less than the system time (or -1), right ?
Further based on this Token."cancelling" state
{{org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.RenewalTimerTask}}
can determine to renew and create new timer task?
> Potential race between renew and cancel in DelegationTokenRenwer
> -
>
> Key: YARN-2919
> URL: https://issues.apache.org/jira/browse/YARN-2919
> Project: Hadoop YARN
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Naganarasimha G R
>Priority: Critical
> Attachments: YARN-2919.20141209-1.patch
>
>
> YARN-2874 fixes a deadlock in DelegationTokenRenewer, but there is still a
> race because of which a renewal in flight isn't interrupted by a cancel.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258191#comment-14258191 ] Hudson commented on YARN-2837: -- FAILURE: Integrated in Hadoop-Yarn-trunk #785 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/785/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, > YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLocks in RMStateStore<->ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258195#comment-14258195
]
Hudson commented on YARN-2946:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #785 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/785/])
YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith
Sharmaks (jianhe: rev 4f18018b7abb405f38d269da096ec9fd80c81d9a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreAMRMTokenEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreEventType.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTMasterKeyEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
> DeadLocks in RMStateStore<->ZKRMStateStore
> --
>
> Key: YARN-2946
> URL: https://issues.apache.org/jira/browse/YARN-2946
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Blocker
> Attachments: 0001-YARN-2946.patch, 0001-YARN-2946.patch,
> 0002-YARN-2946.patch, 0003-YARN-2946.patch, 0003-YARN-2946.patch,
> 0004-YARN-2946.patch, RM_BeforeFix_Deadlock_cycle_1.png,
> RM_BeforeFix_Deadlock_cycle_2.png, TestYARN2946.java
>
>
> Found one deadlock in ZKRMStateStore.
> # Initial stage zkClient is null because of zk disconnected event.
> # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
> re establish zookeeper connection either via synconnected or expired event,
> it is highly possible that any other thred can obtain lock on
> {{ZKRMStateStore.this}} from state machine transition events. This cause
> Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2970) NodeLabel operations in RMAdmin CLI get missing in help command.
[
https://issues.apache.org/jira/browse/YARN-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258193#comment-14258193
]
Hudson commented on YARN-2970:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #785 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/785/])
YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command.
(Contributed by Varun Saxena) (junping_du: rev
e1ee0d45ea3315e7fbe1d28a7a35a0ce0c54483b)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
> NodeLabel operations in RMAdmin CLI get missing in help command.
>
>
> Key: YARN-2970
> URL: https://issues.apache.org/jira/browse/YARN-2970
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Junping Du
>Assignee: Varun Saxena
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2970.001.patch, YARN-2970.patch
>
>
> NodeLabel operations in RMAdmin CLI get missing in help command when I am
> debugging YARN-313, we should add them on as other cmds:
> {noformat}
> yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshResources]
> [-refreshSuperUserGroupsConfiguration] [-refreshUserToGroupsMappings]
> [-refreshAdminAcls] [-refreshServiceAcl] [-getGroup [username]]
> [-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]) [-help
> [cmd]]
>-refreshQueues: Reload the queues' acls, states and scheduler specific
> properties.
> ResourceManager will reload the mapred-queues configuration
> file.
>-refreshNodes: Refresh the hosts information at the ResourceManager.
>-refreshResources: Refresh resources of NodeManagers at the
> ResourceManager.
>-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>-refreshUserToGroupsMappings: Refresh user-to-groups mappings
>-refreshAdminAcls: Refresh acls for administration of ResourceManager
>-refreshServiceAcl: Reload the service-level authorization policy file.
> ResoureceManager will reload the authorization policy file.
>-getGroups [username]: Get the groups which given user belongs to.
>-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]):
> Update resource on specific node.
>-help [cmd]: Displays help for the given command or all commands if none
> is specified.
>-addToClusterNodeLabels [label1,label2,label3] (label splitted by ","):
> add to cluster node labels
>-removeFromClusterNodeLabels [label1,label2,label3] (label splitted by
> ","): remove from cluster node labels
>-replaceLabelsOnNode [node1:port,label1,label2 node2:port,label1,label2]:
> replace labels on nodes
>-directlyAccessNodeLabelStore: Directly access node label store, with this
> option, all node label related operations will not connect RM. Instead, they
> will access/modify stored node labels directly. By default, it is false
> (access via RM). AND PLEASE NOTE: if you configured
> yarn.node-labels.fs-store.root-dir to a local directory (instead of NFS or
> HDFS), this option will only work when the command run on the machine where
> RM is running.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2940) Fix new findbugs warnings in rest of the hadoop-yarn components
[ https://issues.apache.org/jira/browse/YARN-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258194#comment-14258194 ] Hudson commented on YARN-2940: -- FAILURE: Integrated in Hadoop-Yarn-trunk #785 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/785/]) CHANGES.txt: add YARN-2940 (junping_du: rev 20674255ad891bb693ecb2faa5bdb37477073dac) * hadoop-yarn-project/CHANGES.txt YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) (junping_du: rev d468c9aaf15e977f12e6214b47996412c907e069) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/QueueCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/RegistryAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java > Fix new findbugs warnings in rest of the hadoop-yarn components > --- > > Key: YARN-2940 > URL: https://issues.apache.org/jira/browse/YARN-2940 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Li Lu > Fix For: 2.7.0 > > Attachments: YARN-2940-121014-1.patch, YARN-2940-121014.patch, > YARN-2940-122314.patch, findbugs_report_yarn.zip > > > Fix findbugs warnings in the following YARN components: > hadoop-yarn-applications-distributedshell > hadoop-yarn-applications-unmanaged-am-launcher > hadoop-yarn-server-web-proxy > hadoop-yarn-registry > hadoop-yarn-server-common > hadoop-yarn-client -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258196#comment-14258196 ] Hudson commented on YARN-2937: -- FAILURE: Integrated in Hadoop-Yarn-trunk #785 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/785/]) YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. (zjshen: rev 41a548a916d4248164cb9495320f123ec215d70e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java > Fix new findbugs warnings in hadoop-yarn-nodemanager > > > Key: YARN-2937 > URL: https://issues.apache.org/jira/browse/YARN-2937 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.7.0 > > Attachments: YARN-2937.001.patch, YARN-2937.002.patch, > YARN-2937.003.patch, YARN-2937.004.patch, YARN-2937.005.patch, > YARN-2937.006.patch, YARN-2937.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2940) Fix new findbugs warnings in rest of the hadoop-yarn components
[ https://issues.apache.org/jira/browse/YARN-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258187#comment-14258187 ] Hudson commented on YARN-2940: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #51 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/51/]) CHANGES.txt: add YARN-2940 (junping_du: rev 20674255ad891bb693ecb2faa5bdb37477073dac) * hadoop-yarn-project/CHANGES.txt YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) (junping_du: rev d468c9aaf15e977f12e6214b47996412c907e069) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/QueueCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/RegistryAdminService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java > Fix new findbugs warnings in rest of the hadoop-yarn components > --- > > Key: YARN-2940 > URL: https://issues.apache.org/jira/browse/YARN-2940 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Li Lu > Fix For: 2.7.0 > > Attachments: YARN-2940-121014-1.patch, YARN-2940-121014.patch, > YARN-2940-122314.patch, findbugs_report_yarn.zip > > > Fix findbugs warnings in the following YARN components: > hadoop-yarn-applications-distributedshell > hadoop-yarn-applications-unmanaged-am-launcher > hadoop-yarn-server-web-proxy > hadoop-yarn-registry > hadoop-yarn-server-common > hadoop-yarn-client -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[ https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258189#comment-14258189 ] Hudson commented on YARN-2937: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #51 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/51/]) YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. (zjshen: rev 41a548a916d4248164cb9495320f123ec215d70e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java > Fix new findbugs warnings in hadoop-yarn-nodemanager > > > Key: YARN-2937 > URL: https://issues.apache.org/jira/browse/YARN-2937 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.7.0 > > Attachments: YARN-2937.001.patch, YARN-2937.002.patch, > YARN-2937.003.patch, YARN-2937.004.patch, YARN-2937.005.patch, > YARN-2937.006.patch, YARN-2937.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258184#comment-14258184 ] Hudson commented on YARN-2837: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #51 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/51/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml > Timeline server needs to recover the timeline DT when restarting > > > Key: YARN-2837 > URL: https://issues.apache.org/jira/browse/YARN-2837 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, > YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch > > > Timeline server needs to recover the stateful information when restarting as > RM/NM/JHS does now. So far the stateful information only includes the > timeline DT. Without recovery, the timeline DT of the existing YARN apps is > not long valid, and cannot be renewed any more after the timeline server is > restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLocks in RMStateStore<->ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258188#comment-14258188
]
Hudson commented on YARN-2946:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #51 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/51/])
YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith
Sharmaks (jianhe: rev 4f18018b7abb405f38d269da096ec9fd80c81d9a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AMRMTokenSecretManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTMasterKeyEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreEventType.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreAMRMTokenEvent.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreRMDTEvent.java
> DeadLocks in RMStateStore<->ZKRMStateStore
> --
>
> Key: YARN-2946
> URL: https://issues.apache.org/jira/browse/YARN-2946
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Rohith
>Assignee: Rohith
>Priority: Blocker
> Attachments: 0001-YARN-2946.patch, 0001-YARN-2946.patch,
> 0002-YARN-2946.patch, 0003-YARN-2946.patch, 0003-YARN-2946.patch,
> 0004-YARN-2946.patch, RM_BeforeFix_Deadlock_cycle_1.png,
> RM_BeforeFix_Deadlock_cycle_2.png, TestYARN2946.java
>
>
> Found one deadlock in ZKRMStateStore.
> # Initial stage zkClient is null because of zk disconnected event.
> # When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
> re establish zookeeper connection either via synconnected or expired event,
> it is highly possible that any other thred can obtain lock on
> {{ZKRMStateStore.this}} from state machine transition events. This cause
> Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2970) NodeLabel operations in RMAdmin CLI get missing in help command.
[
https://issues.apache.org/jira/browse/YARN-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258186#comment-14258186
]
Hudson commented on YARN-2970:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #51 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/51/])
YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command.
(Contributed by Varun Saxena) (junping_du: rev
e1ee0d45ea3315e7fbe1d28a7a35a0ce0c54483b)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
* hadoop-yarn-project/CHANGES.txt
> NodeLabel operations in RMAdmin CLI get missing in help command.
>
>
> Key: YARN-2970
> URL: https://issues.apache.org/jira/browse/YARN-2970
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Junping Du
>Assignee: Varun Saxena
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: YARN-2970.001.patch, YARN-2970.patch
>
>
> NodeLabel operations in RMAdmin CLI get missing in help command when I am
> debugging YARN-313, we should add them on as other cmds:
> {noformat}
> yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshResources]
> [-refreshSuperUserGroupsConfiguration] [-refreshUserToGroupsMappings]
> [-refreshAdminAcls] [-refreshServiceAcl] [-getGroup [username]]
> [-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]) [-help
> [cmd]]
>-refreshQueues: Reload the queues' acls, states and scheduler specific
> properties.
> ResourceManager will reload the mapred-queues configuration
> file.
>-refreshNodes: Refresh the hosts information at the ResourceManager.
>-refreshResources: Refresh resources of NodeManagers at the
> ResourceManager.
>-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>-refreshUserToGroupsMappings: Refresh user-to-groups mappings
>-refreshAdminAcls: Refresh acls for administration of ResourceManager
>-refreshServiceAcl: Reload the service-level authorization policy file.
> ResoureceManager will reload the authorization policy file.
>-getGroups [username]: Get the groups which given user belongs to.
>-updateNodeResource [NodeID] [MemSize] [vCores] ([OvercommitTimeout]):
> Update resource on specific node.
>-help [cmd]: Displays help for the given command or all commands if none
> is specified.
>-addToClusterNodeLabels [label1,label2,label3] (label splitted by ","):
> add to cluster node labels
>-removeFromClusterNodeLabels [label1,label2,label3] (label splitted by
> ","): remove from cluster node labels
>-replaceLabelsOnNode [node1:port,label1,label2 node2:port,label1,label2]:
> replace labels on nodes
>-directlyAccessNodeLabelStore: Directly access node label store, with this
> option, all node label related operations will not connect RM. Instead, they
> will access/modify stored node labels directly. By default, it is false
> (access via RM). AND PLEASE NOTE: if you configured
> yarn.node-labels.fs-store.root-dir to a local directory (instead of NFS or
> HDFS), this option will only work when the command run on the machine where
> RM is running.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2684) FairScheduler should tolerate queue configuration changes across RM restarts
[ https://issues.apache.org/jira/browse/YARN-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258169#comment-14258169 ] Rohith commented on YARN-2684: -- While going through writing test cases for queue removed scenario, found different behavior that even leafqueue does not exist in fair-scheduler, queue got created!!! Any configuration am I missing? > FairScheduler should tolerate queue configuration changes across RM restarts > > > Key: YARN-2684 > URL: https://issues.apache.org/jira/browse/YARN-2684 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, resourcemanager >Affects Versions: 2.5.1 >Reporter: Karthik Kambatla >Assignee: Rohith >Priority: Critical > > YARN-2308 fixes this issue for CS, this JIRA is to fix it for FS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-810) Support CGroup ceiling enforcement on CPU
[
https://issues.apache.org/jira/browse/YARN-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258131#comment-14258131
]
Hadoop QA commented on YARN-810:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689009/YARN-810-5.patch
against trunk revision 4f18018.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 15 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6187//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6187//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6187//console
This message is automatically generated.
> Support CGroup ceiling enforcement on CPU
> -
>
> Key: YARN-810
> URL: https://issues.apache.org/jira/browse/YARN-810
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.1.0-beta, 2.0.5-alpha
>Reporter: Chris Riccomini
>Assignee: Sandy Ryza
> Attachments: YARN-810-3.patch, YARN-810-4.patch, YARN-810-5.patch,
> YARN-810.patch, YARN-810.patch
>
>
> Problem statement:
> YARN currently lets you define an NM's pcore count, and a pcore:vcore ratio.
> Containers are then allowed to request vcores between the minimum and maximum
> defined in the yarn-site.xml.
> In the case where a single-threaded container requests 1 vcore, with a
> pcore:vcore ratio of 1:4, the container is still allowed to use up to 100% of
> the core it's using, provided that no other container is also using it. This
> happens, even though the only guarantee that YARN/CGroups is making is that
> the container will get "at least" 1/4th of the core.
> If a second container then comes along, the second container can take
> resources from the first, provided that the first container is still getting
> at least its fair share (1/4th).
> There are certain cases where this is desirable. There are also certain cases
> where it might be desirable to have a hard limit on CPU usage, and not allow
> the process to go above the specified resource requirement, even if it's
> available.
> Here's an RFC that describes the problem in more detail:
> http://lwn.net/Articles/336127/
> Solution:
> As it happens, when CFS is used in combination with CGroups, you can enforce
> a ceiling using two files in cgroups:
> {noformat}
> cpu.cfs_quota_us
> cpu.cfs_period_us
> {noformat}
> The usage of these two files is documented in more detail here:
> https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.html
> Testing:
> I have tested YARN CGroups using the 2.0.5-alpha implementation. By default,
> it behaves as described above (it is a soft cap, and allows containers to use
> more than they asked for). I then tested CFS CPU quotas manually with YARN.
> First, you can see that CFS is in use in the CGroup, based on the file names:
> {noformat}
> [criccomi@eat1-qa464 ~]$ sudo -u app ls -l /cgroup/cpu/hadoop-yarn/
> total 0
> -r--r--r-- 1 app app 0 Jun 13 16:46 cgroup.procs
> drwxr-xr-x 2 app app 0 Jun 13 17:08 container_1371141151815_0004_01_02
> -rw-r--r-- 1 app app 0 Jun 13 16:46 cpu.cfs_period_us
> -rw-r--r-- 1 app app 0 Jun 13 16:46 cpu.cfs_quota_us
> -rw-r--r-- 1 app
