[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735448#comment-14735448 ] zhihai xu commented on YARN-4096: - +1. Committing it in. > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735497#comment-14735497 ] Hudson commented on YARN-4096: -- FAILURE: Integrated in Hadoop-trunk-Commit #8416 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8416/]) YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe. (zxu: rev 16b9037dc1300b8bdbe54ba7cd47c53fe16e93d8) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.7.2 > > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735537#comment-14735537 ] Hudson commented on YARN-4096: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1095 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1095/]) YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe. (zxu: rev 16b9037dc1300b8bdbe54ba7cd47c53fe16e93d8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.7.2 > > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735490#comment-14735490 ] zhihai xu commented on YARN-4096: - thanks Jason for the contribution! Committed it to branch-2.7.2, branch-2 and trunk. > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735704#comment-14735704 ] Hudson commented on YARN-4096: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2307 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2307/]) YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe. (zxu: rev 16b9037dc1300b8bdbe54ba7cd47c53fe16e93d8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.7.2 > > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735672#comment-14735672 ] Hudson commented on YARN-4096: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #357 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/357/]) YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe. (zxu: rev 16b9037dc1300b8bdbe54ba7cd47c53fe16e93d8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * hadoop-yarn-project/CHANGES.txt > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.7.2 > > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735791#comment-14735791 ] Hudson commented on YARN-4096: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2284 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2284/]) YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe. (zxu: rev 16b9037dc1300b8bdbe54ba7cd47c53fe16e93d8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.7.2 > > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[ https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735837#comment-14735837 ] Hudson commented on YARN-4096: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #345 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/345/]) YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe. (zxu: rev 16b9037dc1300b8bdbe54ba7cd47c53fe16e93d8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * hadoop-yarn-project/CHANGES.txt > App local logs are leaked if log aggregation fails to initialize for the app > > > Key: YARN-4096 > URL: https://issues.apache.org/jira/browse/YARN-4096 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.7.2 > > Attachments: YARN-4096.001.patch > > > If log aggregation fails to initialize for an application then the local logs > will never be deleted. This is similar to YARN-3476 except this is a failure > when log aggregation tries to initialize the app-specific log aggregator > rather than a failure during a log upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[
https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723628#comment-14723628
]
Hadoop QA commented on YARN-4096:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 40s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 21s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 16s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 31s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 45m 4s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12753307/YARN-4096.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / cf83156 |
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8951/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8951/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8951/console |
This message was automatically generated.
> App local logs are leaked if log aggregation fails to initialize for the app
>
>
> Key: YARN-4096
> URL: https://issues.apache.org/jira/browse/YARN-4096
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, nodemanager
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-4096.001.patch
>
>
> If log aggregation fails to initialize for an application then the local logs
> will never be deleted. This is similar to YARN-3476 except this is a failure
> when log aggregation tries to initialize the app-specific log aggregator
> rather than a failure during a log upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4096) App local logs are leaked if log aggregation fails to initialize for the app
[
https://issues.apache.org/jira/browse/YARN-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724577#comment-14724577
]
zhihai xu commented on YARN-4096:
-
Hi [~jlowe], It is a good catch. It is nice to make an useless variable
{{logAggregationDisabled}} become useful. {{initApp}} will catch
{{YarnRuntimeException}} and send event {{APPLICATION_LOG_HANDLING_FAILED}} to
{{ApplicationImpl}}, {{ApplicationImpl}} will send event
{{LogHandlerAppFinishedEvent}} to clean up the local application logs
{{doAppLogAggregationPostCleanUp}} when application finished.
+1 for the patch.
> App local logs are leaked if log aggregation fails to initialize for the app
>
>
> Key: YARN-4096
> URL: https://issues.apache.org/jira/browse/YARN-4096
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, nodemanager
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-4096.001.patch
>
>
> If log aggregation fails to initialize for an application then the local logs
> will never be deleted. This is similar to YARN-3476 except this is a failure
> when log aggregation tries to initialize the app-specific log aggregator
> rather than a failure during a log upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
