[
https://issues.apache.org/jira/browse/YARN-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568261#comment-16568261
]
Bibin A Chundatt edited comment on YARN-8617 at 8/3/18 2:40 PM:
----------------------------------------------------------------
{{AppLogAggregatorImpl#uploadLogsForContainers}} all succesfully uploaded logs
will be deleted at local.
{code}
Set<Path> uploadedFilePathsInThisCycle =
aggregator.doContainerLogAggregation(logAggregationFileController,
appFinished, finishedContainers.contains(container));
if (uploadedFilePathsInThisCycle.size() > 0) {
uploadedLogsInThisCycle = true;
List<Path> uploadedFilePathsInThisCycleList = new ArrayList<>();
uploadedFilePathsInThisCycleList.addAll(uploadedFilePathsInThisCycle);
deletionTask = new FileDeletionTask(delService,
this.userUgi.getShortUserName(), null,
uploadedFilePathsInThisCycleList);
}
{code}
deletion task gets scheduled for deletion in deletionService too.
{{yarn.nodemanager.remote-app-log-dir}} per node, during each cycle the upload
is done based on timestamp to hdfs.
If you have JHS started it should deleted older files.
refer :{{AggregatedLogDeletionService}}
was (Author: bibinchundatt):
{{AppLogAggregatorImpl#uploadLogsForContainers}} all succesfully uploaded logs
will be deleted at local.
{code}
Set<Path> uploadedFilePathsInThisCycle =
aggregator.doContainerLogAggregation(logAggregationFileController,
appFinished, finishedContainers.contains(container));
if (uploadedFilePathsInThisCycle.size() > 0) {
uploadedLogsInThisCycle = true;
List<Path> uploadedFilePathsInThisCycleList = new ArrayList<>();
uploadedFilePathsInThisCycleList.addAll(uploadedFilePathsInThisCycle);
deletionTask = new FileDeletionTask(delService,
this.userUgi.getShortUserName(), null,
uploadedFilePathsInThisCycleList);
}
{code}
deletion task gets scheduled for deletion in deletionService too.
{{yarn.nodemanager.remote-app-log-dir}} per node during each cycle the upload
is done based on timestamp to hdfs. If you have JHS started it should refer
:{{AggregatedLogDeletionService}}
> Aggregated Application Logs accumulates for long running jobs
> -------------------------------------------------------------
>
> Key: YARN-8617
> URL: https://issues.apache.org/jira/browse/YARN-8617
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation
> Affects Versions: 2.7.4
> Reporter: Prabhu Joseph
> Priority: Major
>
> Currently AggregationDeletionService will delete older aggregated log files
> once when they are complete. This will cause logs to accumulate for Long
> Running Jobs like Llap, Spark Streaming.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
