[
https://issues.apache.org/jira/browse/YARN-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253603#comment-16253603
]
Jason Lowe commented on YARN-7500:
----------------------------------
bq. Currently LogAggregation deletes the application logs based on start time
of the job.
The log aggregation deletion service does not do anything based on the
application start or completion time. It only is checking modification
timestamps in the filesystem:
{code}
private static boolean shouldDeleteLogDir(FileStatus dir, long
cutoffMillis,
FileSystem fs) {
boolean shouldDelete = true;
try {
for(FileStatus node: fs.listStatus(dir.getPath())) {
if(node.getModificationTime() >= cutoffMillis) {
shouldDelete = false;
break;
}
}
{code}
So if any entry within the log directory has a modification timestamp that is
newer than the log retention it will not delete the entire log directory.
> LogAggregation DeletionService should consider completedTime for long running
> jobs
> ----------------------------------------------------------------------------------
>
> Key: YARN-7500
> URL: https://issues.apache.org/jira/browse/YARN-7500
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 2.7.3
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
>
> Currently LogAggregation deletes the application logs based on start time of
> the job. For long running jobs (started before
> yarn.log-aggregation.retain-seconds), say it is failed yesterday for some
> reason and we won't have the job logs today for debugging.
> Better to consider the completedTime of the job as part of the deletion
> condition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
