Haibo Chen created YARN-4766: -------------------------------- Summary: NM should not aggregate logs older than the retention policy Key: YARN-4766 URL: https://issues.apache.org/jira/browse/YARN-4766 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation, nodemanager Reporter: Haibo Chen Assignee: Haibo Chen
When a log aggregation fails on the NM the information is for the attempt is kept in the recovery DB. Log aggregation can fail for multiple reasons which are often related to HDFS space or permissions. On restart the recovery DB is read and if an application attempt needs its logs aggregated, the files are scheduled for aggregation without any checks. The log files could be older than the retention limit in which case we should not aggregate them but immediately mark them for deletion from the local file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)