[
https://issues.apache.org/jira/browse/YARN-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162157#comment-14162157
]
Xuan Gong commented on YARN-2583:
---------------------------------
Here is the proposal:
* Add private configuration for number of logs we can save in NM side. We will
delete old logs if the num of logs is larger than this configured value. This
is a temporary solution. The configuration will be deleted once we find a more
scalable method(will be tracked by YARN-2548) to only write a single log file
per LRS.
* jhs contacts RM to check whether app is still running or not. If this app is
still running, we need to keep the app dir, but remove the old logs.
* Remove per-app LogRollingInterval completely and then have NM wake up every
so often and upload log files. In this ticket, we can spin off
LogRollingInterval from AppLogAggregatorImpl. YARN-2651 will be used to track
the changes for other places.
* Enforce the minimal log rolling interval. (3600 seconds will be used as
minimal value)
> Modify the LogDeletionService to support Log aggregation for LRS
> ----------------------------------------------------------------
>
> Key: YARN-2583
> URL: https://issues.apache.org/jira/browse/YARN-2583
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Attachments: YARN-2583.1.patch, YARN-2583.2.patch,
> YARN-2583.3.1.patch, YARN-2583.3.patch
>
>
> Currently, AggregatedLogDeletionService will delete old logs from HDFS. It
> will check the cut-off-time, if all logs for this application is older than
> this cut-off-time. The app-log-dir from HDFS will be deleted. This will not
> work for LRS. We expect a LRS application can keep running for a long time.
> Two different scenarios:
> 1) If we configured the rollingIntervalSeconds, the new log file will be
> always uploaded to HDFS. The number of log files for this application will
> become larger and larger. And there is no log files will be deleted.
> 2) If we did not configure the rollingIntervalSeconds, the log file can only
> be uploaded to HDFS after the application is finished. It is very possible
> that the logs are uploaded after the cut-off-time. It will cause problem
> because at that time the app-log-dir for this application in HDFS has been
> deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
