Zhijie Shen commented on YARN-2583:

1. Let's do it in YARN-2651 (also the config in yarn-default.xml)?
+   * Define how often the logAggregationSerivce uploads container logs in 
+   */
+  public static final String NM_LOG_ROLLING_INTERVAL_SECONDS =
+      NM_PREFIX + "log.rolling-interval-seconds";
+  public static final long DEFAULT_NM_LOG_ROLLING_INTERVAL_SECONDS = -1;
+  private final long rollingInterval;

2. The methods of mock object creation in TestAggregatedLogDeletionService can 
be static.

3. Shall we have a DEBUG flag to break the lower bound for the testing purpose?
+  private static final long MIN_NM_LOG_ROLLING_INTERVAL_SECONDS = 3600;

4. Should the condition be "< 
+    if (configuredRentionSize <= 0) {
+      this.retentionSize =

5. Change the condition to "endwith"?
+                && !next.getPath().getName().contains(
+                    LogAggregationUtils.TMP_FILE_SUFFIX);

> Modify the LogDeletionService to support Log aggregation for LRS
> ----------------------------------------------------------------
>                 Key: YARN-2583
>                 URL: https://issues.apache.org/jira/browse/YARN-2583
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-2583.1.patch, YARN-2583.2.patch, 
> YARN-2583.3.1.patch, YARN-2583.3.patch, YARN-2583.4.patch
> Currently, AggregatedLogDeletionService will delete old logs from HDFS. It 
> will check the cut-off-time, if all logs for this application is older than 
> this cut-off-time. The app-log-dir from HDFS will be deleted. This will not 
> work for LRS. We expect a LRS application can keep running for a long time. 
> Two different scenarios: 
> 1) If we configured the rollingIntervalSeconds, the new log file will be 
> always uploaded to HDFS. The number of log files for this application will 
> become larger and larger. And there is no log files will be deleted.
> 2) If we did not configure the rollingIntervalSeconds, the log file can only 
> be uploaded to HDFS after the application is finished. It is very possible 
> that the logs are uploaded after the cut-off-time. It will cause problem 
> because at that time the app-log-dir for this application in HDFS has been 
> deleted.

This message was sent by Atlassian JIRA

Reply via email to