[
https://issues.apache.org/jira/browse/YARN-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790508#comment-16790508
]
Tarun Parimi commented on YARN-8617:
------------------------------------
Hi [~bibinchundatt],
I was also facing this issue and on testing in my local cluster I observed the
follows:
{quote}1. limit number of files per node
public static final String NM_LOG_AGGREGATION_NUM_LOG_FILES_SIZE_PER_APP
= NM_PREFIX + "log-aggregation.num-log-files-per-app";{quote}
This doesn't seem to work currently for IndexedFileFormat. After the file
exceeds LOG_ROLL_OVER_MAX_FILE_SIZE_GB, a new file is created. But the older
node files can keep on accumulating as long as the app is running. Should we
implement this config for IndexedFileFormat also as a fix?
{quote}For long running service the application folder eg
:user/logs/application_1234 modification time gets updated on every upload
cycle.
This could cause nodefile to remain in hdfs if no new containers are allocated
to same node.{quote}
Should we check and delete nodefiles in AggrgeatedLogDeletionService for
RUNNING apps without the checking the condition appDir.getModificationTime() <
cutoffMillis ?
Doing so will delete the older node files and fix the problem of old node files
getting accumulated.
> Aggregated Application Logs accumulates for long running jobs
> -------------------------------------------------------------
>
> Key: YARN-8617
> URL: https://issues.apache.org/jira/browse/YARN-8617
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation
> Affects Versions: 2.7.4
> Reporter: Prabhu Joseph
> Priority: Major
>
> Currently AggregationDeletionService will delete older aggregated log files
> once when they are complete. This will cause logs to accumulate for Long
> Running Jobs like Llap, Spark Streaming.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
