[
https://issues.apache.org/jira/browse/YARN-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569983#comment-16569983
]
Bibin A Chundatt commented on YARN-8617:
----------------------------------------
[~Prabhu Joseph]
IIUC LogAggregationService -> AggregatorImpl every upload cycle for each node
you will have new files per node
{code}
node3.openstack_45454_1533315091898
node3.openstack_45454_<uploaded time1>
node3.openstack_45454_<uploaded time2>
{code}
{{node3.openstack_45454_1533315091898}} file will not be changed once uploaded.
So AggregationDeletionService should delete the file after log retain time
{code}
Set<Path> uploadedFilePathsInThisCycle =
aggregator.doContainerLogAggregation(logAggregationFileController,
appFinished, finishedContainers.contains(container));
...
logControllerContext.setLogUploadTimeStamp(System.currentTimeMillis());
try {
this.logAggregationFileController.postWrite(logControllerContext);
diagnosticMessage = "Log uploaded successfully for Application: "
+ appId + " in NodeManager: "
+ LogAggregationUtils.getNodeString(nodeId) + " at "
+ Times.format(logControllerContext.getLogUploadTimeStamp())
+ "\n";
{code}
this.logAggregationFileController.postWrite(logControllerContext); --> renames
the file with time stamp
> Aggregated Application Logs accumulates for long running jobs
> -------------------------------------------------------------
>
> Key: YARN-8617
> URL: https://issues.apache.org/jira/browse/YARN-8617
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation
> Affects Versions: 2.7.4
> Reporter: Prabhu Joseph
> Priority: Major
>
> Currently AggregationDeletionService will delete older aggregated log files
> once when they are complete. This will cause logs to accumulate for Long
> Running Jobs like Llap, Spark Streaming.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
