[
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Rand updated YARN-2985:
------------------------------
Attachment: YARN-2985-branch-2-001.patch
Attaching a patch for branch-2. I've tested this experimentally by deploying a
patched Timeline Server to a cluster, running a Spark job on that cluster, and
validating that the aggregated logs disappeared from HDFS after the configured
amount of time had elapsed. The Timeline Server's logs confirm that it
performed the deletion.
I'm not sure how to add tests though. The existing tests for the
{{TestAggregatedLogDeletionService}} are good enough to test that the service
works -- the more interesting thing is verifying that when a Timeline Server is
deployed, log aggregation is enforced for non-MR applications. I don't know how
to test non-MR applications from the hadoop-yarn project tests though.
> YARN should support to delete the aggregated logs for Non-MapReduce
> applications
> --------------------------------------------------------------------------------
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation, nodemanager
> Reporter: Xu Yang
> Assignee: Steven Rand
> Attachments: YARN-2985-branch-2-001.patch
>
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer.
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS,
> but can not delete those logs. Need the NodeManager take over the function of
> aggregated log deletion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
