[
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968609#comment-15968609
]
Steven Rand commented on YARN-2985:
-----------------------------------
[~jlowe], thanks for the thoughtful response. Based on that information, it
seems like the most straightforward way to proceed, at least for branch-2, is
to add a configuration option for running the deletion service in only the
timeline server, and not the JHS. Something like
{{yarn.log-aggregation.run-in-timeline-server}} that defaults to {{false}} for
backcompat, but when set to {{true}}, prevents the JHS from performing
retention, and tells the timeline server to do it instead. Does that seem
reasonable? If so I'll update the patch to do that, but certainly open to
alternatives if there's a better way.
For trunk, I imagine it might be worth just removing retention from the JHS and
moving it to the timeline server entirely, since my understanding is that the
timeline server is supposed to replace the JHS, even for deployments that only
run MR jobs, and 3.0 seems like a reasonable enough point at which to require
the switch from JHS to timeline server. I might be misunderstanding the
relationship between the two though, so please correct me if that doesn't make
sense.
> YARN should support to delete the aggregated logs for Non-MapReduce
> applications
> --------------------------------------------------------------------------------
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation, nodemanager
> Affects Versions: 2.8.0
> Reporter: Xu Yang
> Assignee: Steven Rand
> Attachments: YARN-2985-branch-2-001.patch
>
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer.
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS,
> but can not delete those logs. Need the NodeManager take over the function of
> aggregated log deletion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
