[
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967748#comment-15967748
]
Jason Lowe commented on YARN-2985:
----------------------------------
Based on the description of this JIRA, I think there's some confusion here.
Aggregated logs are deleted for non-MapReduce applications as long as the
deletion service is running, whether that deletion service is hosted by the
MapReduce job history server or somewhere else. That's why the proposed patch
is so small -- it's simply reusing the same code the JHS is already running.
The log deletion service looks at the remote log directory in HDFS. It doesn't
filter the list of application logs it finds there based on whether it thinks
the app is MapReduce or not, rather it just treats them as generic
applications. It happens to run in the MapReduce history server, but it is
_not_ MapReduce-specific. If users don't want to run MapReduce applications
but want to do log aggregtion then they just need to run the MapReduce history
server. They won't use it for MapReduce job history since there are no
MapReduce jobs, but that server will perform aggregated log retention for *all*
applications.
Therefore this JIRA is really about adding the ability to relocate the
aggregated log deletion service from the MapReduce job history server to the
YARN timeline server. We don't want two of these things running in the cluster
if someone has deployed the MapReduce history server and the YARN timeline
server. That could lead to error messages in the logs as one of them goes to
traverse/delete the logs just as the other is already deleting them. However
we also don't want to just rip it out of the MapReduce history server and move
it to the timeline server because the timeline server is still an optional
server in YARN.
So we either need a way for the user to specify where they want the deletion
service to run, whether that's the legacy location in the MapReduce history
server (since they aren't going to run a timeline server which is still an
optional YARN server) or in the timeline server. Or we need to just declare
the timeline server a mandatory server to run (at least for log aggregation
support) and move it from one to the other.
In addition the MapReduce history server supports dynamic refresh of the log
deletion service configs, and it would be nice not to lose that ability when it
is hosted in the timeline server. That could be a separate JIRA unless we're
ripping it out of the JHS. If it can only run in the timeline server then we
would lose refresh functionality unless that JIRA was completed.
As for unit tests, I agree the existing tests for the deletion service cover
the correctness of the service itself, so we just need unit tests for the
timeline server and MapReduce JHS to verify each is starting the deletion
service or not starting the service based on how the cluster is configured.
> YARN should support to delete the aggregated logs for Non-MapReduce
> applications
> --------------------------------------------------------------------------------
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation, nodemanager
> Affects Versions: 2.8.0
> Reporter: Xu Yang
> Assignee: Steven Rand
> Attachments: YARN-2985-branch-2-001.patch
>
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer.
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS,
> but can not delete those logs. Need the NodeManager take over the function of
> aggregated log deletion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
