[
https://issues.apache.org/jira/browse/YARN-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-2942:
--------------------------------
Attachment: YARN-2942.001.patch
I've just uploaded YARN-2942.001.patch, which adds unit tests and has been
rebased on the current trunk. It's still missing the ZK/Curator security stuff.
I ran into an unfortunate problem with the current trunk, which has
HADOOP-11492 that updated Curator to 2.7.1. That caused a problem with the
{{ChildReaper}}, which was using something that requires Guava 15+ and we're
only using 11.0.2. As a workaround, I copied ChildReaper into hadoop-common,
and made a minor modification to allow it to work with Guava 11.
> Aggregated Log Files should be compacted
> ----------------------------------------
>
> Key: YARN-2942
> URL: https://issues.apache.org/jira/browse/YARN-2942
> Project: Hadoop YARN
> Issue Type: New Feature
> Affects Versions: 2.6.0
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Attachments: CompactedAggregatedLogsProposal_v1.pdf,
> CompactedAggregatedLogsProposal_v2.pdf, YARN-2942-preliminary.001.patch,
> YARN-2942-preliminary.002.patch, YARN-2942.001.patch
>
>
> Turning on log aggregation allows users to easily store container logs in
> HDFS and subsequently view them in the YARN web UIs from a central place.
> Currently, there is a separate log file for each Node Manager. This can be a
> problem for HDFS if you have a cluster with many nodes as you’ll slowly start
> accumulating many (possibly small) files per YARN application. The current
> “solution” for this problem is to configure YARN (actually the JHS) to
> automatically delete these files after some amount of time.
> We should improve this by compacting the per-node aggregated log files into
> one log file per application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
