[
https://issues.apache.org/jira/browse/YARN-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-2942:
--------------------------------
Attachment: YARN-2942-preliminary.002.patch
I've just uploaded YARN-2942-preliminary.002.patch, which follows the v2 design
doc. It's basically all there except for unit tests and ZK/Curator security
stuff, which I'm still working on.
Here's some more technical info on the implementation:
- Compacted logs and are placed in the same directory as the aggregated logs,
which has some benefits, including
-- No need to duplicate directory structure
-- {{AggregatedLogDeletionService}} handles cleaning up old compacted logs
without any changes :)
- {{CompactedAggregatedLogFormat}} has a Reader and Writer that handles the
details of reading and writing the compacted logs.
- To simplify some reading code in the {{AggregatedLogsBlock}}, I created a
{{LogFormatReader}} interface, which defines some common methods that both an
{{AggregatedLogFormat.LogReader}} and a
{{CompactedAggregatedLogFormat.LogReader}} both have
- The {{AggregatedLogsBlock}} first tries to read from the compacted log file;
if it can't find it, or can't find the container in the index, or has some
other problem, it will fallback to the aggregated log and have the same
behavior as before
-- The file formats for the aggregated logs and compacted logs are similar
enough that the {{AggregatedLogFormat.ContainerLogsReader}} can be used on
either, so there's no new log file parsing code for that
- Here's the process that the NM goes through (if compaction is enabled):
-- After the {{AppLogAggregatorImpl}} is done uploading aggregated log files,
it will then try to acquire the Curator lock for the current application
-- Then it will append its log file
-- Then it will delete its aggregated log file
-- Then it will release the lock
It would be great if I could get some feedback on the current patch so far.
> Aggregated Log Files should be compacted
> ----------------------------------------
>
> Key: YARN-2942
> URL: https://issues.apache.org/jira/browse/YARN-2942
> Project: Hadoop YARN
> Issue Type: New Feature
> Affects Versions: 2.6.0
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Attachments: CompactedAggregatedLogsProposal_v1.pdf,
> CompactedAggregatedLogsProposal_v2.pdf, YARN-2942-preliminary.001.patch,
> YARN-2942-preliminary.002.patch
>
>
> Turning on log aggregation allows users to easily store container logs in
> HDFS and subsequently view them in the YARN web UIs from a central place.
> Currently, there is a separate log file for each Node Manager. This can be a
> problem for HDFS if you have a cluster with many nodes as you’ll slowly start
> accumulating many (possibly small) files per YARN application. The current
> “solution” for this problem is to configure YARN (actually the JHS) to
> automatically delete these files after some amount of time.
> We should improve this by compacting the per-node aggregated log files into
> one log file per application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
