Robert Kanter updated YARN-2942:
    Attachment: YARN-2942-preliminary.002.patch

I've just uploaded YARN-2942-preliminary.002.patch, which follows the v2 design 
doc.  It's basically all there except for unit tests and ZK/Curator security 
stuff, which I'm still working on.  

Here's some more technical info on the implementation:
- Compacted logs and are placed in the same directory as the aggregated logs, 
which has some benefits, including
-- No need to duplicate directory structure
-- {{AggregatedLogDeletionService}} handles cleaning up old compacted logs 
without any changes :)
- {{CompactedAggregatedLogFormat}} has a Reader and Writer that handles the 
details of reading and writing the compacted logs.
- To simplify some reading code in the {{AggregatedLogsBlock}}, I created a 
{{LogFormatReader}} interface, which defines some common methods that both an 
{{AggregatedLogFormat.LogReader}} and a 
{{CompactedAggregatedLogFormat.LogReader}} both have
- The {{AggregatedLogsBlock}} first tries to read from the compacted log file; 
if it can't find it, or can't find the container in the index, or has some 
other problem, it will fallback to the aggregated log and have the same 
behavior as before
-- The file formats for the aggregated logs and compacted logs are similar 
enough that the {{AggregatedLogFormat.ContainerLogsReader}} can be used on 
either, so there's no new log file parsing code for that
- Here's the process that the NM goes through (if compaction is enabled):
-- After the {{AppLogAggregatorImpl}} is done uploading aggregated log files, 
it will then try to acquire the Curator lock for the current application
-- Then it will append its log file
-- Then it will delete its aggregated log file
-- Then it will release the lock

It would be great if I could get some feedback on the current patch so far.

> Aggregated Log Files should be compacted
> ----------------------------------------
>                 Key: YARN-2942
>                 URL: https://issues.apache.org/jira/browse/YARN-2942
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.6.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: CompactedAggregatedLogsProposal_v1.pdf, 
> CompactedAggregatedLogsProposal_v2.pdf, YARN-2942-preliminary.001.patch, 
> YARN-2942-preliminary.002.patch
> Turning on log aggregation allows users to easily store container logs in 
> HDFS and subsequently view them in the YARN web UIs from a central place.  
> Currently, there is a separate log file for each Node Manager.  This can be a 
> problem for HDFS if you have a cluster with many nodes as you’ll slowly start 
> accumulating many (possibly small) files per YARN application.  The current 
> “solution” for this problem is to configure YARN (actually the JHS) to 
> automatically delete these files after some amount of time.  
> We should improve this by compacting the per-node aggregated log files into 
> one log file per application.

This message was sent by Atlassian JIRA

Reply via email to