Robert Kanter updated YARN-2942:
    Attachment: YARN-2942.001.patch

I've just uploaded YARN-2942.001.patch, which adds unit tests and has been 
rebased on the current trunk.  It's still missing the ZK/Curator security stuff.

I ran into an unfortunate problem with the current trunk, which has 
HADOOP-11492 that updated Curator to 2.7.1.  That caused a problem with the 
{{ChildReaper}}, which was using something that requires Guava 15+ and we're 
only using 11.0.2.  As a workaround, I copied ChildReaper into hadoop-common, 
and made a minor modification to allow it to work with Guava 11.

> Aggregated Log Files should be compacted
> ----------------------------------------
>                 Key: YARN-2942
>                 URL: https://issues.apache.org/jira/browse/YARN-2942
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.6.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: CompactedAggregatedLogsProposal_v1.pdf, 
> CompactedAggregatedLogsProposal_v2.pdf, YARN-2942-preliminary.001.patch, 
> YARN-2942-preliminary.002.patch, YARN-2942.001.patch
> Turning on log aggregation allows users to easily store container logs in 
> HDFS and subsequently view them in the YARN web UIs from a central place.  
> Currently, there is a separate log file for each Node Manager.  This can be a 
> problem for HDFS if you have a cluster with many nodes as you’ll slowly start 
> accumulating many (possibly small) files per YARN application.  The current 
> “solution” for this problem is to configure YARN (actually the JHS) to 
> automatically delete these files after some amount of time.  
> We should improve this by compacting the per-node aggregated log files into 
> one log file per application.

This message was sent by Atlassian JIRA

Reply via email to