Dongwook Kwon commented on YARN-3929:

The reason is we already had similar tool as log-aggregator out of hadoop, not 
only for YARN, it was designed for Hadoop 1 which didn't have native 
log-aggregation feature, in our cluster,  each node has daemon that 
periodically checks application log in local and push to S3, it works fine even 
with 2000 nodes, the issue we have now is with YARN's log-aggregation, as you 
can imagine, 2 systems tries to do the same things, and other internal users 
want to use YARN's log-aggregation for such as HUE or "yarn logs 
--applicationId" command, and we still need to support Hadoop 1, so whenever  
cluster turns on YARN's log-aggregation, we don't have application log for 
troubleshooting. This has been an issue for long and simple solution for our 
team is making this optional as I suggested, I agree, for most of use cases, it 
may not be useful, so I make default as cleaning up and make sure test catch it.

> Uncleaning option for local app log files with log-aggregation feature
> ----------------------------------------------------------------------
>                 Key: YARN-3929
>                 URL: https://issues.apache.org/jira/browse/YARN-3929
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: log-aggregation
>    Affects Versions: 2.4.0, 2.6.0
>            Reporter: Dongwook Kwon
>            Priority: Minor
>         Attachments: YARN-3929.02.patch
> Although it makes sense to delete local app log files once AppLogAggregator 
> copied all files into remote location(HDFS), I have some use cases that need 
> to leave local app log files after it's copied to HDFS. Mostly it's for own 
> backup purpose. I would like to use log-aggregation feature of YARN and want 
> to back up app log files too. Without this option, files has to copy from 
> HDFS to local again. 

This message was sent by Atlassian JIRA

Reply via email to