Jason Lowe commented on YARN-3089:

bq. But if the user does not set any includePattern/excludePattern in 
LogAggregationContext, and does not want to roll-over the logs, all container 
logs will be written into the files with the same name(run the command, such as 
1>>stdout, 2>>stderr ). After we aggregate the logs into HDFS, the 
stdout/stderr will be deleted. In this case, this LRS app is affected.

This sounds pretty bad.  If the app doesn't roll its own logs but is normally 
not very chatty such that it isn't much of an issue, we're going to blow away 
the apps logs every log-aggregation.roll-monitoring-interval-seconds interval?  
That interval isn't configurable per app, so I can see cases where someone puts 
an app on the cluster that isn't super long running, doesn't roll its own logs, 
but then this feature comes along and uploads a partial log and removes the 
active log files.  In other words, fixing this bug may actually start deleting 
logs we shouldn't delete on clusters using the LCE.  However that's a separate 
JIRA from this, as this is focusing on making the behavior of the LCE 
consistent with the default executor wrt. deletes.

> LinuxContainerExecutor does not handle file arguments to deleteAsUser
> ---------------------------------------------------------------------
>                 Key: YARN-3089
>                 URL: https://issues.apache.org/jira/browse/YARN-3089
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Eric Payne
>            Priority: Blocker
>         Attachments: YARN-3089.v1.txt, YARN-3089.v2.txt, YARN-3089.v3.txt
> YARN-2468 added the deletion of individual logs that are aggregated, but this 
> fails to delete log files when the LCE is being used.  The LCE native 
> executable assumes the paths being passed are paths and the delete fails.

This message was sent by Atlassian JIRA

Reply via email to