Xuan Gong commented on YARN-2468:

bq. If LogContext is not specified, we're running into the traditional log 
handling case, right? We will still have a combined log file identified by the 
node id? Or node id will always be the directory, and there exists only one 
file under it?

node id will always be the directory, and there exists only one file under it

bq. Let's say if work-preserving NM restarting happens, NM is going to forget 
all the uploaded logs files, and redo everything, right?

If NM restarts happens, it will upload all logs which are previous uploaded, 
but not deleted. 
I think that we can solve this problem in separate ticket, because this ticket 
is the first step to solve Log handling for LRS. 

bq. LogContext doesn't need to be in ApplicatonSubmissionContext, because 
ApplicatonSubmissionContext contains ContainerLaunchContext. LogContext is 
container related stuff, such that ContainerLaunchContext should be the best 
place. Concurrently, we can have one context for all containers. Maybe in the 
future we can think of setting different LogContext for each individual 


bq. In getFilteredLogFiles, the logic is that if the log file matches the 
include pattern, it will be added first, and if then if it matches the exclude 
pattern, it will be removed. Shall we do the sanity check to make sure we can 
not include and exclude the same pattern, otherwise, the semantics is a bit 

Add more explanation in javaDoc.

Uploaded a new patch to address all comments.

> Log handling for LRS
> --------------------
>                 Key: YARN-2468
>                 URL: https://issues.apache.org/jira/browse/YARN-2468
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-2468.1.patch, YARN-2468.2.patch, YARN-2468.3.patch, 
> YARN-2468.3.rebase.2.patch, YARN-2468.3.rebase.patch, YARN-2468.4.1.patch, 
> YARN-2468.4.patch, YARN-2468.5.patch
> Currently, when application is finished, NM will start to do the log 
> aggregation. But for Long running service applications, this is not ideal. 
> The problems we have are:
> 1) LRS applications are expected to run for a long time (weeks, months).
> 2) Currently, all the container logs (from one NM) will be written into a 
> single file. The files could become larger and larger.

This message was sent by Atlassian JIRA

Reply via email to