Vinod Kumar Vavilapalli commented on YARN-2468:

The overall proposal is fine.

bq. 3. Change the log layout. Currently, all container logs (for the same NM) 
will be written into a single file. This does not work for LRS. We will create 
a directory (named as node id of the NM), under this directory, every time when 
AppLogAggregatorImpl starts to upload container logs; it will create a file 
(named as node_id + timestamp). 
We already have the too-many-files problem, this solution will make it worse. 
I'll give that this is a hard problem to solve - need to think about it a 
little more.

> Log handling for LRS
> --------------------
>                 Key: YARN-2468
>                 URL: https://issues.apache.org/jira/browse/YARN-2468
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-2468.1.patch
> Currently, when application is finished, NM will start to do the log 
> aggregation. But for Long running service applications, this is not ideal. 
> The problems we have are:
> 1) LRS applications are expected to run for a long time (weeks, months).
> 2) Currently, all the container logs (from one NM) will be written into a 
> single file. The files could become larger and larger.

This message was sent by Atlassian JIRA

Reply via email to