[
https://issues.apache.org/jira/browse/YARN-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114754#comment-14114754
]
Xuan Gong commented on YARN-2468:
---------------------------------
We will rely on user’s log application (such as log4j) to do the rollover for
the logs. We have already exposed an environment variable LOG_DIR that the
users can use to set up their log application. We will provide a log service
that will upload all the qualified logs periodically.
Several changes in this patch:
1. Create a logContext (add into ApplicationSubmissionContext), which includes
include_patterns, exclude_patterns (Those are used to filter the logs) and
intervals (defines how often log aggregation service uploads the container
logs).
2. AppLogAggregatorImpl will upload container logs periodically instead of
waiting for the application to finish.
3. Change the log layout. Currently, all container logs (for the same NM) will
be written into a single file. This does not work for LRS. We will create a
directory (named as node id of the NM), under this directory, every time when
AppLogAggregatorImpl starts to upload container logs; it will create a file
(named as node_id + timestamp).
> Log handling for LRS
> --------------------
>
> Key: YARN-2468
> URL: https://issues.apache.org/jira/browse/YARN-2468
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Xuan Gong
> Assignee: Xuan Gong
>
> Currently, when application is finished, NM will start to do the log
> aggregation. But for Long running service applications, this is not ideal.
> The problems we have are:
> 1) LRS applications are expected to run for a long time (weeks, months).
> 2) Currently, all the container logs (from one NM) will be written into a
> single file. The files could become larger and larger.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
