[
https://issues.apache.org/jira/browse/YARN-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312315#comment-15312315
]
Jason Lowe commented on YARN-5193:
----------------------------------
Main thing to watch out for here is additional load to the namenode.
Originally log aggregation used to aggregate containers as they completed, but
that caused nodemanagers to hold open files for every application it had
aggregated at least one container for the duration of the application. The
lease renewal load on the namenode was significant, so it was switched to
aggregate at the end as a workaround.
> For long running services, aggregate logs when a container completes instead
> of when the app completes
> ------------------------------------------------------------------------------------------------------
>
> Key: YARN-5193
> URL: https://issues.apache.org/jira/browse/YARN-5193
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Siddharth Seth
>
> For a long running service, containers will typically not complete very
> often. However, when a container completes - it would be useful to aggregate
> the logs right then, instead of waiting for the app to complete.
> This will allow the command line log tool to lookup containers for an app
> from the log file index itself, instead of having to go and talk to YARN.
> Talking to YARN really only works if ATS is enabled, and YARN is configured
> to publish container information to ATS (That may not always be the case -
> since this can overload ATS quite fast).
> There's some added benefits like cleaning out local disk space early, instead
> of waiting till the app completes. (There's probably a separate jira
> somewhere about cleanup of container for long running services anyway)
> cc [~vinodkv], [~xgong]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
