[
https://issues.apache.org/jira/browse/YARN-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313775#comment-15313775
]
Rohith Sharma K S commented on YARN-4953:
-----------------------------------------
bq. The main issue with aggregating as containers complete is the additional
load on the namenode
Right, this is major issue in large cluster.
Since log rolling is supported I think it is worth to delete aggregated
completed container log folders when log rolling is enabled. Any potential
issues with it. Thoughts?
> Delete completed container log folder when rolling log aggregation is enabled
> -----------------------------------------------------------------------------
>
> Key: YARN-4953
> URL: https://issues.apache.org/jira/browse/YARN-4953
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: Rohith Sharma K S
> Assignee: Rohith Sharma K S
>
> There would be potential bottle neck when cluster is running with very large
> number of containers on the same NodeManager for single application. The
> linux limits the subfolders count to 32K. If number of containers is greater
> than 32K for an application, there would be container launch failure. At this
> point of time, there are no more containers can be launched in this node.
> Currently log folders are deleted after app is finished. Rolling log
> aggregation aggregates logs to hdfs periodically.
> I think if aggregation is completed for finished containers, then clean up
> can be done i.e deleting log folder for finished containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
