Rohith Sharma K S created YARN-4953:
---------------------------------------
Summary: Delete completed container log folder when rolling log
aggregation is enabled
Key: YARN-4953
URL: https://issues.apache.org/jira/browse/YARN-4953
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S
There would be potential bottle neck when cluster is running with very large
number of containers on the same NodeManager for single application. The linux
limits the subfolders count to 32K. If number of containers is greater than 32K
for an application, there would be container launch failure. At this point of
time, there are no more containers can be launched in this node.
Currently log folders are deleted after app is finished. Rolling log
aggregation aggregates logs to hdfs periodically.
I think if aggregation is completed for finished containers, then clean up can
be done i.e deleting log folder for finished containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
