[
https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671777#comment-13671777
]
Jason Lowe commented on YARN-742:
---------------------------------
No, this is a 0.23 cluster, and YARN-24 did not go into branch-0.23.
The problem is not verifyAndCreateRemoteLogDir, rather it's createAppDir. That
unconditionally tries to mkdir and setPermission each of the three log levels
(user, user/logs, and user/logs/appID). The mkdir isn't so bad since it
already exists, but the setPermission always occurs and that causes a write
operation on the namenode. That's three write operations per application, per
node. In this cluster's case, that's a lot of operations due to the average
number of nodes used by the applications and number of applications per day.
> Log aggregation causes a lot of redundant setPermission calls
> -------------------------------------------------------------
>
> Key: YARN-742
> URL: https://issues.apache.org/jira/browse/YARN-742
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.7, 2.0.4-alpha
> Reporter: Kihwal Lee
> Assignee: Jason Lowe
>
> In one of our clusters, namenode RPC is spending 45% of its time on serving
> setPermission calls. Further investigation has revealed that most calls are
> redundantly made on /mapred/logs/<user>/logs. Also mkdirs calls are made
> before this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
