[jira] [Updated] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-742: -- Fix Version/s: (was: 3.0.0) > Log aggregation causes a lot of redundant setPermission calls > - > > Key: YARN-742 > URL: https://issues.apache.org/jira/browse/YARN-742 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Jason Lowe > Fix For: 2.1.0-beta, 0.23.9 > > Attachments: YARN-742-1.branch-0.23.patch, YARN-742-1.patch, > YARN-742.patch > > > In one of our clusters, namenode RPC is spending 45% of its time on serving > setPermission calls. Further investigation has revealed that most calls are > redundantly made on /mapred/logs//logs. Also mkdirs calls are made > before this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated YARN-742: Hadoop Flags: Reviewed > Log aggregation causes a lot of redundant setPermission calls > - > > Key: YARN-742 > URL: https://issues.apache.org/jira/browse/YARN-742 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Jason Lowe > Fix For: 3.0.0, 2.1.0-beta, 0.23.9 > > Attachments: YARN-742-1.branch-0.23.patch, YARN-742-1.patch, > YARN-742.patch > > > In one of our clusters, namenode RPC is spending 45% of its time on serving > setPermission calls. Further investigation has revealed that most calls are > redundantly made on /mapred/logs//logs. Also mkdirs calls are made > before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-742: Attachment: YARN-742-1.branch-0.23.patch Thanks for the review, Kihwal, and apologies with the patch issues. Here's a patch for branch-0.23 > Log aggregation causes a lot of redundant setPermission calls > - > > Key: YARN-742 > URL: https://issues.apache.org/jira/browse/YARN-742 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Jason Lowe > Attachments: YARN-742-1.branch-0.23.patch, YARN-742-1.patch, > YARN-742.patch > > > In one of our clusters, namenode RPC is spending 45% of its time on serving > setPermission calls. Further investigation has revealed that most calls are > redundantly made on /mapred/logs//logs. Also mkdirs calls are made > before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-742: Attachment: YARN-742-1.patch Updated the patch. > Log aggregation causes a lot of redundant setPermission calls > - > > Key: YARN-742 > URL: https://issues.apache.org/jira/browse/YARN-742 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Jason Lowe > Attachments: YARN-742-1.patch, YARN-742.patch > > > In one of our clusters, namenode RPC is spending 45% of its time on serving > setPermission calls. Further investigation has revealed that most calls are > redundantly made on /mapred/logs//logs. Also mkdirs calls are made > before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-742: Attachment: YARN-742.patch Patch to walk back up the app log dir path to check for the existence of a directory before blindly proceeding to create it. It gets out early if it finds a path that exists. In addition to the unit test, I manually tested this on a single-node cluster verifying the directories are created iff they are missing and permissions are set iff necessary due to a too-restrictive umask. > Log aggregation causes a lot of redundant setPermission calls > - > > Key: YARN-742 > URL: https://issues.apache.org/jira/browse/YARN-742 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Jason Lowe > Attachments: YARN-742.patch > > > In one of our clusters, namenode RPC is spending 45% of its time on serving > setPermission calls. Further investigation has revealed that most calls are > redundantly made on /mapred/logs//logs. Also mkdirs calls are made > before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
