[jira] [Comment Edited] (HDFS-13136) Avoid taking FSN lock while doing group member lookup for FSD permission check
[ https://issues.apache.org/jira/browse/HDFS-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469290#comment-16469290 ] Yongjun Zhang edited comment on HDFS-13136 at 5/9/18 6:50 PM: -- HI [~xyao], Thanks for your work here, could it be Resolved since it's committed? I saw it's in branch-3.0 which will target for 3.0.3. Thanks. was (Author: yzhangal): HI [~xyao], Thanks for your work here, could it be Resolved since it's committed? > Avoid taking FSN lock while doing group member lookup for FSD permission check > -- > > Key: HDFS-13136 > URL: https://issues.apache.org/jira/browse/HDFS-13136 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HDFS-13136-branch-3.0.001.patch, > HDFS-13136-branch-3.0.002.patch, HDFS-13136.001.patch, HDFS-13136.002.patch > > > Namenode has FSN lock and FSD lock. Most of the namenode operations need to > take FSN lock first and then FSD lock. The permission check is done via > FSPermissionChecker at FSD layer assuming FSN lock is taken. > The FSPermissionChecker constructor invokes callerUgi.getGroups() that can > take seconds sometimes. There are external cache scheme such SSSD and > internal cache scheme for group lookup. However, the delay could still occur > during cache refresh, which causes severe FSN lock contentions and > unresponsive namenode issues. > Checking the current code, we found that getBlockLocations(..) did it right > but some methods such as getFileInfo(..), getContentSummary(..) did it wrong. > This ticket is open to ensure the group lookup for permission checker is > outside the FSN lock. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13136) Avoid taking FSN lock while doing group member lookup for FSD permission check
[
https://issues.apache.org/jira/browse/HDFS-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364469#comment-16364469
]
Xiaoyu Yao edited comment on HDFS-13136 at 2/14/18 5:29 PM:
Thanks [~szetszwo] for the review. Update patch v2 that fixed the unit test
failures in
{code}
hadoop.hdfs.server.namenode.TestAuditLogger and
hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands
{code}
Now that the getPermissionChecker() is moved out of the FSN lock, the test
mocks are updated to reach deeper to get the expected exception and the audit
log entry. The delta from v1 to v2 is the two unit test changes above. The
other two failures cannot repro.
was (Author: xyao):
Thanks [~szetszwo] for the review. Update patch v2 that fixed the unit test
failures in
hadoop.hdfs.server.namenode.TestAuditLogger and
hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands
Now that the getPermission checker is moved out of the FSN lock, the test mocks
are updated to reach deeper to get the expected exception and the audit log
entry.
The other two failures cannot repro.
> Avoid taking FSN lock while doing group member lookup for FSD permission check
> --
>
> Key: HDFS-13136
> URL: https://issues.apache.org/jira/browse/HDFS-13136
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HDFS-13136.001.patch, HDFS-13136.002.patch
>
>
> Namenode has FSN lock and FSD lock. Most of the namenode operations need to
> take FSN lock first and then FSD lock. The permission check is done via
> FSPermissionChecker at FSD layer assuming FSN lock is taken.
> The FSPermissionChecker constructor invokes callerUgi.getGroups() that can
> take seconds sometimes. There are external cache scheme such SSSD and
> internal cache scheme for group lookup. However, the delay could still occur
> during cache refresh, which causes severe FSN lock contentions and
> unresponsive namenode issues.
> Checking the current code, we found that getBlockLocations(..) did it right
> but some methods such as getFileInfo(..), getContentSummary(..) did it wrong.
> This ticket is open to ensure the group lookup for permission checker is
> outside the FSN lock.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
