[
https://issues.apache.org/jira/browse/HDFS-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-5010:
-----------------------------
Description:
According to the "worst" jstack of a busy namenode I took, there were 29
BLOCKED handler threads. All of them were blocked on the UGI class lock. Here
is the breakdown:
- 2 ensureInitialized() - from non static synchronized methods. HADOOP-9748
will unblock these.
- 27 getCurrentUser()
Among the 27 threads that were blocked at getCurrentUser(),
- 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most
namenode RPC methods
- 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
- 1 NameNodeRpcServer.mkdirs
I think FSPermissionChecker can be modified to be created with a passed in UGI.
FSNamesystem can the one already stored in RPC server by calling
getRemoteUser(). This will eliminate a bulk of getCurrentUser() calls from
namenode RPC handlers. A similar change can be made to mkdirs. Block token
generation is not as straightforward. Even without it we can eliminate majority
of the calls.
was:
According to the "worst" jstack of a busy namenode I took, there were 29
BLOCKED handler threads. All of them were blocked on the UGI class lock. Here
is the breakdown:
- 2 ensureInitialized() - from non static synchronized methods. This Jira will
unblock these.
- 27 getCurrentUser()
Among the 27 threads that were blocked at getCurrentUser(),
- 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most
namenode RPC methods
- 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
- 1 NameNodeRpcServer.mkdirs
I think FSPermissionChecker can be modified to be created with a passed in UGI.
FSNamesystem can the one already stored in RPC server by calling
getRemoteUser(). This will eliminate a bulk of getCurrentUser() calls from
namenode RPC handlers. A similar change can be made to mkdirs. Block token
generation is not as straightforward. Even without it we can eliminate majority
of the calls.
> Reduce the frequency of getCurrentUser() calls from namenode
> ------------------------------------------------------------
>
> Key: HDFS-5010
> URL: https://issues.apache.org/jira/browse/HDFS-5010
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, performance
> Affects Versions: 2.1.0-beta, 0.23.9
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Attachments: HDFS-5010.patch
>
>
> According to the "worst" jstack of a busy namenode I took, there were 29
> BLOCKED handler threads. All of them were blocked on the UGI class lock. Here
> is the breakdown:
> - 2 ensureInitialized() - from non static synchronized methods. HADOOP-9748
> will unblock these.
> - 27 getCurrentUser()
> Among the 27 threads that were blocked at getCurrentUser(),
> - 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most
> namenode RPC methods
> - 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
> - 1 NameNodeRpcServer.mkdirs
> I think FSPermissionChecker can be modified to be created with a passed in
> UGI. FSNamesystem can the one already stored in RPC server by calling
> getRemoteUser(). This will eliminate a bulk of getCurrentUser() calls from
> namenode RPC handlers. A similar change can be made to mkdirs. Block token
> generation is not as straightforward. Even without it we can eliminate
> majority of the calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
