[
https://issues.apache.org/jira/browse/HBASE-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526145#comment-13526145
]
Ted Yu commented on HBASE-7295:
-------------------------------
There were so many TestHBaseFsck failures on Jenkins. I was able to reproduce
the following:
testFixAssignmentsAndNoHdfsChecking(org.apache.hadoop.hbase.util.TestHBaseFsck):
expected:<[NOT_DEPLOYED, HOLE_IN_REGION_CHAIN]> but was:<[MULTI_DEPLOYED,
NOT_DEPLOYED, MULTI_DEPLOYED, NOT_DEPLOYED, HOLE_IN_REGION_CHAIN]>
However, I saw the following in test output.
{code}
2012-12-06 20:51:32,370 WARN [PRI IPC Server handler 0 on 57913]
hdfs.DFSClient$DFSInputStream(2232): Short circuit access failed
org.apache.hadoop.security.AccessControlException:
org.apache.hadoop.security.AccessControlException: Can't continue with
getBlockLocalPathInfo() authorization. The user zhihyu.hfs.0 is not allowed to
call getBlockLocalPathInfo
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:513)
at org.apache.hadoop.hdfs.DFSClient.access$800(DFSClient.java:78)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2228)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2381)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorModtime(FSTableDescriptors.java:429)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorModtime(FSTableDescriptors.java:414)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:169)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:132)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:3403)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.security.AccessControlException: Can't continue with
getBlockLocalPathInfo() authorization. The user zhihyu.hfs.0 is not allowed to
call getBlockLocalPathInfo
at
org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockLocalPathAccess(DataNode.java:1849)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1859)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
{code}
Not sure if the above is related to test failure.
> Contention in HBaseClient.getConnection
> ---------------------------------------
>
> Key: HBASE-7295
> URL: https://issues.apache.org/jira/browse/HBASE-7295
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.3
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Fix For: 0.94.3
>
> Attachments: 7295-0.94.txt, 7295-0.94-v2.txt, 7295-trunk.txt
>
>
> HBaseClient.getConnection() synchronizes on the connections object. We found
> severe contention on a thrift gateway which was fanning out roughly 3000+
> calls per second to hbase region servers. The thrift gateway had 2000+
> threads for handling incoming connections. Threads were blocked on the
> syncrhonized block - we set ipc.pool.size to 200. Since we are using
> RoundRobin/ThreadLocal pool only - its not necessary to synchronize on
> connections - it might lead to cases where we might go slightly over the
> ipc.max.pool.size() but the additional connections would timeout after
> maxIdleTime - underlying PoolMap connections object is thread safe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
