[
https://issues.apache.org/jira/browse/HBASE-16773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15550185#comment-15550185
]
Enis Soztutar commented on HBASE-16773:
---------------------------------------
Worth mentioning that we have seen this in a cluster with 20 priority handlers,
and once this happened the region server was deadlocked.
As for the patch, can you please use the Table or Connection obtained from the
coprocessor environment rather than passing the region.
> AccessController should access local region if possible
> -------------------------------------------------------
>
> Key: HBASE-16773
> URL: https://issues.apache.org/jira/browse/HBASE-16773
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: 16773.v2.txt
>
>
> We observed the following in the stack trace of region server on a 1.1.2
> cluster:
> {code}
> "PriorityRpcServer.handler=19,queue=1,port=60200" #225 daemon prio=5
> os_prio=0 tid=0x00007fb562296000 nid=0x81c0 runnable [0x00007fb509a27000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000003d4dfd770> (a sun.nio.ch.Util$2)
> - locked <0x00000003d4dfd760> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000003d4dfd648> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> - locked <0x00000003d7dae180> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)
> - locked <0x00000003d5c7edc0> (a
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)
> - locked <0x00000003d5c7edc0> (a
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1199)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:32627)
> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:854)
> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:845)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:862)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:828)
> at
> org.apache.hadoop.hbase.security.access.AccessControlLists.getPermissions(AccessControlLists.java:461)
> at
> org.apache.hadoop.hbase.security.access.AccessController.updateACL(AccessController.java:260)
> at
> org.apache.hadoop.hbase.security.access.AccessController.postPut(AccessController.java:1661)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$32.call(RegionCoprocessorHost.java:940)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:936)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3287)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2902)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2844)
> {code}
> There were 20 threads stuck in the retrieval of permissions.
> AccessController shouldn't use Connection if getPermissions() can be
> satisfied by accessing local hbase:acl region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
