Gary Helmling created HBASE-18141: ------------------------------------- Summary: Regionserver fails to shutdown when abort triggered in RegionScannerImpl during RPC call Key: HBASE-18141 URL: https://issues.apache.org/jira/browse/HBASE-18141 Project: HBase Issue Type: Bug Components: regionserver, security Affects Versions: 1.3.1 Reporter: Gary Helmling Assignee: Gary Helmling Priority: Critical Fix For: 1.3.2
When an abort is triggered within the RPC call path by HRegion.RegionScannerImpl, AccessController is incorrectly apply the RPC caller identity in the RegionServerObserver.preStopRegionServer() hook. This leaves the regionserver in a non-responsive state, where its regions are not reassigned and it returns exceptions for all requests. When an abort is triggered on the server side, we should not allow a coprocessor to reject the abort at all. Here is a sample stack trace: {noformat} 17/05/25 06:10:29 FATAL regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.security.token.TokenProvider] 17/05/25 06:10:29 WARN regionserver.HRegionServer: The region server did not stop org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'rpcuser' (global, action=ADMIN) at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:548) at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522) at org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501) at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:86) at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execShutdown(RegionServerCoprocessorHost.java:300) at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:82) at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2118) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2125) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.abortRegionServer(HRegion.java:6326) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleFileNotFound(HRegion.java:6319) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5941) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6084) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5858) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2649) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2320) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) {noformat} I haven't yet evaluated which other release branches this might apply to. I have a patch currently in progress, which I will post as soon as I complete a test case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)