didn't paste the full stack, sorry
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: kylin_metadata,,1535172845912.5d388edae3b13f9f1d6a709720cc5378. is closing at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8612) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2772) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7364) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7323) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7316) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2263) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36609) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2354) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.readResponse(BlockingRpcConnection.java:600) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:334) ... 1 more On Tue, Aug 28, 2018 at 7:59 AM Sonny Heer <[email protected]> wrote: > while using EMR and autoscaling during multiple cube builds. > > Some builds intermittently fail with the following exception and halt that > cube build. Typically restarting the build completes successfully. > > --------------- > > 5d388edae3b13f9f1d6a709720cc5378. is closing > > at > org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8612) > > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2772) > > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7364) > > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7323) > > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7316) > > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2263) > > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36609) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2354) > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > > > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) > > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94) > > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409) > > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405) > > at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) > > at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) > > at > org.apache.hadoop.hbase.ipc.BlockingRpcConnection.readResponse(BlockingRpcConnection.java:600) > > at > org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:334) > > ... 1 more > > ----------------- > > > Anyone run into this and know the cause? I'm assuming because of the > autoscaling the region START/END change as the cluster grows shrinks in > turn this may change where the metadata table is hosted... > > Also a ticket for moving metadata off of hbase here: > https://issues.apache.org/jira/browse/KYLIN-3488 > > Thanks > >
