I currently have an EMR cluster that's running a continuous ingest. I'd
like to spin up read-only clusters with Spark and Zeppelin to query
with. I've gotten the replica up and running with Spark but when an
executor tries to query HBase it's throwing NotServingRegionExceptions.
"""
18/07/10 23:23:36 INFO RpcRetryingCaller: Call exception, tries=10,
retries=35, started=38532 ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.NotServingRegionException: Region
hbase:meta,,1 is not online on
ip-10-0-24-63.ec2.internal,16020,1531253339025
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
"""
Which makes sense because on an EMR read replica the metadata table
should be something like hbase:meta_j-2ZMF9CFOOBAR,,1 not hbase:meta,,1.
Further hbase:meta_j-2ZMF9CFOOBAR,,1 is available on
ip-10-0-24-63.ec2.internal. Does anyone know why this is happening or
how to fix it?
Thanks,
--
Austin L. Heyne
