To expand on this, I'm also having the inverse issue. I had to take down
our main HBase today and now when I try to run hbck it is trying to look
for the hbase:meta,,1 table on a region server that is serving a read
replica metadata table and failing.
It seems like something is messed up on HBase knowing which metadata
table to use when and where that metadata table is located. I image this
is all state that should be maintained in zookeeper but I don't know
where things are going wrong.
-Austin
On 07/10/2018 07:47 PM, Austin Heyne wrote:
I currently have an EMR cluster that's running a continuous ingest.
I'd like to spin up read-only clusters with Spark and Zeppelin to
query with. I've gotten the replica up and running with Spark but when
an executor tries to query HBase it's throwing
NotServingRegionExceptions.
"""
18/07/10 23:23:36 INFO RpcRetryingCaller: Call exception, tries=10,
retries=35, started=38532 ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.NotServingRegionException: Region
hbase:meta,,1 is not online on
ip-10-0-24-63.ec2.internal,16020,1531253339025
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
"""
Which makes sense because on an EMR read replica the metadata table
should be something like hbase:meta_j-2ZMF9CFOOBAR,,1 not
hbase:meta,,1. Further hbase:meta_j-2ZMF9CFOOBAR,,1 is available on
ip-10-0-24-63.ec2.internal. Does anyone know why this is happening or
how to fix it?
Thanks,
--
Austin L. Heyne
