To expand on this, I'm also having the inverse issue. I had to take down our main HBase today and now when I try to run hbck it is trying to look for the hbase:meta,,1 table on a region server that is serving a read replica metadata table and failing.

It seems like something is messed up on HBase knowing which metadata table to use when and where that metadata table is located. I image this is all state that should be maintained in zookeeper but I don't know where things are going wrong.


On 07/10/2018 07:47 PM, Austin Heyne wrote:
I currently have an EMR cluster that's running a continuous ingest. I'd like to spin up read-only clusters with Spark and Zeppelin to query with. I've gotten the replica up and running with Spark but when an executor tries to query HBase it's throwing NotServingRegionExceptions.

18/07/10 23:23:36 INFO RpcRetryingCaller: Call exception, tries=10, retries=35, started=38532 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on ip-10-0-24-63.ec2.internal,16020,1531253339025     at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(     at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(     at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(     at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(
    at org.apache.hadoop.hbase.ipc.RpcExecutor$     at org.apache.hadoop.hbase.ipc.RpcExecutor$

Which makes sense because on an EMR read replica the metadata table should be something like hbase:meta_j-2ZMF9CFOOBAR,,1 not hbase:meta,,1. Further hbase:meta_j-2ZMF9CFOOBAR,,1 is available on ip-10-0-24-63.ec2.internal. Does anyone know why this is happening or how to fix it?


Austin L. Heyne

Reply via email to