Hi Ted,
Thanks for your reply. I am using Hbase 0.98.14. I have used hbck, but for some
(unknown) reason it has not always resolved inconsistencies.
I have been able to get around these issues so far by deleting ZK entries for
the offending table and restarting Hbase. But I am not sure what causes them in
the first place and if I can avoid those issues through code or not. Also, upon
getting these exceptions is it a good idea to retry the operation. I think
Hbase fails only after it exhausts its attempts so retrying may not be helpful?
Here are 3 logs snippets:
1. TableNamespaceManager isn't ready to serve:
Fri Nov 13 17:47:19 IST 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@44726f67,org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):java.io.IOException:
org.apache.hadoop.hbase.master.TableNamespaceManager isn'tready to serve
atorg.apache.hadoop.hbase.master.TableNamespaceManager.getNamespaceTable(TableNamespaceManager.java:112)
atorg.apache.hadoop.hbase.master.TableNamespaceManager.list(TableNamespaceManager.java:211)
atorg.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3473)
atorg.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3367)
2. TableExistsException:
Caused by: org.apache.hadoop.hbase.TableExistsException:
org.apache.hadoop.hbase.TableExistsException: ldmns:exDocStoreat
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:300)at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:106)at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58)...
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3403)at
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:632)at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:523)
3. TableNotEnabledException:
Caused by: org.apache.hadoop.hbase.TableNotEnabledException:
ldmns:DataDomain_stage is disabled. at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:1139)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:963)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:74)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:833) at
org.apache.hadoop.hbase.client.HTable.get(HTable.java:810) at
org.apache.hadoop.hbase.client.HTable.get(HTable.java:842) at
com.thinkaurelius.titan.diskstorage.hbase.HBaseKeyColumnValueStore.getHelper(HBaseKeyColumnValueStore.java:155)
From: Ted Yu <[email protected]>
To: "[email protected]" <[email protected]>; Sumit Nigam
<[email protected]>
Sent: Sunday, November 15, 2015 10:50 AM
Subject: Re: About exceptions
bq. TableNotEnabledExceptionTableNotFoundExceptionIOException
Can you show log snippets where these exceptions occurred ?Which release of
hbase are you using ?
Have you run hbck to repair the inconsistencies ?
See http://hbase.apache.org/book.html#hbck.in.depth
Cheers
On Sat, Nov 14, 2015 at 8:42 PM, Sumit Nigam <[email protected]>
wrote:
Hi,
There are some exceptions which I face intermittently with Hbase and I thought
some help from experts online can really help me. These are:
TableNotEnabledExceptionTableNotFoundExceptionIOException -
TableNamespaceManager isn't ready to serve
One of the reasons I can see for this seems to be zookeeper and Hbase/ Hdfs
data being out of sync due to an unclean shutdown.
So, my questions are these:
1. Are these exceptions only related to unclean shutdowns?2. Do I need to
explicitly handle them and retry the operation again because they also seem to
indicate that it is some race condition between trying to access a table vs
Hbase enabling them?
Any help is greatly appreciated.
Thanks,Sumit
