Sumit: For #1, I have seen a similar issue (HBASE-14190, though on hbase 1.x release). If you have debug logging enabled, please pastebin relevant master log snippet so that we can take a closer look.
For #2, I am bit confused - I didn't find CreateTableProcedure.java in 0.98 branch. To my knowledge, CreateTableProcedure is only in hbase 1 release. Did you see the stack trace in master log ? For #3, there could be various reasons a table was not enabled. You can trace the table assignment in master log, check log from hbase:meta server to see if you can find some clue. bq. Hbase fails only after it exhausts its attempts so retrying may not be helpful? Your understanding should be correct. I want to bring your attention to HBASE-12070 which helps you fix ZK inconsistencies. Cheers On Sun, Nov 15, 2015 at 12:29 AM, Sumit Nigam <[email protected]> wrote: > Hi Ted, > > Thanks for your reply. I am using Hbase 0.98.14. I have used hbck, but for > some (unknown) reason it has not always resolved inconsistencies. > > I have been able to get around these issues so far by deleting ZK entries > for the offending table and restarting Hbase. But I am not sure what causes > them in the first place and if I can avoid those issues through code or > not. Also, upon getting these exceptions is it a good idea to retry the > operation. I think Hbase fails only after it exhausts its attempts so > retrying may not be helpful? > > > Here are 3 logs snippets: > > 1. TableNamespaceManager isn't ready to serve: > > Fri Nov 13 17:47:19 IST 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@44726f67, > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): > java.io.*IOException*: org.apache.hadoop.hbase.master.*TableNamespaceManager > isn't ready to serve* > at > org.apache.hadoop.hbase.master.TableNamespaceManager.getNamespaceTable(TableNamespaceManager.java:112) > at > org.apache.hadoop.hbase.master.TableNamespaceManager.list(TableNamespaceManager.java:211) > at > org.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3473) > at > org.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3367) > > > > 2. TableExistsException: > > Caused by: org.apache.hadoop.hbase.TableExistsException: > org.apache.hadoop.hbase.*TableExistsException: ldmns:exDocStore* > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:300) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:106) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58) > ... > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90) > at > org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3403) > at > org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:632) > at org.apache.hadoop.hbase.client.HBaseAdmin.*createTable* > (HBaseAdmin.java:523) > > > 3. TableNotEnabledException: > > Caused by: org.apache.hadoop.hbase.*TableNotEnabledException*: > ldmns:DataDomain_stage is disabled. > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:1139) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:963) > at > org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:74) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:833) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:810) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:842) > at > com.thinkaurelius.titan.diskstorage.hbase.HBaseKeyColumnValueStore.getHelper(HBaseKeyColumnValueStore.java:155) > > ------------------------------ > *From:* Ted Yu <[email protected]> > *To:* "[email protected]" <[email protected]>; Sumit Nigam < > [email protected]> > *Sent:* Sunday, November 15, 2015 10:50 AM > *Subject:* Re: About exceptions > > bq. TableNotEnabledExceptionTableNotFoundExceptionIOException > > Can you show log snippets where these exceptions occurred ? > Which release of hbase are you using ? > > Have you run hbck to repair the inconsistencies ? > > See http://hbase.apache.org/book.html#hbck.in.depth > > Cheers > > > > On Sat, Nov 14, 2015 at 8:42 PM, Sumit Nigam <[email protected] > > wrote: > > Hi, > There are some exceptions which I face intermittently with Hbase and I > thought some help from experts online can really help me. These are: > TableNotEnabledExceptionTableNotFoundExceptionIOException - > TableNamespaceManager isn't ready to serve > > One of the reasons I can see for this seems to be zookeeper and Hbase/ > Hdfs data being out of sync due to an unclean shutdown. > So, my questions are these: > 1. Are these exceptions only related to unclean shutdowns?2. Do I need to > explicitly handle them and retry the operation again because they also seem > to indicate that it is some race condition between trying to access a table > vs Hbase enabling them? > Any help is greatly appreciated. > Thanks,Sumit > > > > >
