Thank you Ted.
I was unaware of both those issues. The issue with these exceptions is that
they are intermittent and do not replicate easily. So, let me see if I can
replicate it with trace enabled. For #1, should retrying be attempted? Or
possibly, if we increase #retries from our end, is there a chance that it may
get past the issue? I like the idea of master having a WAL (HBASE-14190) to
find/ fix such inconsistencies.
#2 That trace showed up in a hbase client.
#3 unclean shutdown is possibly one case? I do not explicitly enable/ disable
tables. So, I assume those reasons may be related to Hbase code? And any advise
on if I can somehow avoid it in first place?
Thanks,Sumit
From: Ted Yu <[email protected]>
To: Sumit Nigam <[email protected]>
Cc: "[email protected]" <[email protected]>
Sent: Sunday, November 15, 2015 3:34 PM
Subject: Re: About exceptions
Sumit:For #1, I have seen a similar issue (HBASE-14190, though on hbase 1.x
release).If you have debug logging enabled, please pastebin relevant master log
snippet so that we can take a closer look.
For #2, I am bit confused - I didn't find CreateTableProcedure.java in 0.98
branch. To my knowledge, CreateTableProcedure is only in hbase 1 release.Did
you see the stack trace in master log ?
For #3, there could be various reasons a table was not enabled.You can trace
the table assignment in master log, check log from hbase:meta server to see if
you can find some clue.
bq. Hbase fails only after it exhausts its attempts so retrying may not be
helpful?
Your understanding should be correct.
I want to bring your attention to HBASE-12070 which helps you fix ZK
inconsistencies.
Cheers
On Sun, Nov 15, 2015 at 12:29 AM, Sumit Nigam <[email protected]> wrote:
Hi Ted,
Thanks for your reply. I am using Hbase 0.98.14. I have used hbck, but for some
(unknown) reason it has not always resolved inconsistencies.
I have been able to get around these issues so far by deleting ZK entries for
the offending table and restarting Hbase. But I am not sure what causes them in
the first place and if I can avoid those issues through code or not. Also, upon
getting these exceptions is it a good idea to retry the operation. I think
Hbase fails only after it exhausts its attempts so retrying may not be helpful?
Here are 3 logs snippets:
1. TableNamespaceManager isn't ready to serve:
Fri Nov 13 17:47:19 IST 2015,
org.apache.hadoop.hbase.client.RpcRetryingCaller@44726f67,org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):java.io.IOException:
org.apache.hadoop.hbase.master.TableNamespaceManager isn'tready to serve
atorg.apache.hadoop.hbase.master.TableNamespaceManager.getNamespaceTable(TableNamespaceManager.java:112)
atorg.apache.hadoop.hbase.master.TableNamespaceManager.list(TableNamespaceManager.java:211)
atorg.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3473)
atorg.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:3367)
2. TableExistsException:
Caused by: org.apache.hadoop.hbase.TableExistsException:
org.apache.hadoop.hbase.TableExistsException: ldmns:exDocStoreat
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:300)at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:106)at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58)...
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3403)at
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:632)at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:523)
3. TableNotEnabledException:
Caused by: org.apache.hadoop.hbase.TableNotEnabledException:
ldmns:DataDomain_stage is disabled. at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:1139)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:963)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:74)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:833) at
org.apache.hadoop.hbase.client.HTable.get(HTable.java:810) at
org.apache.hadoop.hbase.client.HTable.get(HTable.java:842) at
com.thinkaurelius.titan.diskstorage.hbase.HBaseKeyColumnValueStore.getHelper(HBaseKeyColumnValueStore.java:155)
From: Ted Yu <[email protected]>
To: "[email protected]" <[email protected]>; Sumit Nigam
<[email protected]>
Sent: Sunday, November 15, 2015 10:50 AM
Subject: Re: About exceptions
bq. TableNotEnabledExceptionTableNotFoundExceptionIOException
Can you show log snippets where these exceptions occurred ?Which release of
hbase are you using ?
Have you run hbck to repair the inconsistencies ?
See http://hbase.apache.org/book.html#hbck.in.depth
Cheers
On Sat, Nov 14, 2015 at 8:42 PM, Sumit Nigam <[email protected]>
wrote:
Hi,
There are some exceptions which I face intermittently with Hbase and I thought
some help from experts online can really help me. These are:
TableNotEnabledExceptionTableNotFoundExceptionIOException -
TableNamespaceManager isn't ready to serve
One of the reasons I can see for this seems to be zookeeper and Hbase/ Hdfs
data being out of sync due to an unclean shutdown.
So, my questions are these:
1. Are these exceptions only related to unclean shutdowns?2. Do I need to
explicitly handle them and retry the operation again because they also seem to
indicate that it is some race condition between trying to access a table vs
Hbase enabling them?
Any help is greatly appreciated.
Thanks,Sumit
