Hi Mohit, We should not delete znode's manually which will cause inconsistencies like region may be shown as online on master, but it wont be on region server. That's put is failing in your case. Master restart will bring back your cluster to normal state(recovery any failures in enable/disable). Even hbck also wont solve this problem.
FYI, Presently discussion is going on this issue. You can follow jira associated with this issue at https://issues.apache.org/jira/browse/HBASE-6469 On Thu, Sep 27, 2012 at 8:11 PM, Mohit Anchlia <[email protected]>wrote: > Thanks everyone for the input, it's helpful. I did remove the znode from > /hbase/table/SESSIONID_TIMELINE and after that I was able to list the > table. At that point I tried to do a "put" but when I did a put I got a > message NoRegionServer online. I looked in the logs and it says the Failed > to open region server at nodexxx. When I went to nodexxx it complains > something about unable to run testcompression. > > I setup SNAPPY compression on my table and I also ran SNAPPY compression > test which was successful. Not sure what's going on in the cluster. > On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq <[email protected]> > wrote: > > > Hello Mohit, > > > > It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies for the > > typo. For rest of the things, I feel Ramkrishna sir has provided a good > and > > proper explanation. Please let us know if you still have any doubt or > > question. > > > > Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to share > > space with you people. > > > > Regards, > > Mohammad Tariq > > > > > > > > On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan < > > [email protected]> wrote: > > > > > Hi Mohith > > > First of all thanks to Tariq for his replies. > > > > > > Just to add on, > > > Basically HBase uses the Zookeeper to know the status of the cluster > > like > > > the no of tables enabled, disabled and deleted. > > > Enabled and deleted states are handled bit different in the 0.94 > version. > > > > > > ZK is used for various region assignments. > > > > > > Also the ZK is used to track the Active master and standby master. > > > > > > As you understand correctly that the master is responsible for the > > overall > > > maintenance of the no of tables and their respective states, it seeks > the > > > help of ZK to do it and that is where the states are persisted. > > > > > > Also there are few cases where the enable and disable table are having > > some > > > issues due to some race conditions in the 0.92 versions, In the latest > > > version we are trying to resolve them. > > > You can attach the master and RS logs to identify exactly what caused > > this > > > problem in your case which will be really help ful so that I can be > fixed > > > in > > > the kernel. > > > > > > Regards > > > Ram > > > > > > > -----Original Message----- > > > > From: Mohit Anchlia [mailto:[email protected]] > > > > Sent: Thursday, September 27, 2012 5:09 AM > > > > To: [email protected] > > > > Subject: Re: disable table > > > > > > > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll > > > > restart > > > > hbase and see if it works. > > > > > > > > One thing I don't understand is why is zookeeper holding information > > > > about > > > > this table if it is enabled or disabled? Wouldn't this information be > > > > with > > > > master? > > > > > > > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia > > > > <[email protected]>wrote: > > > > > > > > > I don't see path like /hbase/SESSIONID_TIMELINE > > > > > This is what I see > > > > > > > > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table > > > > > [SESSIONID_TIMELINE] > > > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table > > > > > > > > > > cZxid = 0x1000000fe > > > > > ctime = Mon Sep 10 15:31:45 PDT 2012 > > > > > mZxid = 0x1000000fe > > > > > mtime = Mon Sep 10 15:31:45 PDT 2012 > > > > > pZxid = 0x5000008f1 > > > > > cversion = 3 > > > > > dataVersion = 0 > > > > > aclVersion = 0 > > > > > ephemeralOwner = 0x0 > > > > > dataLength = 0 > > > > > numChildren = 1 > > > > > > > > > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq > > > > <[email protected]>wrote: > > > > > > > > > >> In order to delete a znode you have to go to the ZK shell and > issue > > > > the > > > > >> "delete" command along with the required path. For example : > > > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit > > > > the ZK > > > > >> homepage at : zookeeper.apache.org > > > > >> > > > > >> Actually when we try to fetch data from an Hbase table, the client > > > > or app > > > > >> first contacts the ZK to get the location of server holding the > > > > >> -ROOT- table. From this we come to know about the server hosting > the > > > > >> .META. > > > > >> table. This tells us the location of the server which actually > holds > > > > the > > > > >> rows of interest. Because of some reasons the znode which was > > > > holding this > > > > >> info has either faced some catastrophe or lost the info associated > > > > with > > > > >> this particular table. Or sometimes the znode remains unable to > keep > > > > >> itself > > > > >> updated with the latest changes. That could also be a probable > > > > reason. We > > > > >> should always keep in mind that ZK is the centralized service that > > > > >> actually > > > > >> coordinating everything behind the scene. As a result, any problem > > > > to the > > > > >> ZK quorum means problem with Hbase custer. > > > > >> > > > > >> Regards, > > > > >> Mohammad Tariq > > > > >> > > > > >> > > > > >> > > > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia > > > > <[email protected] > > > > >> >wrote: > > > > >> > > > > >> > Thanks! I do see Inconsistency. How do I remove the znode. And > > > > also > > > > >> could > > > > >> > you please help me understand how this might have happened? > > > > >> > > > > > >> > > > > > >> > ERROR: Region > > > > >> > > > > > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. > not > > > > >> > deployed on any region server. > > > > >> > > > > > >> > > > > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq > > > > <[email protected]> > > > > >> > wrote: > > > > >> > > > > > >> > > A possible reason could be that the znode associated with this > > > > >> particular > > > > >> > > table is not behaving properly. In such case, you can try the > > > > >> following: > > > > >> > > > > > > >> > > Stop Hbase > > > > >> > > Stop ZK > > > > >> > > Take a backup of ZK data > > > > >> > > Restart ZK > > > > >> > > Remove the znode > > > > >> > > Start Hbase again > > > > >> > > > > > > >> > > After this hopefully your table would be enabled. > > > > >> > > > > > > >> > > Regards, > > > > >> > > Mohammad Tariq > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq > > > > <[email protected]> > > > > >> > > wrote: > > > > >> > > > > > > >> > > > Yes. Also have a look at the logs of the problematic region > if > > > > hbck > > > > >> > shows > > > > >> > > > any inconsistency. > > > > >> > > > > > > > >> > > > Regards, > > > > >> > > > Mohammad Tariq > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia < > > > > >> [email protected] > > > > >> > > >wrote: > > > > >> > > > > > > > >> > > >> Which node should I look at for logs? Is this the master > > > > node? I'll > > > > >> > try > > > > >> > > >> hbck. > > > > >> > > >> > > > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq < > > > > >> [email protected]> > > > > >> > > >> wrote: > > > > >> > > >> > > > > >> > > >> > Hello Mohit, > > > > >> > > >> > > > > > >> > > >> > Try hbck once and see if it shows any inconsistency. > > > > Also, > > > > >> you > > > > >> > can > > > > >> > > >> try > > > > >> > > >> > restarting your cluster and deleting the table again. > > > > Having a > > > > >> look > > > > >> > at > > > > >> > > >> the > > > > >> > > >> > logs could also be useful. > > > > >> > > >> > > > > > >> > > >> > Regards, > > > > >> > > >> > Mohammad Tariq > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia < > > > > >> > > [email protected] > > > > >> > > >> > >wrote: > > > > >> > > >> > > > > > >> > > >> > > When I try to disable table I get: > > > > >> > > >> > > > > > > >> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE' > > > > >> > > >> > > ERROR: > org.apache.hadoop.hbase.TableNotEnabledException: > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException: > > > > >> > SESSIONID_TIMELINE > > > > >> > > >> > > Here is some help for this command: > > > > >> > > >> > > Start disable of named table: e.g. "hbase> disable > 't1'" > > > > >> > > >> > > > > > > >> > > >> > > But then I try to enable I get: > > > > >> > > >> > > > > > > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE' > > > > >> > > >> > > ERROR: > org.apache.hadoop.hbase.TableNotDisabledException: > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException: > > > > >> > > SESSIONID_TIMELINE > > > > >> > > >> > > Here is some help for this command: > > > > >> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'" > > > > >> > > >> > > > > > > >> > > >> > > I've tried flush, major_compaction also. I tseems it's > > > > stuck in > > > > >> > > >> > > inconsistent state. Could someone point me to correct > > > > >> direction? I > > > > >> > > am > > > > >> > > >> > using > > > > >> > > >> > > 92.1 > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > >
