There's a reason why disabling takes time, if you delete rows from .META. you might end up in an inconsistent situation and we'll have a hard time helping you :)
So HBASE-3557 is what you want. Regarding your current issue, RIT is "region in transition" meaning that the region is in a state recognized by the master as either moving from one region server to another, or just closing. Normally after disabling a table its regions all left their in transition state, but by deleting the rows directly in .META. who knows exactly what's the state of your table? J-D On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar <[email protected]> wrote: > Hi J-D, > > Here is the tail of HMaster log: > > 2011-03-29 23:48:51,155 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:52,158 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:53,161 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:54,163 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:55,164 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:56,166 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:57,168 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:58,170 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:48:59,172 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:49:00,174 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:49:01,176 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:49:02,178 DEBUG > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > region to clear regions in transition; > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > ts=1301422405271 > 2011-03-29 23:49:02,178 ERROR > org.apache.hadoop.hbase.master.handler.TableEventHandler: Error manipulating > table AcContact > java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for > region to leave region > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions > at > org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60) > at > org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > What is "clearing regions in transition"? What could be the issue? > > I was told before too that deleting from META isn't recommended. But we were > facing the disable problem way too often. So we thought we'd make it > in-built. What other alternatives do I have? > > Thx, > Hari > > On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans > <[email protected]>wrote: > >> The 60 secs timeout means that the client was waiting on the master >> for some operation but the master took longer than 60 secs to do it, >> so its log should be the next place too look for something whack. >> >> BTW deleting the rows from .META. directly is probably the worst thing >> you can do. >> >> J-D >> >> On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar >> <[email protected]> wrote: >> > Here is the stack trace: >> > >> > 11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection, >> > connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to >> > server hadoop2/192.168.1.111:2181 >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection >> established >> > to hadoop2/192.168.1.111:2181, initiating session >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment >> complete >> > on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c, >> > negotiated timeout = 180000 >> > 11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row >> > AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. >> > 11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries >> from >> > META. Retrying to create table >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table >> even >> > after Cleaning Meta entries >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: >> > ************************************************************* >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/ >> > 192.168.1.57:60000 failed on socket timeout exception: >> > java.net.SocketTimeoutException: 60000 millis timeout while waiting for >> > channel to be ready for read. ch : >> java.nio.channels.SocketChannel[connected >> > local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000] >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: >> > >> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802) >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775) >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >> > $Proxy4.createTable(Unknown Source) >> > >> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341) >> > >> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303) >> > >> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227) >> > >> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown >> > Source) >> > com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown >> Source) >> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> > java.lang.reflect.Method.invoke(Method.java:597) >> > org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> > >> > The "CleanFromMeta" function catches IOException and deletes all rows >> from >> > .META. We had added this in the exception catch block because we used to >> > face the "Table taking too long to be disabled" exception often. It seems >> > the rows in META already get created when the IOException is thrown. >> > CleanFromMeta cleans .META. and then I try again to create the table, >> after >> > which I get the socket timeout exception. >> > >> > I am using Hadoop v0.20.2, r911707. Can this be a reason for this error? >> I >> > get the message "You are currently running the HMaster without HDFS >> append >> > support enabled. This may result in data loss. Please see the HBase >> > wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for >> > details." on the HBase Master UI. >> > >> > On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar >> > <[email protected]>wrote: >> > >> >> Hi Stack >> >> >> >> yes the tablename is AcContact. The tableName variable was wrong. >> >> Fixed it now but I still get the same error. Schema is just something >> >> created by parsing an XML file which has stuff like column family >> >> name, compression type etc so I guess it doesn't have much to do with >> >> version. Except that I had to change the bloom filter variable to >> >> String ( used to be boolean in 0.20.6). I will paste the stack trace >> >> asap >> >> >> >> hari >> >> >> >> On Monday, March 28, 2011, Stack <[email protected]> wrote: >> >> > Can I see more of the stack track please Hari and is AcContact the >> >> > table you are creating? Is the schema you've saved aside one you >> >> > created with 0.20 hbase? I don't think it matters but asking just in >> >> > case. >> >> > St.Ack >> >> > >> >> > On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar >> >> > <[email protected]> wrote: >> >> >> Hi, >> >> >> >> >> >> I am trying to create table in hbase v0.90.1 and I get the following >> >> error: >> >> >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket >> connection >> >> to >> >> >> server hadoop2/192.168.1.111:2181 >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection >> >> established >> >> >> to hadoop2/192.168.1.111:2181, initiating s >> >> >> ession >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment >> >> complete >> >> >> on server hadoop2/192.168.1.111:2181, sess >> >> >> ionid = 0x12efc946d66000b, negotiated timeout = 180000 >> >> >> 11/03/28 18:39:52 INFO >> >> client.HConnectionManager$HConnectionImplementation: >> >> >> Closed zookeeper sessionid=0x12efc946d6600 >> >> >> 0b >> >> >> 11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: >> 0x12efc946d66000b >> >> >> closed >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down >> >> >> 11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException: >> No >> >> >> server address listed in .META. for region >> >> >> AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while >> >> creating >> >> >> table: Table1 >> >> >> >> >> >> This is the code I am using: >> >> >> ..... >> >> >> HTableDescriptor desc = >> >> CreateTableByXML.convertSchemaToDescriptor(schema); >> >> >> try { >> >> >> hbaseAdmin.createTable(desc); >> >> >> } catch (IOException e) { >> >> >> CreateTableByXML.LOG.error("Caught IOException: " + >> e.getMessage() >> >> >> + " while creating table: " + tableName); >> >> >> .... >> >> >> ..... >> >> >> >> >> >> >> >> >> It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it >> >> started >> >> >> giving this error. Any ideas? >> >> >> >> >> >> Hari >> >> >> >> >> > >> >> >> > >> >
