Yep I know, I think I was also on the mailing list thread that inspired HBASE-3557 :) But what can I do in the current version better than that?
In any case, I do that only when I catch IOException. So why am I getting this IOException. I deleted the hbase folder in HDFS and tried doing everything from start. Here is the HMaster log before the IOException was thrown: http://pastebin.com/x1BUuPpQ thanks, Hari On Wed, Mar 30, 2011 at 12:21 AM, Jean-Daniel Cryans <[email protected]>wrote: > There's a reason why disabling takes time, if you delete rows from > .META. you might end up in an inconsistent situation and we'll have a > hard time helping you :) > > So HBASE-3557 is what you want. > > Regarding your current issue, RIT is "region in transition" meaning > that the region is in a state recognized by the master as either > moving from one region server to another, or just closing. Normally > after disabling a table its regions all left their in transition > state, but by deleting the rows directly in .META. who knows exactly > what's the state of your table? > > J-D > > On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar > <[email protected]> wrote: > > Hi J-D, > > > > Here is the tail of HMaster log: > > > > 2011-03-29 23:48:51,155 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:52,158 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:53,161 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:54,163 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:55,164 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:56,166 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:57,168 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:58,170 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:48:59,172 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:49:00,174 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:49:01,176 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:49:02,178 DEBUG > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on > > region to clear regions in transition; > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, > > ts=1301422405271 > > 2011-03-29 23:49:02,178 ERROR > > org.apache.hadoop.hbase.master.handler.TableEventHandler: Error > manipulating > > table AcContact > > java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for > > region to leave region > > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions > > at > > > org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60) > > at > > > org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66) > > at > > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > What is "clearing regions in transition"? What could be the issue? > > > > I was told before too that deleting from META isn't recommended. But we > were > > facing the disable problem way too often. So we thought we'd make it > > in-built. What other alternatives do I have? > > > > Thx, > > Hari > > > > On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans < > [email protected]>wrote: > > > >> The 60 secs timeout means that the client was waiting on the master > >> for some operation but the master took longer than 60 secs to do it, > >> so its log should be the next place too look for something whack. > >> > >> BTW deleting the rows from .META. directly is probably the worst thing > >> you can do. > >> > >> J-D > >> > >> On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar > >> <[email protected]> wrote: > >> > Here is the stack trace: > >> > > >> > 11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client > connection, > >> > connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection > >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection > to > >> > server hadoop2/192.168.1.111:2181 > >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection > >> established > >> > to hadoop2/192.168.1.111:2181, initiating session > >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment > >> complete > >> > on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c, > >> > negotiated timeout = 180000 > >> > 11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row > >> > AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. > >> > 11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries > >> from > >> > META. Retrying to create table > >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create > table > >> even > >> > after Cleaning Meta entries > >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: > >> > ************************************************************* > >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/ > >> > 192.168.1.57:60000 failed on socket timeout exception: > >> > java.net.SocketTimeoutException: 60000 millis timeout while waiting > for > >> > channel to be ready for read. ch : > >> java.nio.channels.SocketChannel[connected > >> > local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000] > >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: > >> > > >> > org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802) > >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775) > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) > >> > $Proxy4.createTable(Unknown Source) > >> > > >> > org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341) > >> > > >> > org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303) > >> > > >> > org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227) > >> > > >> > com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown > >> > Source) > >> > com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown > >> Source) > >> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > java.lang.reflect.Method.invoke(Method.java:597) > >> > org.apache.hadoop.util.RunJar.main(RunJar.java:156) > >> > > >> > The "CleanFromMeta" function catches IOException and deletes all rows > >> from > >> > .META. We had added this in the exception catch block because we used > to > >> > face the "Table taking too long to be disabled" exception often. It > seems > >> > the rows in META already get created when the IOException is thrown. > >> > CleanFromMeta cleans .META. and then I try again to create the table, > >> after > >> > which I get the socket timeout exception. > >> > > >> > I am using Hadoop v0.20.2, r911707. Can this be a reason for this > error? > >> I > >> > get the message "You are currently running the HMaster without HDFS > >> append > >> > support enabled. This may result in data loss. Please see the HBase > >> > wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for > >> > details." on the HBase Master UI. > >> > > >> > On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar > >> > <[email protected]>wrote: > >> > > >> >> Hi Stack > >> >> > >> >> yes the tablename is AcContact. The tableName variable was wrong. > >> >> Fixed it now but I still get the same error. Schema is just something > >> >> created by parsing an XML file which has stuff like column family > >> >> name, compression type etc so I guess it doesn't have much to do with > >> >> version. Except that I had to change the bloom filter variable to > >> >> String ( used to be boolean in 0.20.6). I will paste the stack trace > >> >> asap > >> >> > >> >> hari > >> >> > >> >> On Monday, March 28, 2011, Stack <[email protected]> wrote: > >> >> > Can I see more of the stack track please Hari and is AcContact the > >> >> > table you are creating? Is the schema you've saved aside one you > >> >> > created with 0.20 hbase? I don't think it matters but asking just > in > >> >> > case. > >> >> > St.Ack > >> >> > > >> >> > On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar > >> >> > <[email protected]> wrote: > >> >> >> Hi, > >> >> >> > >> >> >> I am trying to create table in hbase v0.90.1 and I get the > following > >> >> error: > >> >> >> > >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket > >> connection > >> >> to > >> >> >> server hadoop2/192.168.1.111:2181 > >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection > >> >> established > >> >> >> to hadoop2/192.168.1.111:2181, initiating s > >> >> >> ession > >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment > >> >> complete > >> >> >> on server hadoop2/192.168.1.111:2181, sess > >> >> >> ionid = 0x12efc946d66000b, negotiated timeout = 180000 > >> >> >> 11/03/28 18:39:52 INFO > >> >> client.HConnectionManager$HConnectionImplementation: > >> >> >> Closed zookeeper sessionid=0x12efc946d6600 > >> >> >> 0b > >> >> >> 11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: > >> 0x12efc946d66000b > >> >> >> closed > >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down > >> >> >> 11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught > IOException: > >> No > >> >> >> server address listed in .META. for region > >> >> >> AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while > >> >> creating > >> >> >> table: Table1 > >> >> >> > >> >> >> This is the code I am using: > >> >> >> ..... > >> >> >> HTableDescriptor desc = > >> >> CreateTableByXML.convertSchemaToDescriptor(schema); > >> >> >> try { > >> >> >> hbaseAdmin.createTable(desc); > >> >> >> } catch (IOException e) { > >> >> >> CreateTableByXML.LOG.error("Caught IOException: " + > >> e.getMessage() > >> >> >> + " while creating table: " + tableName); > >> >> >> .... > >> >> >> ..... > >> >> >> > >> >> >> > >> >> >> It used to work fine in v0.20.6. I upgraded today to v0.90.1 and > it > >> >> started > >> >> >> giving this error. Any ideas? > >> >> >> > >> >> >> Hari > >> >> >> > >> >> > > >> >> > >> > > >> > > >
