hadoop3 was trying to open it, but it seems like it's not able to. It really looks like: https://issues.apache.org/jira/browse/HBASE-3669
You should also check what's going on on that slave. J-D On Tue, Mar 29, 2011 at 12:12 PM, Hari Sreekumar <[email protected]> wrote: > Yep I know, I think I was also on the mailing list thread that inspired > HBASE-3557 :) But what can I do in the current version better than that? > > In any case, I do that only when I catch IOException. So why am I getting > this IOException. I deleted the hbase folder in HDFS and tried doing > everything from start. Here is the HMaster log before the IOException was > thrown: http://pastebin.com/x1BUuPpQ > > thanks, > Hari > > On Wed, Mar 30, 2011 at 12:21 AM, Jean-Daniel Cryans > <[email protected]>wrote: > >> There's a reason why disabling takes time, if you delete rows from >> .META. you might end up in an inconsistent situation and we'll have a >> hard time helping you :) >> >> So HBASE-3557 is what you want. >> >> Regarding your current issue, RIT is "region in transition" meaning >> that the region is in a state recognized by the master as either >> moving from one region server to another, or just closing. Normally >> after disabling a table its regions all left their in transition >> state, but by deleting the rows directly in .META. who knows exactly >> what's the state of your table? >> >> J-D >> >> On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar >> <[email protected]> wrote: >> > Hi J-D, >> > >> > Here is the tail of HMaster log: >> > >> > 2011-03-29 23:48:51,155 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:52,158 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:53,161 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:54,163 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:55,164 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:56,166 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:57,168 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:58,170 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:48:59,172 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:49:00,174 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:49:01,176 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:49:02,178 DEBUG >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on >> > region to clear regions in transition; >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING, >> > ts=1301422405271 >> > 2011-03-29 23:49:02,178 ERROR >> > org.apache.hadoop.hbase.master.handler.TableEventHandler: Error >> manipulating >> > table AcContact >> > java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for >> > region to leave region >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions >> > at >> > >> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60) >> > at >> > >> org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66) >> > at >> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:662) >> > >> > What is "clearing regions in transition"? What could be the issue? >> > >> > I was told before too that deleting from META isn't recommended. But we >> were >> > facing the disable problem way too often. So we thought we'd make it >> > in-built. What other alternatives do I have? >> > >> > Thx, >> > Hari >> > >> > On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans < >> [email protected]>wrote: >> > >> >> The 60 secs timeout means that the client was waiting on the master >> >> for some operation but the master took longer than 60 secs to do it, >> >> so its log should be the next place too look for something whack. >> >> >> >> BTW deleting the rows from .META. directly is probably the worst thing >> >> you can do. >> >> >> >> J-D >> >> >> >> On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar >> >> <[email protected]> wrote: >> >> > Here is the stack trace: >> >> > >> >> > 11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client >> connection, >> >> > connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection >> >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection >> to >> >> > server hadoop2/192.168.1.111:2181 >> >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection >> >> established >> >> > to hadoop2/192.168.1.111:2181, initiating session >> >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment >> >> complete >> >> > on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c, >> >> > negotiated timeout = 180000 >> >> > 11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row >> >> > AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. >> >> > 11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries >> >> from >> >> > META. Retrying to create table >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create >> table >> >> even >> >> > after Cleaning Meta entries >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: >> >> > ************************************************************* >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/ >> >> > 192.168.1.57:60000 failed on socket timeout exception: >> >> > java.net.SocketTimeoutException: 60000 millis timeout while waiting >> for >> >> > channel to be ready for read. ch : >> >> java.nio.channels.SocketChannel[connected >> >> > local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000] >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: >> >> > >> >> >> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802) >> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775) >> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >> >> > $Proxy4.createTable(Unknown Source) >> >> > >> >> >> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341) >> >> > >> >> >> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303) >> >> > >> >> >> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227) >> >> > >> >> >> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown >> >> > Source) >> >> > com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown >> >> Source) >> >> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> > >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> >> > >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> > java.lang.reflect.Method.invoke(Method.java:597) >> >> > org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> >> > >> >> > The "CleanFromMeta" function catches IOException and deletes all rows >> >> from >> >> > .META. We had added this in the exception catch block because we used >> to >> >> > face the "Table taking too long to be disabled" exception often. It >> seems >> >> > the rows in META already get created when the IOException is thrown. >> >> > CleanFromMeta cleans .META. and then I try again to create the table, >> >> after >> >> > which I get the socket timeout exception. >> >> > >> >> > I am using Hadoop v0.20.2, r911707. Can this be a reason for this >> error? >> >> I >> >> > get the message "You are currently running the HMaster without HDFS >> >> append >> >> > support enabled. This may result in data loss. Please see the HBase >> >> > wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for >> >> > details." on the HBase Master UI. >> >> > >> >> > On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar >> >> > <[email protected]>wrote: >> >> > >> >> >> Hi Stack >> >> >> >> >> >> yes the tablename is AcContact. The tableName variable was wrong. >> >> >> Fixed it now but I still get the same error. Schema is just something >> >> >> created by parsing an XML file which has stuff like column family >> >> >> name, compression type etc so I guess it doesn't have much to do with >> >> >> version. Except that I had to change the bloom filter variable to >> >> >> String ( used to be boolean in 0.20.6). I will paste the stack trace >> >> >> asap >> >> >> >> >> >> hari >> >> >> >> >> >> On Monday, March 28, 2011, Stack <[email protected]> wrote: >> >> >> > Can I see more of the stack track please Hari and is AcContact the >> >> >> > table you are creating? Is the schema you've saved aside one you >> >> >> > created with 0.20 hbase? I don't think it matters but asking just >> in >> >> >> > case. >> >> >> > St.Ack >> >> >> > >> >> >> > On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar >> >> >> > <[email protected]> wrote: >> >> >> >> Hi, >> >> >> >> >> >> >> >> I am trying to create table in hbase v0.90.1 and I get the >> following >> >> >> error: >> >> >> >> >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket >> >> connection >> >> >> to >> >> >> >> server hadoop2/192.168.1.111:2181 >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection >> >> >> established >> >> >> >> to hadoop2/192.168.1.111:2181, initiating s >> >> >> >> ession >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment >> >> >> complete >> >> >> >> on server hadoop2/192.168.1.111:2181, sess >> >> >> >> ionid = 0x12efc946d66000b, negotiated timeout = 180000 >> >> >> >> 11/03/28 18:39:52 INFO >> >> >> client.HConnectionManager$HConnectionImplementation: >> >> >> >> Closed zookeeper sessionid=0x12efc946d6600 >> >> >> >> 0b >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: >> >> 0x12efc946d66000b >> >> >> >> closed >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down >> >> >> >> 11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught >> IOException: >> >> No >> >> >> >> server address listed in .META. for region >> >> >> >> AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while >> >> >> creating >> >> >> >> table: Table1 >> >> >> >> >> >> >> >> This is the code I am using: >> >> >> >> ..... >> >> >> >> HTableDescriptor desc = >> >> >> CreateTableByXML.convertSchemaToDescriptor(schema); >> >> >> >> try { >> >> >> >> hbaseAdmin.createTable(desc); >> >> >> >> } catch (IOException e) { >> >> >> >> CreateTableByXML.LOG.error("Caught IOException: " + >> >> e.getMessage() >> >> >> >> + " while creating table: " + tableName); >> >> >> >> .... >> >> >> >> ..... >> >> >> >> >> >> >> >> >> >> >> >> It used to work fine in v0.20.6. I upgraded today to v0.90.1 and >> it >> >> >> started >> >> >> >> giving this error. Any ideas? >> >> >> >> >> >> >> >> Hari >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> > >> >
