Hi J-D,
Here is the tail of HMaster log:
2011-03-29 23:48:51,155 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:52,158 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:53,161 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:54,163 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:55,164 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:56,166 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:57,168 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:58,170 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:59,172 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:00,174 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:01,176 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error manipulating
table AcContact
java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
region to leave region
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
What is "clearing regions in transition"? What could be the issue?
I was told before too that deleting from META isn't recommended. But we were
facing the disable problem way too often. So we thought we'd make it
in-built. What other alternatives do I have?
Thx,
Hari
On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans <[email protected]>wrote:
> The 60 secs timeout means that the client was waiting on the master
> for some operation but the master took longer than 60 secs to do it,
> so its log should be the next place too look for something whack.
>
> BTW deleting the rows from .META. directly is probably the worst thing
> you can do.
>
> J-D
>
> On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
> <[email protected]> wrote:
> > Here is the stack trace:
> >
> > 11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection,
> > connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server hadoop2/192.168.1.111:2181
> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
> established
> > to hadoop2/192.168.1.111:2181, initiating session
> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
> > negotiated timeout = 180000
> > 11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
> > AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
> > 11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries
> from
> > META. Retrying to create table
> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table
> even
> > after Cleaning Meta entries
> > 11/03/28 18:48:02 FATAL create.CreateTableByXML:
> > *************************************************************
> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
> > 192.168.1.57:60000 failed on socket timeout exception:
> > java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> > channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected
> > local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
> > 11/03/28 18:48:02 FATAL create.CreateTableByXML:
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> > $Proxy4.createTable(Unknown Source)
> >
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
> >
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
> >
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
> >
> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
> > Source)
> > com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
> Source)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > java.lang.reflect.Method.invoke(Method.java:597)
> > org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >
> > The "CleanFromMeta" function catches IOException and deletes all rows
> from
> > .META. We had added this in the exception catch block because we used to
> > face the "Table taking too long to be disabled" exception often. It seems
> > the rows in META already get created when the IOException is thrown.
> > CleanFromMeta cleans .META. and then I try again to create the table,
> after
> > which I get the socket timeout exception.
> >
> > I am using Hadoop v0.20.2, r911707. Can this be a reason for this error?
> I
> > get the message "You are currently running the HMaster without HDFS
> append
> > support enabled. This may result in data loss. Please see the HBase
> > wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
> > details." on the HBase Master UI.
> >
> > On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
> > <[email protected]>wrote:
> >
> >> Hi Stack
> >>
> >> yes the tablename is AcContact. The tableName variable was wrong.
> >> Fixed it now but I still get the same error. Schema is just something
> >> created by parsing an XML file which has stuff like column family
> >> name, compression type etc so I guess it doesn't have much to do with
> >> version. Except that I had to change the bloom filter variable to
> >> String ( used to be boolean in 0.20.6). I will paste the stack trace
> >> asap
> >>
> >> hari
> >>
> >> On Monday, March 28, 2011, Stack <[email protected]> wrote:
> >> > Can I see more of the stack track please Hari and is AcContact the
> >> > table you are creating? Is the schema you've saved aside one you
> >> > created with 0.20 hbase? I don't think it matters but asking just in
> >> > case.
> >> > St.Ack
> >> >
> >> > On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
> >> > <[email protected]> wrote:
> >> >> Hi,
> >> >>
> >> >> I am trying to create table in hbase v0.90.1 and I get the following
> >> error:
> >> >>
> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
> connection
> >> to
> >> >> server hadoop2/192.168.1.111:2181
> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
> >> established
> >> >> to hadoop2/192.168.1.111:2181, initiating s
> >> >> ession
> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
> >> complete
> >> >> on server hadoop2/192.168.1.111:2181, sess
> >> >> ionid = 0x12efc946d66000b, negotiated timeout = 180000
> >> >> 11/03/28 18:39:52 INFO
> >> client.HConnectionManager$HConnectionImplementation:
> >> >> Closed zookeeper sessionid=0x12efc946d6600
> >> >> 0b
> >> >> 11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session:
> 0x12efc946d66000b
> >> >> closed
> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
> >> >> 11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException:
> No
> >> >> server address listed in .META. for region
> >> >> AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
> >> creating
> >> >> table: Table1
> >> >>
> >> >> This is the code I am using:
> >> >> .....
> >> >> HTableDescriptor desc =
> >> CreateTableByXML.convertSchemaToDescriptor(schema);
> >> >> try {
> >> >> hbaseAdmin.createTable(desc);
> >> >> } catch (IOException e) {
> >> >> CreateTableByXML.LOG.error("Caught IOException: " +
> e.getMessage()
> >> >> + " while creating table: " + tableName);
> >> >> ....
> >> >> .....
> >> >>
> >> >>
> >> >> It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it
> >> started
> >> >> giving this error. Any ideas?
> >> >>
> >> >> Hari
> >> >>
> >> >
> >>
> >
>
