Ah, yea. I have this error in the regionserver log:
2011-03-30 10:09:26,534 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.63 MB,
free=196.71 MB, max=198.34 MB, blocks=0, accesses=0, hits=0, hitRatio=�%,
cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%, evictions=0,
evicted=0, evictedPerRun=NaN
2011-03-30 10:09:37,328 DEBUG
org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000ms
elapsed
2011-03-30 10:09:46,369 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
region: AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,369 DEBUG
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing
open of AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,370 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x12f02e9bdec0007 Attempting to transition node
95652cfd0dab6dcf3be0c837c539f8d3 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2011-03-30 10:09:46,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x12f02e9bdec0007 Successfully transitioned node
95652cfd0dab6dcf3be0c837c539f8d3 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2011-03-30 10:09:46,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region: REGION => {NAME =>
'AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.', STARTKEY =>
'', ENDKEY => '', ENCODED => 95652cfd0dab6dcf3be0c837c539f8d3, TABLE =>
{{NAME => 'AcContact', FAMILIES => [{NAME => 'Data', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1000', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}]}}
2011-03-30 10:09:46,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Instantiated AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,373 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of region=AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
java.io.IOException: Compression algorithm 'lzo' previously failed test.
at
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:2555)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2544)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:262)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:94)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Will try to fix LZO here and let you know..
On Wed, Mar 30, 2011 at 1:24 AM, Jean-Daniel Cryans <[email protected]>wrote:
> hadoop3 was trying to open it, but it seems like it's not able to. It
> really looks like: https://issues.apache.org/jira/browse/HBASE-3669
>
> You should also check what's going on on that slave.
>
> J-D
>
> On Tue, Mar 29, 2011 at 12:12 PM, Hari Sreekumar
> <[email protected]> wrote:
> > Yep I know, I think I was also on the mailing list thread that inspired
> > HBASE-3557 :) But what can I do in the current version better than that?
> >
> > In any case, I do that only when I catch IOException. So why am I getting
> > this IOException. I deleted the hbase folder in HDFS and tried doing
> > everything from start. Here is the HMaster log before the IOException was
> > thrown: http://pastebin.com/x1BUuPpQ
> >
> > thanks,
> > Hari
> >
> > On Wed, Mar 30, 2011 at 12:21 AM, Jean-Daniel Cryans <
> [email protected]>wrote:
> >
> >> There's a reason why disabling takes time, if you delete rows from
> >> .META. you might end up in an inconsistent situation and we'll have a
> >> hard time helping you :)
> >>
> >> So HBASE-3557 is what you want.
> >>
> >> Regarding your current issue, RIT is "region in transition" meaning
> >> that the region is in a state recognized by the master as either
> >> moving from one region server to another, or just closing. Normally
> >> after disabling a table its regions all left their in transition
> >> state, but by deleting the rows directly in .META. who knows exactly
> >> what's the state of your table?
> >>
> >> J-D
> >>
> >> On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar
> >> <[email protected]> wrote:
> >> > Hi J-D,
> >> >
> >> > Here is the tail of HMaster log:
> >> >
> >> > 2011-03-29 23:48:51,155 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:52,158 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:53,161 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:54,163 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:55,164 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:56,166 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:57,168 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:58,170 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:48:59,172 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:49:00,174 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:49:01,176 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:49:02,178 DEBUG
> >> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
> >> > region to clear regions in transition;
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
> state=OPENING,
> >> > ts=1301422405271
> >> > 2011-03-29 23:49:02,178 ERROR
> >> > org.apache.hadoop.hbase.master.handler.TableEventHandler: Error
> >> manipulating
> >> > table AcContact
> >> > java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
> >> > region to leave region
> >> > AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in
> transitions
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
> >> > at
> >> >
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> >
> >> > What is "clearing regions in transition"? What could be the issue?
> >> >
> >> > I was told before too that deleting from META isn't recommended. But
> we
> >> were
> >> > facing the disable problem way too often. So we thought we'd make it
> >> > in-built. What other alternatives do I have?
> >> >
> >> > Thx,
> >> > Hari
> >> >
> >> > On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans <
> >> [email protected]>wrote:
> >> >
> >> >> The 60 secs timeout means that the client was waiting on the master
> >> >> for some operation but the master took longer than 60 secs to do it,
> >> >> so its log should be the next place too look for something whack.
> >> >>
> >> >> BTW deleting the rows from .META. directly is probably the worst
> thing
> >> >> you can do.
> >> >>
> >> >> J-D
> >> >>
> >> >> On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
> >> >> <[email protected]> wrote:
> >> >> > Here is the stack trace:
> >> >> >
> >> >> > 11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client
> >> connection,
> >> >> > connectString=hadoop2:2181 sessionTimeout=180000
> watcher=hconnection
> >> >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket
> connection
> >> to
> >> >> > server hadoop2/192.168.1.111:2181
> >> >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
> >> >> established
> >> >> > to hadoop2/192.168.1.111:2181, initiating session
> >> >> > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
> >> >> complete
> >> >> > on server hadoop2/192.168.1.111:2181, sessionid =
> 0x12efc946d66000c,
> >> >> > negotiated timeout = 180000
> >> >> > 11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
> >> >> > AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
> >> >> > 11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table
> entries
> >> >> from
> >> >> > META. Retrying to create table
> >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create
> >> table
> >> >> even
> >> >> > after Cleaning Meta entries
> >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML:
> >> >> > *************************************************************
> >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
> >> >> > 192.168.1.57:60000 failed on socket timeout exception:
> >> >> > java.net.SocketTimeoutException: 60000 millis timeout while waiting
> >> for
> >> >> > channel to be ready for read. ch :
> >> >> java.nio.channels.SocketChannel[connected
> >> >> > local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
> >> >> > 11/03/28 18:48:02 FATAL create.CreateTableByXML:
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
> >> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
> >> >> >
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> >> >> > $Proxy4.createTable(Unknown Source)
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
> >> >> >
> >> >>
> >>
> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
> >> >> > Source)
> >> >> > com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
> >> >> Source)
> >> >> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >> >
> >> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >> >
> >> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >> > java.lang.reflect.Method.invoke(Method.java:597)
> >> >> > org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >> >> >
> >> >> > The "CleanFromMeta" function catches IOException and deletes all
> rows
> >> >> from
> >> >> > .META. We had added this in the exception catch block because we
> used
> >> to
> >> >> > face the "Table taking too long to be disabled" exception often. It
> >> seems
> >> >> > the rows in META already get created when the IOException is
> thrown.
> >> >> > CleanFromMeta cleans .META. and then I try again to create the
> table,
> >> >> after
> >> >> > which I get the socket timeout exception.
> >> >> >
> >> >> > I am using Hadoop v0.20.2, r911707. Can this be a reason for this
> >> error?
> >> >> I
> >> >> > get the message "You are currently running the HMaster without HDFS
> >> >> append
> >> >> > support enabled. This may result in data loss. Please see the HBase
> >> >> > wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
> >> >> > details." on the HBase Master UI.
> >> >> >
> >> >> > On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
> >> >> > <[email protected]>wrote:
> >> >> >
> >> >> >> Hi Stack
> >> >> >>
> >> >> >> yes the tablename is AcContact. The tableName variable was wrong.
> >> >> >> Fixed it now but I still get the same error. Schema is just
> something
> >> >> >> created by parsing an XML file which has stuff like column family
> >> >> >> name, compression type etc so I guess it doesn't have much to do
> with
> >> >> >> version. Except that I had to change the bloom filter variable to
> >> >> >> String ( used to be boolean in 0.20.6). I will paste the stack
> trace
> >> >> >> asap
> >> >> >>
> >> >> >> hari
> >> >> >>
> >> >> >> On Monday, March 28, 2011, Stack <[email protected]> wrote:
> >> >> >> > Can I see more of the stack track please Hari and is AcContact
> the
> >> >> >> > table you are creating? Is the schema you've saved aside one
> you
> >> >> >> > created with 0.20 hbase? I don't think it matters but asking
> just
> >> in
> >> >> >> > case.
> >> >> >> > St.Ack
> >> >> >> >
> >> >> >> > On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
> >> >> >> > <[email protected]> wrote:
> >> >> >> >> Hi,
> >> >> >> >>
> >> >> >> >> I am trying to create table in hbase v0.90.1 and I get the
> >> following
> >> >> >> error:
> >> >> >> >>
> >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
> >> >> connection
> >> >> >> to
> >> >> >> >> server hadoop2/192.168.1.111:2181
> >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
> >> >> >> established
> >> >> >> >> to hadoop2/192.168.1.111:2181, initiating s
> >> >> >> >> ession
> >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session
> establishment
> >> >> >> complete
> >> >> >> >> on server hadoop2/192.168.1.111:2181, sess
> >> >> >> >> ionid = 0x12efc946d66000b, negotiated timeout = 180000
> >> >> >> >> 11/03/28 18:39:52 INFO
> >> >> >> client.HConnectionManager$HConnectionImplementation:
> >> >> >> >> Closed zookeeper sessionid=0x12efc946d6600
> >> >> >> >> 0b
> >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session:
> >> >> 0x12efc946d66000b
> >> >> >> >> closed
> >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut
> down
> >> >> >> >> 11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught
> >> IOException:
> >> >> No
> >> >> >> >> server address listed in .META. for region
> >> >> >> >> AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
> while
> >> >> >> creating
> >> >> >> >> table: Table1
> >> >> >> >>
> >> >> >> >> This is the code I am using:
> >> >> >> >> .....
> >> >> >> >> HTableDescriptor desc =
> >> >> >> CreateTableByXML.convertSchemaToDescriptor(schema);
> >> >> >> >> try {
> >> >> >> >> hbaseAdmin.createTable(desc);
> >> >> >> >> } catch (IOException e) {
> >> >> >> >> CreateTableByXML.LOG.error("Caught IOException: " +
> >> >> e.getMessage()
> >> >> >> >> + " while creating table: " + tableName);
> >> >> >> >> ....
> >> >> >> >> .....
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> It used to work fine in v0.20.6. I upgraded today to v0.90.1
> and
> >> it
> >> >> >> started
> >> >> >> >> giving this error. Any ideas?
> >> >> >> >>
> >> >> >> >> Hari
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
