The 60 secs timeout means that the client was waiting on the master for some operation but the master took longer than 60 secs to do it, so its log should be the next place too look for something whack.
BTW deleting the rows from .META. directly is probably the worst thing you can do. J-D On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar <[email protected]> wrote: > Here is the stack trace: > > 11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection, > connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to > server hadoop2/192.168.1.111:2181 > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection established > to hadoop2/192.168.1.111:2181, initiating session > 11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment complete > on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c, > negotiated timeout = 180000 > 11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row > AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. > 11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries from > META. Retrying to create table > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table even > after Cleaning Meta entries > 11/03/28 18:48:02 FATAL create.CreateTableByXML: > ************************************************************* > 11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/ > 192.168.1.57:60000 failed on socket timeout exception: > java.net.SocketTimeoutException: 60000 millis timeout while waiting for > channel to be ready for read. ch : java.nio.channels.SocketChannel[connected > local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000] > 11/03/28 18:48:02 FATAL create.CreateTableByXML: > org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802) > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775) > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) > $Proxy4.createTable(Unknown Source) > org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341) > org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303) > org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227) > com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown > Source) > com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown Source) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > java.lang.reflect.Method.invoke(Method.java:597) > org.apache.hadoop.util.RunJar.main(RunJar.java:156) > > The "CleanFromMeta" function catches IOException and deletes all rows from > .META. We had added this in the exception catch block because we used to > face the "Table taking too long to be disabled" exception often. It seems > the rows in META already get created when the IOException is thrown. > CleanFromMeta cleans .META. and then I try again to create the table, after > which I get the socket timeout exception. > > I am using Hadoop v0.20.2, r911707. Can this be a reason for this error? I > get the message "You are currently running the HMaster without HDFS append > support enabled. This may result in data loss. Please see the HBase > wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for > details." on the HBase Master UI. > > On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar > <[email protected]>wrote: > >> Hi Stack >> >> yes the tablename is AcContact. The tableName variable was wrong. >> Fixed it now but I still get the same error. Schema is just something >> created by parsing an XML file which has stuff like column family >> name, compression type etc so I guess it doesn't have much to do with >> version. Except that I had to change the bloom filter variable to >> String ( used to be boolean in 0.20.6). I will paste the stack trace >> asap >> >> hari >> >> On Monday, March 28, 2011, Stack <[email protected]> wrote: >> > Can I see more of the stack track please Hari and is AcContact the >> > table you are creating? Is the schema you've saved aside one you >> > created with 0.20 hbase? I don't think it matters but asking just in >> > case. >> > St.Ack >> > >> > On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar >> > <[email protected]> wrote: >> >> Hi, >> >> >> >> I am trying to create table in hbase v0.90.1 and I get the following >> error: >> >> >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket connection >> to >> >> server hadoop2/192.168.1.111:2181 >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection >> established >> >> to hadoop2/192.168.1.111:2181, initiating s >> >> ession >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment >> complete >> >> on server hadoop2/192.168.1.111:2181, sess >> >> ionid = 0x12efc946d66000b, negotiated timeout = 180000 >> >> 11/03/28 18:39:52 INFO >> client.HConnectionManager$HConnectionImplementation: >> >> Closed zookeeper sessionid=0x12efc946d6600 >> >> 0b >> >> 11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: 0x12efc946d66000b >> >> closed >> >> 11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down >> >> 11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException: No >> >> server address listed in .META. for region >> >> AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while >> creating >> >> table: Table1 >> >> >> >> This is the code I am using: >> >> ..... >> >> HTableDescriptor desc = >> CreateTableByXML.convertSchemaToDescriptor(schema); >> >> try { >> >> hbaseAdmin.createTable(desc); >> >> } catch (IOException e) { >> >> CreateTableByXML.LOG.error("Caught IOException: " + e.getMessage() >> >> + " while creating table: " + tableName); >> >> .... >> >> ..... >> >> >> >> >> >> It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it >> started >> >> giving this error. Any ideas? >> >> >> >> Hari >> >> >> > >> >
