Hi Lars, Thank you so much Lars for your perfect solution, it worked, you have been a life saver here, worked like a charm. Thanks again Lars.
Regards Pankaj Misra ________________________________________ From: lars hofhansl [[email protected]] Sent: Wednesday, October 17, 2012 11:43 PM To: [email protected] Subject: Re: HBase Issue after Namenode Formating Can you try clearing all the state in ZK as well? (stop ZK, wipe the ZK data directory, restart ZK) ----- Original Message ----- From: Pankaj Misra <[email protected]> To: "[email protected]" <[email protected]> Cc: Sent: Wednesday, October 17, 2012 11:09 AM Subject: RE: HBase Issue after Namenode Formating Hi All, I did some more fact finding around this issue, and it seems to me that the Master is not getting initialized and hence I am not able to create table. I tried connecting via the Java client as well, which gave me the following exception. I am not sure what is causing Master to not initialize at all. I am not able to figure out, where am I going wrong, and would request help. Thanks. java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0} at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56) at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72) Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0} at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247) at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44) Also, the error as seen in zookeeper log for the above session is 2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51602 which had sessionid 0x13a6fd74b50000c 2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224) at java.lang.Thread.run(Thread.java:662) 2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51515 which had sessionid 0x13a6fd74b500009 Thanks and Regards Pankaj Misra ________________________________________ From: Pankaj Misra Sent: Wednesday, October 17, 2012 10:30 PM To: [email protected] Subject: RE: HBase Issue after Namenode Formating Hi All, I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine. Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below hbase(main):004:0> create 'testrun','samplecf' ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400) In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master. Additionally I see the following error in the HBase Master logs 2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry 2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory 2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this master's address, localhost,60000,1350487401859; master was restarted? Deleting node. 2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: No master available. Notifying waiting threads 2012-10-17 21:15:33,084 INFO org.apache.hadoop.hbase.master.HMaster: Cluster went down before this master became active 2012-10-17 21:15:33,084 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads Calling for help, as I think I am missing something here. Thanks and Regards Pankaj Misra ________________________________ Impetus Ranked in the Top 50 India's Best Companies to Work For 2012. Impetus webcast 'Designing a Test Automation Framework for Multi-vendor Interoperable Systems' available at http://lf1.me/0E/. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference. ________________________________ Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012. Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference. ________________________________ Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012. Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
