Hey Andy, This looks like the log from the regions server. Any chance you can post the log for the HMaster?
-Joey On Fri, Jul 15, 2011 at 12:43 AM, Zhong, Andy <[email protected]>wrote: > St.Ack, > > It's weird. I did Hbase upgrade from Hbase 0.20.3 to Hbase 0.90.2 by > following "replace the Hadoop JAR file from Hadoop 0.20.2 release with the > sync-supporting Hadoop JAR file found in HBase 0.90.2". It was suceesfully > before, but now we are facing error from Hbase shell: NativeException: > org.apache.hadoop.hbase.MasterNotRunningException: > hofvmecln15.ecom.sears.com:60000, even though the processes of Hmaster and > Region are running, and 60010 can be accessible via 60010. > > Could you please help to find out the possible root cause? Below is the > logging messages from the hbase.log from Master Node: > > 2011-07-14 22:54:55,903 INFO zookeeper.ZooKeeper - Client > environment:user.dir=/appl/hbase/share/hbase-0.90.2 > 2011-07-14 22:54:55,904 INFO zookeeper.ZooKeeper - Initiating client > connection, connectString=hofvmecln19:2181,hofvmecln18:2181,hofvmecln17:2181 > sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@51b4a74b > 2011-07-14 22:54:55,905 INFO zookeeper.ClientCnxn - > zookeeper.disableAutoWatchReset is false > 2011-07-14 22:54:55,932 INFO zookeeper.ClientCnxn - Attempting connection > to server hofvmecln18/151.149.116.46:2181 > 2011-07-14 22:54:55,938 INFO zookeeper.ClientCnxn - Priming connection to > java.nio.channels.SocketChannel[connected > local=/151.149.119.93:52156remote=hofvmecln18/ > 151.149.116.46:2181] > 2011-07-14 22:54:55,939 INFO zookeeper.ClientCnxn - Server connection > successful > 2011-07-14 22:54:56,030 INFO regionserver.HRegionServer - Got ZooKeeper > event, state: SyncConnected, type: None, path: null > 2011-07-14 22:54:56,202 INFO regionserver.HRegionServer - Telling master > at hofvmecln15.ecom.sears.com:60000 that we are up > 2011-07-14 22:54:56,407 WARN ipc.HBaseServer - IPC Server listener on > 60000: readAndProcess threw exception java.io.EOFException. Count of bytes > read: 0 > java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at java.io.DataInputStream.readUTF(DataInputStream.java:592) > at java.io.DataInputStream.readUTF(DataInputStream.java:547) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:122) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:978) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:946) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2011-07-14 22:54:56,412 WARN regionserver.HRegionServer - Unable to > connect to master. Retrying. Error was: > java.io.IOException: Call to > hofvmecln15.ecom.sears.com/151.149.119.93:60000 failed on local exception: > java.io.EOFException > at > org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:757) > at > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:727) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) > at $Proxy0.getProtocolVersion(Unknown Source) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512) > at > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:423) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1296) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:504) > at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448) > > > thanks, > Andy Zhong > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Stack > Sent: Wednesday, June 08, 2011 11:53 AM > To: [email protected] > Subject: Re: Hadoop/HBase Upgrade Suggestion > > On Wed, Jun 8, 2011 at 9:25 AM, Zhong, Sheng <[email protected]> > wrote: > > Could anyone give me suggestion for Hadoop/HBase upgrade? We're > > currently using apache hadoop 0.20.2 + hbase 0.20.3 + zookeeper-3.2.2. > > Has anyone done with latest stable version of hadoop-0.20.203.0rc1 + > > Hbase 0.90.2, and will Hbase 0.90.2 have compatible issue with > > hadoop-0.20.203.0rc1? I am very appreciated by any help and suggestion. > > > > hadoop-0.20.203 does not have append support so its not recommended that > you run hbase on it. Here is our (a little stale at this point > -- see recent thread) story regards hadoop versions > http://hbase.apache.org/book/notsoquick.html#hadoop > > St.Ack > > This message, including any attachments, is the property of Sears Holdings > Corporation and/or one of its subsidiaries. It is confidential and may > contain proprietary or legally privileged information. If you are not the > intended recipient, please delete it without reading the contents. Thank > you. > -- Joseph Echeverria Cloudera, Inc. 443.305.9434
