That stack trace seems to come from the thread that tries to kill the master, not the master itself (they both log at the same place). Since the master doesn't answer, it's safe to kill -9 it. Next time make sure you don't run out of space ;)
J-D On Wed, May 19, 2010 at 4:16 AM, Ted Yu <[email protected]> wrote: > Hi, > We use hbase 0.20.3 > The server running master ran out of disk space. > I cleaned up some space: > /dev/mapper/VolGroup00-LogVol00 > 303346048 268141776 19795120 94% / > > But I still cannot stop hbase - non-stop dot printing: > [sjc1-qa-certiq1.sjc1:hadoop 26345]stop-hbase.sh > stopping master........ > > Here is from master log: > 2010-05-19 03:53:44,033 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master > got 10.201.8.247:60000 > 2010-05-19 03:53:44,034 INFO > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster > attempt 9 of 10 failed; no more retrying. > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308) > at > org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:831) > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:712) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) > at $Proxy0.getProtocolVersion(Unknown Source) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:341) > at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72) > at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1258) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282) > 2010-05-19 03:53:44,034 ERROR org.apache.hadoop.hbase.master.HMaster: master > is not running > 2010-05-19 03:53:44,035 INFO org.apache.zookeeper.ZooKeeper: Closing > session: 0x128ad17448a007d > 2010-05-19 03:53:44,035 INFO org.apache.zookeeper.ClientCnxn: Closing > ClientCnxn for session: 0x128ad17448a007d > 2010-05-19 03:53:44,039 INFO org.apache.zookeeper.ClientCnxn: Exception > while closing send thread for session 0x128ad17448a007d : Read error rc = -1 > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > 2010-05-19 03:53:44,144 INFO org.apache.zookeeper.ClientCnxn: Disconnecting > ClientCnxn for session: 0x128ad17448a007d > 2010-05-19 03:53:44,144 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x128ad17448a007d closed > 2010-05-19 03:53:44,144 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection with > ZooKeeper > > 202,1-4 99% > > Here is jps output: > [sjc1-qa-certiq1.sjc1:hadoop 26341]jps > 2287 Jps > 23365 JobTracker > 23561 HMaster > > Has anyone seen similar problem ? > > Thanks >
