Please refer to HBASE-2570 where you can find jstack for HMaster. Thanks
On Wed, May 19, 2010 at 9:26 AM, Jean-Daniel Cryans <[email protected]>wrote: > That stack trace seems to come from the thread that tries to kill the > master, not the master itself (they both log at the same place). Since > the master doesn't answer, it's safe to kill -9 it. Next time make > sure you don't run out of space ;) > > J-D > > On Wed, May 19, 2010 at 4:16 AM, Ted Yu <[email protected]> wrote: > > Hi, > > We use hbase 0.20.3 > > The server running master ran out of disk space. > > I cleaned up some space: > > /dev/mapper/VolGroup00-LogVol00 > > 303346048 268141776 19795120 94% / > > > > But I still cannot stop hbase - non-stop dot printing: > > [sjc1-qa-certiq1.sjc1:hadoop 26345]stop-hbase.sh > > stopping master........ > > > > Here is from master log: > > 2010-05-19 03:53:44,033 DEBUG > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode > /hbase/master > > got 10.201.8.247:60000 > > 2010-05-19 03:53:44,034 INFO > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster > > attempt 9 of 10 failed; no more retrying. > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > > at > > > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > > at > > > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308) > > at > > > org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:831) > > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:712) > > at > > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333) > > at $Proxy0.getProtocolVersion(Unknown Source) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:341) > > at > org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72) > > at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1258) > > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282) > > 2010-05-19 03:53:44,034 ERROR org.apache.hadoop.hbase.master.HMaster: > master > > is not running > > 2010-05-19 03:53:44,035 INFO org.apache.zookeeper.ZooKeeper: Closing > > session: 0x128ad17448a007d > > 2010-05-19 03:53:44,035 INFO org.apache.zookeeper.ClientCnxn: Closing > > ClientCnxn for session: 0x128ad17448a007d > > 2010-05-19 03:53:44,039 INFO org.apache.zookeeper.ClientCnxn: Exception > > while closing send thread for session 0x128ad17448a007d : Read error rc = > -1 > > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > > 2010-05-19 03:53:44,144 INFO org.apache.zookeeper.ClientCnxn: > Disconnecting > > ClientCnxn for session: 0x128ad17448a007d > > 2010-05-19 03:53:44,144 INFO org.apache.zookeeper.ZooKeeper: Session: > > 0x128ad17448a007d closed > > 2010-05-19 03:53:44,144 DEBUG > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection > with > > ZooKeeper > > > > 202,1-4 99% > > > > Here is jps output: > > [sjc1-qa-certiq1.sjc1:hadoop 26341]jps > > 2287 Jps > > 23365 JobTracker > > 23561 HMaster > > > > Has anyone seen similar problem ? > > > > Thanks > > >
