Hi,
I am using HBASE 0.92.1+156 thought CDH 4.1.1 version. In my test cluster with
5 region servers, disk space utilization is reached to 84.39 % ( DFS is still
remaining 17,22 GB out of 469GB configured storage).
4 out 5 region servers went down today morning. Everything else is running
fine. Going through the logs, I found same error message in NameServer.log,
HMaster log and all 4 region server logs before they went down.
Summary of log message is -> File /hbase/xyz could only be replicated to 0
nodes instead of minReplication (=1). There are 5 datanode(s) running and no
node(s) are excluded in this operation.
I am trying to investigate why these region servers went down when there is
still space left on the disk.
Please help,
Saurabh.
Detailed error message from ->
NameNode ->
2013-01-23 06:05:37,797 ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE) cause:java.io.IOException:
File
/hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef
could only be replicated to 0 nodes instead of minReplication (=1). There are
5 datanode(s) running and no node(s) are excluded in this operation.
2013-01-23 06:05:37,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler
11 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
10.42.105.91:53474: error: java.io.IOException: File
/hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef
could only be replicated to 0 nodes instead of minReplication (=1). There are
5 datanode(s) running and no node(s) are excluded in this operation.
java.io.IOException: File
/hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef
could only be replicated to 0 nodes instead of minReplication (=1). There are
5 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
2013-01-23 06:05:38,300 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault: Not
able to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on
org.apache.commons.logging.impl.Log4JLogger
2013-01-23 06:05:38,300 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.42.105.91:50010 to delete
[blk_-890878099230127293_174462]
2013-01-23 06:05:38,300 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.42.105.112:50010 to delete
[blk_7536395638665332051_174484, blk_-890878099230127293_174462,
blk_-7077139403028631837_174485, blk_-1592777521358427003_174452,
blk_1853843283421613770_174486, blk_-4035888711536060826_174456]
2013-01-23 06:05:38,561 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
addStoredBlock: blockMap updated: 10.42.105.112:50010 is added to
blk_-7288320074167043222_174501{blockUCState=COMMITTED, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.42.105.112:50010|RBW]]} size 67108864
2013-01-23 06:05:38,562 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault: Not
able to place enough replicas, still in need of 3 to reach 3
HMaster Node ->
2013-01-23 06:05:37,962 ERROR org.apache.hadoop.hbase.master.HMaster: Region
server ^@^@vm-8aa9-fe74.nam.nsroot.net,60020,1358896553048 reported a fatal
error:
ABORTING region server vm-8aa9-fe74.nam.nsroot.net,60020,1358896553048: Replay
of HLog required. Forcing server shutdown
Cause:
org.apache.hadoop.hbase.DroppedSnapshotException: region:
elf_log,153299:1358937530530:1430447649612607:vm-cb03-2277.nam.nsroot.net:,1358938680796.d38a7249b7be279f6e443c382f4dcd98.
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1297)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1181)
at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1123)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:400)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:374)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:243)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef
could only be replicated to 0 nodes instead of minReplication (=1). There are
5 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
at org.apache.hadoop.ipc.Client.call(Client.java:1160)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy13.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:290)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2013-01-23 06:05:42,395 INFO
org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral
node deleted, processing expiration
[vm-8aa9-fe74.nam.nsroot.net,60020,1358896553048]
2013-01-23 06:05:42,406 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs
for vm-8aa9-fe74.nam.nsroot.net,60020,1358896553048
Region Server Node ->
2013-01-23 06:05:42,453 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close
file
/hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/hbase/elf_log/d38a7249b7be279f6e443c382f4dcd98/.tmp/1f87bc4bff3540189c83771cd5d0c4ef
could only be replicated to 0 nodes instead of minReplication (=1). There are
5 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
at org.apache.hadoop.ipc.Client.call(Client.java:1160)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy13.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:290)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy15.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1150)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1003)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463)
2013-01-23 06:05:42,454 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook:
Shutdown hook finished.
