Thanks Todd, Yups… the problem was Disk Space….. I checked the NameNode UI which mentioned DFS Remaining as 0%
Is it possible to get some more detailed exception message in logs because, none of the logs (DataNode or Client) mention about this problem. ? May be the NameNode UI also can indicate the same….. Just like to know where the point: > 5*BLOCK_SIZE is documented…. :-( Thanks again... Cheers, Subroto Sanyal On Jun 19, 2012, at 6:22 PM, Todd Lipcon wrote: > Hi Subroto, > > Anything in the DN logs? Maybe the one DN is out of disk space? You > need to have 5*BLOCK_SIZE free. > > -Todd > > On Tue, Jun 19, 2012 at 1:29 AM, Subroto <ssan...@datameer.com> wrote: >> Hi, >> >> When DFS cluster; >> The datanodes keep on dumping the logs: >> 2012-06-19 04:05:04,842 INFO >> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding >> block pool BP-1233167859-10.10.35.8-1340093005960 >> 2012-06-19 04:05:04,861 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool >> BP-1233167859-10.10.35.8-1340093005960 (storage id >> DS-1255092928-10.244.15.192-50010-1340093104449) service to >> ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning >> handshake with NN >> 2012-06-19 04:05:04,979 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool >> BP-1233167859-10.10.35.8-1340093005960 (storage id >> DS-1255092928-10.244.15.192-50010-1340093104449) service to >> ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully >> registered with NN >> 2012-06-19 04:05:04,980 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode >> ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using >> DELETEREPORT_INTERVAL of 300000 msec BLOCKREPORT_INTERVAL of 21600000msec >> Initial delay: 0msec; heartBeatInterval=3000 >> 2012-06-19 04:05:05,258 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool >> BP-1233167859-10.10.35.8-1340093005960 (storage id >> DS-1255092928-10.244.15.192-50010-1340093104449) service to >> ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim >> ACTIVE state with txid=24 >> 2012-06-19 04:05:05,258 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE >> Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id >> DS-1255092928-10.244.15.192-50010-1340093104449) service to >> ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 >> 2012-06-19 04:05:05,394 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks >> took 1 msec to generate and 135 msecs for RPC and NN processing >> 2012-06-19 04:05:05,394 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, >> processed >> command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859 >> 2012-06-19 04:05:05,396 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block >> Verification Scanner initialized with interval 504 hours for block pool >> BP-1233167859-10.10.35.8-1340093005960. >> 2012-06-19 04:05:05,400 INFO >> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added >> bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new >> size=1 >> 2012-06-19 04:05:09,845 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:14,848 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:19,849 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:24,850 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:29,851 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:34,852 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:39,853 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:44,854 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> 2012-06-19 04:05:49,855 INFO >> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new >> period : work left in prev period : 0.00% >> >> The DFSClient operations are also not taking place properly: >> 2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer >> Exception >> java.io.IOException: File /hbase/hbase.version could only be replicated to 0 >> nodes instead of minReplication (=1). There are 1 datanode(s) running and >> no node(s) are excluded in this operation. >> at >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470) >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292) >> at >> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) >> >> Please let me know if there is any configuration which needs to be set….. >> The hadoop version being used is: 2.0.0 >> >> >> >> Cheers, >> Subroto Sanyal > > > > -- > Todd Lipcon > Software Engineer, Cloudera
