Hello
On my namenode logs, I am seeing these error lines quite frequently. What
is the reason behind this and how to overcome this? I have three data nodes
and hdfs dfsadmin -report says 3 Live Datanodes.
Kafka Connect is responsible of writing records from kafka topic to hdfs
here. The data produce speed on kafka is high. 17 million records per hour.
We have given a 30 minute rotate time interval to kafka connect to write
the records on hdfs.
2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: *BLOCK*
NameNode.addBlock: file
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
fileId=16394 for DFSClient_NONMAPREDUCE_-1139929652_127
2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK*
getAdditionalBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
inodeId 16394 for DFSClient_NONMAPREDUCE_-1139929652_127
2017-08-01 09:12:55,217 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed
to choose remote rack (location = ~/default-rack), fallback to local rack
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:702)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:582)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:350)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:216)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:113)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:128)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1567)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange: DIR*
FSDirectory.addBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
with blk_1073741825_1001 block is added to the in-memory file system
2017-08-01 09:12:55,220 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073741825_1001{UCState=UNDER_CONSTRUCTION,
truncateBlock=null, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW],
ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]}
for
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange:
persistNewBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
with new block blk_1073741825_1001{UCState=UNDER_CONSTRUCTION,
truncateBlock=null, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW],
ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]},
current total block count is 1
Thanks
Nishant Verma
