[
https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675139#comment-13675139
]
Ravi Prakash commented on HDFS-1172:
------------------------------------
I am able to consistently reproduce this issue with the following command on an
80 node cluster:
hadoop jar
$HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar
SliveTest -baseDir /user/someUser/slive -duration 120 -dirSize 122500 -files
122500 -maps 560 -reduces 1 -seed 1 -ops 100 -readSize 1048576,1048576
-writeSize 1048576,1048576 -appendSize 1048576,1048576 -replication 1,1
-blockSize 1024,1024 -delete 0,uniform -create 100,uniform -mkdir 0,uniform
-rename 0,uniform -append 0,uniform -ls 0,uniform -read 0,uniform
This litters the task logs with the NotReplicatedYetException
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1268)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:469)
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
> Key: HDFS-1172
> URL: https://issues.apache.org/jira/browse/HDFS-1172
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
> Fix For: 0.24.0
>
> Attachments: HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt,
> replicateBlocksFUC1.patch, replicateBlocksFUC1.patch, replicateBlocksFUC.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't
> find an existing JIRA. It often happens that we see the NN schedule
> replication on the last block of files very quickly after they're completed,
> before the other DNs in the pipeline have a chance to report the new block.
> This results in a lot of extra replication work on the cluster, as we
> replicate the block and then end up with multiple excess replicas which are
> very quickly deleted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
