[
https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HDFS-1172:
-----------------------------------
Attachment: HDFS-1172.009.patch
I attached updated patch as 009.
* fixed intermittent failures of
{{TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate}}
** Changed the number of DataNodes of MiniDFSCluster to 3.
** Set {{blockReceivedDelayForTestsSetting}} for only 1 DataNode to get enough
time window in which file is completed but at lease one of the replicas is not
reported.
** Got rid of randomizing sleep time in
{{BPServiceActor#delayBeforeBlockReceivedForTests}}.
* It turned out that some other JIRAs added fixes needed here.
** {{BlockManager#hasEnoughEffectiveReplicas}} added by HDFS-8938 takes pending
replicas into account.
** {{numCurrentReplica}} in {{BlockManager#addStoredBlock}} was fixed to take
pending replicas into account by HDFS-8623.
* Test failures below are not related and already filed/fixed.
** TestLazyWriter: HDFS-9067
** TestLazyPersistLockedMemory: HDFS-9073
** TestJMXGet: HDFS-9072
** TestLazyPersistReplicaPlacement: HDFS-9074
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
> Key: HDFS-1172
> URL: https://issues.apache.org/jira/browse/HDFS-1172
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
> Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch,
> HDFS-1172.009.patch, HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt,
> replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't
> find an existing JIRA. It often happens that we see the NN schedule
> replication on the last block of files very quickly after they're completed,
> before the other DNs in the pipeline have a chance to report the new block.
> This results in a lot of extra replication work on the cluster, as we
> replicate the block and then end up with multiple excess replicas which are
> very quickly deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
