[
https://issues.apache.org/jira/browse/HDFS-12044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12044:
---------------------------------
Attachment: HDFS-12044.02.patch
Hi, [~andrew.wang]
NN limits the speed of putting such tasks to DN's queue at speed of
{{maxReplicationStreams - xmitsInProcess}}, so that when DN has active
reconstruction tasks taking {{xmitsInProcess}}, NN will adaptively slow down
the enqueue process and will not increase the queue length infinitely. I think
this is how the non-EC block recovery works today without an {{Executor}}.
I agree that it would provide more guarantee if we explicitly limit the
parallelism from the Executor, so I changed patch {{02}} to use an unbounded
queue in the Executor to hold submitted re-construction tasks in {{02}} patch.
Do you think it is sufficient, [~andrew.wang]
> Mismatch between BlockManager#maxReplicatioStreams and
> ErasureCodingWorker.stripedReconstructionPool pool size causes slow and burst
> recovery.
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-12044
> URL: https://issues.apache.org/jira/browse/HDFS-12044
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
> Affects Versions: 3.0.0-alpha3
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12044.00.patch, HDFS-12044.01.patch,
> HDFS-12044.02.patch
>
>
> {{ErasureCodingWorker#stripedReconstructionPool}} is with {{corePoolSize=2}}
> and {{maxPoolSize=8}} as default. And it rejects more tasks if the queue is
> full.
> When {{BlockManager#maxReplicationStream}} is larger than
> {{ErasureCodingWorker#stripedReconstructionPool#corePoolSize/maxPoolSize}},
> for example, {{maxReplicationStream=20}} and {{corePoolSize=2 ,
> maxPoolSize=8}}. Meanwhile, NN sends up to {{maxTransfer}} reconstruction
> tasks to DN for each heartbeat, and it is calculated in {{FSNamesystem}}:
> {code}
> final int maxTransfer = blockManager.getMaxReplicationStreams() -
> xmitsInProgress;
> {code}
> However, at any giving time,
> {{{ErasureCodingWorker#stripedReconstructionPool}} takes 2 {{xmitInProcess}}.
> So for each heartbeat in 3s, NN will send about {{20-2 = 18}} reconstruction
> tasks to the DN, and DN throw away most of them if there were 8 tasks in the
> queue already. So NN needs to take longer to re-consider these blocks were
> under-replicated to schedule new tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
