[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162204#comment-16162204
]
Andrew Wang commented on HDFS-12412:
------------------------------------
Thanks for working on this Eddy. I did a grep for the removed config key:
{noformat}
-> % ag "stripedread.threads"
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
140: 1. `dfs.datanode.ec.reconstruction.stripedread.threads` - Number of
concurrent reader threads. Default value is 20 threads.
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
3052: <name>dfs.datanode.ec.reconstruction.stripedread.threads</name>
{noformat}
We need to update these as well. Otherwise, +1 LGTM!
> Remove ErasureCodingWorker.stripedReadPool
> ------------------------------------------
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
> Affects Versions: 3.0.0-alpha3
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
