[
https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521517#comment-14521517
]
Kai Zheng commented on HDFS-7348:
---------------------------------
Thanks [~hitliuyi], [~zhz] and [~libo-intel] for the great discussion! Looks
like we have already come to good plans.
bq.Does it save CPU to decode in big chunks? Kai Zheng Could you advise?
Sorry I just noticed this. Yes you're right, as Yi also noted, it's good to
allocate big native buffers for ISA-L coders to outperform greatly. We have
test data that indicate using about 32MB chunk size ISA-L coder can work the
very best.
I agree it's good to decouple sync-and-decode unit from the chunk/cell size in
a schema, and make it configurable. Yes it might not be good to do it in the
entire block level as doing so may make DN exhaust in memory and not reliable.
We should be able to enforce a memory usage threshold limit for recovery tasks.
As some dedicated DNs have powerful CPU cores it's good to distribute recovery
work to them, so very likely on such DNs there are more than one recovering
tasks concurrently executing.
> Erasure Coding: striped block recovery
> --------------------------------------
>
> Key: HDFS-7348
> URL: https://issues.apache.org/jira/browse/HDFS-7348
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Reporter: Kai Zheng
> Assignee: Yi Liu
> Attachments: ECWorker.java, HDFS-7348.001.patch
>
>
> This JIRA is to recover one or more missed striped block in the striped block
> group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
