[
https://issues.apache.org/jira/browse/HDFS-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-7344:
----------------------------
Description:
According to HDFS-7285 and the design, this handles DataNode side extension and
related support for Erasure Coding, and implements ECWorker. The handled use
cases:
* Converting replica blocks into stripping erasure coding blocks;
* Converting replica blocks into non-stripping erasure coding blocks;
* Recovering erased stripping erasure coding blocks;
* Recovering erased non-stripping erasure coding blocks;
* On-demand recovering serving client request for erased non-stripping erasure
coding blocks.
It generally needs to restore BlockGroup and schema information from coding
commands from NameNode or other entities, and construct specific coding work to
execute. The required block reader, writer, either local or remote, encoder and
decoder, will be implemented separately as sub-tasks.
This JIRA will track all the linked sub-tasks, and is responsible for general
discussions and integration for ECWorker. It won't resolve until all the
related tasks are done.
was:
According to HDFS-7285 and the design, this handles DataNode side extension and
related support for Erasure Coding, and implements ECWorker. It mainly covers
the following aspects, and separate tasks may be opened to handle each of them.
* Process encoding work, calculating parity blocks as specified in block groups
and codec schema;
* Process decoding work, recovering data blocks according to block groups and
codec schema;
* Handle client requests for passive recovery blocks data and serving data on
demand while reconstructing;
* Write parity blocks according to storage policy.
> Erasure Coding worker and support in DataNode
> ---------------------------------------------
>
> Key: HDFS-7344
> URL: https://issues.apache.org/jira/browse/HDFS-7344
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Reporter: Kai Zheng
> Assignee: Li Bo
> Attachments: HDFS ECWorker Design.pdf, hdfs-ec-datanode.0108.zip,
> hdfs-ec-datanode.0108.zip
>
>
> According to HDFS-7285 and the design, this handles DataNode side extension
> and related support for Erasure Coding, and implements ECWorker. The handled
> use cases:
> * Converting replica blocks into stripping erasure coding blocks;
> * Converting replica blocks into non-stripping erasure coding blocks;
> * Recovering erased stripping erasure coding blocks;
> * Recovering erased non-stripping erasure coding blocks;
> * On-demand recovering serving client request for erased non-stripping
> erasure coding blocks.
> It generally needs to restore BlockGroup and schema information from coding
> commands from NameNode or other entities, and construct specific coding work
> to execute. The required block reader, writer, either local or remote,
> encoder and decoder, will be implemented separately as sub-tasks.
> This JIRA will track all the linked sub-tasks, and is responsible for general
> discussions and integration for ECWorker. It won't resolve until all the
> related tasks are done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
