[
https://issues.apache.org/jira/browse/HDFS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394077#comment-14394077
]
Kai Zheng commented on HDFS-7715:
---------------------------------
Hi Jack,
It's not easy for others to follow if we insert replying here and there. Please
just Comment, instead Reply, for your non-trivial comments. Thanks.
For your concern, I thought for now let's focus on how to get it work in both
raw erasure coder and erasure coder levels, for ECBlock, and ECBlockGroup, as
we did for RS codes. I'm working on how to adapt and apply erasure codec and
coder in HDFS. Basically we will extend ECBlock and have something like
DFSECBlock like what you just did. For the HH specific, your question comes to
how to read chunks from a block. In RS and other codes, it's simple, just reads
chunk by chunk in sequence; In HH, it needs to divide and deal with the two
sub-strips in the block, so there must be some offset and len to book keep for
each chunk. Considering such purpose, we may need to add {{readChunk}} method
in ECBlock class so HH coder can customize the behavior. Do you think it would
work ? Just my quick thought.
> Implement the Hitchhiker erasure coding algorithm
> -------------------------------------------------
>
> Key: HDFS-7715
> URL: https://issues.apache.org/jira/browse/HDFS-7715
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: jack liuquan
> Attachments: 7715-hitchhikerXOR-v2.patch,
> HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch
>
>
> [Hitchhiker |
> http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is
> a new erasure coding algorithm developed as a research project at UC
> Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45%
> during data reconstruction. This JIRA aims to introduce Hitchhiker to the
> HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
