[
https://issues.apache.org/jira/browse/HDFS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382849#comment-14382849
]
Kai Zheng commented on HDFS-7715:
---------------------------------
Hi Rashmi,
Thanks for your confirm.
bq.By using raw coders, do you mean first encoding using RS completely and then
modify it for Hitchhiker?
Exactly.
bq.There will be some performance impact if done this way since the encoder
will have to process all the data twice.
Good concern ! In your implementation, how did you avoid that, not RS first
then some fix for some parities ? You mean you perform the RS and HH specific
math calculation in the same procedure ? Sorry I didn't check it out yet.
For the idea of re-using existing raw coders, I thought it should be OK for
encoding, since the performance can benefit from native implementation of the
reused raw coders. What concerned me a lot is about recovering. Do you have
some thought about how to perform the decoding/recovering if we attempt reusing
the existing raw coders ? I have to admit that I haven't any idea yet.
> Implement the Hitchhiker erasure coding algorithm
> -------------------------------------------------
>
> Key: HDFS-7715
> URL: https://issues.apache.org/jira/browse/HDFS-7715
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: jack liuquan
> Attachments: HDFS-7715-hhxor-decoder.patch,
> HDFS-7715-hhxor-encoder.patch
>
>
> [Hitchhiker |
> http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is
> a new erasure coding algorithm developed as a research project at UC
> Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45%
> during data reconstruction. This JIRA aims to introduce Hitchhiker to the
> HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
