[
https://issues.apache.org/jira/browse/HDFS-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354402#comment-14354402
]
Rashmi Vinayak commented on HDFS-7715:
--------------------------------------
Hi Zhe,
Thanks for adding these questions to the JIRA!
Hi Jack,
Please find my responses below:
1. The jointEncodeSubPacketsNew() has doXor as one its inputs. This method does
Hitchhiker-XOR+ if the input doXor is True and it does Hitchhiker-nonXor if the
input doXor is False.
2. Yes, the matrix 'generatorMatrixForParity' is used for Hitchhiker-nonXor. It
is used in calculating the piggybacks added, and this has to match the
generator matrix of the underlying RS code. The current hard-coded definition
is the generator matrix of the underlying RS code employed in HDFS-RAID.
3. Yes, the current value is hard-coded for (10,4) which are the parameters
used by Facebook's version of HDFS-RAID. I have obtained these values by
feeding unit vectors as inputs to the RS encoder as explained in Section
'5.2.2. Encoder' (paragraph 2) of the [Hitchhiker
paper|http://eecs.berkeley.edu/~rashmikv/papers/Hitchhiker_SIGCOMM14.pdf]. The
same method can be employed for any general values (k,r). I think it might be a
good idea to first consider only (10,4), and then add the generalization.
> Implement the Hitchhiker erasure coding algorithm
> -------------------------------------------------
>
> Key: HDFS-7715
> URL: https://issues.apache.org/jira/browse/HDFS-7715
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: jack liuquan
>
> [Hitchhiker |
> http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is
> a new erasure coding algorithm developed as a research project at UC
> Berkeley. It has been shown to reduce network traffic and disk I/O by 25% and
> 45% during data reconstruction. This JIRA aims to introduce Hitchhiker to the
> HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
