[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143778#comment-15143778
]
Zhe Zhang commented on HDFS-9603:
---------------------------------
Thanks Rui for the work. With this JIRA and HDFS-8020, I think we should
revisit the {{ECBlockGroup}} => {{ECBlock}} => {{ECChunk}}, and
{{ErasureCodingStep}} abstractions. Also pinging [~lewuathe] for the discussion.
IIUC the purpose of these abstractions (instead of directly using raw coders)
is to support more advanced algorithms such as LRC and Hitchhiker. [~drankye]
Could you help clarify if Hitchhiker needs the {{ECBlockGroup}}? Or only LRC
potentially needs it? If HH don't need it, I suggest we keep using raw coders
for now. We should still make the change to create different coders (RS vs.
XOR) based on configuration.
Right now, without a clear need for the {{ECBlockGroup}} abstraction, we are
basically creating a {{ECBlockGroup}} from the input buffers
({{prepareBlkGroup}}), just for the purpose of taking out the buffers again
later ({{encode}}).
I'm also suggesting the above to prepare for the forthcoming release of EC
(either 2.9 or 3.0). With LRC looking unlikely to be implemented before the
release, I think we should focus on:
# Supporting multiple EC policies
# hflush / hsync
# Fully integrate ISA-L coder, new Java coder, and HH coder. Ideally with more
performance tests on HH.
Thoughts?
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---------------------------------------------------------------
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Rui Li
> Assignee: Rui Li
> Attachments: HDFS-9603.1.patch, HDFS-9603.2.patch, HDFS-9603.3.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
