[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272065#comment-14272065
]
Kai Zheng commented on HDFS-7337:
---------------------------------
Hi [~andrew.wang], thanks for your comments and sorry for my late response.
bq. Could you generate normal plaintext diffs rather than a zip? We might also
want to reorganize things into existing packages. The rawcoder stuff could go
somewhere in hadoop-common for instance. We could move the block grouper
classes into blockmanagement. etc.
Yes I will provide diff or patch format when attaching the new revision.
I have discussed with Uma, Zhe and Weihua also about how to organize the bundle
of new codes. Looks like we all agree to move rawcoder classes to
hadoop-common. About the block grouper in this codec codes, it's not about
block placement, but only for codec specific logics. As discussed above and Zhe
also agreed, we would need to support plugin of modules regarding how to form a
block group for an ec code algorithm. The block grouper here is for such and
taken care of by the high level construct ErasureCodec. Please kindly review my
above comments for Zhe so let me know if anything I'm not going in the right
way.
bq. I see mixed tabs and spaces, we do spaces only in Hadoop.
Sorry this messy. I will absolutely clean up and follow the styles when
breakdown and submit patches for the sub tasks.
bq. Since the LRC stuff is still up in the air, could we defer everything
related to that to a later JIRA?
I agree. I added the LRC* stuffs just to make sure I'm keeping the kind of
codes like LRC in mind so that the codec framework is general enough and we
won't involve into having to redesign when considering to support such code
algorithms. I won't submit any LRC related formal patches before we're
confirmed about legal stuff.
bq. In RSBlockGrouper, using ExtendedBlockId is overkill, since the bpid is the
same for everything
I'm happy to know that about bpid. Thanks.
Will address the left comments later.
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Kai Zheng
> Attachments: HDFS-7337-prototype-v1.patch,
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
> PluggableErasureCodec.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
