[
https://issues.apache.org/jira/browse/HDFS-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229446#comment-15229446
]
Kai Zheng commented on HDFS-7661:
---------------------------------
Thanks all for the great discussions and hard work. Sorry for my interruption.
The proposed way that sends all cellBuffers to all parity DNs additionally
sounds to work because all the real data in question or to be flushed are
replicated to the parity DNs thus the parity cells can be computed on DNs on
demand. On the other hand, this will complicate the effort to support advanced
erasure codecs because any codec/coder will then need to aware the trick played
here.
I understand there is no easy solution to this problem. I'm wondering the
involved complexity and overhead for the support may be too much because we
don't have any practical solid use cases that requires hflush/hsync in striping
mode. Looks like HBase workload did be mentioned, but I doubt if there is any
benchmark that proves striping will be suitable for HBase. Any HBase fellow
could cast some thoughts here? How about making hflush/hsync api as
NOT-SUPPORTED or TO-BE-IMPLEMENTED for striping files? This may resolve a
blocker for the 3.0 release, and when any practical use cases or requirement
for such, this effort can then be revisited.
> [umbrella] support hflush and hsync for erasure coded files
> -----------------------------------------------------------
>
> Key: HDFS-7661
> URL: https://issues.apache.org/jira/browse/HDFS-7661
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
> Reporter: Tsz Wo Nicholas Sze
> Assignee: GAO Rui
> Attachments: EC-file-flush-and-sync-steps-plan-2015-12-01.png,
> HDFS-7661-unitTest-wip-trunk.patch, HDFS-7661-wip.01.patch,
> HDFS-EC-file-flush-sync-design-v20160323.pdf,
> HDFS-EC-file-flush-sync-design-version1.1.pdf, Undo-Log-Design-20160406.jpg
>
>
> We also need to support hflush/hsync and visible length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
