[
https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104567#comment-14104567
]
Jing Zhao commented on HDFS-3689:
---------------------------------
bq. It might be better not to have a "transparent" feature that contains
performance surprises, but instead have something that both writers and readers
must knowingly adopt.
Yes, if I understand correctly, both the variable block length and the sparse
files require that the writers and readers both know the changes. And the
current implementation is not an exact "transparent" feature since 1) the
writer needs to call either the new append API or hsync(END_BLOCK) to generate
a variable-length block, and 2) the reader can identify the block length.
bq. of course they should. Sparse property should be maintained in metadata (i
think)
Thanks for the response, [~octo47]! My concern here is that if by doing sparse
files we finally still need to add extra metadata to both DN (and data transfer
protocol) and NN (to support sparse block copy triggered by tools like distcp),
we may loose the beauty and simplicity of the sparse file idea. And based on
the discussion between [~sureshms] and [~cutting], we still require the
assumption that both writers and readers must knowingly adopt.
> Add support for variable length block
> -------------------------------------
>
> Key: HDFS-3689
> URL: https://issues.apache.org/jira/browse/HDFS-3689
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs-client, namenode
> Affects Versions: 3.0.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch
>
>
> Currently HDFS supports fixed length blocks. Supporting variable length block
> will allow new use cases and features to be built on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
