[
https://issues.apache.org/jira/browse/HDFS-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15875368#comment-15875368
]
Yiqun Lin edited comment on HDFS-11435 at 2/21/17 6:03 AM:
-----------------------------------------------------------
This seems interesting! Hi [~manojg], do you have any solutions in your mind
for reporting the length of open files to NN now? As I see in HDFS-11402, Jing
mentioned the way to utilize the DN heartbeats. Are you plan to do on this way?
From my understanding on this, the heartbeat is heavy now. So We did the
DataNode Lifeline. This maybe what we are concerning here.
was (Author: linyiqun):
This seems interesting! Hi [~manojg], do you have any solutions in your mind
for reporting the length of open files to NN now? As I see in HDFS-11402, Jing
mentioned the way to utilize the DN heartbeats. Are you plan to do on this way?
> NameNode should track open for write files lengths more frequent than on
> newer block allocations
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-11435
> URL: https://issues.apache.org/jira/browse/HDFS-11435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Manoj Govindassamy
> Assignee: Manoj Govindassamy
>
> *Problem:*
> Currently the length of an open for write / Under construction file is
> updated on the NameNode only when
> # Block boundary: On block boundaries and upon allocation of new Block,
> NameNode gets to know the file growth and the file length catches up
> # hsync(SyncFlag.UPDATE_LENGTH): Upon Client apps invoking a hsync on the
> write stream with a special flag, DataNodes send an incremental block report
> with the latest file length which NameNode uses it to update its meta data.
> # First hflush() on the new Block: Upon Client apps doing first time hflush()
> on an every new Block, DataNodes notifies NameNode about the latest file
> length.
> # Output stream close: Forces DataNodes update NameNode about the file length
> after data persistence and proper acknowledgements in the pipeline.
> So, lengths for open for write files are usually a lot less than the length
> seen by the DN/client. Highly preferred to have NameNode not lagging in file
> lengths by order of Block size for under construction files and to have more
> frequent, scalable update mechanism for these open file lengths.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
