[
https://issues.apache.org/jira/browse/HDFS-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877278#comment-15877278
]
Yiqun Lin commented on HDFS-11435:
----------------------------------
Thanks [~manojg] for the confirming. I suppose this should be similar to
HDFS-11194 which adding the {{SlowPeersReport}} in heartbeat to aggregate peer
performance metrics on NameNode. There is also some discussion of heartbeat
protocol. You can take a look that JIRA and may be helpful to you.
> NameNode should track open for write files lengths more frequent than on
> newer block allocations
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-11435
> URL: https://issues.apache.org/jira/browse/HDFS-11435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Manoj Govindassamy
> Assignee: Manoj Govindassamy
>
> *Problem:*
> Currently the length of an open for write / Under construction file is
> updated on the NameNode only when
> # Block boundary: On block boundaries and upon allocation of new Block,
> NameNode gets to know the file growth and the file length catches up
> # hsync(SyncFlag.UPDATE_LENGTH): Upon Client apps invoking a hsync on the
> write stream with a special flag, DataNodes send an incremental block report
> with the latest file length which NameNode uses it to update its meta data.
> # First hflush() on the new Block: Upon Client apps doing first time hflush()
> on an every new Block, DataNodes notifies NameNode about the latest file
> length.
> # Output stream close: Forces DataNodes update NameNode about the file length
> after data persistence and proper acknowledgements in the pipeline.
> So, lengths for open for write files are usually a lot less than the length
> seen by the DN/client. Highly preferred to have NameNode not lagging in file
> lengths by order of Block size for under construction files and to have more
> frequent, scalable update mechanism for these open file lengths.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
