[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532761#comment-17532761
]
Ayush Saxena commented on HDFS-16569:
-------------------------------------
The client has location, that is why the block went into the committed state,
once the client reports then only the block moves to committed state, once the
ibr is received then it marks the block complete.
{quote}Since client already has the last block locations, it's not necessary to
rely on the ibr from DN when closing file.
{quote}
You mean to say, since the client has already reported the block location, no
need to wait for the datanode to report it again, right? So, the answer is No,
that is per design, the namenode doesn't blindly trust a client, and there are
other reasons as well. So, it requires at least one datanode to confirm that it
has received the block with same amount of data as the client claims.
If you are seeing this log a lot, then most probably your cluster is slow, that
IBR's are reaching late. Try this config
{{dfs.namenode.file.close.num-committed-allowed}}, this should allow you to
close the file with committed blocks.
> Consider attaching block location info from client when closing a completed
> file
> --------------------------------------------------------------------------------
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Yuanbo Liu
> Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
