[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532765#comment-17532765
]
Yuanbo Liu commented on HDFS-16569:
-----------------------------------
[~hexiaoqiao] Thanks for your comment.
The process is here:
1. client get block location from nn
2. setup pipeline and write block data
3. when the block is finished, then retry multiple times to make sure DN has
sent RECEIVED_BLOCK to NN by IBR
4. when batch ibr is enabled (the interval is usually 300ms), client will not
mark block as finshed before 300ms(even longer if DN is very busy)
5. repeat step3,4 until last block is finshed, then client try to close file
multiple times(default is 5 times with around 15 seconds).
6. If DN is busy, or ibr batch is not triggerred when interval is not reached,
We can see log 'is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in
file' in namenode
As we know, client already gets the location info from nn, and gets acks from
dn, it seems not necessary to rely on ibr to mark block as completed. So we can
send block locations to NN in step3
> Consider attaching block location info from client when closing a completed
> file
> --------------------------------------------------------------------------------
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Yuanbo Liu
> Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
