[
https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434941#comment-17434941
]
Bryan Beaudreault commented on HDFS-16261:
------------------------------------------
Just to close the loop on that last comment, it did not make sense to implement
this in the DataNode. It was theoretically possible, but became very messy due
to how access of the volumeMap is managed there. I also think implementing in
the NameNode is more inline with the general HDFS architecture, where the
NameNode owns the state around block locations and pending work.
I've had the grace period running on a couple internal clusters for a couple
weeks now. I just submitted a PR for review.
> Configurable grace period around invalidation of replaced blocks
> ----------------------------------------------------------------
>
> Key: HDFS-16261
> URL: https://issues.apache.org/jira/browse/HDFS-16261
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Bryan Beaudreault
> Assignee: Bryan Beaudreault
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When a block is moved with REPLACE_BLOCK, the new location is recorded in the
> NameNode and the NameNode instructs the old host to in invalidate the block
> using DNA_INVALIDATE. As it stands today, this invalidation is async but
> tends to happen relatively quickly.
> I'm working on a feature for HBase which enables efficient healing of
> locality through Balancer-style low level block moves (HBASE-26250). One
> issue is that HBase tends to keep open long running DFSInputStreams and
> moving blocks from under them causes lots of warns in the RegionServer and
> increases long tail latencies due to the necessary retries in the DFSClient.
> One way I'd like to fix this is to provide a configurable grace period on
> async invalidations. This would give the DFSClient enough time to refresh
> block locations before hitting any errors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
