Stephen O'Donnell created HDFS-15415:
----------------------------------------
Summary: Reduce locking in Datanode DirectoryScanner
Key: HDFS-15415
URL: https://issues.apache.org/jira/browse/HDFS-15415
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.4.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
In HDFS-15406, we have a small change to greatly reduce the runtime and locking
time of the datanode DirectoryScanner. They may be room for further improvement
here:
1. These lines of code in DirectoryScanner#scan(), obtain a snapshot of the
finalized blocks from memory, and then sort them, under the DN lock. However
the blocks are stored in a sorted structure (FoldedTreeSet) and hence the sort
should be unnecessary.
{code}
final List<ReplicaInfo> bl = dataset.getFinalizedBlocks(bpid);
Collections.sort(bl); // Sort based on blockId
{code}
2. From the scan step, we have captured a snapshot of what is on disk. After
calling `dataset.getFinalizedBlocks(bpid);` as above we have taken a snapshot
of in memory. The two snapshots are never 100% in sync as things are always
changing as the disk is scanned.
We are only comparing finalized blocks, so they should not really change:
* If a block is deleted after our snapshot, our snapshot will not see it and
that is OK.
* A finalized block could be appended. If that happens both the genstamp and
length will change, but that should be handled by reconcile when it calls
`FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being
appended after they have been scanned from disk, but before they have been
compared with memory.
My suspicion is that we can do all the comparison work outside of the lock and
checkAndUpdate() re-checks any differences later under the lock on a block by
block basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
