[ https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199691#comment-17199691 ]
Stephen O'Donnell edited comment on HDFS-15415 at 9/21/20, 10:27 PM: --------------------------------------------------------------------- I will commit the branch-3.3 patch tomorrow if there are no objections. Its basically identical to the trunk patch. The conflict is caused by the line: {code} Trunk: try (AutoCloseableLock lock = dataset.acquireDatasetReadLock()) { branch-3.3: try (AutoCloseableLock lock = dataset.acquireDatasetLock()) { {code} The change simply removes the lock from the entire block in the same way as the trunk patch. Waiting for HDFS-15583 to get committed before this can be backported to branch-3.2. was (Author: sodonnell): I will commit the branch-3.3 patch tomorrow if there are no objections. Its basically identical to the trunk patch. The conflict is caused by the line: {code} Trunk: try (AutoCloseableLock lock = dataset.acquireDatasetReadLock()) { branch-3.3: try (AutoCloseableLock lock = dataset.acquireDatasetLock()) { {code} The change simply removes the lock from the entire block. Waiting for HDFS-15583 to get committed before this can be backported to branch-3.2. > Reduce locking in Datanode DirectoryScanner > ------------------------------------------- > > Key: HDFS-15415 > URL: https://issues.apache.org/jira/browse/HDFS-15415 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 3.4.0 > Reporter: Stephen O'Donnell > Assignee: Stephen O'Donnell > Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15415.001.patch, HDFS-15415.002.patch, > HDFS-15415.003.patch, HDFS-15415.004.patch, HDFS-15415.005.patch, > HDFS-15415.branch-3.3.001.patch > > > In HDFS-15406, we have a small change to greatly reduce the runtime and > locking time of the datanode DirectoryScanner. They may be room for further > improvement. > From the scan step, we have captured a snapshot of what is on disk. After > calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in > memory. The two snapshots are never 100% in sync as things are always > changing as the disk is scanned. > We are only comparing finalized blocks, so they should not really change: > * If a block is deleted after our snapshot, our snapshot will not see it and > that is OK. > * A finalized block could be appended. If that happens both the genstamp and > length will change, but that should be handled by reconcile when it calls > `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being > appended after they have been scanned from disk, but before they have been > compared with memory. > My suspicion is that we can do all the comparison work outside of the lock > and checkAndUpdate() re-checks any differences later under the lock on a > block by block basis. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org