[
https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17203511#comment-17203511
]
Stephen O'Donnell commented on HDFS-15415:
------------------------------------------
Updated a patch for branch-3.1. With this patch on 3.1, the diff between it and
the commited patch of 3.2 is:
{code}
$ git diff backport-HDFS-15415-3.1..origin/branch-3.2
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
<snipped>
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
@@ -37,9 +37,9 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
-import org.apache.commons.lang.time.FastDateFormat;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
+import org.apache.commons.lang3.time.FastDateFormat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.StorageType;
@@ -58,7 +58,8 @@
*/
@InterfaceAudience.Private
public class DirectoryScanner implements Runnable {
- private static final Log LOG = LogFactory.getLog(DirectoryScanner.class);
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DirectoryScanner.class);
private static final int MILLIS_PER_SECOND = 1000;
private static final String START_MESSAGE =
"Periodic Directory Tree Verification scan"
@@ -163,7 +164,7 @@ public String toString() {
/**
* Merges {@code that} ScanInfoPerBlockPool into this one
*
- * @param the ScanInfoPerBlockPool to merge
+ * @param that ScanInfoPerBlockPool to merge
*/
public void addAll(ScanInfoPerBlockPool that) {
if (that == null) return;
@@ -459,9 +460,9 @@ private void scan() {
d++;
if (d < blockpoolReport.length) {
- // There may be multiple on-disk records for the same block, don't
increment
- // the memory record pointer if so.
- ScanInfo nextInfo = blockpoolReport[Math.min(d,
blockpoolReport.length - 1)];
+ // There may be multiple on-disk records for the same block,
+ // don't increment the memory record pointer if so.
+ ScanInfo nextInfo = blockpoolReport[d];
if (nextInfo.getBlockId() != info.getBlockId()) {
++m;
}
{code}
This patch only changes the scan method, and note the only difference between
the two files is the change added by HDFS-13829 on branch-3.2.
> Reduce locking in Datanode DirectoryScanner
> -------------------------------------------
>
> Key: HDFS-15415
> URL: https://issues.apache.org/jira/browse/HDFS-15415
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.4.0
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-15415.001.patch, HDFS-15415.002.patch,
> HDFS-15415.003.patch, HDFS-15415.004.patch, HDFS-15415.005.patch,
> HDFS-15415.branch-3.1.001.patch, HDFS-15415.branch-3.2.001.patch,
> HDFS-15415.branch-3.2.002.patch, HDFS-15415.branch-3.3.001.patch
>
>
> In HDFS-15406, we have a small change to greatly reduce the runtime and
> locking time of the datanode DirectoryScanner. They may be room for further
> improvement.
> From the scan step, we have captured a snapshot of what is on disk. After
> calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in
> memory. The two snapshots are never 100% in sync as things are always
> changing as the disk is scanned.
> We are only comparing finalized blocks, so they should not really change:
> * If a block is deleted after our snapshot, our snapshot will not see it and
> that is OK.
> * A finalized block could be appended. If that happens both the genstamp and
> length will change, but that should be handled by reconcile when it calls
> `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being
> appended after they have been scanned from disk, but before they have been
> compared with memory.
> My suspicion is that we can do all the comparison work outside of the lock
> and checkAndUpdate() re-checks any differences later under the lock on a
> block by block basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
