[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=592667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-592667
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 04/May/21 11:07
Start Date: 04/May/21 11:07
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r625692545
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##
@@ -315,6 +316,41 @@ public DirectoryScanner(FsDatasetSpi dataset,
Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+int reconcileBatchSize =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+if (reconcileBatchSize < 0) {
Review comment:
change it to <=0
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##
@@ -315,6 +316,41 @@ public DirectoryScanner(FsDatasetSpi dataset,
Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+int reconcileBatchSize =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+if (reconcileBatchSize < 0) {
+ LOG.warn("Invalid value configured for " +
+ "dfs.datanode.reconcile.blocks.batch.size, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchSize =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT;
+}
+
+reconcileBlocksBatchSize = reconcileBatchSize;
+
+long reconcileBatchInterval =
+conf.getTimeDuration(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT,
+TimeUnit.MILLISECONDS);
+
+if (reconcileBatchInterval < 0) {
Review comment:
change it to <=0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 592667)
Time Spent: 2h 20m (was: 2h 10m)
> Make DirectoryScanner reconcile blocks batch size and interval between batch
> configurable.
> --
>
> Key: HDFS-15934
> URL: https://issues.apache.org/jira/browse/HDFS-15934
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> HDFS-14476 Make this batch to avoid lock too much time, but different cluster
> has different demand, we should make batch size and batch interval
> configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=584143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584143
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 08:25
Start Date: 16/Apr/21 08:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-821010516
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 237m 0s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 325m 37s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 4b1137f6cfbd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[ https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=584015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584015 ] ASF GitHub Bot logged work on HDFS-15934: - Author: ASF GitHub Bot Created on: 16/Apr/21 02:59 Start Date: 16/Apr/21 02:59 Worklog Time Spent: 10m Work Description: qizhu-lucas commented on pull request #2833: URL: https://github.com/apache/hadoop/pull/2833#issuecomment-820872414 Thanks very much for @ayushtkn patient review. I have updated above in latest PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 584015) Time Spent: 2h (was: 1h 50m) > Make DirectoryScanner reconcile blocks batch size and interval between batch > configurable. > -- > > Key: HDFS-15934 > URL: https://issues.apache.org/jira/browse/HDFS-15934 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > HDFS-14476 Make this batch to avoid lock too much time, but different cluster > has different demand, we should make batch size and batch interval > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=583573=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-583573
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 15/Apr/21 16:43
Start Date: 15/Apr/21 16:43
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r614224919
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##
@@ -315,6 +316,40 @@ public DirectoryScanner(FsDatasetSpi dataset,
Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+int reconcileBatchSize =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+if (reconcileBatchSize < 0) {
+ LOG.warn(
+ "Invalid value configured for
dfs.datanode.reconcile.blocks.batch.size, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchSize =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT;
+}
+
+reconcileBlocksBatchSize = reconcileBatchSize;
+
+long reconcileBatchInterval =
+conf.getLong(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
Review comment:
This should use ``conf.getTimeDuration``, If you need some reference can
check HDFS-15107, so as how to add support for time units.
##
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##
@@ -873,6 +873,22 @@
+
+ dfs.datanode.reconcile.blocks.batch.size
+ 1000
+ HDFS-14476 to enable running reconcile to checkAndUpdate with
batch,
Review comment:
No need for jira id in the description, and this doesn't enable/disable,
it is just to specify the batch size.
##
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##
@@ -873,6 +873,22 @@
+
+ dfs.datanode.reconcile.blocks.batch.size
+ 1000
+ HDFS-14476 to enable running reconcile to checkAndUpdate with
batch,
+setting this to define batch size.
+
+
+
+
+ dfs.datanode.reconcile.blocks.batch.interval
+ 2000
+ Setting this to define interval between
+batches defined in dfs.datanode.reconcile.blocks.batch.size.
+
Review comment:
Some grammatical error, Please rectify
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##
@@ -315,6 +316,40 @@ public DirectoryScanner(FsDatasetSpi dataset,
Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+int reconcileBatchSize =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+if (reconcileBatchSize < 0) {
+ LOG.warn(
+ "Invalid value configured for
dfs.datanode.reconcile.blocks.batch.size, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchSize =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT;
+}
+
+reconcileBlocksBatchSize = reconcileBatchSize;
+
+long reconcileBatchInterval =
+conf.getLong(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
+
+if (reconcileBatchInterval < 0) {
+ LOG.warn(
+ "Invalid value configured for
dfs.datanode.reconcile.blocks.batch.interval, " +
+ "should be greater than 0, Using default.");
+ reconcileBatchInterval =
+ DFSConfigKeys.
+ DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT;
+}
+
+reconcileBlocksBatchInterval = reconcileBatchInterval;
}
Review comment:
You need to format this code properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 583573)
Time Spent: 1h 50m (was: 1h 40m)
> Make DirectoryScanner reconcile blocks batch size and interval between batch
> configurable.
>
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=577602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-577602
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 06/Apr/21 13:50
Start Date: 06/Apr/21 13:50
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-814135509
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 55s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 484 unchanged
- 0 fixed = 490 total (was 484) |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 228m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 326m 59s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.datanode.TestBlockRecovery2 |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests |
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[ https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=577409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-577409 ] ASF GitHub Bot logged work on HDFS-15934: - Author: ASF GitHub Bot Created on: 06/Apr/21 08:24 Start Date: 06/Apr/21 08:24 Worklog Time Spent: 10m Work Description: qizhu-lucas commented on pull request #2833: URL: https://github.com/apache/hadoop/pull/2833#issuecomment-813931427 @ayushtkn Thanks a lot for your review and good suggestions. I have updated in latest PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 577409) Time Spent: 1.5h (was: 1h 20m) > Make DirectoryScanner reconcile blocks batch size and interval between batch > configurable. > -- > > Key: HDFS-15934 > URL: https://issues.apache.org/jira/browse/HDFS-15934 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > HDFS-14476 Make this batch to avoid lock too much time, but different cluster > has different demand, we should make batch size and batch interval > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[ https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=577406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-577406 ] ASF GitHub Bot logged work on HDFS-15934: - Author: ASF GitHub Bot Created on: 06/Apr/21 08:23 Start Date: 06/Apr/21 08:23 Worklog Time Spent: 10m Work Description: qizhu-lucas commented on a change in pull request #2833: URL: https://github.com/apache/hadoop/pull/2833#discussion_r607637557 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java ## @@ -836,6 +836,10 @@ public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600; public static final String DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY = "dfs.datanode.directoryscan.threads"; public static final int DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT = 1; + public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE = "dfs.datanode.reconcile.blocks.batch.size"; + public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT = 1000; + public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL = "dfs.datanode.reconcile.blocks.batch.interval"; + public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT = 2000; Review comment: Fixed it in latest patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 577406) Time Spent: 1h (was: 50m) > Make DirectoryScanner reconcile blocks batch size and interval between batch > configurable. > -- > > Key: HDFS-15934 > URL: https://issues.apache.org/jira/browse/HDFS-15934 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > HDFS-14476 Make this batch to avoid lock too much time, but different cluster > has different demand, we should make batch size and batch interval > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[ https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=577408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-577408 ] ASF GitHub Bot logged work on HDFS-15934: - Author: ASF GitHub Bot Created on: 06/Apr/21 08:23 Start Date: 06/Apr/21 08:23 Worklog Time Spent: 10m Work Description: qizhu-lucas commented on a change in pull request #2833: URL: https://github.com/apache/hadoop/pull/2833#discussion_r607637557 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java ## @@ -836,6 +836,10 @@ public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600; public static final String DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY = "dfs.datanode.directoryscan.threads"; public static final int DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT = 1; + public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE = "dfs.datanode.reconcile.blocks.batch.size"; + public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT = 1000; + public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL = "dfs.datanode.reconcile.blocks.batch.interval"; + public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT = 2000; Review comment: Fixed it in latest PR. ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ## @@ -873,6 +873,22 @@ + + dfs.datanode.reconcile.blocks.batch.size + 1000 + Run reconcile to checkAndUpdate with batch, Review comment: Changed in latest PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 577408) Time Spent: 1h 20m (was: 1h 10m) > Make DirectoryScanner reconcile blocks batch size and interval between batch > configurable. > -- > > Key: HDFS-15934 > URL: https://issues.apache.org/jira/browse/HDFS-15934 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > HDFS-14476 Make this batch to avoid lock too much time, but different cluster > has different demand, we should make batch size and batch interval > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=577407=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-577407
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 06/Apr/21 08:23
Start Date: 06/Apr/21 08:23
Worklog Time Spent: 10m
Work Description: qizhu-lucas commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r607637716
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##
@@ -316,6 +317,18 @@ public DirectoryScanner(FsDatasetSpi dataset,
Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+reconcileBlocksBatchSize =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+reconcileBlocksBatchInterval =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
Review comment:
Fixed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 577407)
Time Spent: 1h 10m (was: 1h)
> Make DirectoryScanner reconcile blocks batch size and interval between batch
> configurable.
> --
>
> Key: HDFS-15934
> URL: https://issues.apache.org/jira/browse/HDFS-15934
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> HDFS-14476 Make this batch to avoid lock too much time, but different cluster
> has different demand, we should make batch size and batch interval
> configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=576499=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-576499
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 03/Apr/21 21:50
Start Date: 03/Apr/21 21:50
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#discussion_r606714508
##
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##
@@ -873,6 +873,22 @@
+
+ dfs.datanode.reconcile.blocks.batch.size
+ 1000
+ Run reconcile to checkAndUpdate with batch,
Review comment:
can you add some recheck the descriptions for both the configs, the
first line in. both is same, and doesn't make sense to me, recheck once.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
##
@@ -316,6 +317,18 @@ public DirectoryScanner(FsDatasetSpi dataset,
Configuration conf) {
masterThread =
new ScheduledThreadPoolExecutor(1, new Daemon.DaemonFactory());
+
+reconcileBlocksBatchSize =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT);
+
+reconcileBlocksBatchInterval =
+conf.getInt(DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL,
+DFSConfigKeys.
+DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT);
Review comment:
Add a validation for these configs, if ``reconcileBlocksBatchSize`` and
``reconcileBlocksBatchInterval`` is less than one use default. and add a warn
log message if these values are incorrect something like:
Invalid value configured for < config name>, should be greater than 0, Using
default.
In the end add an Info log for the values being used.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##
@@ -836,6 +836,10 @@
public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT =
21600;
public static final String DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY =
"dfs.datanode.directoryscan.threads";
public static final int DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT = 1;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE =
"dfs.datanode.reconcile.blocks.batch.size";
+ public static final int DFS_DATANODE_RECONCILE_BLOCKS_BATCH_SIZE_DEFAULT
= 1000;
+ public static final String DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL =
"dfs.datanode.reconcile.blocks.batch.interval";
+ public static final int
DFS_DATANODE_RECONCILE_BLOCKS_BATCH_INTERVAL_DEFAULT = 2000;
Review comment:
Can we add support for time units?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 576499)
Time Spent: 50m (was: 40m)
> Make DirectoryScanner reconcile blocks batch size and interval between batch
> configurable.
> --
>
> Key: HDFS-15934
> URL: https://issues.apache.org/jira/browse/HDFS-15934
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> HDFS-14476 Make this batch to avoid lock too much time, but different cluster
> has different demand, we should make batch size and batch interval
> configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[ https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=576165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-576165 ] ASF GitHub Bot logged work on HDFS-15934: - Author: ASF GitHub Bot Created on: 02/Apr/21 16:33 Start Date: 02/Apr/21 16:33 Worklog Time Spent: 10m Work Description: qizhu-lucas edited a comment on pull request #2833: URL: https://github.com/apache/hadoop/pull/2833#issuecomment-809439075 @Hexiaoqiao @ayushtkn @liuml07 Could you help review this, when you are free? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 576165) Time Spent: 40m (was: 0.5h) > Make DirectoryScanner reconcile blocks batch size and interval between batch > configurable. > -- > > Key: HDFS-15934 > URL: https://issues.apache.org/jira/browse/HDFS-15934 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > HDFS-14476 Make this batch to avoid lock too much time, but different cluster > has different demand, we should make batch size and batch interval > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[
https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=573697=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573697
]
ASF GitHub Bot logged work on HDFS-15934:
-
Author: ASF GitHub Bot
Created on: 29/Mar/21 20:06
Start Date: 29/Mar/21 20:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2833:
URL: https://github.com/apache/hadoop/pull/2833#issuecomment-809675715
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 55s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 484 unchanged
- 0 fixed = 488 total (was 484) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 234m 32s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 320m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2833/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2833 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[ https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=573534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573534 ] ASF GitHub Bot logged work on HDFS-15934: - Author: ASF GitHub Bot Created on: 29/Mar/21 14:45 Start Date: 29/Mar/21 14:45 Worklog Time Spent: 10m Work Description: qizhu-lucas commented on pull request #2833: URL: https://github.com/apache/hadoop/pull/2833#issuecomment-809439075 @Hexiaoqiao @ayushtkn Could you help review this, when you are free? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 573534) Time Spent: 20m (was: 10m) > Make DirectoryScanner reconcile blocks batch size and interval between batch > configurable. > -- > > Key: HDFS-15934 > URL: https://issues.apache.org/jira/browse/HDFS-15934 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > HDFS-14476 Make this batch to avoid lock too much time, but different cluster > has different demand, we should make batch size and batch interval > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.
[ https://issues.apache.org/jira/browse/HDFS-15934?focusedWorklogId=573531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573531 ] ASF GitHub Bot logged work on HDFS-15934: - Author: ASF GitHub Bot Created on: 29/Mar/21 14:44 Start Date: 29/Mar/21 14:44 Worklog Time Spent: 10m Work Description: qizhu-lucas opened a new pull request #2833: URL: https://github.com/apache/hadoop/pull/2833 …erval between batch configurable. ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 573531) Remaining Estimate: 0h Time Spent: 10m > Make DirectoryScanner reconcile blocks batch size and interval between batch > configurable. > -- > > Key: HDFS-15934 > URL: https://issues.apache.org/jira/browse/HDFS-15934 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > HDFS-14476 Make this batch to avoid lock too much time, but different cluster > has different demand, we should make batch size and batch interval > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
