[
https://issues.apache.org/jira/browse/HDFS-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946649#comment-16946649
]
Hudson commented on HDFS-14859:
-------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17502 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17502/])
HDFS-14859. Prevent unnecessary evaluation of costly operation (ayushsaxena:
rev 91320b446171013ad47783d7400d646d2d71ca3d)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java
> Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when
> dfs.namenode.safemode.min.datanodes is not zero
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-14859
> URL: https://issues.apache.org/jira/browse/HDFS-14859
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.1.0, 3.3.0, 3.1.4
> Reporter: Srinivasu Majeti
> Assignee: Srinivasu Majeti
> Priority: Major
> Labels: block
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14859.001.patch, HDFS-14859.002.patch,
> HDFS-14859.003.patch, HDFS-14859.004.patch, HDFS-14859.005.patch,
> HDFS-14859.006.patch, HDFS-14859.007.patch
>
>
> There have been improvements like HDFS-14171 and HDFS-14632 to the
> performance issue caused from getNumLiveDataNodes calls per block. The
> improvement has been only done w.r.t dfs.namenode.safemode.min.datanodes
> paramter being set to 0 or not.
> {code}
> private boolean areThresholdsMet() {
> assert namesystem.hasWriteLock();
> - int datanodeNum =
> blockManager.getDatanodeManager().getNumLiveDataNodes();
> + // Calculating the number of live datanodes is time-consuming
> + // in large clusters. Skip it when datanodeThreshold is zero.
> + int datanodeNum = 0;
> + if (datanodeThreshold > 0) {
> + datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> + }
> synchronized (this) {
> return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
> }
> {code}
> I feel above logic would create similar situation of un-necessary evaluations
> of getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes paramter is
> set > 0 even though "blockSafe >= blockThreshold" is false for most of the
> time in NN startup safe mode. We could do something like below to avoid this
> {code}
> private boolean areThresholdsMet() {
> assert namesystem.hasWriteLock();
> synchronized (this) {
> return blockSafe >= blockThreshold && (datanodeThreshold > 0)?
> blockManager.getDatanodeManager().getNumLiveDataNodes() >=
> datanodeThreshold : true;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
