[
https://issues.apache.org/jira/browse/HDFS-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936994#comment-16936994
]
Arpit Agarwal commented on HDFS-14859:
--------------------------------------
+1 for the patch. Please address the checkstyle warnings.
The javac warning is unrelated to your patch and can be ignored. The UT
failures also look unrelated.
> Prevent Un-necessary evaluation of costly operation getNumLiveDataNodes when
> dfs.namenode.safemode.min.datanodes is not zero
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-14859
> URL: https://issues.apache.org/jira/browse/HDFS-14859
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.1.0, 3.3.0, 3.1.4
> Reporter: Srinivasu Majeti
> Assignee: Srinivasu Majeti
> Priority: Major
> Labels: block
> Attachments: HDFS-14859.001.patch, HDFS-14859.002.patch,
> HDFS-14859.003.patch, HDFS-14859.004.patch, HDFS-14859.005.patch,
> HDFS-14859.006.patch
>
>
> There have been improvements like HDFS-14171 and HDFS-14632 to the
> performance issue caused from getNumLiveDataNodes calls per block. The
> improvement has been only done w.r.t dfs.namenode.safemode.min.datanodes
> paramter being set to 0 or not.
> {code}
> private boolean areThresholdsMet() {
> assert namesystem.hasWriteLock();
> - int datanodeNum =
> blockManager.getDatanodeManager().getNumLiveDataNodes();
> + // Calculating the number of live datanodes is time-consuming
> + // in large clusters. Skip it when datanodeThreshold is zero.
> + int datanodeNum = 0;
> + if (datanodeThreshold > 0) {
> + datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> + }
> synchronized (this) {
> return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
> }
> {code}
> I feel above logic would create similar situation of un-necessary evaluations
> of getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes paramter is
> set > 0 even though "blockSafe >= blockThreshold" is false for most of the
> time in NN startup safe mode. We could do something like below to avoid this
> {code}
> private boolean areThresholdsMet() {
> assert namesystem.hasWriteLock();
> synchronized (this) {
> return blockSafe >= blockThreshold && (datanodeThreshold > 0)?
> blockManager.getDatanodeManager().getNumLiveDataNodes() >=
> datanodeThreshold : true;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
