[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027999#comment-15027999
]
Mingliang Liu commented on HDFS-9436:
-------------------------------------
Thank you very much for the detailed explanation. This is very helpful to me to
understand the benchmark.
# The {{numOpsRequired}} default value 10 and number of datanodes default 10
together seem OK. So perhaps we don't need to change {{numOpsRequired}} here?
# My second concern in last comment was that {{TinyDatanode#blocks}} is an
empty ArrayList with initial capacity. In {{TinyDatanode#addBlock()}} first
statement, the {{if(nrBlocks == blocks.size()) {}} will always be true. I
believe this is wrong in current {{trunk}} code. We should either fill the
{{blocks}} with dummy report in {{TinyDatanode()}} constructor, or use initial
capacity instead of blocks.size() in the above _if_ statement (we should
replace {{ArrayList#set}} with {{ArrayList#add}} as well). The overall change
is like following (not tested):
{code}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
index 4db4da0..0ff7893 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
@@ -929,6 +929,7 @@ long executeOp(int daemonId, int inputIdx, String ignore)
int nrBlocks; // actual number of blocks
BlockListAsLongs blockReportList;
final int dnIdx;
+ private final int blockCapacity;
private static int getNodePort(int num) throws IOException {
int port = 1 + num;
@@ -939,6 +940,7 @@ private static int getNodePort(int num) throws IOException {
TinyDatanode(int dnIdx, int blockCapacity) throws IOException {
this.dnIdx = dnIdx;
this.blocks = new ArrayList<BlockReportReplica>(blockCapacity);
+ this.blockCapacity = blockCapacity;
this.nrBlocks = 0;
}
@@ -996,22 +998,22 @@ void sendHeartbeat() throws IOException {
}
boolean addBlock(Block blk) {
- if(nrBlocks == blocks.size()) {
+ if(nrBlocks == blockCapacity) {
if(LOG.isDebugEnabled()) {
LOG.debug("Cannot add block: datanode capacity = " + blocks.size());
}
return false;
}
- blocks.set(nrBlocks, new BlockReportReplica(blk));
+ blocks.add(new BlockReportReplica(blk));
nrBlocks++;
return true;
}
void formBlockReport() {
// fill remaining slots with blocks that do not exist
- for (int idx = blocks.size()-1; idx >= nrBlocks; idx--) {
+ for (int idx = blockCapacity - 1; idx >= nrBlocks; idx--) {
Block block = new Block(blocks.size() - idx, 0, 0);
- blocks.set(idx, new BlockReportReplica(block));
+ blocks.add(new BlockReportReplica(block));
}
blockReportList = BlockListAsLongs.EMPTY;
}
{code}
Scroll down if not fully shown.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
> ----------------------------------------------------------------------------
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Priority: Minor
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
