Hi Mark,
Your NNBench output does not look OK to me. Below is an example NNBench cmd
line I used some time ago and its output.
Btw, what I found useful in testing the disk IO performance for namenode
operations (others might find this unusual) is the TestEditLog unittest. I
would modify the test as appropriate and run the simulation. This was a quick
micro-benchmark I used to compare a couple of devices. Note that you'll have to
modify the test in trunk and remove the line with
EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you're not
testing the real thing.
hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation create_write
-maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20 -bytesPerChecksum 1
-numberOfFiles 500 -replicationFactorPerFile 1
-------------- NNBench -------------- :
Version: NameNode Benchmark 0.4
Date & time: 2012-12-18 07:29:20,832
Test Operation: create_write
Start time: 2012-12-18 07:25:36,111
Maps to run: 32
Reduces to run: 1
Block Size (bytes): 1
Bytes to write: 20
Bytes per checksum: 1
Number of files: 500
Replication factor: 1
Successful file operations: 16000
# maps that missed the barrier: 0
# exceptions: 0
TPS: Create/Write/Close: 153
Avg exec time (ms): Create/Write/Close: 301.6148125
Avg Lat (ms): Create/Write: 68.25625
Avg Lat (ms): Close: 233.245
RAW DATA: AL Total #1: 1092100
RAW DATA: AL Total #2: 3731920
RAW DATA: TPS Total (ms): 4825837
RAW DATA: Longest Map Time (ms): 208692.0
RAW DATA: Late maps: 0
RAW DATA: # of exceptions: 0
Hope this helps,
Ivan
From: Mark Kerzner [mailto:[email protected]]
Sent: Wednesday, June 05, 2013 5:16 PM
To: Hadoop User
Subject: Re: How to test the performance of NN?
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more
efficient NN? - I am experimenting with Kove<http://kove.com/> device,
persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see if
this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install just to
get experience running it. Then I will run it on a 3-node cluster without Kove,
and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should watch
for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench: Version:
NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date & time:
2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test Operation:
open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start time:
2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size (bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of files:
1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: # exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS: Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms): Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Open:
Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read: NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #1:
4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total (ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic
<[email protected]<mailto:[email protected]>> wrote:
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance
numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension
are you trying to optimize? Depending on this, people on this list might be
able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the
standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me
anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
