Awesome advice, Ivan! It works for me now. Thank you, Mark
On Thu, Jun 6, 2013 at 12:18 AM, Ivan Mitic <[email protected]> wrote: > Hi Mark,**** > > ** ** > > Your NNBench output does not look OK to me. Below is an example NNBench > cmd line I used some time ago and its output.**** > > ** ** > > Btw, what I found useful in testing the disk IO performance for namenode > operations (others might find this unusual) is the TestEditLog unittest. I > would modify the test as appropriate and run the simulation. This was a > quick micro-benchmark I used to compare a couple of devices. Note that > you’ll have to modify the test in trunk and remove the line with > EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you’re > not testing the real thing.**** > > ** ** > > ** ** > > hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation > create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20 > -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1**** > > ** ** > > ** ** > > -------------- NNBench -------------- :**** > > Version: NameNode Benchmark 0.4**** > > Date & time: 2012-12-18 07:29:20,832**** > > ** ** > > Test Operation: create_write**** > > Start time: 2012-12-18 07:25:36,111**** > > Maps to run: 32**** > > Reduces to run: 1**** > > Block Size (bytes): 1**** > > Bytes to write: 20**** > > Bytes per checksum: 1**** > > Number of files: 500**** > > Replication factor: 1**** > > Successful file operations: 16000**** > > ** ** > > # maps that missed the barrier: 0**** > > # exceptions: 0**** > > ** ** > > TPS: Create/Write/Close: 153**** > > Avg exec time (ms): Create/Write/Close: 301.6148125**** > > Avg Lat (ms): Create/Write: 68.25625**** > > Avg Lat (ms): Close: 233.245**** > > ** ** > > RAW DATA: AL Total #1: 1092100**** > > RAW DATA: AL Total #2: 3731920**** > > RAW DATA: TPS Total (ms): 4825837**** > > RAW DATA: Longest Map Time (ms): 208692.0**** > > RAW DATA: Late maps: 0**** > > RAW DATA: # of exceptions: 0**** > > ** ** > > Hope this helps,**** > > Ivan**** > > ** ** > > *From:* Mark Kerzner [mailto:[email protected]] > *Sent:* Wednesday, June 05, 2013 5:16 PM > *To:* Hadoop User > *Subject:* Re: How to test the performance of NN?**** > > ** ** > > Hi, Ivan,**** > > thank you for your willingness to help. First, what do I mean by a more > efficient NN? - I am experimenting with Kove <http://kove.com/> device, > persistent memory. I want to do one of the two things:**** > > 1. Use it for FSDirectory, with the expectation of very fast writes, to > see if this improves performance.**** > > 2. Use it for BlockMaps, not for performance, but for fault-tolerance.**** > > To prepare for this, I want to run some benchmarks on the given install > just to get experience running it. Then I will run it on a 3-node cluster > without Kove, and then with Kove.**** > > Here is what I got from nnbench, and I am trying to find what I should > watch for to measure NN performance. Does this all make sense?**** > > Thank you, > Mark > > > 13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- > : > 13/06/05 16:50:16 INFO hdfs.NNBench: > Version: NameNode Benchmark 0.4 > 13/06/05 16:50:16 INFO hdfs.NNBench: Date & > time: 2013-06-05 16:50:16,412 > 13/06/05 16:50:16 INFO hdfs.NNBench: > 13/06/05 16:50:16 INFO hdfs.NNBench: Test > Operation: open_read > 13/06/05 16:50:16 INFO hdfs.NNBench: Start > time: 2013-06-05 16:48:34,792 > 13/06/05 16:50:16 INFO hdfs.NNBench: Maps to > run: 12 > 13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to > run: 6 > 13/06/05 16:50:16 INFO hdfs.NNBench: Block Size > (bytes): 1 > 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to > write: 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per > checksum: 1 > 13/06/05 16:50:16 INFO hdfs.NNBench: Number of > files: 1000 > 13/06/05 16:50:16 INFO hdfs.NNBench: Replication > factor: 3 > 13/06/05 16:50:16 INFO hdfs.NNBench: Successful file > operations: 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: > 13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the > barrier: 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: # > exceptions: 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: > 13/06/05 16:50:16 INFO hdfs.NNBench: TPS: > Open/Read: 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms): > Open/Read: 0.0 > 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): > Open: Infinity > 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read: > NaN > 13/06/05 16:50:16 INFO hdfs.NNBench: > 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total > #1: 4665 > 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total > #2: 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total > (ms): 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time > (ms): 0.0 > 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late > maps: 0 > 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of > exceptions: 0**** > > ** ** > > ** ** > > On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <[email protected]> wrote:** > ** > > Hi Mark,**** > > **** > > NNBench is a namenode load test. Output of the test is the set of > performance numbers, like transactions per second, average latency of > operations, etc.**** > > **** > > What do you mean by trying to create a more efficient namenode? What > dimension are you trying to optimize? Depending on this, people on this > list might be able to guide you to something more appropriate. **** > > **** > > Hope this helps,**** > > Ivan **** > > **** > > *From:* Mark Kerzner [mailto:[email protected]] > *Sent:* Wednesday, June 05, 2013 3:17 PM > *To:* Hadoop User > *Subject:* How to test the performance of NN?**** > > **** > > Hi, > > I am trying to create a more efficient namenode, and for that I need to > the standard distribution, and then compare it to my version. > > Which benchmark should I run? I am doing nnbench, but it is not telling me > anything about performance, only about potential failures. > > Thank you.**** > > Sincerely, > Mark**** > > ** ** >
