Hi, Ivan, thank you for your willingness to help. First, what do I mean by a more efficient NN? - I am experimenting with Kove <http://kove.com/> device, persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see if this improves performance. 2. Use it for BlockMaps, not for performance, but for fault-tolerance. To prepare for this, I want to run some benchmarks on the given install just to get experience running it. Then I will run it on a 3-node cluster without Kove, and then with Kove. Here is what I got from nnbench, and I am trying to find what I should watch for to measure NN performance. Does this all make sense? Thank you, Mark 13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- : 13/06/05 16:50:16 INFO hdfs.NNBench: Version: NameNode Benchmark 0.4 13/06/05 16:50:16 INFO hdfs.NNBench: Date & time: 2013-06-05 16:50:16,412 13/06/05 16:50:16 INFO hdfs.NNBench: 13/06/05 16:50:16 INFO hdfs.NNBench: Test Operation: open_read 13/06/05 16:50:16 INFO hdfs.NNBench: Start time: 2013-06-05 16:48:34,792 13/06/05 16:50:16 INFO hdfs.NNBench: Maps to run: 12 13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to run: 6 13/06/05 16:50:16 INFO hdfs.NNBench: Block Size (bytes): 1 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to write: 0 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per checksum: 1 13/06/05 16:50:16 INFO hdfs.NNBench: Number of files: 1000 13/06/05 16:50:16 INFO hdfs.NNBench: Replication factor: 3 13/06/05 16:50:16 INFO hdfs.NNBench: Successful file operations: 0 13/06/05 16:50:16 INFO hdfs.NNBench: 13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the barrier: 0 13/06/05 16:50:16 INFO hdfs.NNBench: # exceptions: 0 13/06/05 16:50:16 INFO hdfs.NNBench: 13/06/05 16:50:16 INFO hdfs.NNBench: TPS: Open/Read: 0 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms): Open/Read: 0.0 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Open: Infinity 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read: NaN 13/06/05 16:50:16 INFO hdfs.NNBench: 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #1: 4665 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #2: 0 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total (ms): 0 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms): 0.0 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late maps: 0 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of exceptions: 0 On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <[email protected]> wrote: > Hi Mark,**** > > ** ** > > NNBench is a namenode load test. Output of the test is the set of > performance numbers, like transactions per second, average latency of > operations, etc.**** > > ** ** > > What do you mean by trying to create a more efficient namenode? What > dimension are you trying to optimize? Depending on this, people on this > list might be able to guide you to something more appropriate. **** > > ** ** > > Hope this helps,**** > > Ivan **** > > ** ** > > *From:* Mark Kerzner [mailto:[email protected]] > *Sent:* Wednesday, June 05, 2013 3:17 PM > *To:* Hadoop User > *Subject:* How to test the performance of NN?**** > > ** ** > > Hi, > > I am trying to create a more efficient namenode, and for that I need to > the standard distribution, and then compare it to my version. > > Which benchmark should I run? I am doing nnbench, but it is not telling me > anything about performance, only about potential failures. > > Thank you.**** > > Sincerely, > Mark**** >
