interesting. FWIW the work on formally specify (in the Computer Science notion of "formally") is in HADOOP-9361; the HCFS work being driven by redhat is more about testing.
Some extra ideas on benchmarking # something to assess performance of cross FS operations # it'd be nice to have something that would let you experiment with different hardware options in that NN # the gridmix3 MapReduce benchmarks can collect job use stats to generate synthetic workloads. Maybe we could derive something similar from NN metrics, so that we could build up a better pool of operations on different workloads (e.g. HBase, Hive + Tez) and apply them. # there's work needed on scalability tests across filesystems; for the '9361 tests I'm making them per-FS programmable for options like max #of files in a directory test, max filesize etc -any additions there would be welcome On 4 September 2013 22:27, Erik Paulson <epaul...@unit1127.com> wrote: > Hello all - > > As part of a side project, I've been interested in HDFS benchmarking, > particularly of the Namenode. To get started, I tried to track down a > number of different benchmarks and collect a few observations about each. > I've put together a list here: > > http://epaulson.github.io/HadoopInternals/benchmarks.html > > The benchmarks I included were: > DFSIO > DFSIO-e > NNBench and NNBenchWithoutMR > S-Live > LoadGenerator > NNThroughputBenchmark > TestEditLog > MStress, from Quantcast > Ohio State Microbenchmarks > SWIM > > (I also wrote a bit about what else I'd like to see in a NN benchmark) > > I'd appreciate any corrections, feedback, and pointers to code that I > missed! > > Thanks! > > -Erik > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
