Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The following page has been changed by stack: http://wiki.apache.org/hadoop/Hbase/NewFileFormat/Performance The comment on the change is: Adding in rfile w/ 64k buffers ------------------------------------------------------------------------------ Numbers comparing MapFile and RFile (TFile+mods has dropped from the running for the moment anyways). The code used running tests is available over in [http://github.com/ryanobjc/hbase-rfile/tree/rfile github]. I did following on local filesystem and on 4node hdfs: - {{{$ ./bin/hadoop org.apache.hadoop.hbase.MapFilePerformanceEvaluation ; ./bin/hadoop org.apache.hadoop.hbase.RFilePerformanceEvaluation}}} + {{{$ ./bin/hadoop org.apache.hadoop.hbase.MapFilePerformanceEvaluation + $ ./bin/hadoop org.apache.hadoop.hbase.RFilePerformanceEvaluation}}} + + For more context, see [wiki:Hbase/NewFileFormat New File Format]. == Local Filesystem == Macosx, 10 byte cells and keys. @@ -151, +154 @@ 09/02/06 22:10:48 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows. 09/02/06 22:10:50 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows took 2625ms.}}} + == HDFS 1k cells == + + === MapFile === + {{{09/02/06 22:28:58 INFO hbase.MapFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows. + 09/02/06 22:28:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable + 09/02/06 22:28:58 INFO compress.CodecPool: Got brand-new compressor + 09/02/06 22:28:58 INFO compress.CodecPool: Got brand-new compressor + 09/02/06 22:29:13 INFO hbase.MapFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows took 14915ms. + 09/02/06 22:29:13 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows. + 09/02/06 22:29:46 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows took 32558ms. + 09/02/06 22:29:46 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows. + 09/02/06 22:33:55 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows took 249211ms. + 09/02/06 22:33:55 INFO hbase.MapFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows. + 09/02/06 22:37:49 INFO hbase.MapFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows took 234521ms. + 09/02/06 22:37:49 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows. + 09/02/06 22:37:52 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows took 2827ms.}}} + + === RFile 64k buffers === + {{{09/02/06 22:37:53 INFO hbase.RFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows. + 09/02/06 22:37:56 INFO hbase.RFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows took 3083ms. + 09/02/06 22:37:56 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows. + 09/02/06 22:38:24 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows took 27405ms. + 09/02/06 22:38:24 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows. + 09/02/06 22:41:24 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows took 180332ms. + 09/02/06 22:41:24 INFO hbase.RFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows. + 09/02/06 22:44:20 INFO hbase.RFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows took 175614ms. + 09/02/06 22:44:20 INFO hbase.RFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows. + 09/02/06 22:44:23 INFO hbase.RFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows took 2840ms.}}} +
