[
https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913540#comment-13913540
]
Jean-Marc Spaggiari commented on HBASE-10592:
---------------------------------------------
Works well for me in 0.96
With your patch:
{code}
hbase@hbasetest1:~$ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation
--presplit=6 --rows=2097152 randomWrite 1
2014-02-26 15:22:28,980 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-02-26 15:22:29,799 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x52c03865 connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:30,360 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x5814caa8 connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:30,391 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x5814caa8 connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:32,137 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x5814caa8 connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:32,190 INFO [main] hbase.PerformanceEvaluation: Table created
with 6 splits
2014-02-26 15:22:32,197 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x5814caa8 connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:22:32,221 INFO [main] hbase.PerformanceEvaluation: Start class
org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest at offset 0 for
2097152 rows
2014-02-26 15:22:32,298 INFO [main] hbase.PerformanceEvaluation: Timed test
starting in thread main
2014-02-26 15:22:52,933 INFO [main] hbase.PerformanceEvaluation:
0/209715/2097152
2014-02-26 15:23:06,678 INFO [main] hbase.PerformanceEvaluation:
0/419430/2097152
2014-02-26 15:23:22,174 INFO [main] hbase.PerformanceEvaluation:
0/629145/2097152
2014-02-26 15:23:39,249 INFO [main] hbase.PerformanceEvaluation:
0/838860/2097152
2014-02-26 15:23:54,392 INFO [main] hbase.PerformanceEvaluation:
0/1048575/2097152
2014-02-26 15:24:07,971 INFO [main] hbase.PerformanceEvaluation:
0/1258290/2097152
2014-02-26 15:24:26,551 INFO [main] hbase.PerformanceEvaluation:
0/1468005/2097152
2014-02-26 15:24:38,495 INFO [main] hbase.PerformanceEvaluation:
0/1677720/2097152
2014-02-26 15:24:51,138 INFO [main] hbase.PerformanceEvaluation:
0/1887435/2097152
2014-02-26 15:25:09,124 INFO [main] hbase.PerformanceEvaluation:
0/2097150/2097152
2014-02-26 15:25:09,360 INFO [main] hbase.PerformanceEvaluation: Finished
class org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest in 157058ms
at offset 0 for 2097152 rows (13,17 MB/s)
2014-02-26 15:25:09,360 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing master protocol:
MasterService
2014-02-26 15:25:09,360 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x14426a37e621805
{code}
Without the patch
{code}
hbase@hbasetest1:~$ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation
--presplit=6 --rows=2097152 randomWrite 1
2014-02-26 15:29:57,987 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-02-26 15:29:58,663 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2014-02-26 15:29:58,832 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x545aae15 connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:29:59,400 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x4f7540bd connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:29:59,436 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x4f7540bd connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:30:00,665 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x4f7540bd connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:30:00,706 INFO [main] hbase.PerformanceEvaluation: Table created
with 6 splits
2014-02-26 15:30:00,719 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x4f7540bd connecting to ZooKeeper
ensemble=hbasetest1.distparser.com:2181
2014-02-26 15:30:00,765 INFO [main] hbase.PerformanceEvaluation: Start class
org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest at offset 0 for
2097152 rows
2014-02-26 15:30:00,827 INFO [main] hbase.PerformanceEvaluation: Timed test
starting in thread main
2014-02-26 15:30:20,306 INFO [main] hbase.PerformanceEvaluation:
0/209715/2097152
2014-02-26 15:30:34,670 INFO [main] hbase.PerformanceEvaluation:
0/419430/2097152
2014-02-26 15:30:48,351 INFO [main] hbase.PerformanceEvaluation:
0/629145/2097152
2014-02-26 15:31:04,139 INFO [main] hbase.PerformanceEvaluation:
0/838860/2097152
2014-02-26 15:31:19,112 INFO [main] hbase.PerformanceEvaluation:
0/1048575/2097152
2014-02-26 15:31:31,711 INFO [main] hbase.PerformanceEvaluation:
0/1258290/2097152
2014-02-26 15:31:50,578 INFO [main] hbase.PerformanceEvaluation:
0/1468005/2097152
2014-02-26 15:32:03,861 INFO [main] hbase.PerformanceEvaluation:
0/1677720/2097152
2014-02-26 15:32:16,284 INFO [main] hbase.PerformanceEvaluation:
0/1887435/2097152
2014-02-26 15:32:32,913 INFO [main] hbase.PerformanceEvaluation:
0/2097150/2097152
2014-02-26 15:32:33,245 INFO [main] hbase.PerformanceEvaluation: Finished
class org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest in 152415ms
at offset 0 for 2097152 rows (13,57 MB/s)
2014-02-26 15:32:33,245 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing master protocol:
MasterService
2014-02-26 15:32:33,246 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x14426a37e621818
{code}
Similar output, similar result. Looked at the code too.
+1
> Refactor PerformanceEvaluation tool
> -----------------------------------
>
> Key: HBASE-10592
> URL: https://issues.apache.org/jira/browse/HBASE-10592
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.96.2, 0.98.1, 0.99.0
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
> Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch,
> HBASE-10592.00.patch, HBASE-10592.01-0.96.patch, HBASE-10592.01-0.98.patch,
> HBASE-10592.01.patch
>
>
> PerfEval is kind of a mess. It's painful to add new features because the test
> options are itemized and passed as parameters to internal methods.
> Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode
> is a chore because of it.
> This patch refactors the tool. Options are now passed around to methods and
> such as a POJO instead of one-by-one. Get rid of accessors that don't help
> anyone. On the mapreduce side, serialization is now handled using json
> (jackson is a dependency anyway) instead of the hand-rolled regex we used
> before. Also do away with custom InputSplit and FileFormat, instead using
> Text and NLineInputFormat. On the local mode side, combine 1 client and N
> clients into the same implementation. That implementation now uses an
> ExecutorService, so we can later decouple number of client workers from
> number of client tasks. Finally, drop a bunch of confusing local state,
> instead use the new TestOptions POJO as a parameter to static methods.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
