[
https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-4908:
-------------------------------
Attachment: D549.2.patch
mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port
from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Partially addressing Stack's comments (I will address the rest of them in the
next version of this diff). The load test seems to work fine on a dev cluster.
I also tried out RestartMetaTest. The idea is to start a local HBase cluster
with multiple regionservers on different ports, load some data, kill a
regionserver, restart it, and do some checks. I like the of a "process-based
local HBase cluster" because it opens up a possibility for a whole new range of
unit tests (integration tests?) where we are allowed to kill processes for
real, but cannot inspect the internals of those processes. This also allows us
to test cluster management scripts (the hbase script, hbase-daemon.sh, etc.)
from Java itself. Of course, we need to make sure we don't leave those
processes hanging, but that can be addressed in the shutdown hook of the unit
test (or @After handler).
RestartMetaTest still needs some work to make it robust and usable, and we
probably want to make it a unit test.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java
src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
> Key: HBASE-4908
> URL: https://issues.apache.org/jira/browse/HBASE-4908
> Project: HBase
> Issue Type: Sub-task
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D549.1.patch, D549.2.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded
> load generator and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered
> that it has some features that I have not tried yet (some kind of a kill
> test, and some way to run HBase as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest
> command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a
> load test in our five-node dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn
> load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression
> GZIP
> I will be using this code to load-test the delta encoding patch and making
> fixes, but I am submitting the patch for early feedback. I will probably try
> out its other functionality and comment on how it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
