On Wed, Nov 2, 2011 at 8:11 PM, Stack <st...@duboce.net> wrote:
> On Wed, Nov 2, 2011 at 5:31 PM, Roman Shaposhnik <r...@apache.org> wrote:
>> As I mentioned before, I'd love to expand on that. But so far
>> I haven't heard anybody willing to contribute to:
>>
>> https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-artifacts/hbase/
>>
>
> I'm volunteering to help out Roman. A suite of tests that run against
> a cluster to verify stuff is basically as good/bad as it has always
> been is I think top priority for the project and plugs in nicely with
> the work nkeywal is doing refactoring our unit tests.
I'd love to see that happen! Here's what we have to offer from the
Bigtop side of things:
1. A place to commit these tests:
https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-artifacts/hbase/
2. An [evolving] integration test library/framework called iTest
(based on JUnit/Maven):
https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-execution/
https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-test-framework/
3. A way to rapidly build a desired Hadoop stack and deploy it to
an on-demand
cluster:
https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-deploy/puppet/
4. A Jenkins server with enough of slaves in place to tie #1-#3
together and collate the
test results:
http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/
Writing tests is not difficult at all (just take a look at existing
ones and you should
see how its done). We also need to improve the DOCs on everything that's
related to iTest, but that hasn't happened yet -- so if you're interested feel
free to email me personally of head over to bigtop-...@incubator.apache.org.
What we're missing in Bigtop is deep domain knowledge of the components
and their integration testing needs. The existing tests are basically for the
features that we cared about for one reason or another (e.g. Compression
codecs, Pig/HBase interop, etc.) and they are not an HBase view of integration
testing needs. But hey -- its better than nothing! ;-)
Also, a low-hanging-fruit sort of approach that we've taken with Hadoop
and Pig is to try and re-use existing unit tests in the context of Integration
testing. A good example here is TestCLI which runs a whole bunch of
FS operations and compares results to the golden file. This is the type of
test that we can run against a real cluster (baring some minor modifications
that went upstream eventually). Thus if there's a subset of HBase unit
tests that we can reuse in such a context -- that'll be very nice.
Thanks,
Roman.
