I a agree with these three levels of testing, and that at the very least we should keep unit tests separated from the system/integration tests. One huge advantage would be to quickly run unit tests that hopefully are more predictable, and thus avoid intermittent test fails.
Bobby just mentioned but I had already in mind that it would be useful to create different maven profiles for integration and unit tests. I have done something similar in a different project and it works really well. If we agree that it is something we want to implement here, I will create a JIRA ticket for this and get it done. Please let me know. Thanks, Hugo > On Nov 5, 2015, at 11:23 AM, Bobby Evans <[email protected]> wrote: > > I totally agree. Too many of our "unit" tests are integration tests and end > up spinning up an entire local-mode cluster. > I personally would like to see three levels of testing. > 1) true unit tests. They only touch the code under test and very little > else. The should be what you get when you run mvn test2) integration tests. > These should spin up local mode clusters and modify configs/etc to get a > decent set of more white box tests. The should run as a part of trivis-ci, > and probably should run by enabling a special profile.3) Sanity Integration > Tests. ducktape looks like a great fit here. I would love to see us spin up > an few different scenarios for testing, with/without security. Talking to > Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc. > These would be run frequently but not necessarily a part of CI initially. > - Bobby > > > On Wednesday, November 4, 2015 7:21 PM, Harsha <[email protected]> wrote: > > > Hi All, > As community we are growing and adding new and exciting > features to Storm and also we've ever growing connector which > only helps in storm adoption. One thing we've severely lacking > is system tests. There are unit tests which acts as > integration tests to storm-core but there are no system wide > integration tests that can spin up kafka nodes and storm , > hbase and run a topology that can make sure the data is > getting into hbase. > We at Hortonworks use our test topologies run some of these > tests but this integration code to spin up vms or nodes is hard > to share. In apache kafka we are using ducktape to write system > tests so far its working out good. If there are any other > frameworks you've in mind we can definitely take a look. But as > a community we need start looking at > https://github.com/confluentinc/ducktape or similar frameworks > to start writing systems tests. Ducktape makes it easy to run > in a vm or any other infrastructure. Appreciate any feedback > on this. > > Thanks, > Harsha > >
