+1, this seems like a great idea Art. I'll be glad to help with this.
> On Jan 31, 2015, at 5:25 PM, artnaseef <[email protected]> wrote: > > *Background* > As we all know, ActiveMQ tests take a very long time to execute, and a > number of the tests have been unreliable. Personally, it was ActiveMQ that > lead me to learn maven's "skip test" features early on because I would not > wait hours to get a small change into a build during development, nor did I > want to spend time (as a user of ActiveMQ - this is before I was a > committer) trying to resolve test failures. This contradicts industry > best-practices. > A large, world-class product such as ActiveMQ relies, wisely, on automated > tests to ensure regression bugs are not introduced with new feature > additions and bug fixes. > ActiveMQ unit tests really contains a variety of test types, such as: > Use Case tests > Load tests > Bug Reproduction tests > Unit Tests that target a single class (i.e. True Unit Tests)* > * I personally haven't seen any of these in ActiveMQ, but I suspect/hope > that some exist. > > *Goals* > Improve the automated testing with the following result in mind: > Developer cycle time: reduce the amount of time to execute tests when making > changes to ActiveMQ code so that a developer gets a reasonable expectation > that changes are valid without drawing the development cycle out into > excessive time periods > Continued quality: continue to cover all of the test scenarios currently > covered > Release validation: continue to provide very thorough validation for > releases at the expense of significant, additional test time > Reliability: improve reliability of tests for the release process so that > it is reasonable to expect a good build every time for quality releases > Verify units: improve overall quality of the system through more thorough > testing of each unit (class) so that small bugs in units do not turn into > hard-to-find bugs in the system (I found one of these just walking through > code yesterday - it will get fixed soon) > > *Ideas* > Identify junit tests that are really integration or load tests and separate > as such > > Create an artifact for integration tests, move integration tests there and > run them during the verify phase > Create an artifact for load tests, move load tests there and also run > during the verify phase (interim) > Stand up a true load-test environment and move load tests into that > environment and out of the main build process > Create true unit tests with high code coverage > Move tests into separate artifacts to reduce the number of tests, and > therefore the amount of time necessary to resume tests from a failure point > (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc) > Review problematic tests and determine ways to resolve (e.g. replace with > new tests using different approaches, use the load test environment, etc) > > *Wrap-Up* > The steps above will move ActiveMQ testing toward a solid, reliable test > framework without creating gaps in quality. > Unless alternative plans are defined, I am going to enter Jira tickets for > these and start moving in this direction. Volunteers are welcome and > greatly appreciated. > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
