*Background* As we all know, ActiveMQ tests take a very long time to execute, and a number of the tests have been unreliable. Personally, it was ActiveMQ that lead me to learn maven's "skip test" features early on because I would not wait hours to get a small change into a build during development, nor did I want to spend time (as a user of ActiveMQ - this is before I was a committer) trying to resolve test failures. This contradicts industry best-practices. A large, world-class product such as ActiveMQ relies, wisely, on automated tests to ensure regression bugs are not introduced with new feature additions and bug fixes. ActiveMQ unit tests really contains a variety of test types, such as: Use Case tests Load tests Bug Reproduction tests Unit Tests that target a single class (i.e. True Unit Tests)* * I personally haven't seen any of these in ActiveMQ, but I suspect/hope that some exist.
*Goals* Improve the automated testing with the following result in mind: Developer cycle time: reduce the amount of time to execute tests when making changes to ActiveMQ code so that a developer gets a reasonable expectation that changes are valid without drawing the development cycle out into excessive time periods Continued quality: continue to cover all of the test scenarios currently covered Release validation: continue to provide very thorough validation for releases at the expense of significant, additional test time Reliability: improve reliability of tests for the release process so that it is reasonable to expect a good build every time for quality releases Verify units: improve overall quality of the system through more thorough testing of each unit (class) so that small bugs in units do not turn into hard-to-find bugs in the system (I found one of these just walking through code yesterday - it will get fixed soon) *Ideas* Identify junit tests that are really integration or load tests and separate as such Create an artifact for integration tests, move integration tests there and run them during the verify phase Create an artifact for load tests, move load tests there and also run during the verify phase (interim) Stand up a true load-test environment and move load tests into that environment and out of the main build process Create true unit tests with high code coverage Move tests into separate artifacts to reduce the number of tests, and therefore the amount of time necessary to resume tests from a failure point (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc) Review problematic tests and determine ways to resolve (e.g. replace with new tests using different approaches, use the load test environment, etc) *Wrap-Up* The steps above will move ActiveMQ testing toward a solid, reliable test framework without creating gaps in quality. Unless alternative plans are defined, I am going to enter Jira tickets for these and start moving in this direction. Volunteers are welcome and greatly appreciated. -- View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
