+1 Art, I had some other ideas on another thread, reading the general concept it sounds like a possible way forward to making AMQ testing easier.
Cheers, J On Sun, Feb 1, 2015 at 12:34 AM, Clebert Suconic <[email protected]> wrote: > We have been through that exercise already on activemq-6. Perhaps it > would be a better use of resources if we worked towards activemq-6? > > With hornetq (now activemq-6) we differentiated unit-tests, > integration tests, soak tests and performance tests. > > When you do a build you won't do the entire testsuite to validate a > simple build.. it's just a matter of minutes and simple unit tests. > > With integration tests (we could find a better name for it) we do more > extensive tests and even then we limit certain things as good > practices. (like using more latches and less Sleeps.. unless you > really have to. It's impossible to run all the profiles on every > time.. Like you can't force someone to run a 10Hours soak test to make > a simple build... we have profiles for that. > > Best practices is the key on keeping the testsuite clean. As I said on > my first paragraph, maybe we should concentrate efforts on activemq-6, > Improving what we have on activemq-6 repo is a better bet IMO. > > We can keep this discussion open if you need help on any specifics on > coding and running the profiles we have. > > > On Sat, Jan 31, 2015 at 8:35 PM, Mark Frazier <[email protected]> wrote: >> +1, this seems like a great idea Art. >> >> I'll be glad to help with this. >> >> >>> On Jan 31, 2015, at 5:25 PM, artnaseef <[email protected]> wrote: >>> >>> *Background* >>> As we all know, ActiveMQ tests take a very long time to execute, and a >>> number of the tests have been unreliable. Personally, it was ActiveMQ that >>> lead me to learn maven's "skip test" features early on because I would not >>> wait hours to get a small change into a build during development, nor did I >>> want to spend time (as a user of ActiveMQ - this is before I was a >>> committer) trying to resolve test failures. This contradicts industry >>> best-practices. >>> A large, world-class product such as ActiveMQ relies, wisely, on automated >>> tests to ensure regression bugs are not introduced with new feature >>> additions and bug fixes. >>> ActiveMQ unit tests really contains a variety of test types, such as: >>> Use Case tests >>> Load tests >>> Bug Reproduction tests >>> Unit Tests that target a single class (i.e. True Unit Tests)* >>> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope >>> that some exist. >>> >>> *Goals* >>> Improve the automated testing with the following result in mind: >>> Developer cycle time: reduce the amount of time to execute tests when making >>> changes to ActiveMQ code so that a developer gets a reasonable expectation >>> that changes are valid without drawing the development cycle out into >>> excessive time periods >>> Continued quality: continue to cover all of the test scenarios currently >>> covered >>> Release validation: continue to provide very thorough validation for >>> releases at the expense of significant, additional test time >>> Reliability: improve reliability of tests for the release process so that >>> it is reasonable to expect a good build every time for quality releases >>> Verify units: improve overall quality of the system through more thorough >>> testing of each unit (class) so that small bugs in units do not turn into >>> hard-to-find bugs in the system (I found one of these just walking through >>> code yesterday - it will get fixed soon) >>> >>> *Ideas* >>> Identify junit tests that are really integration or load tests and separate >>> as such >>> >>> Create an artifact for integration tests, move integration tests there and >>> run them during the verify phase >>> Create an artifact for load tests, move load tests there and also run >>> during the verify phase (interim) >>> Stand up a true load-test environment and move load tests into that >>> environment and out of the main build process >>> Create true unit tests with high code coverage >>> Move tests into separate artifacts to reduce the number of tests, and >>> therefore the amount of time necessary to resume tests from a failure point >>> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc) >>> Review problematic tests and determine ways to resolve (e.g. replace with >>> new tests using different approaches, use the load test environment, etc) >>> >>> *Wrap-Up* >>> The steps above will move ActiveMQ testing toward a solid, reliable test >>> framework without creating gaps in quality. >>> Unless alternative plans are defined, I am going to enter Jira tickets for >>> these and start moving in this direction. Volunteers are welcome and >>> greatly appreciated. >>> >>> >>> >>> -- >>> View this message in context: >>> http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html >>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. > > > > -- > Clebert Suconic > http://community.jboss.org/people/[email protected] > http://clebertsuconic.blogspot.com
