The choice to fix, refactor, or remove test cases should be reasonably straight forward on a case by case basis - the real challenge in my mind is the volume to be reviewed.
Perhaps the AMQ community could parcel the test cases into small sets, each tracked by a Jira task. These sets could then be posted into a community page tracking, showing which ones have been reviewed, which are under review, and which ones have not been touched. The reason I'd like to see a table tracking these test case set reviews is that it would provide new contributors an easy way to see where they could jump in and help out -- much like the old Servicemix community wish page (That's how I was able to jump in and start helping effectively back in the day). Many hands making the work light. The over head of having the tracking table, Jiras, and co-ordination should be offset by having the work spread well over many people, and providing new contributors a great way to start interacting with the community. Cheers, Jamie On Sat, Jan 31, 2015 at 9:03 PM, artnaseef <[email protected]> wrote: > *Overview* > Defining a consistent approach to tests for releases will help us both > near-term and long-term come to agreement on (a) how to maintain quality > releases, and (b) how to improve the tests in a way that serves the needs of > releases. > > As a general practice, tests that are unreliable raise a major question - > just how valuable are the tests? With enough unreliable tests, can we ever > expect a single build to complete successfully? > > How can we ensure the quality of ActiveMQ is maintained, and tests are > safeguarding the solution from the introduction of bugs, in light of these > tests? > > *Ideally* > Putting some ideals here so we have the "end in mind" (Stephen Covey) -- > i.e. so they can help us move in the right direction overall. These are > definitely not feasible within any reasonable timeframe. > > Putting on my "purist" hat -- ideally, we would analyze every test to > determine the possibility of FALSE-NEGATIVES *and* FALSE-POSITIVES generated > by the test. From there, it would be possible to look for methods of > distinguishing false-negatives and false-positives (for example, by > reviewing logs) and improving the tests so they hopefully never end in false > results. > > Another ideal approach - return to the drawing board and define all of the > test scenarios needed to ensure ActiveMQ operates properly, then determine > the most reliable way to cover those test scenarios. Discard redundant > tests and replace unreliable ones with reliable ones. > > *Approach for Releases* > Back to the focus of this thread - let's define an acceptable approach to > the release. Here is an idea to get the discussion started: > > - Run the build with the Maven "-fn" flag (fail-none), then review all > failed tests and determine a course of action for each: > - Re-run the test if there is reason (preferably a clear, documented > reason) to believe the failure was a false-negative (e.g. a test that > times-out too aggressively) > - Declare the failure a bug (or at least, a suspected bug), create a Jira > entry, and resolve > - Replace the test with a more reliable alternative that addresses the > same underlying concern as the original test > > *Call for Feedback* > To move this discussion forward, please provide as much negative feedback as > necessary and, at the same time, please provide reasoning or ideas that can > help move things forward. Criticism (unactionable feedback) is discouraging > and unwelcome. On a similar note - the practice of throwing out "-1" votes, > even for small, easily-addressed issues, without any offer to assist is > getting old. I dream of seeing "-1, file <x> needs an update; I'll take > care of that myself right now." > > *Wrap-Up* > Let's get this solved, continue with frequent releases, and then move > forward in improving ActiveMQ and enjoying the results! > > Expect another thread soon with ideas on improving the tests in general. > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/DISCUSS-Releases-and-Testing-tp4690763.html > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
