-----Original Message----- >From: Frank Schönheit <[EMAIL PROTECTED]> >Sent: Jul 18, 2006 6:21 AM >To: [email protected] >Subject: [qa-dev] Proposal: Classification of regression issues > >In IssueZilla, we currently have the keyword regression, which, >according to the definition [1], "can be set if functionality definitely >worked in an older version but is broken in a newer version". For >various reasons detailed below, this keyword alone sometimes is not >sufficient for issue classification. > >Specifically, there are, in real, three categories of regression issues >we're interested in: > >1.Functionality which is different from the same functionality in a >previous version, but the difference is intentional, since the >respective feature has been overworked, and specified differently than >before. > This is not a regression and the actual cause should be changed to a different cause of the problem.
>2.Functionality which is different from the same functionality in a >previous version, and should be fixed. However, the issue is considered >too minor to be a stopper for the next (micro) release. > This is an issue. However, it should not be lost that this affected at least one user and maybe more. Setting the target to the next major release. This means for a problem found in 2.0.3, the target should at least be 2.1. >3.Functionality which is different from the same functionality in a >previous version, and is serious enough to be a stopper for the next >(micro) release. Again this is a regression issue and should have a target of the next micro release. This means for a problem found in 2.0.3, the target IS 2.0.4. This stops the next release until fixed. >One could argue that 1. could be clearly "classified" by removing the >"regression" keyword (let aside for the moment different perceptions >about whether the intentional change was a change to the better), and 2. >and 3. could be distinguished by the target milestone field. > NO! The regression keyword should remain. Why? So that someone who works the code knows that their changes caused a problem and they can quickly work back and find the point where the problem occured and effect a faster fix. >Thus, our proposal: > >For the three categories mentioned above, the issues are handled as follows: > >1. After adding a justification (i.e. saying that the "regression" is >not considered a regression, but an implementation of a new/changed >concept), the issue is handled as usual. No particular keyword is to be >set. Any already existent regression keyword is to be revoked. > Definately. And the issue should be closed as works as designed, or change in functionality. >In case that the changed functionality/behaviour is controversial, this >should be discussed in the project and if needed, escalated to the >project lead (next step of escalation would be project leads list). > This should be discussed BEFORE implementation, not after. This will allow for faster responses and if possible, this could be posted to the user forum. Something like: Change in functionality of Calc's add function. Then a short discussion of the change in functionality. A discussion can then be pursued as to if users want this change. If it is decided that the change must happen, then it happens. >2. The keyword "regression" is added to the issue (if not already >present), together with a comment stating that the issue is not >considered a blocker for the next release. The target is set to the next >train release (e.g. "OOo 2.x") or "OOo later", depending on the >judgement of the issue's severity. After that, the issue is handled as >usual. In particular, depending on user feedback (e.g. votes), it might >be decided to fix the issue in the next release. > Never use OO Later as the issue will get 'lost'. Set the target to the next major dot release (say 2.1) or next major release (3.0). >3. The (newly-introduced) keyword "release_blocker" is added to the >issue, the target is set to the next minor release (2.0.<something>). > >The keyword definitions would be as follows: > regression: > This keyword can be set if functionality definately worked in an > older version, but doesn't in a newer version. It is very helpful > to give detailed information from which version to which version the > functionality broke. If this issue is considered a blocker for the > next release, the OpenOffice.org team might decide to indicate this > by setting the "release_blocker" keyword. > > release_blocker: > This keyword denotes issues which describe a regression, i.e. > functionality which worked in an older version, but is broken in a > newer version. In opposite to the "regression" keyword, > "release_blocker" marks issues which are considered a blocker for > the next maintanance release. This is adding a level of complexity that is unnecessary. Setting the target is all that is needed along with a comment that the issue is blocking or non-blocking. If an issue is determined to be blocking then any and all release candidates are blocked until one of three actions are taken: 1. An effective fix is found and implemented that causes no further problems. 2. A determination by the user community that the regression is acceptable and a fix can wait until the next minor release. 3. No acceptable fix can be found that does not cause major problems to occur elsewhere in the program can be found in a reasonable time period (one month after the problem is discovered.) >Advantages of this approach: > >- The classification of regression issues takes place in one IssueZilla > field only: whether an issue has the "regression", the > "release_blocker", or none of those keywords determines the > category. > (for the nitpickers amongst us: Sure, "no keyword" could also mean "no > regression at all", i.e. indicate a "normal" issue, but this shouldn't > matter for our purpose.) > Thus, the data is less prone to human errors. > >- The number of issues flagged with "release_blocker" can be used as > one measure for the readiness of OpenOffice.org for a micro release. > >- When re-targeting issues becomes necessary, its clear that > "release_blocker" issues can *not* be re-targeted. Currently, this > decision would require a close examination of the argumentations in > the issue, because it is not clear whether, for instance, a regression > issue targeted to "2.0.4" has this target more intentionally or more > accidentally (and both cases are common :) . With the new keyword, > it's easily clear that changing the target of a "release_blocker" > issue is a no-go. > (And we could easily introduce a query which finds issues which have > been re-targeted wrongly.) Issues that block a release, regardless of their category (be it a regression or other) should be marked as a release-blocker. This should be separate from the actual cause/category. Thus all issues that block a release can be handled as separate entities rather than creating another issue just to handle release blocker issues. Under no circumstances should the category be changed just to mark an issue as a release blocker. Also, users should NOT be able to set this field, but rather this action should be allowed only by a QA rep or higher. James M. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
