-----Original Message----- >From: Frank Schönheit - Sun Microsystems Germany <[EMAIL PROTECTED]> >Sent: Jul 18, 2006 11:11 PM >To: [email protected] >Subject: Re: [qa-dev] Proposal: Classification of regression issues > >Hi James, > >>>2.Functionality which is different from the same functionality in a >>>previous version, and should be fixed. However, the issue is considered >>>too minor to be a stopper for the next (micro) release. >> >> This is an issue. However, it should not be lost that this affected at >> least one >> user and maybe more. Setting the target to the next major release. This >> means >> for a problem found in 2.0.3, the target should at least be 2.1. > >As a general rule, this won't work. There will always be regressions, >there will always be other isses, there will always be new features to >implement, and there will always be to few people to fix them all. > >The proposal is exactly to introduce a policy which does *not* >automatically require to fix regessions in the next release. For every >bug, there's at least one user (probably more) which is affected by it. >If we would use your argueing, we would have to target *all* bugs to >2.1, which is simply senseless. > >>>3.Functionality which is different from the same functionality in a >>>previous version, and is serious enough to be a stopper for the next >>>(micro) release. >> >> Again this is a regression issue and should have a target of the >> next micro release. This means for a problem found in 2.0.3, the >> target IS 2.0.4. This stops the next release until fixed. > >Isn't this exactly what I proposed later in my mail for this category? :) > Yes. I wrote this before I read through the entire message. This should be considered a 'second' for this action.
>>>One could argue that 1. could be clearly "classified" by removing the >>>"regression" keyword (let aside for the moment different perceptions >>>about whether the intentional change was a change to the better), and 2. >>>and 3. could be distinguished by the target milestone field. >> >> NO! The regression keyword should remain. Why? So that someone who >> works the code knows that their changes caused a problem and they can >> quickly work back and find the point where the problem occured and effect >> a faster fix. > >Read again, please. I said a possible "regessions" could be removed for >category-1-issues. Those are the issues which describe intentional >changes. Thus, nobody will go back and simply restore the old behaviour >here. Except when during a discussion, it's decided that the change in >fact was to the worse, and needs to be reverted. But in this case, the >regression keyword is not needed to trigger the reversion. This was also stated in what I said. If the problem is NOT a regression, then we recategorize the problem. If it truely is a regression, then the category should remain regression and the triage process is applied as to the impact of the regression and is it a 'show stopper' problem. > >>>In case that the changed functionality/behaviour is controversial, this >>>should be discussed in the project and if needed, escalated to the >>>project lead (next step of escalation would be project leads list). >> >> This should be discussed BEFORE implementation, not after. > >Yes. And there should be no crime in the world. Last time I checked, >both things were still happening all the time :) > I stated that as the ideal, not the reality. >Of course discussing a change before actually doing it is better. This >alrady happens a lot (read [email protected] to be >up-to-date with what is planned to be changed, and join the respective >discussions.). Ok. I can do this. >Nonetheless, people will sometimes not notice the beforehand >discussions, or no compromise might be found during a discussion (in >which case it's the project lead's authority to decide), or whatever. >There always will be cases where a change made in best belief is still >controversion, and thus might result in filed issues. > This happens. Discussion cannot always bring the result everyone is happy with. >> Never use OO Later as the issue will get 'lost'. Set the target to the next >> major dot release (say 2.1) or next major release (3.0). > >No, that's against current targeting policy: "2.x" is for issues which >are planned to be fixed in one of the next releases, "2.0.x" and "3.0" >is for issues which are *definitely* planned to be fixed in this >particular release. > As a real world software tester, I can state that if issues are marked "fix later" they don't get fixed. If you place a target version on the problem, it gets looked at and considered. Not every problem targeted for a certain version will get fixed in that version. I realize this and we can always retarget a problem to a later version (and this happens all the time with OpenOffice.org.) >Also, here again holds that this wouldn't scale. You could do the same >argueing for every issue: Don't target it to "OOo Later" or it will get >lost. > >As a matter of fact, we have more issues than active developers to fix >them, and I believe this won't change. Saying that everything which is >an issue must be fixed in the next release is meaningless, as it won't >work. This holds for regressions as well as non-regressions. This is true for all software. There are more problems found than can be fixed for a given release. Can all of them be fixed? No. However, you set a goal and try to make it. If I pick up an issue, I try to at least discover where it the system is broken and identify it. This makes it easier for the developers to fix it. > >>> release_blocker: >>> This keyword denotes issues which describe a regression, i.e. >>> functionality which worked in an older version, but is broken in a >>> newer version. In opposite to the "regression" keyword, >>> "release_blocker" marks issues which are considered a blocker for >>> the next maintanance release. >> >> This is adding a level of complexity that is unnecessary. Setting the >> target is all that is needed along with a comment that the issue is >> blocking or non-blocking. > >Sorry, but this isn't sufficient. As outline in my mail, targets are >prone to human errors. There are multiple scenarios where people change >targets of issues - with legitimate reasons as far as non-blocking >issues are concerned. However, if the "blocking" fact is indicated in an >comment only, then this will certainly be overlooked in a certain >percentage of cases, and the issue will be errornously retargeted. After >that, there's no certainity to get the issue back to the target it deserves. > >If we would have a "blocker" keyword, both problems would vanish: Wrong >retargeting would be less probable, and finding those wrongly retargeted >issues would be a matter of a simple query. > Yes. However, where is this going to go? >> If an issue is determined to be blocking >> then any and all release candidates are blocked until one of three actions >> are taken: >> >> 1. An effective fix is found and implemented that causes no further problems. >> 2. A determination by the user community that the regression is acceptable >> and a fix can wait until the next minor release. > >Which effectively moves the issue into category 2 (regression, but >non-blocker). Yes, this is true. This is why we have triage teams. A user can state, "I cannot use OpenOffice.org because of this problem" but in the big picture this is the only user affected. Would you expend a large amount of effort to fix the problem? No. However, if 100 users were affected or a large installation of OpenOffice.org were, you would fix the problem quickly. > >> 3. No acceptable fix can be found that does not cause major problems to >> occur elsewhere in the program can be found in a reasonable time period (one >> month after the problem is discovered.) >> ... > >> Issues that block a release, regardless of their category (be it a regression >> or other) should be marked as a release-blocker. This should be separate >> from the actual cause/category. Thus all issues that block a release can be >> handled as separate entities rather than creating another issue just to >> handle >> release blocker issues. Under no circumstances should the category be >> changed >> just to mark an issue as a release blocker. > >This would be fullfilled with removing the "regression" aspect from the >"release_blocker" keyword, wouldn't it? >That is, the current proposal says that a "release_blocker" is always a >"regression", too - which indeed might be way too strict. If we'd >introduce "release_blocker" with the obvious semantics, then it could be >applied to regression and non-regression issues, which would make more >sense. Also, adding "release_blocker" then would keep the "regression" >keyword. > I agree with your assessment of the use of the words 'release blocker'. A new problem could block a release, not just regressions. >> Also, users should NOT be able to set this field, but rather this >> action should be allowed only by a QA rep or higher. > >Yes, this would be desirable. Unfortunately, this is beyond our control. >IssueZilla does not have this fine-grained permissions, so effectively, >nearly everybody (at least everybody with the "canconfirm" privilege) >can set arbitrary keywords. I was not aware of the limitations of IZ. Thus, this field should be resettable by the triage team. >We should establish a policy that "release_blocker" is to be set by >certain (class of) users only, but we cannot technically enforce it. >However, I don't consider this a real problem: We would use queries for >all "release_blocker" issues on a regular basis, and here (at the >latest), it would be noticed if somebody wrongly set that keyword. Ok. I know that policies are difficult to enforce when the underlying software cannot support them. However, this does not stop the triage team from correcting improperly prioritized issues. James M. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
