Thorsten,
I decided to wait until 3.0 was released and settled in before sending
anything more. FWIW, here is a contribution to the issue process debate.
Mike
OpenOffice Quality
OpenOffice seems to have proportionally more open issues than other open
source projects. It sometimes feels like an exponential growth, though
it probably isn't. Nevertheless, there is a sense in which the project
might be described as drowning in its own issues. This note identifies
some of the problems and suggests three practical ways which might
improve the processes.
Perceived problems with issue handling
*
The development team have a massive ongoing task to assess new
issues and allocate them appropriately to future releases. For
many issues, this assessment is repeated multiple times.
*
Issues tend to be treated individually, which makes it difficult
to see how they relate to each other and to planned developments.
Even when development takes place on the code in question, fixing
related issues is often a matter of chance rather than planning,
so even if the work to fix some existing issues happened to be a
very minor addition to planned development work, which in some
cases it would be, in practice the chance of taking advantage of
this is relatively small.
*
Arguably, the vast majority of 'ordinary' users would prefer a
product with fewer defects, rather than new functionality, much of
which most of them will never use. On the other hand, developers
naturally prefer to add functionality, because it is generally
much more satisfying and often a whole lot easier than fixing code
developed by others which is not adequately documented and hard to
understand. In an open source project, developers have the final
say, which for Open Office tends to set the balance on too much
new functionality and too little quality improvement.
*
Despite Writer arguably having a technical edge over Word,
corporate IT departments of large multi-nationals each supporting
thousands of desktops (the majority English language, PC, MS
Office) will not adopt OpenOffice with its current quality^1
<#sdfootnote1sym>. To become the package of choice in this
influential area, the number of bugs, particularly P2 bugs, would
have to be drastically reduced. The current issue handling process
is unlikely to deliver this even in the very long term.
*
In order to assess reported or discovered defects, qa team members
generally gain a good understanding of an issue. For P1 and some
P2 issues, ie for those issues which have to be fixed in the next
couple of releases, the subsequent processes generally work well.
For the rest, qa expertise gained during investigation is at best
only indirectly used to set implementation target and it is
something of a lottery which of some P2 and virtually all P3/P4/P5
issues are fixed and which are not.
/Summary of suggestions/
*
*Emulating the Stopper meta-issue process* -- a way of formally
handling defects related to development work
*
*The Magic Sorting Hat* -- a proposal for an adjustment in initial
target setting to more closely reflect community needs
*
*Priority 2 Defects* -- a more concentrated effort on serious
defects with the objective of a major reduction in numbers at (eg)
each dot2 release
Emulating the Stopper meta-issue process
The design process for development work allows any interested person to
contribute. However, these discussions typically do not focus on
identified defects in the same code areas.
One process that works exceptionally well is the stopper meta-issue,
used to manage the release of each OpenOffice version. Similar
meta-issues could bring together both new work and related defects in a
particular area. It would be open to anyone to suggest what defects to
include and up to the developer to decide which defects could sensibly
be accepted and resolved. The advantage to the developer is that a study
of related defects would generally inform and enhance the design. The
advantage for the community is that quality would improve as more
defects were resolved. An advantage for both parties would be the
facilitation of a debate on the proper balance between fixing specific
defects and developing new functionality.
The Magic Sorting Hat
It would be good to have a magic way to allocate every issue to the
right target, OOo later for Slytherin defects and the next release for
Griffindor. In the absence of that, we must depend on mere Muggles.^2
<#sdfootnote2sym> But at least we might endeavour to minimise effort and
maximise the application of available expertise. To do so, we could take
a different perspective on what the target on an issue means. When a
developer sets a target of one of the next couple of releases, the
target implies allocation of resources and at least the intention to
meet the target. In practice though, these issues are often reallocated.
Sometimes this reallocation can be extensive, for example the large
number reallocated from 3.0 to 3.1 and looking at the long list for 3.1,
another significant reallocation would not be a surprise. Thus, a target
is a target and not a commitment, even when set by a developer and the
sky doesn't fall in when a target is changed.
Now, let us suppose there was a perfect way of allocating newly verified
defects to the 'correct' release, taking into account available
resources, the needs of the community and a proper balance between
developing functionality and correcting defects. When the development
team began to plan a release, they would find the allocation already
done. An advantage? I think so.
We do not of course have such a way, but we do have a resource which
could do something that would come closer than at present.
When an issue is being verified, the qa person may spend a considerable
time researching it, often up to several hours. At least, that is what
happens to me. Inevitably, a certain 'feel' develops about how important
the issue is and how difficult it might be to fix. This is a depth of
understanding arguably beyond what is likely in a quick trawl through
issues to make allocations. At present this understanding is arguably
not put to best use. The contention is that to invite the qa team to
make an initial allocation is very likely to be a better use of
resources, more accurate (ie reflecting real user need) and the nearest
we will ever have to a Magic Sorting Hat.
The advantage for the development team is that there would no longer be
a need to trawl through the untargetted issues to find the important
ones, but with no reduction in control, since issues with targets that
cannot be met can have their targets changed. The advantages for the
community is that defects would no longer be in a limbo with no apparent
response to the raising of the issue and that there would be a better
mechanism for highlighting the more important. A target of OOO later is
better than no target at all. The advantage for the qa team is that they
would feel that their expertise was being better used.
This process would avoid the double initial assessment of many issues
and to the extent that the qa assessments were accurate, would
materially help developers.
Guidelines would be needed for the qa team when setting targets, perhaps
most important that the process would be only for setting an initial
target and never for changing a target already set.
Priority 2 Defects
By definition, P2 defects are serious, but in practice whether or not a
P2 defect is fixed depends on its target rather than its priority. In
particular, P2 defects with target OOO later seems a questionable
combination. Overall OpenOffice quality could be significantly improved
by fixing a majority of P2 defects before moving on to the next major
release.
I would like to canvas support for regular efforts to reduce the number
of outstanding P2 defects during the cycle of each major release, for
example it might be worthwhile targeting as many as possible of the P2
defects to say the dot2 stage (3.2, 4.2...) with the objective of
eliminating a large majority of P2 defects by, say, the 4.2 release.
You have only to look at the statistics to conclude that something isn't
working as well as it might.
P2 defect list
P2 Defects: New, Started or Reopened as at 14 October 2008
Target
Defects
blank/not determined
65
OOO later
56
3.x
109
3.2
3
3.1
63
3.0.1
21
2.4.2
1
DevTools
9
There were also 75 Unconfirmed P2 defects with a blank target.
Implementation
Generally there are reactions against procedural change, especially when
responsibilities shift. It is also by non means certain that a new
procedure will have the impact predicted or that it will not introduce
unexpected and unwelcome side effects. On the other hand the current
procedure is arguably not fully fit for purpose. One option might be to
experiment with the changes proposed, for example a limited number of
nominated qa team members might operate the 'magic sorting hat' for a
period, after which the impact could be assessed and a decision made
whether to continue or readjust.
1 <#sdfootnote1anc>This is a considered professional judgement based on
27 years work in one of those departments.
2 <#sdfootnote2anc>With apologies to JK Rowling
Mike Hall wrote:
Thorsten,
Thanks. I understand all of that, but nevertheless it's arguable that
the process does not work as well as it could.
I accept your challenge "If you have a proposal..." I have a glimmer
of an idea and some (actually quite a lot) of relevant experience.
Give me a couple of weeks while we are away to mull it over and write
something sensible down.
Mike
Thorsten Ziehm wrote:
Hi Mike,
[...]
Currently there isn't a better process or any other solution, I know. If
you have a proposal how to handle it better with the existing resources
write it down and propose it here. But I do not see any chance to set
Sun QA engineers as default as in the past. If they work on all incoming
issues in time, they will not have the time to work on new incoming
features/bug fixes etc. So the help from community is need to work on
one of the topics regularly.
[...]
