Scott Kitterman wrote:
> I think the only path to better tested releases is higher quality test
> releases.
<and later>
> Personally, I think it means we need to be doing a much better job of testing
> and bug fixing as developers.
<...>
> The moral of the story is that my project had increased in complexity beyond
> what my QA approach could sustain and I had to re-engineer QA, even at the
> cost of some features. Once I had a QA approach that matched the complexity
> of the project, then release quality went up again.
>
> I suspect that we may be in a similar position with Ubuntu. We need to
> radically rethink testing and how test results get back into fixes. I
> believe that Ubuntu has gotten more complex and we need to match our test/fix
> methodology to match. I would like to hear ideas on the subject.
There is currently a battery of automated tests run, testing the
installability of packages, compilation of packages, policy compliance
of packages, file conflicts of packages, dependency maps of packages,
etc. As developers we would do well to track these more carefully,
and try to ensure that fewer packages appear in any of these lists.
Most of these lists are linked from http://qa.ubuntuwire.com.
Additionally, there is coordinated testing done of each testing
release, with a clear set of test procedures. Adding more test cases
(especially flavour-specific test cases), and getting more people to
run through these procedures (currently only about 5 or 6 people run
most of them) would be a great help towards ensuring that the releases
meet quality goals. Further information and image results are
available from http://iso.qa.ubuntu.com/qatracker/build/all/all
Setting up an automatic install / upgrade / remove / purge tester
would be good, perhaps using piuparts or similar infrastructure,
although this requires considerable resources in terms of local
storage and processing power. There are likely also several other
automatable targets in terms of package state or archive consistency
that would benefit from additional tools to track them.
> I'm not certain, however, that we need more bugs. We need better bugs and we
> need more fixes. Suggestions?
In recent times there seems to be some disconnect between the
presence of bugs and the presence of developers working on these bugs.
While we all tend to be busy much of the time, perhaps there are ways
that we can improve the view of bugs in need of attention, or
otherwise help understand which bugs are likely to be perceived as
painful to users at release time. I think the recent decision to
align the use of bug importance with the SRU criteria will help us all
to better communicate what bugs need to be fixed now, and better
understand whether a given release is likely to have issues.
Further, I'd like to see more bugs. I hear lots of stories of
people who don't want to bother filing a bug, and would rather
complain in their blogs, on the forums, in IRC, or even in the mailing
lists: helping encourage lots of bug filing helps reduce these cases,
and presents a better picture of what is working, and what is broken.
While this necessarily means more bugs that are really support
requests, or duplicate, or require upstream decisions, or need
completely new code, it also means that we can develop metrics to
understand what is sought, and try to meet that.
We should also be better about chasing bugs that are fixed. There
are a number of bugs marked fixed upstream, or fixed in Debian, or
with patches. There are plenty of others for which there are good
fixes in Fedora, SuSE, Gentoo, etc. At least in those cases where we
can track that a given bug exists in Ubuntu and a fix is available, we
ought assiduously chase these: the more quickly we can go from a fix
being available somewhere to a fix being available in Ubuntu, the more
likely we are to be informed of a fix being available somewhere, and
the better chance we have of a relatively small number of bugs.
--
Emmet HIKORY
--
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
