As some of you have guessed, I've picked up a couple of other
problems, which caused me to slow down on this particular issue. In
the revision below, I've incorporated some feedback from Charlie
LaBrec, who's been involved with defect management at Sun, both as an
architect and as a Solaris engineer dealing with actual OS bugs.
(I'll go reread the Bugzilla thread from a couple of weeks ago, but if
anyone wants to send me edits, I'll mention that
hg clone ssh://[EMAIL PROTECTED]/hg/website/spec
will get you your own copy of the document so that you could send
actual patches.)
Otherwise, we can get onto candidate evaluation.
- Stephen
----
| vim:set expandtab:
OpenSolaris
DEFECT TRACKING SYSTEM REQUIREMENTS [DRAFT]
Stephen Hahn
1. Summary
This document identifies and explains the requirements for a defect
tracking system (DTS) solution to be used for open development.
Software development at SMI, given that SMI is already offering
value-added products and services atop externally developed
software, has been utilizing a historical DTS that has not been
designed with an adaptable structure. The requirements are grouped
into three sets of decreasing importance. It outlines a number of
specific evaluations that will be used to determine whether there is
an existing candidate DTS capable of meeting the requirements.
2. Discussion
2.1. Constraints
As with most of the problems in transitioning from proprietary to
open development, initial inspection might lead one to conclude that
the defect tracking problem is overconstrained. In the course of
reaching a coherent and ordered set of requirements for defect
tracking, we explicitly identify some of the legacy constraints, and
eliminate their influence on a potential solution.
2.1.1. Anecdotal problems
The present DTS solution at SMI, "Bugster", is the latest in a
sequence of custom and semi-custom DTS installations. Ignoring the
specifics of its technical implementation, Bugster presents three
major problems for open development scenarios:
* Per-account charge. The current licensing of the software
requires that each active account requires a paid user license to
an external software vendor. Given that we expect on the order of
5000 active contributing developers the OpenSolaris and open JDK
communities, an ongoing variable cost of this kind is problematic.
* Performance and scalability. Similarly, the current system is
implemented as an HA single clustered installation, capable of
supporting 1200 concurrent active users.
* Scrubbing and proprietary information boundaries. The information
classification of the existing bug corpus is vague and
inconsistent across the numerous development groups within SMI.
Revisiting the current defects for possible publication is
uneconomic, other than on an as-needed basis. The lack of a clear
boundary has made some employees apprehensive about any defect
publication.
2.1.2. Integration aspects
* Upstream/downstream. The OpenSolaris program is an additional
step in a progression from an operating environment composed
entirely of components created and/or maintained by a single
organization to one constructed from a wide variety of components
originating from multiple organizations and individuals. Where
those components have defect state tracked in their own DTS, it
would be economically beneficial to have that state represented in
the community DTS used for the operating environment effort, both
to minimize duplicate efforts and to maintain accurate status on
the upstream component.
* Other SMI systems. In contrast to the source code management
replacement, the Bugster DTS provides data to other information
systems within Sun Microsystems. Although the burden of
confidential data management rests with Sun (for that data it
considers confidential), the DTS API must meet or exceed the
current API provided by Bugster.
2.2. Ideal outcome sequence
Unfortunately, migrations from one DTS to another are expensive, at
least in the experience of the Solaris/OpenSolaris bug corpus. A
realistic idealized outcome would be three-phased, after the
candidate DTS is selected and implemented:
Phase 1. Two systems. Bugs could be filed in either system, and
integrations could refer to bugs in either system. Duplicates
would be encouraged to be closed in favour of the community DTS.
Work on a toolset to minimize the cost of such closures and
migrations could begin. In principle, a Phase 1 deployment is
possible--for recording purposes--in one to three months after
selection. Some tools integration work is required to use the
selected candidate for commits and defect resolution.
Phase 2. Deprecated second system. Integrations must refer only to
bugs in the community DTS, but bugs could be filed in either system.
New bug categories and classifications are created only in the
community DTS. Client toolsets and distribution-specific systems
[1] are expected to have adjusted to the community DTS capabilities.
After a successful exit from Phase 1, Phase 2 would run until the
legacy DTS is not deemed significant by the OpenSolaris Community.
Phase 3. Single system. All relevant public defect data is present
in the community DTS.
3. Requirements
As with previous requirements documents [2], the requirements described
below arise from a number of distinct classes: some are social, in
that the requirement is believed necessary for successful use in the
community; some are technical, in that the requirement is believed
necessary to successfully produce software in a multi-project,
multi-committer, multi-site development organization; and some are
economic, in that the requirement is attempting to describe
attributes that would limit the costs of the ongoing use of the
tool.
In an attempt to use neutral terms, we use the phrase "candidate
DTS" to describe the DTS solution we are evaluating and "legacy DTS"
to refer to the DTS solution in use (inside Sun, with partial
visibility via bugs.opensolaris.org) at present.
The requirements are ranked by necessity, using the terminology
proposed in IEEE Std 830-1998 [3]. We incorporate many of the
requirements from Wesolowski [4]. Many of the requirements are
derived from the earlier evaluation of distributed source code
management solutions [2].
3.1. "Essential" requirements
E0. Open source
To be considered for use by the OpenSolaris community, the
candidate DTS is expected to be available in source form under
an OSI-approved license. At this point in time, the community
expects to be able to operate directly one or more distinct
instances of the DTS.
E1. Universal access
With the exceptions identified below, all defect data pertaining
to OpenSolaris's open components and imported open components
must be available for search, view, and modify operations to
participants. Data must become available to all users at the
same time, unless restricted by an information management
agreement (such as those used in industry collaborations around
security vulnerabilities). All Participants must be potentially
eligible for any of the generally recognised defect workflow
roles, such as responsible engineer (RE) and initial evaluator
(IE), without reference to their affiliations outside the
Community.
Operations will use the Participant's OpenSolaris
credentials as a representation of the Participant's identity.
The candidate DTS should provide an API that permits the
addition of other identity representations, such as OpenID [5].
E2. Selective differentiated access
A mechanism must exist to indicate that a defect is private to a
set of Participants, and no part of it may be made accessible to
Participants not in that set, even if the defect's subcategory
| would otherwise cause it to be accessible. (DTS Administrators
| will have access to all defects.)
This requirement is to allow for security coordination, and
similar efforts. (Requests for differentiated access are
expected to be managed by the Tools Group, the Board, or a
designee. Distributions are expected to keep customer
confidential data external to the Community DTS.)
| E3. Configurable notification
All mail or other messages sent by the DTS to user-supplied
addresses (such as but not limited to a defect's interest list or a
subcategory's initial evaluator) containing defect data must enforce
the confidentiality of defects described in E2. Messaging on
defect transitions is expected to be configurable; one approach
would be a message template system.
Messaging and notification is expected to be delivered as
| electronic mail. Support for additional formats and protocols
| is, in general, optional; support for some form of syndication
| is desirable.
E4. Access control enforcement
The DTS must enforce the access restrictions described in E2 and
E3. It is not the intent of this specification to require any
particular method of enforcement.
|
| Once hosted at opensolaris.org, it is expected that defect
| content is covered by the Terms of Use governing the site [6].
E5. Interface stability and completeness
The storage representation, command line interfaces, network
protocols, and hooks interfaces should be documented and have
some level of declared commitment. The state of the storage
representation and the operations that modify it should be well
defined, so that use with advanced file system capabilities can
be assessed for hazards. (For example, consistent use with file
system snapshot capabilities.)
Use of the candidate DTS with advanced file system capabilities
should be defined. (For example: Can ZFS clones be used to
| back up the defect corpus, or is consistency enforced above the
| file system layer?)
E6. Standard operations and transactions
The candidate DTS is expected to support a variety of standard
operations on defect reports. The following list of
transactions are to be assessed and documented by the evaluating
engineer:
- create defect
- modify defect content
- modify defect metadata
- place defect in resolved state
- restore defect from resolved to unresolved state
- associate defect with external resource
- associate defect with another DTS-tracked defect
- associate defect with tag/text label
| - associate defect with a list of parties desiring defect update
| notifications
- change defect category
Each of these transactions should correctly modify the defect
state and send appropriate notifications.
Equivalency (another operation or set of operations that might
be used in place of an operation not specifically supported),
and the reasons for the omission of a transaction, are to be
assessed and documented by the evaluating engineer.
Additional operations supported by the candidate DTS on defect
reports should be documented by the evaluating engineer.
E7. Scalability
The candidate DTS must have one or more deployment
methods that allow the DTS to carry large query loads. The
latency between a defect report update and its accessibility to
be queried must be documented for such scenarios.
E8. Interface extensibility
Beyond the requirements of E3, an extensible interface, so that
OpenSolaris-specific tools might be integrated with DTS
operations is desired. Such an interface might be composed of a
documented "hooks" interface, a documented library interface, or
some other modular approach. An extensive hooks interface, with
hook evaluations able to terminate operations, is a strongly
desired attribute in a candidate DTS; a candidate DTS with such
an interface will be considered to meet fully this requirement.
Such interfaces are expected to allow the automation of specific
operations, such as the mass state updates performed by a
consolidation's C-Team.
It should be possible, via the interfaces or extension
facilities, to perform administrative operations on the DTS:
creation, modification, and closure of defect categories (or
other defect classifications);
creation, modification, and deletion of DTS user accounts (or
other user or group relationships); and similar operations.
E9. Stable defect representation
Specifically extending E3 and E8, the candidate DTS will have a
stable, formatted representation of a defect report. A
candidate DTS that emitted defect reports in a documented plain
text or XML format would satisfy this requirement.
E10. Hierarchical defect classification
It appears that a candidate DTS, to represent the current defect
corpus, must provide some means of classification that can
represent at least a product-category-subcategory hierarchy.
(Although a candidate DTS could meet this requirement with only
generic defect metadata ("tagging"), it is expected that queries
will regularly use the defect classifications to narrow the
query results, leading to an effective performance criterion.)
| E11. Non-hierarchical defect classification
|
| Beyond E10, it appears that a candidate DTS should have one or
| more free-form means of attaching classification information to
| a defect. Examples would include one or more keywords fields, a
| set of name-value attributes, or a tagging facility.
|
| E12. Concurrent change detection, basic
|
| A common occurrence in the legacy DTS is that two developers
| simultaneously begin editing a defect record and that, in the
| race to commit, one set of changes is lost. At a minimum, a
| candidate DTS should notify the second developer that his or her
| changes have not been committed. (Silently overwriting the
| changes is not considered change detection.)
|
| E13. Unicode data representation
|
| As the community expands, it seems evident that a set of defects
| will have associate program input and output from other locales,
| meaning that accurate representation in the defect's data fields
| is required. A candidate DTS that consistently uses Unicode
| representations should meet this requirement.
|
| E14. Stable per-defect URIs
|
| The candidate DTS should provide a unique URI for each defect
| represented in the system, or at very minimum, a unique ID that
| can be used as the basis for such a URI. The unique component
| is expected to have some association with the product or
| products tracked by the candidate DTS.
|
| A candidate DTS that provides a monotonically increasing URI
| component, such as a "counter", would satisfy this requirement.
|
| E15. Versioned or audited defect history
|
| In the case of erroneous changes--accidental or malicious--it
| should be possible to recover the previous contents of a defect
| record without recourse to a site's archive (although the
| details in satisfying E5 and E9 may make recovery via file
| system capabilities straightforward).
|
2.2. "Conditional" requirements
| C16. Ease of use
The candidate DTS should be easy to install in a reasonably
self-contained fashion. In principle, shipment in an
OpenSolaris consolidation should be possible with a finite
investment of resources, meaning that the candidate DTS does
not have a complicated makefile system, has dependencies that
can be easily managed, etc.
The primary interfaces should be understandable based on the
interfaces and documentation to a user familiar with
DTS concepts.
| C17. Tool community health
The community or author of the candidate DTS needs to be active
and engaged with their user population. The ability of the
candidate DTS's community to absorb, directly or through a
liaison, the defects and feature requests of the OpenSolaris
community should be estimated, preferably by a direct inquiry to
the candidate DTS community.
| C18. OpenSolaris community implementation expertise
One or more contributors within the OpenSolaris community need
to be able to assess potential defects in the implementation of
the candidate DTS and potentially participate in the development
of new features or supporting tools for the candidate DTS.
| C19. Concurrent change detection, enhanced
|
| Beyond E12, it is desired that the candidate DTS provide an
| adjusted defect editing view to the second modifying developer,
| such that his or her changes can be applied to the latest
| version of the defect's contents.
|
| C20. Transactional operations and corruption recovery
The operations on the candidate's defect datastore should have
defined semantics, in particular identifying non-atomic
transactions and mechanisms for recovery from a corrupted
datastore. This requirement may be met by the underlying
datastore technology.
| C21. Search capability
An integrated search capability of the text representation of
the defect corpus is a desired feature in the candidate DTS. It
is possible to construct a search capability, given E3 and E9,
but an facility integrated with the various client components is
| preferred. (Additionally, by virtue of satisfying E8, it should
| also be possible to construct a structured search capability
| using other software components.)
|
| Search capabilities on specific defect attributes, such as those
| given via satisfaction of E10 and E11, are also desirable.
2.3 "Optional" requirements
| O22. Attachment handling
The legacy DTS manages large attachments as an integrated
| feature, to manage system crash dumps, process core files,
application output, etc. related to the failures captured by a
defect report. Since a simple file archival service can meet
these requirements--once restricted access due to
confidentiality is removed--the candidate DTS need not provide
its own attachment handling feature.
3. Evaluations
We anticipate a number of qualitative and quantitative tests to
evaluate the satisfaction of the various requirements, where a
"meets" or "does not meet" result is not applicable.
3.1. Representational and performance criteria
These criteria focus on the ability of the candidate DTS to
represent a large set of distinct, interrelated defects. The
current Sun Microsystems defect corpus contains in the vicinity of
three million defect reports. Evaluations will be performed on a
defect corpus derived from that published on bugs.opensolaris.org; this
corpus will have at least 100 000 defect reports.
The expected set of meaningful operations for performance evaluation
are:
- unresolved defects query against a product or category,
- defect query by keyword or other text label,
- defects resolved by a single responsible engineer,
Various candidate DTSs may provide notions of defect classification,
state, importance, or other attributes. Queries against these
attributes should be evaluated, equivalent to
- defects in either a resolved or an unresolved state, for a given
classification,
- defects addressed in a given build/release
- unresolved defects, ordered by importance
The candidate DTS will be evaluated for data integrity by
interruption of the set of operations by signal and by machine
failure.
The safety of the candidate DTS with respect to file system
capabilities will be evaluated using ZFS snapshot/clone technology
for safe repository copies.
3.2. Implementation criteria
The candidate DTS implementation will be assessed by a design and
code review by an OpenSolaris contributor with expertise in the
implementation language of the candidate DTS.
4. References
[1] S. Hahn, "OpenSolaris: Defect tracking relationships", 2007.
http://blogs.sun.com/sch/entry/opensolaris_defect_tracking_relationships
[2] OpenSolaris Distributed Source Code Management Requirements, 2005.
[3] IEEE Std 830-1998, "IEEE Recommended Practice for Software
Requirements Specifications", 1998.
[4] K. Wesolowski, "Defect Management Requirements in an Open
Development Paradigm", 2006.
[5] OpenID, http://openid.net
|
| [6] OpenSolaris Terms of Use, 2007.
| http://opensolaris.org/os/tou/
--
[EMAIL PROTECTED] http://blogs.sun.com/sch/
_______________________________________________
tools-discuss mailing list
tools-discuss@opensolaris.org
