I've made a number of edits, based on private feedback and the
on-alias responses. Clarifications:
1. My operational concern about access to the entire database was
specifically about not having a multi-gigabyte download. Nothing
else should be read into that point.
2. The relationship between the development branch and the
maintenance branches of any particular consolidation does not seem
well understood. It seems premature to conclude that a multiple
release relationship belongs in the community DTS, versus a
distribution's annotations. We can come back to this, perhaps as
some of the candidate evaluations proceed.
3. It's pretty much reality that it's going to take Sun time to
transition from one DTS to another; I would expect Sun is
obligated to develop tools of the kind Alan C meant, but I don't
believe that a Phase 2 scenario can be avoided, even with such
tools instantaneously materializing.
4. I would like to come back to Jim's point about "what DTSs do"
versus "what Bugster does" after we actually try one or two of
them. That is, unlike other evaluations, I think there's a
sandbox phase to the evaluation, where we agree that, say, a
notification alias is equivalent to both the IE and RM--or that we
don't agree, and need to formalize that ownership relation in our
requirements. (I suppose this point generalizes #2 above.)
- Stephen
----
OpenSolaris
DEFECT TRACKING SYSTEM REQUIREMENTS [DRAFT]
| Stephen Hahn
1. Summary
This document identifies and explains the requirements for a defect
tracking system (DTS) solution to be used for open development.
Software development at SMI, given that SMI is already offering
value-added products and services atop externally developed
software, has been utilizing a historical DTS that has not been
designed with an adaptable structure. The requirements are grouped
into three sets of decreasing importance. It outlines a number of
specific evaluations that will be used to determine whether there is
an existing candidate DTS capable of meeting the requirements.
2. Discussion
2.1. Constraints
As with most of the problems in transitioning from proprietary to
open development, initial inspection might lead one to conclude that
the defect tracking problem is overconstrained. In the course of
reaching a coherent and ordered set of requirements for defect
tracking, we explicitly identify some of the legacy constraints, and
eliminate their influence on a potential solution.
2.1.1. Anecdotal problems
The present DTS solution at SMI, "Bugster", is the latest in a
sequence of custom and semi-custom DTS installations. Ignoring the
specifics of its technical implementation, Bugster presents three
major problems for open development scenarios:
* Per-account charge. The current licensing of the software
requires that each active account requires a paid user license to
an external software vendor. Given that we expect on the order of
5000 active contributing developers the OpenSolaris and open JDK
communities, an ongoing variable cost of this kind is problematic.
* Performance and scalability. Similarly, the current system is
implemented as an HA single clustered installation, capable of
supporting 1200 concurrent active users.
* Scrubbing and proprietary information boundaries. The information
classification of the existing bug corpus is vague and
inconsistent across the numerous development groups within SMI.
Revisiting the current defects for possible publication is
uneconomic, other than on an as-needed basis. The lack of a clear
boundary has made some employees apprehensive about any defect
publication.
2.1.2. Integration aspects
| * Upstream/downstream. The OpenSolaris program is an additional
| step in a progression from an operating environment composed
| entirely of components created and/or maintained by a single
| organization to one constructed from a wide variety of components
| originating from multiple organizations and individuals. Where
| those components have defect state tracked in their own DTS, it
| would be economically beneficial to have that state represented in
| the community DTS used for the operating environment effort, both
| to minimize duplicate efforts and to maintain accurate status on
| the upstream component.
* Other SMI systems. In contrast to the source code management
replacement, the Bugster DTS provides data to other information
| systems within Sun Microsystems. Although the burden of
confidential data management rests with Sun (for that data it
considers confidential), the DTS API must meet or exceed the
current API provided by Bugster.
2.2. Ideal outcome sequence
Unfortunately, migrations from one DTS to another are expensive, at
least in the experience of the Solaris/OpenSolaris bug corpus. A
realistic idealized outcome would be three-phased, after the
candidate DTS is selected and implemented:
Phase 1. Two systems. Bugs could be filed in either system, and
integrations could refer to bugs in either system. Duplicates
would be encouraged to be closed in favour of the community DTS.
Work on a toolset to minimize the cost of such closures and
| migrations could begin. In principle, a Phase 1 deployment is
| possible--for recording purposes--in one to three months after
| selection. Some tools integration work is required to use the
| selected candidate for commits and defect resolution.
Phase 2. Deprecated second system. Integrations must refer only to
bugs in the community DTS, but bugs could be filed in either system.
New bug categories and classifications are created only in the
community DTS. Client toolsets and distribution-specific systems
[1] are expected to have adjusted to the community DTS capabilities.
| After a successful exit from Phase 1, Phase 2 would run until the
| legacy DTS is not deemed significant by the OpenSolaris Community.
Phase 3. Single system. All relevant public defect data is present
in the community DTS.
3. Requirements
As with previous requirements documents [2], the requirements described
below arise from a number of distinct classes: some are social, in
that the requirement is believed necessary for successful use in the
community; some are technical, in that the requirement is believed
necessary to successfully produce software in a multi-project,
multi-committer, multi-site development organization; and some are
economic, in that the requirement is attempting to describe
attributes that would limit the costs of the ongoing use of the
tool.
In an attempt to use neutral terms, we use the phrase "candidate
DTS" to describe the DTS solution we are evaluating and "legacy DTS"
to refer to the DTS solution in use (inside Sun, with partial
visibility via bugs.opensolaris.org) at present.
The requirements are ranked by necessity, using the terminology
proposed in IEEE Std 830-1998 [3]. We incorporate many of the
requirements from Wesolowski [4]. Many of the requirements are
derived from the earlier evaluation of distributed source code
management solutions [2].
3.1. "Essential" requirements
E0. Open source
To be considered for use by the OpenSolaris community, the
candidate DTS is expected to be available in source form under
an OSI-approved license. At this point in time, the community
expects to be able to operate directly one or more distinct
instances of the DTS.
E1. Universal access
With the exceptions identified below, all defect data pertaining
to OpenSolaris's open components and imported open components
must be available for search, view, and modify operations to
participants. Data must become available to all users at the
same time, unless restricted by an information management
agreement (such as those used in industry collaborations around
security vulnerabilities). All Participants must be potentially
eligible for any of the generally recognised defect workflow
roles, such as responsible engineer (RE) and initial evaluator
(IE), without reference to their affiliations outside the
| Community.
|
| Operations will use the Participant's OpenSolaris
| credentials as a representation of the Participant's identity.
| The candidate DTS should provide an API that permits the
| addition of other identity representations, such as OpenID [5].
E2. Selective differentiated access
A mechanism must exist to indicate that a defect is private to a
set of Participants, and no part of it may be made accessible to
Participants not in that set, even if the defect's subcategory
would otherwise cause it to be accessible.
This requirement is to allow for security coordination, and
similar efforts. (Requests for differentiated access are
expected to be managed by the Tools Group, the Board, or a
designee. Distributions are expected to keep customer
confidential data external to the Community DTS.)
E3. Differentiated, configurable messaging
All mail or other messages sent by the DTS to user-supplied
addresses (such as but not limited to a defect's interest list or a
subcategory's initial evaluator) containing defect data must enforce
the confidentiality of defects described in E2. Messaging on
defect transitions is expected to be configurable; one approach
would be a message template system.
Messaging and notification is expected to be delivered as
| electronic mail. Support for additional protocols is, in
| general, optional; support for some form of syndication is
| desirable.
E4. Access control enforcement
The DTS must enforce the access restrictions described in E2 and
E3. It is not the intent of this specification to require any
particular method of enforcement.
E5. Interface stability and completeness
The storage representation, command line interfaces, network
protocols, and hooks interfaces should be documented and have
some level of declared commitment. The state of the storage
representation and the operations that modify it should be well
defined, so that use with advanced file system capabilities can
be assessed for hazards. (For example, consistent use with file
system snapshot capabilities.)
Use of the candidate DTS with advanced file system capabilities
should be defined. (For example: Can ZFS clones be used to
| back up the defect corpus? Can filesystem ACLs be used to control
access to portions of the repository?)
E6. Standard operations and transactions
| The candidate DTS is expected to support a variety of standard
| operations on defect reports. The following list of
| transactions are to be assessed and documented by the evaluating
| engineer:
- create defect
- modify defect content
- modify defect metadata
- place defect in resolved state
- restore defect from resolved to unresolved state
- associate defect with external resource
- associate defect with another DTS-tracked defect
| - associate defect with tag/text label
- change defect category
Each of these transactions should correctly modify the defect
state and send appropriate notifications.
Equivalency (another operation or set of operations that might
be used in place of an operation not specifically supported),
| and the reasons for the omission of a transaction, are to be
assessed and documented by the evaluating engineer.
|
| Additional operations supported by the candidate DTS on defect
| reports should be documented by the evaluating engineer.
E7. Scalability
The candidate DTS must have one or more deployment
methods that allow the DTS to carry large query loads. The
latency between a defect report update and its accessibility to
be queried must be documented for such scenarios.
| E8. Interface extensibility
Beyond the requirements of E3, an extensible interface, so that
OpenSolaris-specific tools might be integrated with DTS
operations is desired. Such an interface might be composed of a
documented "hooks" interface, a documented library interface, or
some other modular approach. An extensive hooks interface, with
hook evaluations able to terminate operations, is a strongly
desired attribute in a candidate DTS; a candidate DTS with such
an interface will be considered to meet fully this requirement.
| Such interfaces are expected to allow the automation of specific
| operations, such as the mass state updates performed by a
| consolidation's C-Team.
|
| It should be possible, via the interfaces or extension
| facilities, to perform administrative operations on the DTS:
| creation, modification, and closure of defect categories (or
| other defect classifications);
| creation, modification, and deletion of DTS user accounts (or
| other user or group relationships); and similar operations.
|
| E9. Stable defect representation
|
| Specifically extending E3 and E8, the candidate DTS will have a
| stable, formatted representation of a defect report. A
| candidate DTS that emitted defect reports in a documented plain
| text or XML format would satisfy this requirement.
|
| E10. Hierarchical defect classification
|
| It appears that a candidate DTS, to represent the current defect
| corpus, must provide some means of classification that can
| represent at least a product-category-subcategory hierarchy.
| (Although a candidate DTS could meet this requirement with only
| generic defect metadata ("tagging"), it is expected that queries
| will regularly use the defect classifications to narrow the
| query results, leading to an effective performance criterion.)
|
| 2.2. "Conditional" requirements
|
| C11. Ease of use
|
| The candidate DTS should be easy to install in a reasonably
| self-contained fashion. In principle, shipment in an
| OpenSolaris consolidation should be possible with a finite
| investment of resources, meaning that the candidate DTS does
| not have a complicated makefile system, has dependencies that
| can be easily managed, etc.
|
| The primary interfaces should be understandable based on the
| interfaces and documentation to a user familiar with
| DTS concepts.
|
| C12. Stable per-defect URIs
|
| The candidate DTS should provide a unique URI for each defect
| represented in the system, or at very minimum, a unique ID that
| can be used as the basis for such a URI. The unique component
| is expected to have some association with the product or
| products tracked by the candidate DTS.
|
| A candidate DTS that provides a monotonically increasing URI
| component, such as a "counter", would satisify this requirement.
|
| C13. Tool community health
|
| The community or author of the candidate DTS needs to be active
| and engaged with their user population. The ability of the
| candidate DTS's community to absorb, directly or through a
| liaison, the defects and feature requests of the OpenSolaris
| community should be estimated, preferably by a direct inquiry to
| the candidate DTS community.
|
| C14. OpenSolaris community implementation expertise
|
| One or more contributors within the OpenSolaris community need
| to be able to assess potential defects in the implementation of
| the candidate DTS and potentially participate in the development
| of new features or supporting tools for the candidate DTS.
|
| C15. Transactional operations and corruption recovery
The operations on the candidate's defect datastore should have
defined semantics, in particular identifying non-atomic
transactions and mechanisms for recovery from a corrupted
datastore. This requirement may be met by the underlying
datastore technology.
| C16. Search capability
|
| An integrated search capability of the text representation of
| the defect corpus is a desired feature in the candidate DTS. It
| is possible to construct a search capability, given E3 and E9,
| but an facility integrated with the various client components is
| preferred.
|
2.3 "Optional" requirements
| O17. Attachment handling
The legacy DTS manages large attachments as an integrated
feature, to manage system crash dumps, process corefiles,
application output, etc. related to the failures captured by a
defect report. Since a simple file archival service can meet
these requirements--once restricted access due to
confidentiality is removed--the candidate DTS need not provide
its own attachment handling feature.
3. Evaluations
We anticipate a number of qualitative and quantitative tests to
evaluate the satisfaction of the various requirements, where a
"meets" or "does not meet" result is not applicable.
3.1. Representational and performance criteria
These criteria focus on the ability of the candidate DTS to
represent a large set of distinct, interrelated defects. The
| current Sun Microsystems defect corpus contains in the vicinity of
three million defect reports. Evaluations will be performed on a
| defect corpus derived from that published on bugs.opensolaris.org; this
corpus will have at least 100 000 defect reports.
The expected set of meaningful operations for performance evaluation
are:
- unresolved defects query against a product or category,
- defect query by keyword or other text label,
- defects resolved by a single responsible engineer,
| Various candidate DTSs may provide notions of defect classification,
| state, importance, or other attributes. Queries against these
| attributes should be evaluated, equivalent to
|
| - defects in either a resolved or an unresolved state, for a given
| classification,
| - defects addressed in a given build/release
| - unresolved defects, ordered by importance
|
The candidate DTS will be evaluated for data integrity by
interruption of the set of operations by signal and by machine
failure.
The safety of the candidate DTS with respect to file system
capabilities will be evaluated using ZFS snapshot/clone technology
for safe repository copies.
3.2. Implementation criteria
The candidate DTS implementation will be assessed by a design and
code review by an OpenSolaris contributor with expertise in the
implementation language of the candidate DTS.
4. References
[1] S. Hahn, "OpenSolaris: Defect tracking relationships", 2007.
http://blogs.sun.com/sch/entry/opensolaris_defect_tracking_relationships
[2] OpenSolaris Distributed Source Code Management Requirements, 2005.
[3] IEEE Std 830-1998, "IEEE Recommended Practice for Software
Requirements Specifications", 1998.
[4] K. Wesolowski, "Defect Management Requirements in an Open
Development Paradigm", 2006.
|
| [5] OpenID, http://openid.net
--
[EMAIL PROTECTED] http://blogs.sun.com/sch/
_______________________________________________
tools-discuss mailing list
tools-discuss@opensolaris.org
