Thanks for the comments Aryeh.
So rather than having a contribution reviewed and approved when the
contributor wants to start a CfC to Review+Approved (R&A) their
contribution, the idea is to allow any number of edits and contributions
to a test suite until such time the group thinks the test suite is
complete (e.g. after a CR is published) and then start a formal R&A? I
like that idea.
I'm not sure we need to explicitly designate test suite maintainers.
Ideally, someone other than the spec Editor(s) should create the test
cases (e.g. to avoid the "fox guarding the chicken coup"). It seems like
we should grant everyone CRUD rights to the test suites and if that
becomes problematic we can address those issues later.
I'd like to see what others have to say about the original proposal and
Aryeh's comments.
-Art Barstow
On Apr/1/2011 12:15 PM, ext Aryeh Gregor wrote:
On Thu, Mar 31, 2011 at 8:04 AM, Arthur Barstow<[email protected]> wrote:
3. http://www.w3.org/2008/webapps/wiki/Approval - how to start a test case
review, approval process, how to update an approved test case
It looks like every submitted test suite must undergo a CfC, and so
must every update. I'll first say that this is much better than the
way the HTMLWG currently works, which requires explicit endorsement by
a reviewer before approval but provides no way to obtain such review
other than hoping someone will be willing to give it.
But it's also much more cumbersome than the process for changing the
actual specifications. In writing specs, instead, we designate one or
more people as editors for each spec, and let them make changes at
will. Others who have suggestions can use the bug-tracking system,
and contentious issues must be addressed during Last Call before a
specification can become a Candidate Recommendation. This procedure
is well-optimized for the fact that realistically, the large majority
of changes are uncontroversial, so it makes the most sense to assume
that changes are uncontroversial until someone actually objects,
without any formal approval requirements.
I propose that the Working Group follow this model for tests too. For
each specification, we should designate certain people as maintainers
for that specification's tests. They should be allowed to update the
tests and add new tests freely, and should be responsible for
responding to bug reports. At Last Call, we should explicitly ask for
feedback for the test suite as well as the specification, and treat
feedback on tests similarly to feedback on the spec.
I suggest that for starters, the editors of a spec should also be
maintainers of its tests. Additional test maintainers for each spec
can be added by a WG CfC. This recognizes the fact that the editors
of a spec are presumably competent to judge tests, but that many
editors won't want to also review tests and many test reviewers won't
want to edit specs, so there's value in keeping the groups separate.
If the approval model currently on the wiki page is adopted, I predict
that it will become extremely cumbersome to maintain large test suites
for any specification that's not already very stable. Any significant
specification change will require a CfC to change all affected test
cases, which will be a lot of spam to sift through if we have lots of
tests and spec changes. This model also prevents editors who want to
write their own tests from writing the tests in conjunction with the
specification, because of the higher approval bar for tests than for
spec edits. I don't think it makes sense at all.
