I agree the need for clear test suite status is implied and should be
explicit. I've added a new requirement for this to [1]. As to how this
requirement is addressed, perhaps we should adopt/re-use some existing
good practice; otherwise perhaps we can use a Status/Readme file in each
.../tests/ directory.
Besides "WG Approval", you see the need for "maintainer approval".
Additionally, you think there will be scenarios where multiple
submissions make it difficult to know which tests have been reviewed
(e.g. by the maintainer). This could be addressed by the maintainer
creating a separate directory (e.g. "reviewed") and the maintainer would
use it to place copies of test cases (e.g. from submissions directories)
she reviewed (and possibly edited and approved). I don't think the
process as currently defined would preclude the maintainer from doing
something like this and if it becomes a common/good practice, we could
document it. In the case of a single contributor, it's probably not
something that would be needed (but wouldn't necessarily be harmful).
-Art Barstow
[1] http://www.w3.org/2008/webapps/wiki/Testing#Testing_Goals
On Apr/17/2011 7:06 PM, ext Aryeh Gregor wrote:
On Wed, Apr 13, 2011 at 10:02 AM, Arthur Barstow<art.bars...@nokia.com> wrote:
I have updated WebApps' testing process documents to reflect comments
submitted to the initial draft process [1]. As such, this is a Call for
Consensus to agree to the testing process as described in:
http://www.w3.org/2008/webapps/wiki/Testing
http://www.w3.org/2008/webapps/wiki/Submission
http://www.w3.org/2008/webapps/wiki/Approval
http://www.w3.org/2008/webapps/wiki/Harness
Sorry for the lateness of this review -- I was swamped with work and
didn't find the time to respond earlier. I still have a few issues
with the proposed approval procedure. The way it sounds is that tests
can either be non-approved, or approved. Non-approved tests sound
like (I'm not totally clear on this) they're supposed to live in
per-contributor submission/ directories, without anyone else
necessarily having any say over them. Approved tests, on the other
hand, only exist at LC or later, and can't be changed without Working
Group approval. (Which means what? I'm not sure.)
The problem I've seen with the submission/ vs. approved/ approach in
the HTMLWG is that there's no distinction between tests in submission/
that are useful and correct but just haven't been reviewed by anyone,
and tests that aren't complete or correct yet and aren't really
intended for serious use. We should have a clear test suite that's
usable in practice well before LC, even if not all the tests are
reviewed yet.
So what I'd prefer is that the contents of approved/ be under the
control of the maintainer of the test suite, like the editor controls
the spec. If people are submitting tests, the maintainer should be
allowed to approve them without a CfC, at least while the spec is
still a Working Draft. That way we have a single repository from the
beginning that should include all useful tests, instead of having many
tests of varying quality scattered throughout submission/. Once a
test is in approved/, it should be possible to file bug reports on it,
discuss it on the mailing list, etc. It needs to be in a central
location and the responsibility of the working group, not the
submitter, so that the working group can ensure the test suite's
quality from the earliest possible point.
Then a CfC would be whether the WG is okay with the current contents
of approved/ or whether some of the tests should be un-approved. A
CfC shouldn't be necessary to approve tests to begin with, any more
than it's necessary to make an edit to a spec.
