On 9/21/14, 9:00 AM, James Graham wrote:
I am substantially less convinced that tying these tests to the spec
lifecycle makes sense.
Agreed. The only reason it's an issue for me is the lack of
errata-issuance by the W3C and hence the tendency to attempt to enshrine
obviously-wrong things in specifications forever.
The W3C Process encourages people to think of
interoperability as a binary condition; either implementations are
interoperable or they are not.
More interestingly, either the specification is implementable or not.
Again, because once the REC is published everyone goes home and never
touches that document again.
The two implementations condition was there to make sure you didn't end
up with specs that basically reflected one UA's internals and that no
one else could implement....
However the incentives for testing
under such a regime are not well aligned
Yes, absolutely agreed.
I would much prefer to
have a testsuite written by people making a genuine effort to find
errors in implementations even if the result is that no one passes every
single test.
This would be much more helpful, absolutely.
Of course then you need to ask yourself whether the test failures are
just bugs in implementations or fundamental problems with the spec. A
question spec writers are often loath to ask. :(
I'm also dubious that requiring a test for every assertion in the spec
is a standard that we are prepared to hold ourselves to when we ship
code.
Indeed.
Since shipping code is, in the grand scheme of things,
substantially more important than shipping a spec — the former affecting
all our users and the latter only lawyers
I wish specs only affected lawyers, especially given how they are often
created/maintained.
Sadly, they do affect web developers and browser developers, not to
mention other specs.
it doesn't seem at all
reasonable to demand that the specification is held to a higher standard.
Note that I made no such demand, precisely because I think it's unrealistic.
My concrete suggestion is that we, as an organisation, work to achieve
parity between the tests we require to ship our own code and those we
release in ways that can be used to support a spec and, more
importantly, those that can be used verify that different
implementations match up.
This is a good idea, but not terribly relevant to what dbaron should say
in his AC rep capacity, right?
Making this process as low-overhead as
possible is something that I'm working on.
And it's much appreciated!
Note that actually sanely testing something like navigation in
non-browser-specific ways is ... hard. Basic things like "open a
cross-origin window and wait for it to load" aren't really possible. :(
Obviously this isn't going to make a short-term difference for old
features like WindowProxy. I'm not sure what to suggest for those cases
given that we have de-facto shown an unwillingness to invest even
relatively small amounts of effort into reviewing existing tests that
could be part of the HTML testsuite for that feature [1].
Missing reference?
In the longer term, one might hope that bugfixes will produce new
testcases that could be upstreamed, and Servo might need a proper
testsuite to achieve interoperability. Having said that, Servo has so
far not produced a significant number of tests, which has been a little
surprising as they have been implementing some of the core pieces of the
platform which are indeed under-tested. I suspect this is because the
skills, interests and goals of the team are around producing code rather
than tests.
Yep. And because they really haven't been aiming for full web compat
yet, I'd hope, but rather prototyping out some of the parallalelism ideas.
The WG turned into an
unproductive, bureaucratic, nightmare and succeeded in driving out their
core constituency leaving the remaining participants to debate topics of
little consequence.
Yup.
If we were to complain about the lack of testing for HTML
Again, note that I don't think we have any realistic way to complain
about it, for precisely the reasons you list.
This idea of shipping on a date-based schedule isn't actually
obviously bad, as long as you set the expectations correctly, which is
something the W3C will get wrong.
I hope you're wrong (e.g. that the W3C will actually continue fairly
frequent date-based updates to HTML), but I fear you're right.
So yes, I think that, at the moment, "everybody knows" looking at HTML
under w3.org/TR/ is a mistake. Even if they don't it's probably not
*that* important because HTML isn't the most interesting spec right now;
Unless you're Servo, yeah.
-Boris
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
