On 03/29/2016 12:33 AM, Geoffrey Sneddon wrote:
Hey, Given as far as I know you've been at least somewhat involved with every spec that's used wpt as part of its implementation report, let me throw a few questions at you: * How's the lack of versioning been dealt with? (e.g., between HTML5 and tests for features added since HTML5 was forked)
We used wptreport and filtering out test results we were not interested in: https://github.com/w3c/test-results/blob/gh-pages/html/filter.js
* How's the W3C/WHATWG divergence been dealt with? (For specs for which that applies!)
At the time of HTML 5.0, the diff between WHATWG and W3C was pretty much the same as the diff between 5.0 and 5.1, so we just filtered out the results that were only pertinent to 5.1. Other differences didn't have impact on implementations.
I do realize that, while we were able to get away with this approach back then, it wouldn't be so easy if an API was making a breaking change such as changing the return type of a function.
Far as I can tell, neither of them really have been dealt with. If I'm not misreading the few IRs I opened up, they've all assumed that all tests within the respective directories in wpt apply to the spec leaving CR. Furthermore, when it comes to links to the testsuite, they seem to all just link to that directory in the master branch of wpt, which will become increasingly less useful at testing a stable snapshot as time progresses!
We indeed didn't make a snapshot of the tests at the time. I guess if one wanted to run the old html tests, one could checkout an old WPT commit from that time.
There's also the interesting case of a edit to a REC that ends up being first published as a CR, where one would presumably want to add/modify tests relative to the testsuite for the REC while excluding tests for more recent revisions of the spec to exit CR then. As an aside: there's been some discussion in the CSS WG about testsuites recently (uh, my doing, "whoops") which you might have some interest in, some of which deals with issues like testsuites for different versions of a spec.
I'll try to catch up on this front. I'm curious about the use cases for the old test suites. While I can think of some, I'm worried about the added costs for maintaining multiple versions of a test suite. We already have such difficulties for maintaining multiple versions of a spec...
Philippe
