On Wed, Nov 4, 2015 at 11:50 PM, James Graham <ja...@hoppipolla.co.uk> wrote:
> On 04/11/15 10:24, Robert O'Callahan wrote: > >> On Wed, Nov 4, 2015 at 5:14 PM, James Graham <ja...@hoppipolla.co.uk> >> wrote: >> >> On 03/11/15 22:08, Robert O'Callahan wrote: >>> >>> Why not create a Mochitest compatibility layer over testharness.js so >>> >>>> that tests that only use SimpleTest.waitForExplicitFinish(), >>>> SimpleTest.finish(), is(), todo() and ok() can run on Servo with trivial >>>> changes? >>>> >>>> >>> A few reasons I am skeptical of this: >>> >>> * Mochitests test gecko behaviour, not standardised behaviour. These can >>> and do differ. Therefore they are even less trustworthy than the average >>> test. >>> >>> >> I'm not sure what you mean by this. It's easy to write web-platform-tests >> that depend on non-standard behavior. I had to debug one such last week. >> Furthermore Gecko mochitests have been tested against Gecko, which has >> itself been tested on the Web, so Gecko mochitests are far more likely to >> be "correct" than a test someone just wrote and only tested on, say, >> Servo. >> > > I mean that mochitests are used as regression tests; they aren't > necessarily testing the per-spec behaviour (which might match Blink or > Edge, but not Gecko), but whatever the Gecko code implements. Of course > web-platform-tests can be buggy like any other code, but they are at least > intended to test the right thing. I think it's uncommon for mochitests to intentionally be stricter than what the spec requires ... excluding cases where the spec is known to be inadequate. It's easy to be accidentally more strict, but that's true for WPT too. > * Mochitests are really written assuming the full gecko featureset. Given >> >>> the total lack of isolation between asserts, they may behave rather >>> unexpectedly and misleadingly in a browser not having all those features. >>> >>> >> Many use Gecko-specific features but many don't. >> > > Not just use of gecko-specific (or, often spidermonkey-specific) > behaviours, but also the fact that mochitests assume an assert never fails > (because if one does it's a regression that needs to be fixed). In Servo, > which might not implement all the features used, an assert can be expected > to fail for some time until more feature work is done. This is particularly > true at the moment when Servo doesn't implement the window error handler. I > anticipate this impedance mismatch would cause the tests to be less useful > in Servo than they are in Gecko. > That makes sense. > > * In the long term having multiple APIs for writing tests that people have >> >>> to learn in order to read tests is a big net negative. If we allow >>> mochitests to be upstreamed with a shim we should expect other vendors to >>> do the same, and to end up with half a dozen possible test formats. >>> >>> >> That's fair, though it's a problem we introduced when we started adding >> web-platform-tests to the tree. >> >> Put it this way: when Servo needs tests for a feature, I expect it makes >> much more sense to take Gecko mochitests for the feature and convert them >> to web-platform-tests than to write new tests from scratch. >> > > Well sure, I agree that taking mochitests as the input to a test-writing > effort is a good idea. I see this as being very different to blindly > shimming mochitests into the wpt harness. Having said that, however, I > don't think people have complained a lot about lack of test coverage from > wpt except in the areas that it doesn't cover at all i.e. dynamic changes > to layout, or other human interaction. To pick a couple of areas I work on: for CSSOM-Views for example there is practically no coverage at all. The media tests are better but still very limited compared to what we test in mochitests. Rob -- lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr .a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn _______________________________________________ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
