On Thu, Oct 28, 2021 at 10:24 AM Sam Sneddon via webkit-dev < webkit-dev@lists.webkit.org> wrote:
> As part of the ongoing work on GPU Process, we’re interested in adding > support for reftest fuzzy matching (i.e., allowing a certain amount of > tolerance when comparing the generated images). > > Our intention is to match the semantics of WPT’s reftests ( > https://web-platform-tests.org/writing-tests/reftests.html#fuzzy-matching > ): > <meta name=fuzzy content="maxDifference=15;totalPixels=300"> > There are cases where we’ll want to apply these to the tests > unconditionally, for example where varying behaviour is expected across > ports (such as anti-aliasing differences), and in these cases for WPT tests > these annotations should probably be exported upstream. > > The current plan, and work is underway to do this, is to support this > syntax via parsing the HTML in Python when there is a hash mismatch, which > should minimise the performance impact versus always reading this metadata. > > However, this doesn’t entirely suffice. There are cases where we might > want to allow more tolerance on one platform or another, or vary based on > GPU model or driver. As such, this requires not only platform specific > metadata (i.e., similar to that which we have in TestExpectations files > today), but also expectations with finer granularity. > Are we sure we really need that? What are examples of tests that do warrant such a mechanism? Generally, we want to keep our testing infrastructure as simple as possible. One option is to extend the meta content to encode conditional variants, > though this doesn’t work for WPT tests (unless we get buy-in to upstream > these annotations into the upstream repo, though that might be desirable > for the sake of results on wpt.fyi). We would need to be confident that > this wouldn’t become unwieldy however; we wouldn’t want to end up with > something like > (if:port=Apple)maxDifference=1;totalPixels=10,(if:platform=iOS)maxDifference=10;totalPixels=20,(if:port=GTK)maxDifference=10;totalPixels=300. > > Another option is to extend TestExpectations to store more specific data > (though again this might become unwieldy, as we’re unlikely to add new > “platforms” based on every variable we might want to distinguish results > on). This also means the metadata is far away from the test itself, and the > TestExpectations files would continue to grow even further (and we already > have 34k lines of TestExpectations data!). TestExpectations is also a > rather horrible file format to modify the parser of. > I'm fine with either of the above options but I don't think we should introduce this kind of micro syntax if we're going with meta. We should probably specify a platform in a different attribute altogether. e.g. <meta name="fuzzy" content="platforms=mac-bigsur; maxDifference=15; totalPixels=300"> I really hate that WPT is using a micro-syntax for this. Why isn't this simply a different content attribute like this: <meta name="fuzzy" platforms="mac-bigsur" max-difference="15" total-pixels="300"> There is also test-options.json which has most of the same downsides as > TestExpectations, albeit without the pain in modifying the parser. > > Finally, we could add per-test or per-directory files alongside the tests. > (Due to how things work, these could presumably also be in directories in > platform/.) This I think is probably the best option as it keeps the > metadata near the test, without needing to modify the test (which, per > above, is problematic for WPT as we move to automatically exporting > changes). One could imagine either a __dir__-metadata.json (to use a > similar name to how WPT names directory-level metadata files) or a > -expected-fuzzy.json file alongside each test. > Both of these two options seem worse than either encoding in the test or putting in the test expectations. They invent a brand new mechanism to store metadata for tests. We don't want to introduce yet another file / mechanism people need to be aware of. - R. Niwa
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
