On Mon, Aug 1, 2022 at 10:36 AM Vivian Zhi (支文文） <[email protected]> wrote:
> Thanks for valuable feedback! Stephen, Xianzhu, will see if we can add a > filter in result.html to grab those tests in range. > The CL <https://chromium-review.googlesource.com/c/chromium/src/+/3803707> adding pixel diff filter in results.html has landed. Thanks Thorben! In this example results.html <https://test-results.appspot.com/data/layout_results/linux-rel/1086271/blink_web_tests%20%28with%20patch%29/layout-test-results/results.html>, you can examine the pixel results of tests that produced pixel differences matching a particular fuzzy rule in the following steps: 1. Enter pixel difference filter e.g. "channel_max:1-1" in the filter input box; 2. Click "All" button (as we show regressions only by default). You might want to switch to "side-by-side view" and click the image to examine the pixel values. With "channel_max:1-1" we can see all tests that produced pixel differences that can be tolerated with a fuzzy rule like <meta name=fuzzy content="0-1;0-1000000">. There are 70 such tests in the example results.html. All of them look benign to me. So perhaps a universal rule (for non wpt tests) is proper? On the other hand, even if we have such a universal rule, we can only recover 70 tests. Instead of applying the rule automatically, we can also manually modify these tests to include a meta fuzzy rule. > On Mon, Aug 1, 2022 at 8:40 AM Xianzhu Wang <[email protected]> > wrote: > >> On Mon, Aug 1, 2022 at 4:25 AM Stephen Chenney <[email protected]> >> wrote: >> >>> Thanks for investigating the potential for fuzzy matching. >>> >>> Rendering Core continues to oppose a single fuzzy match rule across all >>> web_tests. We have some tests where single pixel differences matter >>> (related to pixel snapping, for example) and a universal fuzzy match would >>> fail to identify problems with those. This came up in practice recently >>> when the GPU team enabled fuzzy matching without telling us, and expected >>> failing tests started passing when they shouldn't. >>> >> >> I think a key difference between the original fuzzy matching rule and the >> rule proposed by Vivian is the ranges. With maxDifference=0-1, we should be >> able to catch most visible single pixel differences. What I'm not sure is >> whether a difference like rgb(1, 0, 0) vs rgb(0, 0, 0) (each component in >> the range of 0-255) should be treated as a failure in some cases. >> >> Maybe specific sub teams have directories they could apply default fuzzy >>> matching to. My guess is that the same directories where it will work will >>> be directories with few failing tests, limiting the impact of a >>> per-directory approach. >>> >>> Is there a way to reproduce the sampling below with a side-by-side >>> comparison of the images? I would find it helpful to look through some of >>> the cases that would pass with <meta name="fuzzy" content="0-1;0-1000">, >>> for example. >>> >> >> A filter by actual maxDifference and totalPixels in results.html will be >> helpful. I can add it when I get time. >> >> Stephen. >>> >>> On Fri, Jul 29, 2022 at 8:20 PM 'Vivian Zhi (支文文）' via blink-dev < >>> [email protected]> wrote: >>> >>>> Hi blink-dev >>>> >>>> I would like to let you know that blink-engprod has added feature >>>> support for non-WPT fuzzy tests. It now allows both non-WPT reftests and >>>> pixel tests to use the same fuzzy matching meta-tags as WPT tests.It also >>>> shows max color channel difference and total number of different pixels >>>> image diff stats in results.html >>>> <https://test-results.appspot.com/data/layout_results/linux-rel/1073794/blink_web_tests%20%28with%20patch%29/layout-test-results/results.html>. >>>> With these capabilities in place, we like to research further to see if we >>>> can set up some general fuzzy match rules, help blink dev identify flaky >>>> tests that can be potentially resolved by adjusting fuzzy matching rules. >>>> Currently there are quite some web tests that are flaky due to a slight >>>> image mismatch, which should have been tolerated. If we setup a general >>>> fuzzy matching rule , something like: >>>> >>>> <meta name="fuzzy" content="0-1;0-1000"> >>>> >>>> Instruct the image comparison web tests that if color channel and pixel >>>> diff fall within the range of the rule, we can ignore the diff and pass the >>>> test.This way we can reduce test flakiness while still maintain test >>>> accuracy without missing a real bug. >>>> >>>> We want to ask you some quick survey questions to help us make design >>>> decisions, whether it makes sense to set up an universal cross-the-board >>>> fuzzy match tolerant rule for all blink web tests, or we should make the >>>> rules more specific to individual test or test sets. >>>> >>>> 1. Is an universal fuzzy match tolerant rule acceptable for the web >>>> tests in your area? >>>> >>>> a). If the answer is yes, what is the acceptable range of max color >>>> channel and pixel diff for your tests? >>>> b) If the answer is no, pls share your reasons. >>>> >>>> 2. Do you prefer fuzzy matching rule adjustment at a per-test or per >>>> test set level based on the pixel difference numbers shown in results.html? >>>> >>>> Here is some sample data help you make choice, we collected data >>>> recently from blink_web_tests result on linux-test builder, the >>>> distribution of color channel maxDifference and totalPixel diff for >>>> failing/flaky blink_web_tests >>>> ( Note: over 70% tests in color channel maxDifference 0-10 range have >>>> maxDifference=1): >>>> >>>> Color Channel maxDifferenece >>>> Range Fail test count >>>> 0-10 98 >>>> 11-100 31 >>>> 101-200 28 >>>> 201-260 111 >>>> totalPixels >>>> Diff Range >>>> Fail test count >>>> 0-100 30 >>>> 100-1000 57 >>>> 1000-10,000 99 >>>> 10,000-100,000 66 >>>> 100,000-1,000,000 16 >>>> >>>> Let me know if you have any questions, looking forward to hearing from >>>> you! >>>> >>>> >>>> Vivian >>>> on behalf of Chrome-Blink-EngProd >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "blink-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPCqkTs-L5u22-Xp5U_LeBdLP%3D%2BTDH1KGv8MTmtKQFRcANCZJg%40mail.gmail.com >>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPCqkTs-L5u22-Xp5U_LeBdLP%3D%2BTDH1KGv8MTmtKQFRcANCZJg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGsbWzRDrX%3Dgz9NNcwpBEOXCxR37p2XwZC3Agm6fdE6%2BFcPhvg%40mail.gmail.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGsbWzRDrX%3Dgz9NNcwpBEOXCxR37p2XwZC3Agm6fdE6%2BFcPhvg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADBxriebkmh_5i4RDqJocTJY3KEdHQX%3Do_tWELssnw3Uwp01ew%40mail.gmail.com.
