Hi, Some of you might have noticed that since a few days ago on the Bugzilla there is a new EWS bubble for GTK WK2 layout tests.
Until now the GTK port was without layout test coverage on the EWS. It has been a challenge to add this testers on the EWS because the GTK port layout tests are usually not 100% green and there are usually unexpected failures and also flakies. This means that the EWS was usually either reporting false positives (due to flakies) or exiting early (because of more than 30 unexpected failures on the clean tree or more than 30 accumulated between the ones caused by patch and the ones in the clean tree) We tried to do extra gardening efforts to try to mark all the flakies and also to try to keep it always green, but for several reasons we couldn't keep it on this always-green status for more than a few consecutive days. Also new flakies would keep appearing. So we ended deploying a different version of the EWS that has a much higher tolerance to pre-existent failures (up to 500 before exiting early) and also that tries hard to discard pre-existent failures and flakies by repeating each failure 10 times with patch and 10 times without it. [1] This new version of the EWS WK2 layout tester is only used on the GTK port for now, and we plan to use it also for the WPE port when we deploy layout tests on the EWS for it. So this has the advantage that this EWS should not report false positives or flakies. Any failure reported by it should be a consistent new failure (fails always with the patch but never without it after 10 runs with patch and 10 without it). If you see it reporting any false positive please let me know it. However, it has also the disadvantage that the view to see the layout test results is a bit confusing because you see there also failures unrelated with your patch (those are either pre-existent failures or flakies) My tip here is to only pay attention to the test failures reported by the EWS as new, which are those that are shown on the Bugzilla bubble when you hover the mouse over it (or those that appear on the e-mail that you receive from the EWS) Perhaps I'm thinking it can be a further improvement to add an extra step to do a final run with the patch just for the list of new failures, in order to present a clean view of the results with only the new failures. This can be also very useful to be able to properly use the script update-test-expectations-from-bugzilla to automatically apply the new test expectations from the EWS locally. Any comments welcome :) Best regards! [1] https://bugs.webkit.org/show_bug.cgi?id=231999 _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
