Re: [webkit-dev] New EWS queue: Stress Test EWS

[Alexey Proskuryakov via webkit-dev]

Hi,

This issue is tracked as https://bugs.webkit.org/show_bug.cgi?id=225126 


- Alexey

> 28 апр. 2021 г., в 3:23 AM, Manuel Rego Casasnovas via webkit-dev 
>  написал(а):
> 
> Hi,
> 
> The stress test EWS has some issue when dealing with testharness.js tests.
> 
> Every now and then it thinks it's a different type of test and it dumps
> the layout tree, and it fails as the actual result has nothing to do
> with a layout tree dump.
> 
> Actually it dumps an empty layout tree:
> layer at (0,0) size 800x600
>   RenderView at (0,0) size 800x600
> layer at (0,0) size 800x600
>   RenderBlock {HTML} at (0,0) size 800x600
> RenderBody {BODY} at (8,8) size 784x584
> 
> Last example I've seen:
> https://ews-build.webkit.org/#/builders/62/builds/1903
> 
> I guess there might be some timing issue or something like that, as it
> looks like it doesn't even load the test properly before comparing the
> results.
> 
> Cheers,
>  Rego
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] New EWS queue: Stress Test EWS

[Manuel Rego Casasnovas via webkit-dev]

Hi,

The stress test EWS has some issue when dealing with testharness.js tests.

Every now and then it thinks it's a different type of test and it dumps
the layout tree, and it fails as the actual result has nothing to do
with a layout tree dump.

Actually it dumps an empty layout tree:
 layer at (0,0) size 800x600
   RenderView at (0,0) size 800x600
 layer at (0,0) size 800x600
   RenderBlock {HTML} at (0,0) size 800x600
 RenderBody {BODY} at (8,8) size 784x584

Last example I've seen:
https://ews-build.webkit.org/#/builders/62/builds/1903

I guess there might be some timing issue or something like that, as it
looks like it doesn't even load the test properly before comparing the
results.

Cheers,
  Rego
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] New EWS queue: Stress Test EWS

[Aakash Jain via webkit-dev]

Hi Everyone,

I am happy to announce that I have added a new EWS queue: Stress Test EWS. From 
now on, you would see a new status bubbles (on Bugzilla bug) named: 
'mac-wk2-stress'. For the patches that add or modify layout tests, this EWS 
will run those added/modified layout-tests repeatedly large number of times 
(currently 100 iterations). It will also run those added/modified layout-tests 
repeatedly in guard-malloc mode. Example: 
https://ews-build.webkit.org/#/builders/62/builds/249. The aim is to find any 
issue with the layout-tests, especially flakiness, or failure when run in a 
certain manner.

Flaky layout-tests are currently a huge problem, as they sometimes cause false 
positives on EWS (and other parts of infrastructure), and slow down our 
infrastructure (as for every test failure, EWS needs to run the test-suite few 
times to ensure the failures are not flaky or pre-existing). Until now, we 
didn't have any good automated way to indicate the test flakiness to the 
engineers. With this EWS, we are adding pre-commit testing for layout-test 
flakiness. We understand that layout tests might be flaky in a variety of ways, 
sometimes in a very specific or unexpected manner. I am looking at covering 
more and more scenarios in this EWS by making incremental improvements to it.

Few ideas to further improve this EWS are:
- Test on more OS/configurations (this EWS is currently using macOS release wk2 
mode)
- Run the test being added/modified repeatedly along-with other layout-tests in 
same directory
- Improve the logic to identify layout-tests, current logic is naive and only 
recognizes html layout-tests

More ideas are welcome.

Thanks
Aakash
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev