Re: [webkit-dev] EWS doesn't lie!
All of this makes sense, but the downsides is that it does not ensure convergence between the ports. If this situation is expected to stay for some time, I wonder whether some middle ground may be found for ports that are build-stable. For instance, having status information on new/modified tests for all ports would be useful and not that expensive. When tests do not pass on some ports, updating the test expectations (and creating bug entries) would be an improvement over the current situation. 2014-02-10 10:25 GMT+01:00 Osztrogonác Csaba : > On 02/10/2014 09:51 AM, youenn fablet wrote: > >> Is it by design that only mac bots run regression tests? Technical >> issue? Lack of resources? >> > > Technically, it's so easy to make an EWS to run layout tests > too with adding a "runTests": true" to the ews.json file: > https://trac.webkit.org/browser/trunk/Tools/Scripts/ > webkitpy/common/config/ews.json > > But in my opinion it wouldn't be a good idea to enable layout tests > on Windows, GTK, EFL EWS bots, because it would make them absolutely > useless and we would lost the information if a patch builds or not. > > To have a quite stable and working tester EWS, the buildbot for the > given platform must be green _almost all the time_. If there is at > least one failing test, the testing is at least twice slower, because > the EWS runs the test with the patch and then without the patch to > check if the list of the failing tests are same. Additionally the > given port must be very stable. If there are any small flakiness, > the EWS wouldn't pass ever and would stuck in an infinite loop. > > The rough true is that now only the Mac platform is stable and green > enough to have tester EWS bots. (There are ~210 +/-5 failures on the > Windows bots from the cstack merge, ~205 +/-10 failures on EFL-WK1 long > time ago, ~80 +/- 2 failures on GTK-WK1 lone time ago, ~ 60 +/- 5 > failures on GTK-WK2, ...) > > Additionally to have tester EWS, port maintainers should have to setup > many new hardware (min. 4-8 machines with 4/8 cores per port to have > acceptable runtime) and EWS runtime would be much more slower than > the runtime of build only EWS bots, because bulding + running tests > take ~ an hour everywhere. > > Ossy > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] EWS doesn't lie!
On 02/10/2014 09:51 AM, youenn fablet wrote: Is it by design that only mac bots run regression tests? Technical issue? Lack of resources? Technically, it's so easy to make an EWS to run layout tests too with adding a "runTests": true" to the ews.json file: https://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/ews.json But in my opinion it wouldn't be a good idea to enable layout tests on Windows, GTK, EFL EWS bots, because it would make them absolutely useless and we would lost the information if a patch builds or not. To have a quite stable and working tester EWS, the buildbot for the given platform must be green _almost all the time_. If there is at least one failing test, the testing is at least twice slower, because the EWS runs the test with the patch and then without the patch to check if the list of the failing tests are same. Additionally the given port must be very stable. If there are any small flakiness, the EWS wouldn't pass ever and would stuck in an infinite loop. The rough true is that now only the Mac platform is stable and green enough to have tester EWS bots. (There are ~210 +/-5 failures on the Windows bots from the cstack merge, ~205 +/-10 failures on EFL-WK1 long time ago, ~80 +/- 2 failures on GTK-WK1 lone time ago, ~ 60 +/- 5 failures on GTK-WK2, ...) Additionally to have tester EWS, port maintainers should have to setup many new hardware (min. 4-8 machines with 4/8 cores per port to have acceptable runtime) and EWS runtime would be much more slower than the runtime of build only EWS bots, because bulding + running tests take ~ an hour everywhere. Ossy ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] EWS doesn't lie!
On Feb 10, 2014, at 12:51 AM, youenn fablet wrote: > Is it by design that only mac bots run regression tests? Certainly not. > Technical issue? I think so; look at http://build.webkit.org/dashboard/; the Mac port is the only one that’s even close to green enough to make running the tests on EWS worthwhile (i.e. insufficiently noisy that new failures are possible to distinguish). > Lack of resources? Could be this too, but I can’t speak for the other ports. It takes a good number of machines to keep up (and Mac starts falling behind as soon as someone introduces a flaky test because of EWS’s slightly odd machinery). > > 2014-01-30 9:05 GMT+01:00 Alexey Proskuryakov : > Hi WebKit hackers, > > It sometimes happens that people land patches despite EWS detecting layout > test regressions, especially when these seem too unlikely to believe. > > In my experience, EWS has been very stable recently, and if tester bubbles > are red, it almost certainly means that the patch is faulty. Even if they > don't turn red, but remain yellow for a long time, there is a good chance > that the patch introduces flaky failures, so EWS can't make up its mind about > exactly which tests regressed. > > If you click on a yellow bubble, that takes you to a page with additional > details, where you can see which tests are failing. I'd like to look into > improving how this information is presented at some point in the future, yet > even now, it shouldn't be too time consuming to check what's going on. > > For reference, we currently have mac and mac-wk2 EWS bots running regression > tests in release mode. Other bots only verify that the patch builds, and > don't run tests. No bots use debug mode as far as I know, so debug-only build > failures and assertions will not be detected. Please run tests locally to > catch as many of those as possible. > > - WBR, Alexey Proskuryakov > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] EWS doesn't lie!
Is it by design that only mac bots run regression tests? Technical issue? Lack of resources? 2014-01-30 9:05 GMT+01:00 Alexey Proskuryakov : > Hi WebKit hackers, > > It sometimes happens that people land patches despite EWS detecting layout > test regressions, especially when these seem too unlikely to believe. > > In my experience, EWS has been very stable recently, and if tester bubbles > are red, it almost certainly means that the patch is faulty. Even if they > don't turn red, but remain yellow for a long time, there is a good chance > that the patch introduces flaky failures, so EWS can't make up its mind > about exactly which tests regressed. > > If you click on a yellow bubble, that takes you to a page with additional > details, where you can see which tests are failing. I'd like to look into > improving how this information is presented at some point in the future, > yet even now, it shouldn't be too time consuming to check what's going on. > > For reference, we currently have mac and mac-wk2 EWS bots running > regression tests in release mode. Other bots only verify that the patch > builds, and don't run tests. No bots use debug mode as far as I know, so > debug-only build failures and assertions will not be detected. Please run > tests locally to catch as many of those as possible. > > - WBR, Alexey Proskuryakov > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] EWS doesn't lie!
Hi WebKit hackers, It sometimes happens that people land patches despite EWS detecting layout test regressions, especially when these seem too unlikely to believe. In my experience, EWS has been very stable recently, and if tester bubbles are red, it almost certainly means that the patch is faulty. Even if they don't turn red, but remain yellow for a long time, there is a good chance that the patch introduces flaky failures, so EWS can't make up its mind about exactly which tests regressed. If you click on a yellow bubble, that takes you to a page with additional details, where you can see which tests are failing. I'd like to look into improving how this information is presented at some point in the future, yet even now, it shouldn't be too time consuming to check what's going on. For reference, we currently have mac and mac-wk2 EWS bots running regression tests in release mode. Other bots only verify that the patch builds, and don't run tests. No bots use debug mode as far as I know, so debug-only build failures and assertions will not be detected. Please run tests locally to catch as many of those as possible. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
