Re: [webkit-dev] Running new/modified tests on EWS bots
On Fri, Mar 20, 2015 at 7:21 PM, Gyuyoung Kim wrote: > Though I really like to support to run tests on EWS like Mac port, it can > often block to land a patch if tests on EFL or GTK aren't green. So I think > we need to find methods which don't annoy patch owner or reviewer. In this > light, if we're able to check if there is different between last test > result and new test result with a patch as Carlos suggested, it seems to be > more realistic approach. If someone tries to support it, I will also help > to try it If I can. > EWS does this already (or at least attempts to do so). Of course, it could still fail if there are flaky tests that aren't marked as such in the test expectations. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Running new/modified tests on EWS bots
Hi, Though I really like to support to run tests on EWS like Mac port, it can often block to land a patch if tests on EFL or GTK aren't green. So I think we need to find methods which don't annoy patch owner or reviewer. In this light, if we're able to check if there is different between last test result and new test result with a patch as Carlos suggested, it seems to be more realistic approach. If someone tries to support it, I will also help to try it If I can. Gyuyoung. On Fri, Mar 20, 2015 at 2:41 AM, Carlos Alberto Lopez Perez < clo...@igalia.com> wrote: > On 19/03/15 16:46, youenn fablet wrote: > > Hi, > > > > Related to the webkit contributor meeting discussion related to ports, > > I would find it useful if EWS bots (gtk, efl, win, ios) were running > > the tests that are modified/created by a patch. > > > > The idea would be to turn yellow the port bubble whenever one of these > > tests do not pass. Results would be uploaded to bugzilla. > > > > This would give an incentive for patch developers to try fixing the > > tests on these ports. > > That may reduce (slightly? noticeably?) port maintainers gardening > effort. > > That would also be valuable when importing test suites. > > > > Any potential issue? objection to move that forward? > > Anyone willing to help? Thoughts? > > > > I think that having an EWS running only the tests that the patch touches > is of limited value because a patch can break unrelated tests. > > However running all the tests requires to have the tree green and this > is not possible for GTK or EFL because it will require more manpower > than the currently available. > > An idea that comes to mind for running all the tests without requiring > to have the tree green is the following: > > 1) EWS either run all the tests without the patch or download from > https://build.webkit.org/results the results from the bot for the > revision if they are available. > 2) EWS run all the tests with the patch. > 3) EWS gets the diff from 1. and 2. > 4) EWS runs only the tests that broke from 1 to 2 several times, in > order to discard the ones that are flaky and only report the ones that > on every run fail now. > > If I'm not overlooking something, I think that this would allow to > identify what tests a given patch breaks and it won't require to have > the tree green, neither it will be affected by flaky tests. > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Running new/modified tests on EWS bots
19 марта 2015 г., в 8:46, youenn fablet написал(а): > Hi, > > Related to the webkit contributor meeting discussion related to ports, > I would find it useful if EWS bots (gtk, efl, win, ios) were running > the tests that are modified/created by a patch. > > The idea would be to turn yellow the port bubble whenever one of these > tests do not pass. Results would be uploaded to bugzilla. > > This would give an incentive for patch developers to try fixing the > tests on these ports. > That may reduce (slightly? noticeably?) port maintainers gardening effort. > That would also be valuable when importing test suites. > > Any potential issue? objection to move that forward? > Anyone willing to help? Thoughts? I think that it's a good idea. The infrastructure added for this could be reused for something that we wanted for a long time - on the Mac, we should run newly added tests multiple times and also with GuardMalloc enabled, to reduce the chances of them being flaky, or uncovering memory corruption. That's a low cost in terms of EWS performance. - Alexey ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Running new/modified tests on EWS bots
On 19/03/15 16:46, youenn fablet wrote: > Hi, > > Related to the webkit contributor meeting discussion related to ports, > I would find it useful if EWS bots (gtk, efl, win, ios) were running > the tests that are modified/created by a patch. > > The idea would be to turn yellow the port bubble whenever one of these > tests do not pass. Results would be uploaded to bugzilla. > > This would give an incentive for patch developers to try fixing the > tests on these ports. > That may reduce (slightly? noticeably?) port maintainers gardening effort. > That would also be valuable when importing test suites. > > Any potential issue? objection to move that forward? > Anyone willing to help? Thoughts? > I think that having an EWS running only the tests that the patch touches is of limited value because a patch can break unrelated tests. However running all the tests requires to have the tree green and this is not possible for GTK or EFL because it will require more manpower than the currently available. An idea that comes to mind for running all the tests without requiring to have the tree green is the following: 1) EWS either run all the tests without the patch or download from https://build.webkit.org/results the results from the bot for the revision if they are available. 2) EWS run all the tests with the patch. 3) EWS gets the diff from 1. and 2. 4) EWS runs only the tests that broke from 1 to 2 several times, in order to discard the ones that are flaky and only report the ones that on every run fail now. If I'm not overlooking something, I think that this would allow to identify what tests a given patch breaks and it won't require to have the tree green, neither it will be affected by flaky tests. signature.asc Description: OpenPGP digital signature ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Running new/modified tests on EWS bots
O Xov, 19-03-2015 ás 16:46 +0100, youenn fablet escribiu: > Related to the webkit contributor meeting discussion related to ports, > I would find it useful if EWS bots (gtk, efl, win, ios) were running > the tests that are modified/created by a patch. > > The idea would be to turn yellow the port bubble whenever one of these > tests do not pass. Results would be uploaded to bugzilla. AFAIK, the only EWS bots running all tests are the Mac ones. It is ok to have EWS running the at least the affected tests and I guess that shouldn't take that much computation power (maybe some development to know which tests need to be run). What would be really nice is having all tests run, but for that I guess tests bots need to be green, which is a pace very difficult to maintain for smaller ports. Br. signature.asc Description: This is a digitally signed message part ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Running new/modified tests on EWS bots
This seems to me like an obvious step to take. I'd much rather know about tests that my patch breaks before pushing rather than afterward. I guess the potential issue would be build power. Michael ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Running new/modified tests on EWS bots
Hi, I'd like this idea basically, it would be great if you or somebody else could work on it. One of my former collegue started to investigate the same thing 1.5 years before, but it was lost among many tasks with higher priorities. Feel free to reuse these bugs and WIP patches: (or file new ones) - https://bugs.webkit.org/show_bug.cgi?id=117831 - and bugs in its "depends on" list In my opinion red tree and flakey tets could cause many problems, because EWS is designed primarily for green tree. But unfortunately only Mac (and now Win release) bots are green almost all the time. I can't remember if EFL or GTK bots were green in the last 1-2 years ever. :-/ Now the tester EWS bots (Mac WK1 and WK2) run all tests. If all tests pass, there is no problem. But if there is only one failure, it tries to build and run tests without the patch. If all tests pass, the patch caused the regression. If there are exactly the same failures on the trunk and with the patch, the bubble becomes green. But flakey failures on the trunk can make EWS crazy. I remember that EWS wasn't able to process any patch for days because of 1-2 flakey failures on the trunk. If we enabled full or partial testing on unstable EWSs, they would become very slow, unstable and we would lost the very quick and stable builder EWSs. Maybe we should have separated builder and builder-and-tester EWS bots to avoid this kind of problems, but we would need more hardware for it. Anyway it would be great if EFL and GTK maintainers could keep the tester bots green almost all the time. I know it isn't an easy task and would need more resource. I remember from the QtWebKit era that 1-2 full-time gardener can be enough to keep the tree green, but it means only minor buildfixes, expected updates, new bug reports and skipping/marking new failures time to time. To increase the stabilility, fix regressions, decrease the number of skipped tests would need much more resource. Of course, I willingly help you, but unfortunately I don't have too much time for this topic nowadays. But feel free to cc me to related bugs and I'll try to help if I can. br, Ossy youenn fablet írta: Hi, Related to the webkit contributor meeting discussion related to ports, I would find it useful if EWS bots (gtk, efl, win, ios) were running the tests that are modified/created by a patch. The idea would be to turn yellow the port bubble whenever one of these tests do not pass. Results would be uploaded to bugzilla. This would give an incentive for patch developers to try fixing the tests on these ports. That may reduce (slightly? noticeably?) port maintainers gardening effort. That would also be valuable when importing test suites. Any potential issue? objection to move that forward? Anyone willing to help? Thoughts? Thanks, Youenn ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Running new/modified tests on EWS bots
Hi, Related to the webkit contributor meeting discussion related to ports, I would find it useful if EWS bots (gtk, efl, win, ios) were running the tests that are modified/created by a patch. The idea would be to turn yellow the port bubble whenever one of these tests do not pass. Results would be uploaded to bugzilla. This would give an incentive for patch developers to try fixing the tests on these ports. That may reduce (slightly? noticeably?) port maintainers gardening effort. That would also be valuable when importing test suites. Any potential issue? objection to move that forward? Anyone willing to help? Thoughts? Thanks, Youenn ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
