Re: [webkit-dev] Requesting feedback about EWS comments on Bugzilla bugs

[Aakash Jain]

> On Jun 16, 2019, at 2:14 PM, Darin Adler  wrote:
> 
>> On Jun 15, 2019, at 9:13 PM, Aakash Jain > > wrote:
>> 
>> 1) Do not upload archive (for layout-test-results) on bugzilla, instead 
>> upload it to another server, unzip it and post a link to the results.html.
>> Pros:
>> a) Engineers won't have to download the attachment, unzip it, look for 
>> failures, and then delete it from their disk. They can simply click the url 
>> to view the results. 
>> b) This approach will also reduce 2 comments per failure to 1 comment. 
>> Currently there are two comments per failure, one for failure details, 
>> second for bugzilla attachment.
> 
> Great improvement to do this. The most confusing thing about build bot 
> comments is all the “creation of attachments” extra text with things like 
> “attachment number” and “patch".
> 
> However, it’s really nice that I can download a directory full of test 
> results easily. I’d like to see the EWS website still have that feature.

Sure, will keep an option to download the test results archive as well.

> 
>> 4) When a patch becomes 'obsolete', tag the corresponding EWS comments as 
>> 'obsolete', so that they will be hidden.
> 
> Incredibly valuable.
> 
>> 5) Do not comment on bugzilla bug at all
> 
> I think this makes sense. I don’t see a reason that test results need to be 
> comments. I think the “red bubble” in EWS already calls someone’s attention 
> to failures.

What do you think about comments for 'Style' failures, is that ok to keep, or 
should we remove them as well?

> If we want to augment it, we should think of what we are aiming at. I do find 
> it useful to see which tests are failing, and when I click on the red bubble 
> I don’t see that information. I have to click once to see the “log of 
> activities” then click on “results”, then see a confusing giant file with 
> lots of other information. At the bottom of that file the one thing I want to 
> know.
> 
> A better hierarchy is to put that “what new tests are failing” summary right 
> t the top and let the logs be fallbacks, not the primary place to see the 
> features.

This brings another interesting question, what page should be displayed on 
clicking an EWS bubble?

a) old EWS style page, e.g.: 
https://webkit-queues.webkit.org/patch/362845/ios-sim-ews 
. This does have 
deficiencies like you mentioned.

b) Buildbot page, e.g.: https://ews-build.webkit.org/#/builders/3/builds/414 
, currently this is new 
EWS behavior (but this doesn't cover the case when there are multiple builds 
(e.g.: retries) for a patch on a given queue).

c) Newly designed page which shows the summary of failures, have link(s) to the 
Buildbot page(s), have link to download test result archives, and maybe summary 
of major build steps which were executed. More feedback regarding the design of 
this page would be useful.

> 
>> instead send email to the author of the patch.
> 
> Why? I don’t think this should send any emails at all, unless the person 
> requested it.

ok

> 
>> Pros: less noisy, also this will allow to include more detailed information 
>> about the failure in email.
> 
> I think the more detailed information should be on the webpage, not in an 
> email.
> 
>> Cons: reviewers would have to click status-bubbles to see the failures, 
>> failure information is not immediately present in the comments.
> 
> I think we should start with this approach, eliminating the comments entirely.

Sure, let's do this. I wouldn't change the old EWS (which is being replaced), 
but new EWS will have this behavior.

> 
> — Darin

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Requesting feedback about EWS comments on Bugzilla bugs

[Aakash Jain]

> On Jun 17, 2019, at 1:52 PM, Keith Rollin  wrote:
> 
>> On Jun 16, 2019, at 11:14, Darin Adler  wrote:
>> 
>> If we want to augment it, we should think of what we are aiming at. I do 
>> find it useful to see which tests are failing, and when I click on the red 
>> bubble I don’t see that information. I have to click once to see the “log of 
>> activities” then click on “results”, then see a confusing giant file with 
>> lots of other information. At the bottom of that file the one thing I want 
>> to know.
> 
> We might want to also start turning those failure into action items. We could 
> have an automatic mechanism that gathers the failures, records them in a 
> database, and then — with sufficient data — makes determinations about the 
> flakiness or other status of the test. It could then mark the test as flaky 
> or raise it as an issue to some responsible (and responsive) party.

Agree. This is the plan. Jonathan Bedard is working on an improved flakiness 
dashboard. Once we have that, EWS will start using it's API to get the test 
flakiness information. That should significantly reduce EWS's false positives 
(and also reduce the number of retries EWS has to do while trying to rule out 
flakiness).

> 
> We could also have a relatively manual process. The failures are surfaced in 
> Bugzilla or in a Bugzilla-accessible page. The engineer posting the patch 
> could then review the failures and mark them as “Flag as flaky”, “Flag as 
> failing and should be fixed by someone else”, “Flag as failing and should be 
> ignored”, etc. These responses could then be turned into action items for 
> some responsible (and responsive) party to address.
> 
> As Michael says, there’s a big issue with ignoring test results. Putting a 
> frictionless process in place to address test results would help make them 
> more effective. When I make a change to an Xcode project and Windows builds 
> throw up errors, that’s not something caused by my immediate patch, but I 
> would like to see the flaky test fixed.
> 
> — Keith
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev