Re: [webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)

2019-11-05 Thread Devin Rousso
Hi,

I may be alone here, but I actually found build/test(s) failure emails to be 
super useful.  The part that I did not find useful was that each time a 
build/test(s) would fail on a particular platform, there would be two 
comments/emails, which furthermore split the useful information into two 
pieces.  IIRC, the first comment/email would contain information about what 
build/test(s) failed, and the second comment/email was just an archive upload 
of the layout test results (if applicable).

I would find it very useful to get an email each time one of the bots had 
something go "wrong", and ideally include a link directly to the 'errors' log 
that Aakash mention in the beginning of this thread (or to the layout test 
results in the case of a test(s) failure).  Right now, if something goes 
"wrong", I have to be the one to discover it and then click through a few pages 
until I find the data that will actually help me solve the problem.

As a patch author, this is useful in letting me not have to remember to check 
my patch once a day in order to see if there are any red bubbles (something 
which I've forgotten to do, leaving a broken patch up for review thinking all 
was fine).  I can upload a patch and ideally leave it be, as I will be notified 
if the build/test(s) fails.

As a patch reviewer, I'm not sure this would be as useful, as I have a saved 
query that I use for finding all patches that need to be reviewed, and on a 
search result page there are no EWS bubbles anyways, so I can't see the info 
there.  I have to click into the bug regardless.  To Ryosuke's point, it may 
still be useful to get an email as a reviewer simply so I know that something 
failed and can potentially remember that fact before I click on a bug in the 
search results page, but I know personally that my mind doesn't really work 
that way :P

Thanks,
Devin

> On Nov 5, 2019, at 09:04, Alexey Proskuryakov  wrote:
> 
> 
> 
>> 4 нояб. 2019 г., в 1:37 PM, Ryosuke Niwa > > написал(а):
>> 
>> 
>> On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov > > wrote:
>> 
>> Can you elaborate on that, how exactly is e-mailing on first failure useful 
>> to reviewers?
>> 
>> Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based 
>> on engineering feedback about noise in bugs and in e-mail, and I 
>> wholeheartedly agree with this feedback. So I think that comments are 
>> generally undesirable.
>> 
>> Since I don't understand what your precise scenario is, I may be make straw 
>> man arguments below, but here are some things that I think make the proposed 
>> behavior unhelpful (add a comment on first failure, or when all EWSes pass).
>> 
>> 1. EWS comments in Bugzilla are so annoying that some people take the 
>> radical step of manually hiding them. EWS history is archived anyway, there 
>> is no need to look into comments for it.
>> 
>> 2. There are often many people CC'ed on the bug to whom EWS data is 
>> irrelevant or even mysterious (e.g. reporters, web developers or 
>> non-reviewers). The noise is a slight annoyance, discouraging further 
>> participation in the project.
>> 
>> 3. I believe that for most reviewers, the mode of operation is one of the 
>> two: (1) do it when pinged directly, or (2) go over the review queue when 
>> one has the time. Getting EWS comments helps neither.
>> 
>> 4. Commenting when all EWSes pass is not very practical - it's too often 
>> that we have some stragglers that take days (or forever). I don't think that 
>> we can make it reliable even if we start actively policing EWS 
>> responsiveness.
>> 
>> 5. The reviewer likely wants to know the state of multiple EWSes if they are 
>> going to wait for EWS at all. What exactly are they going to do after 
>> getting an e-mail that one EWS failed?
>> 
>> I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, 
>> a bug mail will stay in my inbox as one of items to get to.
>> 
>> I can see the usefulness in the somewhat unusual case of a super urgent 
>> patch. We may want multiple people to watch it, so that members of CC list 
>> would go and ask the patch author to update it with more urgency than e-mail 
>> allows for. I think that opt-in is a better mechanism for that, so that 
>> people who opted in would receive information about each EWS data point.
>> 
>> I think there is a value in knowing that a patch isn't ready instead of 
>> having to open the bug to realize that.
> 
> So just to clarify, 
> 
> - a major part of how you get to review bugs is by being CC'ed, and you 
> review them when you have the time to read bugmail;
> - and you don't open the bug in Bugzilla if there is already an EWS failure 
> by the time you read the e-mail where review is requested?
> 
> That's clearly a valid benefit. In my mind, it probably doesn't outweigh the 
> downsides. On the other hand, yours is a voice of someone who reviews way 
> more patches than Maciej and me 

Re: [webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)

2019-11-05 Thread Alexey Proskuryakov


> 4 нояб. 2019 г., в 1:37 PM, Ryosuke Niwa  написал(а):
> 
> 
> On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov  > wrote:
> 
> Can you elaborate on that, how exactly is e-mailing on first failure useful 
> to reviewers?
> 
> Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based 
> on engineering feedback about noise in bugs and in e-mail, and I 
> wholeheartedly agree with this feedback. So I think that comments are 
> generally undesirable.
> 
> Since I don't understand what your precise scenario is, I may be make straw 
> man arguments below, but here are some things that I think make the proposed 
> behavior unhelpful (add a comment on first failure, or when all EWSes pass).
> 
> 1. EWS comments in Bugzilla are so annoying that some people take the radical 
> step of manually hiding them. EWS history is archived anyway, there is no 
> need to look into comments for it.
> 
> 2. There are often many people CC'ed on the bug to whom EWS data is 
> irrelevant or even mysterious (e.g. reporters, web developers or 
> non-reviewers). The noise is a slight annoyance, discouraging further 
> participation in the project.
> 
> 3. I believe that for most reviewers, the mode of operation is one of the 
> two: (1) do it when pinged directly, or (2) go over the review queue when one 
> has the time. Getting EWS comments helps neither.
> 
> 4. Commenting when all EWSes pass is not very practical - it's too often that 
> we have some stragglers that take days (or forever). I don't think that we 
> can make it reliable even if we start actively policing EWS responsiveness.
> 
> 5. The reviewer likely wants to know the state of multiple EWSes if they are 
> going to wait for EWS at all. What exactly are they going to do after getting 
> an e-mail that one EWS failed?
> 
> I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, a 
> bug mail will stay in my inbox as one of items to get to.
> 
> I can see the usefulness in the somewhat unusual case of a super urgent 
> patch. We may want multiple people to watch it, so that members of CC list 
> would go and ask the patch author to update it with more urgency than e-mail 
> allows for. I think that opt-in is a better mechanism for that, so that 
> people who opted in would receive information about each EWS data point.
> 
> I think there is a value in knowing that a patch isn't ready instead of 
> having to open the bug to realize that.

So just to clarify, 

- a major part of how you get to review bugs is by being CC'ed, and you review 
them when you have the time to read bugmail;
- and you don't open the bug in Bugzilla if there is already an EWS failure by 
the time you read the e-mail where review is requested?

That's clearly a valid benefit. In my mind, it probably doesn't outweigh the 
downsides. On the other hand, yours is a voice of someone who reviews way more 
patches than Maciej and me combined these days, so maybe more e-mail is an 
overall benefit to many of the reviewers.

- Alexey



> - R. Niwa
>> 3 нояб. 2019 г., в 6:58 PM, Maciej Stachowiak > > написал(а):
>> 
>> 
>> I think they are useful to actual and potential reviewers. Direct email to 
>> the patch author is not something anyone can Cc themselves on, and is not 
>> archived, so seems like a strictly worse form of communication.
>> 
>>> On Nov 2, 2019, at 9:34 AM, Alexey Proskuryakov >> > wrote:
>>> 
>>> 
>>> My preference is still e-mailing the patch author directly (possibly, also 
>>> having an option to opt in for anyone). Bugzilla comments will always be 
>>> irrelevant for most people CC'ed on the bug, and they are almost always 
>>> undesirable to keep within the discussion flow.
>>> 
>>> - Alexey
>>> 
 1 нояб. 2019 г., в 18:28, Aakash Jain >>> > написал(а):
 
 Sounds good. I prefer the single comment when the first failure occur. 
 That way notification would be sent as soon as the first failure happens.
 
 I'll implement that (assuming it's acceptable to everyone).
 
 Thanks
 Aakash
 
> On Nov 1, 2019, at 8:35 PM, Maciej Stachowiak  > wrote:
> 
> 
> How about only a single comment when the first failure occurs? (Or else 
> when all bots pass, if there is never a failure.)
> 
> This should help the author, the reviewer, and anyone else cc’d, without 
> being too spammy.
> 
>> On Nov 1, 2019, at 5:20 PM, Aakash Jain > > wrote:
>> 
>> Hi Ryosuke,
>> 
>> Many people didn't like the noise by the EWS comments, and we removed 
>> the comments as per previous discussion in: 
>> https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html 
>> .
>> 
>> I agree with your point that having some kind of 

Re: [webkit-dev] iOS EWS behind by 3 days??

2019-11-05 Thread Alexey Proskuryakov


> 4 нояб. 2019 г., в 11:30 PM, Ryosuke Niwa  написал(а):
> 
> Hi all,
> 
> Does anyone know what's happening with iOS EWS?

Not yet, but it's part of a cluster of problems related to interactions between 
simulator and specific macOS host versions.

https://bugs.webkit.org/show_bug.cgi?id=203792 tracks this particular issue.

- Alexey


> They're ~3 days behind now:
> https://ews-build.webkit.org/#/builders/24 
> 
> 
> - R. Niwa
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev