Re: A new testing policy for code landing in mozilla-central

2020-11-12 Thread Brian Grinstead
Hi Jeff, Thanks to help from Marco Castelluccio we now have an artifact in taskcluster that has metadata about all commits that land in m-c, which I've used to gather this data. We're still working on tooling to better understand patterns with different types of bugs & patches, but I'm happy to

Re: A new testing policy for code landing in mozilla-central

2020-11-12 Thread Jeff Muizelaar
Brian, Now that we've been doing this for a while, do we have any aggregated data on the testing tags? i.e. how often are we adding tests vs not. -Jeff On Tue, Sep 15, 2020 at 11:03 AM Brian Grinstead wrote: > > (This is a crosspost from firefox-dev) > > Hi all, > > We’re rolling out a change

Re: A new testing policy for code landing in mozilla-central

2020-09-24 Thread Brian Grinstead
> On Sep 24, 2020, at 7:02 AM, Kartikaya Gupta wrote: > > First - thank you for making these changes! I do find the testing > prompt useful as a reviewer, and it has already resulted in a few > extra tests being added where they probably wouldn't have been > otherwise. > > After living with

Re: A new testing policy for code landing in mozilla-central

2020-09-24 Thread Kartikaya Gupta
First - thank you for making these changes! I do find the testing prompt useful as a reviewer, and it has already resulted in a few extra tests being added where they probably wouldn't have been otherwise. After living with this for a week or so, I have two (relatively minor) papercuts: - I

Re: A new testing policy for code landing in mozilla-central

2020-09-16 Thread Brian Grinstead
> On Sep 16, 2020, at 7:12 AM, surkov.a...@gmail.com > wrote: > > Better test coverage is the best. Agreed, having a policy to force developers > to add tests must be an efficient way to improve test coverage. I worried > though it may reduce amount of contributions from the community since

Re: A new testing policy for code landing in mozilla-central

2020-09-16 Thread surkov.a...@gmail.com
Better test coverage is the best. Agreed, having a policy to force developers to add tests must be an efficient way to improve test coverage. I worried though it may reduce amount of contributions from the community since it complicates the development process: sometimes you have to spend more

Re: A new testing policy for code landing in mozilla-central

2020-09-16 Thread Andrew McCreight
On Wed, Sep 16, 2020 at 5:10 AM Eric Rescorla wrote: > I certainly have some sympathy for this, but I'm not sure that it needs to > be > addressed via tools. What I would generally expect in cases like this is > that the reviewer says "why isn't there a test for X" and the author says > "for

Re: A new testing policy for code landing in mozilla-central

2020-09-16 Thread Eric Rescorla
On Tue, Sep 15, 2020 at 9:28 AM Andrew McCreight wrote: > On Tue, Sep 15, 2020 at 8:03 AM Brian Grinstead > wrote: > > > (This is a crosspost from firefox-dev) > > > > Hi all, > > > > We’re rolling out a change to the review process to put more focus on > > automated testing. This will affect

Re: A new testing policy for code landing in mozilla-central

2020-09-15 Thread Brian Grinstead
> On Sep 15, 2020, at 11:44 AM, Andrew Halberstadt wrote: > > On Tue, Sep 15, 2020 at 12:28 PM Andrew McCreight > wrote: >> I don't know that tests being an official requirement relieves the burden > of guilt of asking for tests, as everybody already knows that tests are > good and that you

Re: A new testing policy for code landing in mozilla-central

2020-09-15 Thread Andrew Halberstadt
On Tue, Sep 15, 2020 at 12:28 PM Andrew McCreight wrote: > I don't know that tests being an official requirement relieves the burden of guilt of asking for tests, as everybody already knows that tests are good and that you should always write tests I think this is largely true at Mozilla, but we

Re: A new testing policy for code landing in mozilla-central

2020-09-15 Thread Brian Grinstead
> On Sep 15, 2020, at 8:48 AM, Jonathan Kew wrote: > > On 15/09/2020 16:03, Brian Grinstead wrote: > > > We’re rolling out a change to the review process to put more focus on > > automated testing. [...] > > As others have said, it's great that we're setting a clearer policy here - >

Re: A new testing policy for code landing in mozilla-central

2020-09-15 Thread Andrew McCreight
On Tue, Sep 15, 2020 at 8:03 AM Brian Grinstead wrote: > (This is a crosspost from firefox-dev) > > Hi all, > > We’re rolling out a change to the review process to put more focus on > automated testing. This will affect you if you review code that lands in > mozilla-central. > > ## TLDR > >

Re: A new testing policy for code landing in mozilla-central

2020-09-15 Thread Jonathan Kew
On 15/09/2020 16:03, Brian Grinstead wrote: > We’re rolling out a change to the review process to put more focus on automated testing. [...] As others have said, it's great that we're setting a clearer policy here - thanks! One thing did strike me as a little surprising, and could perhaps

Re: A new testing policy for code landing in mozilla-central

2020-09-15 Thread Andrew Halberstadt
This change is fantastic, thanks for driving it! Reviewers should not need to feel bad about asking authors to go back and add tests, making this official policy removes that burden of guilt (just like how reviewbot removes the burden of pointing out linting nits). Authors can now grumble at the

A new testing policy for code landing in mozilla-central

2020-09-15 Thread Brian Grinstead
(This is a crosspost from firefox-dev) Hi all, We’re rolling out a change to the review process to put more focus on automated testing. This will affect you if you review code that lands in mozilla-central. ## TLDR Reviewers will now need to add a testing Project Tag in Phabricator when