Re: Too many oranges!

[Ehsan Akhgari]

On 2015-12-29 6:24 PM, Steve Fink wrote:

That makes me think that we're not disagreeing very much. Still some, I
think, but sure -- if assigning a handful of people to work on
intermittent oranges makes the problem go away, then that seems like a
reasonable thing to do. Though if you're taking existing developers, and
I think you'd have to in order to get anywhere, then you'd want to be
sure that stalling off what they're currently working on is not going to
strike them as a ridiculously bad idea.

And even then, I'm not sure of the "tiger team" approach here. We have
that now, and his name is Ehsan. He has good evidence to show that it
doesn't work.


FWIW, the reason that it doesn't work is that my time is limited and 
can't be devoted to this problem enough, and in many cases when I ask 
someone else to help, the answer is that they don't have time.


If more people did this more systematically, I think we can get to a 
good place in the number of oranges, and stay there.



But this sort of problem seems like it's something we have an
institutional disregard for. And any attempt to spot-fix it without
addressing that systemic issue seems less likely to succeed. *That* is
what I understood the true motivation for things like enforced tree
closures was -- not to directly solve the issue, but to force everyone
to care more about it, so that we don't have to do crappy things like
that. But I still don't like it, because it feels like a "you must write
at least 100 lines of code a day" type of thing: it doesn't necessarily
put the pressure at the right place, and it causes a lot of collateral
damage.


I really doubt that the idea of closing trees indefinitely is realistic 
enough to even merit a discussion.  :-)



I didn't really want to get into this (hey, I just wanted to throw out a
suggestion to get people thinking), but to me the problem is that we're
at all ok with allowing the current situation to develop. The sheriffs
have been warning about impending orangopocalypse for as long as I can
remember, and we've all (myself included) largely ignored them. Ehsan
has also brought this up several times, with real data as to how
addressable this is with focused effort, and from what I can tell the
response has basically been a collective "boy, that Ehsan guy is smart
and nice to have around" followed by a continuation of the current way
of doing things.


Yes, exactly.  Our current situation is that people can successfully get 
by pretending that the orange situation is "someone else's problem". 
Look at it this way.  If you decide to ignore an orange bug in your area 
at all costs, you'll have no problem doing that.  Just ignore the bug 
long enough until a) someone else actually fixes it, or b) the test gets 
disabled by a sheriff.


A "tiger team" approach can bring us into a good spot now, but while the 
situation above doesn't change, we'll end up back where we are.  And 
we'll end up having this conversation once again.



To be fair, we *are* doing some important work in the right direction. I
don't know what all of it is, but the treeherder improvements for
identifying and categorizing oranges is really important, as are rr,
chaos mode, web replay, and related work. To me at least, that still
feels like more of the "let's throw a few smart people at the systemic
problem" approach, though. I could be wrong. Maybe we're close enough to
being able to recognize failures and direct them to the appropriate
people, who will then jump up and fix them, that this whole problem will
go away soon.

/me holds breath

/me turns blue


It's true that some of these new initiatives will help debugging 
intermittent failures, they're not strictly needed.  What we need is 
accepting shared responsibility about this issue, and address it like 
any other ongoing technical issue.


Part of this shared responsibility is acknowledging that this is part of 
what MoCo employed engineers need to spend time on, which requires 
support for engineering managers.  As the recent response to bkelly's 
triaging effort shows, there is a lot of goodwill on this issue both 
from engineers and managers, so I hope this time we'll keep being on top 
of this issue.


Cheers,
Ehsan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Steve Fink]

More to the point, I don't think that doing this is "a big hammer 
[that] should be used sparingly." Getting paid to work on Mozilla 
stuff is a huge privilege, and that privilege comes with the duty to 
work on what the organization thinks is most important. The fact that 
it's often most practical to delegate that prioritization to the 
employee makes working at Mozilla (and in the tech industry in 
general) a lot of fun, but employees generally do not have the right 
(morally or legally) to refuse to work on something because it's _not_ 
fun.


While perhaps more difficult to get immediate traction from, it
applies equally well to unpaid volunteers.


I disagree - volunteers are generally free to work on exactly what 
they want. The only leverage that the org has over them is (a) 
accepting or rejecting their contributions and (b) the extent to which 
it expresses appreciation for their work. We should certainly _try_ to 
steer volunteers towards useful things using (a) and (b) above, but we 
cannot order them to do un-fun stuff (like fixing oranges) the way we 
can with employees.



Note that my suggestion isn't really any better; it's just a stick
in place of a carrot. I only suggested it because it feels to me
like it is more likely to produce net positive results. But PTO is
obviously irrelevant to unpaid contributors. Tree closures *are*
globally relevant, but are more likely to drive away unpaid
volunteers than to incent them to help out with unfamiliar
intermittent oranges. The two weeks of enforced quality-related
work at the end of each cycle are better, in that they only
penalize paid staff, but that still feels to me like a top-down
imposition to work in a particular way, one that changes the
flavor of the organization into one that people are less likely to
want to volunteer for.


As noted previously, "ordering everyone to work on X" is very 
different than "ordering some people to work on X". The former is 
clearly flawed, and I think the latter is very sensible.


That makes me think that we're not disagreeing very much. Still some, I 
think, but sure -- if assigning a handful of people to work on 
intermittent oranges makes the problem go away, then that seems like a 
reasonable thing to do. Though if you're taking existing developers, and 
I think you'd have to in order to get anywhere, then you'd want to be 
sure that stalling off what they're currently working on is not going to 
strike them as a ridiculously bad idea.


And even then, I'm not sure of the "tiger team" approach here. We have 
that now, and his name is Ehsan. He has good evidence to show that it 
doesn't work. Admittedly, if you're grabbing one person from each of a 
large set of teams, then you're likely to get better "priority 
diffusion" than you get with one guy working on weekends. Certainly, I'd 
expect that actually fixing the problems is going to require a lot more 
than the small group, so the awareness of the importance is likely to 
spread somewhat.


But this sort of problem seems like it's something we have an 
institutional disregard for. And any attempt to spot-fix it without 
addressing that systemic issue seems less likely to succeed. *That* is 
what I understood the true motivation for things like enforced tree 
closures was -- not to directly solve the issue, but to force everyone 
to care more about it, so that we don't have to do crappy things like 
that. But I still don't like it, because it feels like a "you must write 
at least 100 lines of code a day" type of thing: it doesn't necessarily 
put the pressure at the right place, and it causes a lot of collateral 
damage.


I didn't really want to get into this (hey, I just wanted to throw out a 
suggestion to get people thinking), but to me the problem is that we're 
at all ok with allowing the current situation to develop. The sheriffs 
have been warning about impending orangopocalypse for as long as I can 
remember, and we've all (myself included) largely ignored them. Ehsan 
has also brought this up several times, with real data as to how 
addressable this is with focused effort, and from what I can tell the 
response has basically been a collective "boy, that Ehsan guy is smart 
and nice to have around" followed by a continuation of the current way 
of doing things.


To be fair, we *are* doing some important work in the right direction. I 
don't know what all of it is, but the treeherder improvements for 
identifying and categorizing oranges is really important, as are rr, 
chaos mode, web replay, and related work. To me at least, that still 
feels like more of the "let's throw a few smart people at the systemic 
problem" approach, though. I could be wrong. Maybe we're close enough to 
being able to recognize failures and direct them to the appropriate 
people, who will then jump up and fix them, that this whole problem will 
go away soon.


/me holds breath

/me turns blue

On 

Re: Too many oranges!

[Bobby Holley]

On Tue, Dec 29, 2015 at 11:41 AM, Steve Fink  wrote:

> On 12/22/2015 10:06 AM, L. David Baron wrote:
>
>> On Tuesday 2015-12-22 11:49 -0500, Ehsan Akhgari wrote:
>>
>>> On 2015-12-22 11:18 AM, Kartikaya Gupta wrote:
>>>
 On Mon, Dec 21, 2015 at 3:11 PM, L. David Baron 
 wrote:

> I agree it's definitely gone up recently, and agree that it causes a
> lot of wasted time.  I'm not convinced about closing the tree,
> though; keeping the tree closed for extended periods just leads to
> big backups.
>
> How about everybody reading this message takes a look at the list on
> http://brasstacks.mozilla.com/orangefactor/ and takes one of them to
> fix?  (Or, better, redoes the search filtered on the last 3 days
> instead of last 7.)
>

 I feel like a voluntary approach is likely to have very little effect,
 given the way our goals and priorities are structured. There's very
 little incentive to voluntarily spend time banging your head against a
 wall. That's why I'm more in favor of a forced approach that is
 mandated by managers/product owners/sheriffs (i.e. people who can
 actually tell us to some extent what to do).

>>> I have tried to volunteer some time from my weekends occasionally to look
>>> into the most recurring oranges every few weeks, and usually every time I
>>> manage to figure out a handful of bugs by spending a few hours and as a
>>> result OF would go down in the following week, but then it would go back
>>> up
>>> again.  This is a demotivating task and doesn't really scale to the
>>> magnitude of our orange problem.  I agree with kats that a voluntary
>>> based
>>> approach will not go anywhere.
>>>
>> Managers should definitely be supporting this, and sheriffs (and
>> anybody else with push access) should be backing things out for
>> causing intermittent oranges (even if that's not discovered until
>> days/weeks later).
>>
>> If people don't feel they can fix intermittent oranges during the
>> business day as part of their job, that's a problem.
>>
>
> Hm... it seems like you could take that further. Fixing oranges could be
> considered to be going "above and beyond" your normal responsibilities
> (unless you're just fixing your own!). So for employees, you could convert
> N hours of orange-fixing into up to M hours of additional PTO whenever the
> orange factor > k. :-) If intermittent orange is indeed a large
> project-wide drag on resources (and I believe it is), then the M+N hours of
> regular work "lost" in this way should be more than compensated for by the
> reduced overhead on everyone.
>
> Just a thought from someone who wouldn't have to work through the details.
> And it's crazy enough already that I won't mention the tacked-on idea that
> if unpaid contributors spend time fixing oranges, then we'd send them free
> graphics hardware of the sorts that we have low pre-release coverage on,
> with no strings attached other than "if you give this away, please don't
> remove the sticker that says 'You can help Firefox! Just switch to the beta
> release channel! This card is worth 42 points on Mozilla's Hardware
> Diversity Scoreboard.'")... ;-)
>
> But I don't think having mozilla-inbound/mozilla-central be closed
>> more than it already is is going to help anything.  It will just
>> make people frustrated that they can't land what they've been
>> working on.
>>
>
> Amen. Trying to artificially force this stuff is going in the wrong
> direction. After all, you'd be reducing productivity from the top-down in
> order to improve productivity. It might work, it might not, it might help
> for a while but have long-term negative consequences.
>
> Personally, I feel like getting farther away from our volunteer-driven
> roots is dangerous. Sure, we have lots of paid staff now, but you really
> don't want any more selection pressure to push the overall contributor base
> towards people who are involved for the money and away from people who are
> motivated by the mission.


I don't follow the concerns in this last part. Can you clarify which
proposal you're concerned will take us farther from our volunteer-driven
roots? The part about ordering paid staff to do unpleasant-but-necessary
things, or something else?


>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Steve Fink]

On 12/22/2015 10:06 AM, L. David Baron wrote:

On Tuesday 2015-12-22 11:49 -0500, Ehsan Akhgari wrote:

On 2015-12-22 11:18 AM, Kartikaya Gupta wrote:

On Mon, Dec 21, 2015 at 3:11 PM, L. David Baron  wrote:

I agree it's definitely gone up recently, and agree that it causes a
lot of wasted time.  I'm not convinced about closing the tree,
though; keeping the tree closed for extended periods just leads to
big backups.

How about everybody reading this message takes a look at the list on
http://brasstacks.mozilla.com/orangefactor/ and takes one of them to
fix?  (Or, better, redoes the search filtered on the last 3 days
instead of last 7.)


I feel like a voluntary approach is likely to have very little effect,
given the way our goals and priorities are structured. There's very
little incentive to voluntarily spend time banging your head against a
wall. That's why I'm more in favor of a forced approach that is
mandated by managers/product owners/sheriffs (i.e. people who can
actually tell us to some extent what to do).

I have tried to volunteer some time from my weekends occasionally to look
into the most recurring oranges every few weeks, and usually every time I
manage to figure out a handful of bugs by spending a few hours and as a
result OF would go down in the following week, but then it would go back up
again.  This is a demotivating task and doesn't really scale to the
magnitude of our orange problem.  I agree with kats that a voluntary based
approach will not go anywhere.

Managers should definitely be supporting this, and sheriffs (and
anybody else with push access) should be backing things out for
causing intermittent oranges (even if that's not discovered until
days/weeks later).

If people don't feel they can fix intermittent oranges during the
business day as part of their job, that's a problem.


Hm... it seems like you could take that further. Fixing oranges could be 
considered to be going "above and beyond" your normal responsibilities 
(unless you're just fixing your own!). So for employees, you could 
convert N hours of orange-fixing into up to M hours of additional PTO 
whenever the orange factor > k. :-) If intermittent orange is indeed a 
large project-wide drag on resources (and I believe it is), then the M+N 
hours of regular work "lost" in this way should be more than compensated 
for by the reduced overhead on everyone.


Just a thought from someone who wouldn't have to work through the 
details. And it's crazy enough already that I won't mention the 
tacked-on idea that if unpaid contributors spend time fixing oranges, 
then we'd send them free graphics hardware of the sorts that we have low 
pre-release coverage on, with no strings attached other than "if you 
give this away, please don't remove the sticker that says 'You can help 
Firefox! Just switch to the beta release channel! This card is worth 42 
points on Mozilla's Hardware Diversity Scoreboard.'")... ;-)



But I don't think having mozilla-inbound/mozilla-central be closed
more than it already is is going to help anything.  It will just
make people frustrated that they can't land what they've been
working on.


Amen. Trying to artificially force this stuff is going in the wrong 
direction. After all, you'd be reducing productivity from the top-down 
in order to improve productivity. It might work, it might not, it might 
help for a while but have long-term negative consequences.


Personally, I feel like getting farther away from our volunteer-driven 
roots is dangerous. Sure, we have lots of paid staff now, but you really 
don't want any more selection pressure to push the overall contributor 
base towards people who are involved for the money and away from people 
who are motivated by the mission.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Steve Fink]

On 12/29/2015 11:49 AM, Bobby Holley wrote:



On Tue, Dec 29, 2015 at 11:41 AM, Steve Fink > wrote:


On 12/22/2015 10:06 AM, L. David Baron wrote:

But I don't think having mozilla-inbound/mozilla-central be closed
more than it already is is going to help anything.  It will just
make people frustrated that they can't land what they've been
working on.


Amen. Trying to artificially force this stuff is going in the
wrong direction. After all, you'd be reducing productivity from
the top-down in order to improve productivity. It might work, it
might not, it might help for a while but have long-term negative
consequences.

Personally, I feel like getting farther away from our
volunteer-driven roots is dangerous. Sure, we have lots of paid
staff now, but you really don't want any more selection pressure
to push the overall contributor base towards people who are
involved for the money and away from people who are motivated by
the mission.


I don't follow the concerns in this last part. Can you clarify which 
proposal you're concerned will take us farther from our 
volunteer-driven roots? The part about ordering paid staff to do 
unpleasant-but-necessary things, or something else?


Yes, the part about ordering paid staff to do unpleasant things, or at 
least some proposed mechanisms to do so.


We're getting paid, and so obviously it's within the organization's 
*right* to order us to do stuff that furthers the organization's goals 
at the expense of the individual's. (Which is the suggestion, in the 
case of mandatory tree closures for oranges that a particular developer 
has no hope of addressing personally.) But that's a big hammer, and 
should be used sparingly. The alternative is to motivate staff by 
aligning their goals with the organization's. While perhaps more 
difficult to get immediate traction from, it applies equally well to 
unpaid volunteers.


Note that my suggestion isn't really any better; it's just a stick in 
place of a carrot. I only suggested it because it feels to me like it is 
more likely to produce net positive results. But PTO is obviously 
irrelevant to unpaid contributors. Tree closures *are* globally 
relevant, but are more likely to drive away unpaid volunteers than to 
incent them to help out with unfamiliar intermittent oranges. The two 
weeks of enforced quality-related work at the end of each cycle are 
better, in that they only penalize paid staff, but that still feels to 
me like a top-down imposition to work in a particular way, one that 
changes the flavor of the organization into one that people are less 
likely to want to volunteer for.


Don't get me wrong, I can also argue that it would be suboptimal to only 
have a pile of developers all working on what they feel like, when they 
feel like it, and in the way they feel like it. But the right place is a 
balance between authoritarian and free-for-all, and right now I feel a 
couple of pushes towards excessively authoritarian that bother me.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Bobby Holley]

On Tue, Dec 29, 2015 at 12:23 PM, Steve Fink  wrote:

> On 12/29/2015 11:49 AM, Bobby Holley wrote:
>
>
>
> On Tue, Dec 29, 2015 at 11:41 AM, Steve Fink  wrote:
>
>> On 12/22/2015 10:06 AM, L. David Baron wrote:
>>
>>> But I don't think having mozilla-inbound/mozilla-central be closed
>>> more than it already is is going to help anything.  It will just
>>> make people frustrated that they can't land what they've been
>>> working on.
>>>
>>
>> Amen. Trying to artificially force this stuff is going in the wrong
>> direction. After all, you'd be reducing productivity from the top-down in
>> order to improve productivity. It might work, it might not, it might help
>> for a while but have long-term negative consequences.
>>
>> Personally, I feel like getting farther away from our volunteer-driven
>> roots is dangerous. Sure, we have lots of paid staff now, but you really
>> don't want any more selection pressure to push the overall contributor base
>> towards people who are involved for the money and away from people who are
>> motivated by the mission.
>
>
> I don't follow the concerns in this last part. Can you clarify which
> proposal you're concerned will take us farther from our volunteer-driven
> roots? The part about ordering paid staff to do unpleasant-but-necessary
> things, or something else?
>
>
> Yes, the part about ordering paid staff to do unpleasant things, or at
> least some proposed mechanisms to do so.
>
> We're getting paid, and so obviously it's within the organization's
> *right* to order us to do stuff that furthers the organization's goals at
> the expense of the individual's.
>

Very much so.


> (Which is the suggestion, in the case of mandatory tree closures for
> oranges that a particular developer has no hope of addressing personally.)
>

I think this proposal is flawed enough that it's more or less orthogonal to
concept of ordering a subset of our paid staff to fix oranges.


> But that's a big hammer, and should be used sparingly.
>
>
The alternative is to motivate staff by aligning their goals with the
> organization's.
>

I don't think these differ at all. Saying "your goal for this quarter is X"
is precisely how the organization orders employees to work on X.

More to the point, I don't think that doing this is "a big hammer [that]
should be used sparingly." Getting paid to work on Mozilla stuff is a huge
privilege, and that privilege comes with the duty to work on what the
organization thinks is most important. The fact that it's often most
practical to delegate that prioritization to the employee makes working at
Mozilla (and in the tech industry in general) a lot of fun, but employees
generally do not have the right (morally or legally) to refuse to work on
something because it's _not_ fun.

While perhaps more difficult to get immediate traction from, it applies
> equally well to unpaid volunteers.
>

I disagree - volunteers are generally free to work on exactly what they
want. The only leverage that the org has over them is (a) accepting or
rejecting their contributions and (b) the extent to which it expresses
appreciation for their work. We should certainly _try_ to steer volunteers
towards useful things using (a) and (b) above, but we cannot order them to
do un-fun stuff (like fixing oranges) the way we can with employees.


>
> Note that my suggestion isn't really any better; it's just a stick in
> place of a carrot. I only suggested it because it feels to me like it is
> more likely to produce net positive results. But PTO is obviously
> irrelevant to unpaid contributors. Tree closures *are* globally relevant,
> but are more likely to drive away unpaid volunteers than to incent them to
> help out with unfamiliar intermittent oranges. The two weeks of enforced
> quality-related work at the end of each cycle are better, in that they only
> penalize paid staff, but that still feels to me like a top-down imposition
> to work in a particular way, one that changes the flavor of the
> organization into one that people are less likely to want to volunteer for.
>

As noted previously, "ordering everyone to work on X" is very different
than "ordering some people to work on X". The former is clearly flawed, and
I think the latter is very sensible.


>
> Don't get me wrong, I can also argue that it would be suboptimal to only
> have a pile of developers all working on what they feel like, when they
> feel like it, and in the way they feel like it. But the right place is a
> balance between authoritarian and free-for-all, and right now I feel a
> couple of pushes towards excessively authoritarian that bother me.
>
>
Managers order their reports to do unpleasant things all the time. The
trick to being a good manager is finding the right balance of workload and
incentives to keep everyone happy (enough) so that they're productive,
keeping growing, and don't quit. We have a mixed track record in that area,
but it's fundamentally an implementation 

Re: Too many oranges!

[Gijs Kruitbosch]

On 22/12/2015 17:40, James Graham wrote:

On 22/12/15 17:22, Andrew Halberstadt wrote:

FWIW a summary of top orangefactor[1] oranges are posted regularly
to dev.tree-alerts. Configuring it to also post to
dev.platform is certainly possible if that's what people want. Though I
have a feeling that people will mostly ignore these emails anyway if
we do.


I think previous discussions on this topic have revealed that people
mostly ignore those emails if they're not personally actionable.


This.

I have looked at OF in the past, when I saw that e.g. mochitest-bc was 
orange "a lot of the time", but I found it very difficult to get a 
reasonable list of things that *I* can actually fix - the data is 
"clogged up" with issues from other test suites, other teams, 
infrastructure-related timeouts etc., ... it takes too much effort to 
get it down a prioritized "here are the top 5 bugs that you could 
potentially fix given time+effort" the way that bkelly did in his other 
post, and then it's hard to know how big that part of the problem is - 
you get absolute numbers ("50 oranges last week") but no idea of whether 
that's 90% or .9% of the oranges (for that suite) that week.


Having bkelly (or any other specific person) do this kind of overview 
repeatedly also doesn't scale. It seems like we should be able to 
automate this already based on e.g. bugzilla components of the bug on 
file for the intermittent. I'm worried that waiting on treeherder 
improvements here will mean a long wait. What's the schedule there? How 
hard would it be to add a better filtering system to brass stacks / OF 
in the first part of '16Q1 ?


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Philipp Kewisch]

On 12/22/15 5:38 PM, Ben Kelly wrote:
> I'd rather see us do:
> 
> 1) Raise the visibility of oranges.  Post the most frequent intermittents
> without an owner to dev-platform every N days.

While I am all for raising awareness of oranges, I don't think posting
them to dev-platform is the right thing to do. Aside from others
mentioning they may be ignored, there are also developers subscribed to
the mailing list primarily for hearing about intent to ship/unship and
other high level information about the platform (me included). Test
failures don't fall into this category, I think such messages are better
put on a separate list or at least separated into platform vs Firefox.

Philipp
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Mike Conley]

I would support scheduled time[1] to do maintenance[2] and help improve our
developer tooling and documentation. I'm less sure how to integrate such a
thing in practice.

[1]: A day, a week, heck maybe even a release cycle
[2]: Where maintenance is fixing oranges, closing out papercuts,
refactoring, etc.

On 21 December 2015 at 17:35,  wrote:

> On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta wrote:
> > So, I propose that we create an orangefactor threshold above which the
> > tree should just be closed until people start fixing intermittent
> > oranges. Thoughts?
> >
> > kats
>
> How about regularly scheduled test fix days where everyone drops what they
> are doing and spends a day fixing tests? mc could be closed to everything
> except critical work and test fixes. Managers would be able to opt
> individuals out of this as needed but generally everyone would be expected
> to take part.
>
> Jim
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Jason Duell]

On Tue, Dec 22, 2015 at 11:38 AM, Ben Kelly  wrote:

>
> I'd rather see us do:
>
> 1) Raise the visibility of oranges.  Post the most frequent intermittents
> without an owner to dev-platform every N days.
> 2) Make its someone's job to find owners for top oranges.  I believe RyanVM
> used to do that, but not sure if its still happening now that he has
> changed roles.
>
> Ben
>
>
I'm with bkelly on this one.  Maybe with some additional initial messaging
("War on Orange raised in priority!") too.  I don't think we want to pivot
all work in platform for this.

Jason



>
> >
> > On Tue, Dec 22, 2015 at 7:41 AM Mike Conley  wrote:
> >
> > > I would support scheduled time[1] to do maintenance[2] and help improve
> > our
> > > developer tooling and documentation. I'm less sure how to integrate
> such
> > a
> > > thing in practice.
> > >
> > > [1]: A day, a week, heck maybe even a release cycle
> > > [2]: Where maintenance is fixing oranges, closing out papercuts,
> > > refactoring, etc.
> > >
> > > On 21 December 2015 at 17:35,  wrote:
> > >
> > > > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
> > wrote:
> > > > > So, I propose that we create an orangefactor threshold above which
> > the
> > > > > tree should just be closed until people start fixing intermittent
> > > > > oranges. Thoughts?
> > > > >
> > > > > kats
> > > >
> > > > How about regularly scheduled test fix days where everyone drops what
> > > they
> > > > are doing and spends a day fixing tests? mc could be closed to
> > everything
> > > > except critical work and test fixes. Managers would be able to opt
> > > > individuals out of this as needed but generally everyone would be
> > > expected
> > > > to take part.
> > > >
> > > > Jim
> > > > ___
> > > > dev-platform mailing list
> > > > dev-platform@lists.mozilla.org
> > > > https://lists.mozilla.org/listinfo/dev-platform
> > > >
> > > ___
> > > dev-platform mailing list
> > > dev-platform@lists.mozilla.org
> > > https://lists.mozilla.org/listinfo/dev-platform
> > >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>



-- 

Jason
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Douglas Turner]

In general, we need to make some small but hard changes to our development
process to improve overall quality of Gecko and Firefox.  I don't think
there is any silver bullet and we will probably try things that aren't
universally +1'ed.

We want devs to have autonomy to do what they think is right and this means
not being too prescriptive (e.g., fix bug 123, 456, xyz.  Go reduce the
number of orange tests.).  However, organizationally we need to improve our
over all backlogs, fix the papercuts, reduce test failures, improve
developer productivity, build better tooling etc.  We need to set aside
time to sharpen our axe.

My suggest was to take the n-th milestone was a strawman. I think in terms
of scheduling, setting aside every n-th milestone as a quality release
helps planning a bunch.  When we tell customers (the web, the firefox team,
firefoxos) that feature X will be delivered by Aug, we can build in time to
sharpen our axe.

Another suggest from blassey was to make the last two weeks of the cycle
focused on the above issues.  This is pretty slick as it is a forcing
function again devs crash landing features before an uplift (Not that we'd
ever do that!)

Doug

On Tue, Dec 22, 2015 at 11:58 AM Jason Duell  wrote:

> On Tue, Dec 22, 2015 at 11:38 AM,
>


> Ben Kelly  wrote:
>
>>
>> I'd rather see us do:
>>
>> 1) Raise the visibility of oranges.  Post the most frequent intermittents
>> without an owner to dev-platform every N days.
>> 2) Make its someone's job to find owners for top oranges.  I believe
>> RyanVM
>> used to do that, but not sure if its still happening now that he has
>> changed roles.
>>
>> Ben
>>
>>
> I'm with bkelly on this one.  Maybe with some additional initial messaging
> ("War on Orange raised in priority!") too.  I don't think we want to pivot
> all work in platform for this.
>
> Jason
>
>
>
>>
>> >
>> > On Tue, Dec 22, 2015 at 7:41 AM Mike Conley 
>> wrote:
>> >
>> > > I would support scheduled time[1] to do maintenance[2] and help
>> improve
>> > our
>> > > developer tooling and documentation. I'm less sure how to integrate
>> such
>> > a
>> > > thing in practice.
>> > >
>> > > [1]: A day, a week, heck maybe even a release cycle
>> > > [2]: Where maintenance is fixing oranges, closing out papercuts,
>> > > refactoring, etc.
>> > >
>> > > On 21 December 2015 at 17:35,  wrote:
>> > >
>> > > > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
>> > wrote:
>> > > > > So, I propose that we create an orangefactor threshold above which
>> > the
>> > > > > tree should just be closed until people start fixing intermittent
>> > > > > oranges. Thoughts?
>> > > > >
>> > > > > kats
>> > > >
>> > > > How about regularly scheduled test fix days where everyone drops
>> what
>> > > they
>> > > > are doing and spends a day fixing tests? mc could be closed to
>> > everything
>> > > > except critical work and test fixes. Managers would be able to opt
>> > > > individuals out of this as needed but generally everyone would be
>> > > expected
>> > > > to take part.
>> > > >
>> > > > Jim
>> > > > ___
>> > > > dev-platform mailing list
>> > > > dev-platform@lists.mozilla.org
>> > > > https://lists.mozilla.org/listinfo/dev-platform
>> > > >
>> > > ___
>> > > dev-platform mailing list
>> > > dev-platform@lists.mozilla.org
>> > > https://lists.mozilla.org/listinfo/dev-platform
>> > >
>> > ___
>> > dev-platform mailing list
>> > dev-platform@lists.mozilla.org
>> > https://lists.mozilla.org/listinfo/dev-platform
>> >
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
>
> --
>
> Jason
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Mark Banner]

On 22/12/2015 20:41, Douglas Turner wrote:

My suggest was to take the n-th milestone was a strawman. I think in terms
of scheduling, setting aside every n-th milestone as a quality release
helps planning a bunch.  When we tell customers (the web, the firefox team,
firefoxos) that feature X will be delivered by Aug, we can build in time to
sharpen our axe.


On Hello, we've being attempting to take care of quality slightly 
different. We're working in an agile manner, and for each cycle, we're 
trying to allocate a certain amount of time to fixing bugs, addressing 
technical debt, and other quality issues. I can't remember the exact 
percentages at the moment, but I know we originally discussed something 
like 20%. The remaining 80% is spent on feature work. We're also 
starting to expect/allocate more time for fully implementing features 
after the "initial" implementation is complete.


We've only been trying this over the last couple of months, and with 
other changes going on at the same time, I think its too early to say 
how successful we're being. For me, the positive part, is that we're 
constantly thinking about quality and improving the code base, rather 
than doing it in spurts when it gets very bad.


Mark.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Douglas Turner]

Thanks Ben -- really great.  Let me know you need anything or if things get
stuck!

Doug

On Tue, Dec 22, 2015 at 4:00 PM Ben Kelly  wrote:

> I'll triage the top orange tomorrow and try to find some owners for things.
>
> After the new year I'll set something weekly up to stay on top of things.
> On Dec 22, 2015 2:58 PM, "Jason Duell"  wrote:
>
>>
>>
>> On Tue, Dec 22, 2015 at 11:38 AM, Ben Kelly  wrote:
>>
>>>
>>> I'd rather see us do:
>>>
>>> 1) Raise the visibility of oranges.  Post the most frequent intermittents
>>> without an owner to dev-platform every N days.
>>> 2) Make its someone's job to find owners for top oranges.  I believe
>>> RyanVM
>>> used to do that, but not sure if its still happening now that he has
>>> changed roles.
>>>
>>> Ben
>>>
>>>
>> I'm with bkelly on this one.  Maybe with some additional initial
>> messaging ("War on Orange raised in priority!") too.  I don't think we want
>> to pivot all work in platform for this.
>>
>> Jason
>>
>>
>>
>>>
>>> >
>>> > On Tue, Dec 22, 2015 at 7:41 AM Mike Conley 
>>> wrote:
>>> >
>>> > > I would support scheduled time[1] to do maintenance[2] and help
>>> improve
>>> > our
>>> > > developer tooling and documentation. I'm less sure how to integrate
>>> such
>>> > a
>>> > > thing in practice.
>>> > >
>>> > > [1]: A day, a week, heck maybe even a release cycle
>>> > > [2]: Where maintenance is fixing oranges, closing out papercuts,
>>> > > refactoring, etc.
>>> > >
>>> > > On 21 December 2015 at 17:35,  wrote:
>>> > >
>>> > > > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
>>> > wrote:
>>> > > > > So, I propose that we create an orangefactor threshold above
>>> which
>>> > the
>>> > > > > tree should just be closed until people start fixing intermittent
>>> > > > > oranges. Thoughts?
>>> > > > >
>>> > > > > kats
>>> > > >
>>> > > > How about regularly scheduled test fix days where everyone drops
>>> what
>>> > > they
>>> > > > are doing and spends a day fixing tests? mc could be closed to
>>> > everything
>>> > > > except critical work and test fixes. Managers would be able to opt
>>> > > > individuals out of this as needed but generally everyone would be
>>> > > expected
>>> > > > to take part.
>>> > > >
>>> > > > Jim
>>> > > > ___
>>> > > > dev-platform mailing list
>>> > > > dev-platform@lists.mozilla.org
>>> > > > https://lists.mozilla.org/listinfo/dev-platform
>>> > > >
>>> > > ___
>>> > > dev-platform mailing list
>>> > > dev-platform@lists.mozilla.org
>>> > > https://lists.mozilla.org/listinfo/dev-platform
>>> > >
>>> > ___
>>> > dev-platform mailing list
>>> > dev-platform@lists.mozilla.org
>>> > https://lists.mozilla.org/listinfo/dev-platform
>>> >
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>
>>
>>
>> --
>>
>> Jason
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Ben Kelly]

On Tue, Dec 22, 2015 at 7:29 PM, Douglas Turner  wrote:

> Thanks Ben -- really great.  Let me know you need anything or if things
> get stuck!
>

No problem.  Although I should note David Baron has already NI'd a bunch of
people on the top orange bugs.

Thanks.

Ben


>
> Doug
>
> On Tue, Dec 22, 2015 at 4:00 PM Ben Kelly  wrote:
>
>> I'll triage the top orange tomorrow and try to find some owners for
>> things.
>>
>> After the new year I'll set something weekly up to stay on top of things.
>> On Dec 22, 2015 2:58 PM, "Jason Duell"  wrote:
>>
>>>
>>>
>>> On Tue, Dec 22, 2015 at 11:38 AM, Ben Kelly  wrote:
>>>

 I'd rather see us do:

 1) Raise the visibility of oranges.  Post the most frequent
 intermittents
 without an owner to dev-platform every N days.
 2) Make its someone's job to find owners for top oranges.  I believe
 RyanVM
 used to do that, but not sure if its still happening now that he has
 changed roles.

 Ben


>>> I'm with bkelly on this one.  Maybe with some additional initial
>>> messaging ("War on Orange raised in priority!") too.  I don't think we want
>>> to pivot all work in platform for this.
>>>
>>> Jason
>>>
>>>
>>>

 >
 > On Tue, Dec 22, 2015 at 7:41 AM Mike Conley 
 wrote:
 >
 > > I would support scheduled time[1] to do maintenance[2] and help
 improve
 > our
 > > developer tooling and documentation. I'm less sure how to integrate
 such
 > a
 > > thing in practice.
 > >
 > > [1]: A day, a week, heck maybe even a release cycle
 > > [2]: Where maintenance is fixing oranges, closing out papercuts,
 > > refactoring, etc.
 > >
 > > On 21 December 2015 at 17:35,  wrote:
 > >
 > > > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
 > wrote:
 > > > > So, I propose that we create an orangefactor threshold above
 which
 > the
 > > > > tree should just be closed until people start fixing
 intermittent
 > > > > oranges. Thoughts?
 > > > >
 > > > > kats
 > > >
 > > > How about regularly scheduled test fix days where everyone drops
 what
 > > they
 > > > are doing and spends a day fixing tests? mc could be closed to
 > everything
 > > > except critical work and test fixes. Managers would be able to opt
 > > > individuals out of this as needed but generally everyone would be
 > > expected
 > > > to take part.
 > > >
 > > > Jim
 > > > ___
 > > > dev-platform mailing list
 > > > dev-platform@lists.mozilla.org
 > > > https://lists.mozilla.org/listinfo/dev-platform
 > > >
 > > ___
 > > dev-platform mailing list
 > > dev-platform@lists.mozilla.org
 > > https://lists.mozilla.org/listinfo/dev-platform
 > >
 > ___
 > dev-platform mailing list
 > dev-platform@lists.mozilla.org
 > https://lists.mozilla.org/listinfo/dev-platform
 >
 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org
 https://lists.mozilla.org/listinfo/dev-platform

>>>
>>>
>>>
>>> --
>>>
>>> Jason
>>>
>>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Chris Peterson]

On 12/22/15 9:39 AM, Jonathan Griffin wrote:

If we dedicate a cycle to quality and tests, we should use that opportunity
to figure out what a more viable strategy is longer-term for making sure
these don't get out of hand again, which might include having teams adopt
test, suites, and the intermittent orange counts associated with them, and
being accountable for them in goals and deliverables.


The Firefox team plans for three two-week iterations per release cycle. 
To make quality a long-term commitment, we could dedicate the third 
two-week iteration of each release cycle to just fixing tests and bugs.


This has the added benefit of stablizing Nightly before it merges to Dev 
Edition. People have expressed concerns that Dev Edition, as a "real 
product", should have a higher quality bar than Aurora did.


OTOH, a mozilla-central code freeze will frustrate people and lead to 
more big landings in the first or second iteration of the each release 
cycle. That might be OK because those features would then have 2–5 weeks 
of bake time on Nightly instead of just one. :)

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Karl Tomlinson]

Kartikaya Gupta writes:

> Personally I much prefer the new approach to reporting intermittents.
> It's much easier for me to see at a glance (i.e when the bugs are
> updated with the weekly count) which ones are actually occurring more
> frequently and which bug would be best to spend time on. With the old
> system I just got so much intermittent-failure bugmail that I would
> just ignore it all.

Yes, periodic summaries would be much better than the previous
approach, but not all occurrences are reported.  Sometimes a
search on http://brasstacks.mozilla.com/orangefactor/ is required.

The "Bug Details" search is much faster than the search on the front
page, but it would be more helpful if all occurrences trigger some
sort of periodic report, to indicate whether a search would find
anything.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Douglas Turner]

Mike -- totally supportive of this. I would *love* to see a release cycle
completely dedicated to quality.  We branch again on January 26.  We could
use that cycle to focus on nothing but quality (fixing tests, bug triaging,
no feature development at all).

Thoughts?

On Tue, Dec 22, 2015 at 7:41 AM Mike Conley  wrote:

> I would support scheduled time[1] to do maintenance[2] and help improve our
> developer tooling and documentation. I'm less sure how to integrate such a
> thing in practice.
>
> [1]: A day, a week, heck maybe even a release cycle
> [2]: Where maintenance is fixing oranges, closing out papercuts,
> refactoring, etc.
>
> On 21 December 2015 at 17:35,  wrote:
>
> > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta wrote:
> > > So, I propose that we create an orangefactor threshold above which the
> > > tree should just be closed until people start fixing intermittent
> > > oranges. Thoughts?
> > >
> > > kats
> >
> > How about regularly scheduled test fix days where everyone drops what
> they
> > are doing and spends a day fixing tests? mc could be closed to everything
> > except critical work and test fixes. Managers would be able to opt
> > individuals out of this as needed but generally everyone would be
> expected
> > to take part.
> >
> > Jim
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Mike Conley]

You had me at "quality". :)

On 22/12/2015 11:16 AM, Douglas Turner wrote:
> Mike -- totally supportive of this. I would *love* to see a release
> cycle completely dedicated to quality.  We branch again on January 26. 
> We could use that cycle to focus on nothing but quality (fixing tests,
> bug triaging, no feature development at all).
> 
> Thoughts?
> 
> On Tue, Dec 22, 2015 at 7:41 AM Mike Conley  > wrote:
> 
> I would support scheduled time[1] to do maintenance[2] and help
> improve our
> developer tooling and documentation. I'm less sure how to integrate
> such a
> thing in practice.
> 
> [1]: A day, a week, heck maybe even a release cycle
> [2]: Where maintenance is fixing oranges, closing out papercuts,
> refactoring, etc.
> 
> On 21 December 2015 at 17:35,  > wrote:
> 
> > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
> wrote:
> > > So, I propose that we create an orangefactor threshold above
> which the
> > > tree should just be closed until people start fixing intermittent
> > > oranges. Thoughts?
> > >
> > > kats
> >
> > How about regularly scheduled test fix days where everyone drops
> what they
> > are doing and spends a day fixing tests? mc could be closed to
> everything
> > except critical work and test fixes. Managers would be able to opt
> > individuals out of this as needed but generally everyone would be
> expected
> > to take part.
> >
> > Jim
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org 
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org 
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Kartikaya Gupta]

On Mon, Dec 21, 2015 at 3:11 PM, L. David Baron  wrote:
> I agree it's definitely gone up recently, and agree that it causes a
> lot of wasted time.  I'm not convinced about closing the tree,
> though; keeping the tree closed for extended periods just leads to
> big backups.
>
> How about everybody reading this message takes a look at the list on
> http://brasstacks.mozilla.com/orangefactor/ and takes one of them to
> fix?  (Or, better, redoes the search filtered on the last 3 days
> instead of last 7.)


I feel like a voluntary approach is likely to have very little effect,
given the way our goals and priorities are structured. There's very
little incentive to voluntarily spend time banging your head against a
wall. That's why I'm more in favor of a forced approach that is
mandated by managers/product owners/sheriffs (i.e. people who can
actually tell us to some extent what to do).

kats
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Aaron Klotz]

I like the sound of your ideas and I would like to subscribe to your 
newsletter.


On 12/22/2015 9:16 AM, Douglas Turner wrote:

Mike -- totally supportive of this. I would *love* to see a release cycle
completely dedicated to quality.  We branch again on January 26.  We could
use that cycle to focus on nothing but quality (fixing tests, bug triaging,
no feature development at all).

Thoughts?




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Bobby Holley]

Do we actually need to have everybody working on oranges / quality at once?
Shouldn't we just prioritize it (probably permanently) in a way such that
_some_ members of every team are working on it?

Saying "everybody needs to work on this together" is a socially expedient
way of getting volunteers to do things, but we have a large paid staff and
management/planning structure that is designed to direct some of our
resources to walking and some of them to chewing gum.

Last spring I worked on media quality while other people worked on
features. It went great, and there was far less coordination overhead than
if everyone was trying to work on the same thing.

bholley

On Tue, Dec 22, 2015 at 8:32 AM, Mike Conley  wrote:

> You had me at "quality". :)
>
> On 22/12/2015 11:16 AM, Douglas Turner wrote:
> > Mike -- totally supportive of this. I would *love* to see a release
> > cycle completely dedicated to quality.  We branch again on January 26.
> > We could use that cycle to focus on nothing but quality (fixing tests,
> > bug triaging, no feature development at all).
> >
> > Thoughts?
> >
> > On Tue, Dec 22, 2015 at 7:41 AM Mike Conley  > > wrote:
> >
> > I would support scheduled time[1] to do maintenance[2] and help
> > improve our
> > developer tooling and documentation. I'm less sure how to integrate
> > such a
> > thing in practice.
> >
> > [1]: A day, a week, heck maybe even a release cycle
> > [2]: Where maintenance is fixing oranges, closing out papercuts,
> > refactoring, etc.
> >
> > On 21 December 2015 at 17:35,  > > wrote:
> >
> > > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
> > wrote:
> > > > So, I propose that we create an orangefactor threshold above
> > which the
> > > > tree should just be closed until people start fixing intermittent
> > > > oranges. Thoughts?
> > > >
> > > > kats
> > >
> > > How about regularly scheduled test fix days where everyone drops
> > what they
> > > are doing and spends a day fixing tests? mc could be closed to
> > everything
> > > except critical work and test fixes. Managers would be able to opt
> > > individuals out of this as needed but generally everyone would be
> > expected
> > > to take part.
> > >
> > > Jim
> > > ___
> > > dev-platform mailing list
> > > dev-platform@lists.mozilla.org  dev-platform@lists.mozilla.org>
> > > https://lists.mozilla.org/listinfo/dev-platform
> > >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org  dev-platform@lists.mozilla.org>
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Ehsan Akhgari]

On 2015-12-22 11:18 AM, Kartikaya Gupta wrote:

On Mon, Dec 21, 2015 at 3:11 PM, L. David Baron  wrote:

I agree it's definitely gone up recently, and agree that it causes a
lot of wasted time.  I'm not convinced about closing the tree,
though; keeping the tree closed for extended periods just leads to
big backups.

How about everybody reading this message takes a look at the list on
http://brasstacks.mozilla.com/orangefactor/ and takes one of them to
fix?  (Or, better, redoes the search filtered on the last 3 days
instead of last 7.)



I feel like a voluntary approach is likely to have very little effect,
given the way our goals and priorities are structured. There's very
little incentive to voluntarily spend time banging your head against a
wall. That's why I'm more in favor of a forced approach that is
mandated by managers/product owners/sheriffs (i.e. people who can
actually tell us to some extent what to do).


I have tried to volunteer some time from my weekends occasionally to 
look into the most recurring oranges every few weeks, and usually every 
time I manage to figure out a handful of bugs by spending a few hours 
and as a result OF would go down in the following week, but then it 
would go back up again.  This is a demotivating task and doesn't really 
scale to the magnitude of our orange problem.  I agree with kats that a 
voluntary based approach will not go anywhere.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Ben Kelly]

Does anyone feel the changes to how intermittents are reported to bugs has
affected things?  We used to get a comment for each intermittent, but now
its rolled up into a periodic summary.  Perhaps people feel less urgency to
fix things without the constant bugmail.  Not that I want to advocate
spam...  but it is a recent change.

On Tue, Dec 22, 2015 at 10:40 AM, Mike Conley  wrote:

> I would support scheduled time[1] to do maintenance[2] and help improve our
> developer tooling and documentation. I'm less sure how to integrate such a
> thing in practice.
>
> [1]: A day, a week, heck maybe even a release cycle
> [2]: Where maintenance is fixing oranges, closing out papercuts,
> refactoring, etc.
>
> On 21 December 2015 at 17:35,  wrote:
>
> > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta wrote:
> > > So, I propose that we create an orangefactor threshold above which the
> > > tree should just be closed until people start fixing intermittent
> > > oranges. Thoughts?
> > >
> > > kats
> >
> > How about regularly scheduled test fix days where everyone drops what
> they
> > are doing and spends a day fixing tests? mc could be closed to everything
> > except critical work and test fixes. Managers would be able to opt
> > individuals out of this as needed but generally everyone would be
> expected
> > to take part.
> >
> > Jim
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Kartikaya Gupta]

Personally I much prefer the new approach to reporting intermittents.
It's much easier for me to see at a glance (i.e when the bugs are
updated with the weekly count) which ones are actually occurring more
frequently and which bug would be best to spend time on. With the old
system I just got so much intermittent-failure bugmail that I would
just ignore it all.

kats

On Tue, Dec 22, 2015 at 11:08 AM, Ben Kelly  wrote:
> Does anyone feel the changes to how intermittents are reported to bugs has
> affected things?  We used to get a comment for each intermittent, but now
> its rolled up into a periodic summary.  Perhaps people feel less urgency to
> fix things without the constant bugmail.  Not that I want to advocate
> spam...  but it is a recent change.
>
> On Tue, Dec 22, 2015 at 10:40 AM, Mike Conley  wrote:
>
>> I would support scheduled time[1] to do maintenance[2] and help improve our
>> developer tooling and documentation. I'm less sure how to integrate such a
>> thing in practice.
>>
>> [1]: A day, a week, heck maybe even a release cycle
>> [2]: Where maintenance is fixing oranges, closing out papercuts,
>> refactoring, etc.
>>
>> On 21 December 2015 at 17:35,  wrote:
>>
>> > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta wrote:
>> > > So, I propose that we create an orangefactor threshold above which the
>> > > tree should just be closed until people start fixing intermittent
>> > > oranges. Thoughts?
>> > >
>> > > kats
>> >
>> > How about regularly scheduled test fix days where everyone drops what
>> they
>> > are doing and spends a day fixing tests? mc could be closed to everything
>> > except critical work and test fixes. Managers would be able to opt
>> > individuals out of this as needed but generally everyone would be
>> expected
>> > to take part.
>> >
>> > Jim
>> > ___
>> > dev-platform mailing list
>> > dev-platform@lists.mozilla.org
>> > https://lists.mozilla.org/listinfo/dev-platform
>> >
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Kartikaya Gupta]

On Tue, Dec 22, 2015 at 11:16 AM, Douglas Turner  wrote:
> Mike -- totally supportive of this. I would *love* to see a release cycle
> completely dedicated to quality.  We branch again on January 26.  We could
> use that cycle to focus on nothing but quality (fixing tests, bug triaging,
> no feature development at all).
>
> Thoughts?
>

I'm in favour of this. I'm assuming that the "spring release"
marketing is going to be focused around Firefox 46, so it would be
nice to spend the 47 nightly just doing quality fixes. More generally
it would be great to spend two releases a year (the first ones after
the fall/spring releases) just doing cleanup work.

kats
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[James Graham]

On 22/12/15 17:22, Andrew Halberstadt wrote:

FWIW a summary of top orangefactor[1] oranges are posted regularly
to dev.tree-alerts. Configuring it to also post to
dev.platform is certainly possible if that's what people want. Though I
have a feeling that people will mostly ignore these emails anyway if we do.


I think previous discussions on this topic have revealed that people 
mostly ignore those emails if they're not personally actionable. So 
whilst there are a few people who heroically go through all of the OF 
top bugs and try to get traction on fixes, we would get much better 
results from individual mail to developers or teams showing all the top 
intermittents which they are responsible for.


With that in mind, it's perhaps worth detailing some of the improvements 
we have in mind for treeherder in 2016:


1) Automatic classification of intermittent failures to help sheriffs on 
integration trees, and quickly identify failures that are new on try 
pushes. Potentially, automatic notification when an intermittent 
increases in frequency,


2) A replacement for Orange Factor, based on the same data as the 
autoclassification, that will make it easier to present personalised 
(or, at least path-filtered) lists of top intermittents. These could be 
posted to specific lists where most subscribers would get fully 
actionable information.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Andrew Halberstadt]

On 22/12/15 11:38 AM, Ben Kelly wrote:

On Tue, Dec 22, 2015 at 11:16 AM, Douglas Turner  wrote:


Mike -- totally supportive of this. I would *love* to see a release cycle
completely dedicated to quality.  We branch again on January 26.  We could
use that cycle to focus on nothing but quality (fixing tests, bug triaging,
no feature development at all).

Thoughts?



-1

This has been tried numerous times on fxos without success.  While it might
get oranges down temporarily, its not putting a system in place to address
the problem over the long haul.

I'd rather see us do:

1) Raise the visibility of oranges.  Post the most frequent intermittents
without an owner to dev-platform every N days.
2) Make its someone's job to find owners for top oranges.  I believe RyanVM
used to do that, but not sure if its still happening now that he has
changed roles.

Ben


FWIW a summary of top orangefactor[1] oranges are posted regularly
to dev.tree-alerts. Configuring it to also post to
dev.platform is certainly possible if that's what people want. Though I
have a feeling that people will mostly ignore these emails anyway if we do.

-Andrew

[1] http://brasstacks.mozilla.com/orangefactor/






On Tue, Dec 22, 2015 at 7:41 AM Mike Conley  wrote:


I would support scheduled time[1] to do maintenance[2] and help improve

our

developer tooling and documentation. I'm less sure how to integrate such

a

thing in practice.

[1]: A day, a week, heck maybe even a release cycle
[2]: Where maintenance is fixing oranges, closing out papercuts,
refactoring, etc.

On 21 December 2015 at 17:35,  wrote:


On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta

wrote:

So, I propose that we create an orangefactor threshold above which

the

tree should just be closed until people start fixing intermittent
oranges. Thoughts?

kats


How about regularly scheduled test fix days where everyone drops what

they

are doing and spends a day fixing tests? mc could be closed to

everything

except critical work and test fixes. Managers would be able to opt
individuals out of this as needed but generally everyone would be

expected

to take part.

Jim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Jonathan Griffin]

I think this is a great idea. Although it won't fix the problem long-term,
what it will do is get engineers and especially engineering managers
thinking about the problem, and hopefully understanding it better so they
can incorporate it into future priorities.

There are two fundamental problems that lead to the current state: weak or
no ownership of many tests or suites, so that fixing these oranges is
always somebody else's problem, and a focus on "hard" deliverables which
often leave little or no time to deal with unplanned problems like an
increase in intermittents.

If we dedicate a cycle to quality and tests, we should use that opportunity
to figure out what a more viable strategy is longer-term for making sure
these don't get out of hand again, which might include having teams adopt
test, suites, and the intermittent orange counts associated with them, and
being accountable for them in goals and deliverables.

Jonathan

On Tue, Dec 22, 2015 at 8:16 AM, Douglas Turner  wrote:

> Mike -- totally supportive of this. I would *love* to see a release cycle
> completely dedicated to quality.  We branch again on January 26.  We could
> use that cycle to focus on nothing but quality (fixing tests, bug triaging,
> no feature development at all).
>
> Thoughts?
>
> On Tue, Dec 22, 2015 at 7:41 AM Mike Conley  wrote:
>
> > I would support scheduled time[1] to do maintenance[2] and help improve
> our
> > developer tooling and documentation. I'm less sure how to integrate such
> a
> > thing in practice.
> >
> > [1]: A day, a week, heck maybe even a release cycle
> > [2]: Where maintenance is fixing oranges, closing out papercuts,
> > refactoring, etc.
> >
> > On 21 December 2015 at 17:35,  wrote:
> >
> > > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
> wrote:
> > > > So, I propose that we create an orangefactor threshold above which
> the
> > > > tree should just be closed until people start fixing intermittent
> > > > oranges. Thoughts?
> > > >
> > > > kats
> > >
> > > How about regularly scheduled test fix days where everyone drops what
> > they
> > > are doing and spends a day fixing tests? mc could be closed to
> everything
> > > except critical work and test fixes. Managers would be able to opt
> > > individuals out of this as needed but generally everyone would be
> > expected
> > > to take part.
> > >
> > > Jim
> > > ___
> > > dev-platform mailing list
> > > dev-platform@lists.mozilla.org
> > > https://lists.mozilla.org/listinfo/dev-platform
> > >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Andrew Halberstadt]

+1

I'd prefer to see quality be a perpetually ongoing effort over a
periodic burst. I'd like to see management rotate people into "quality
mode" by explicitly setting goals and deliverables around addressing it.

The problem in the past has been this notion of "greatest impact".
Things like refactorings and fixing tests always get deprioritized
because it's really hard to estimate how beneficial they are to the
overall mission. It was much easier to say shiny new feature X aligns
nicely with our top-line goals.

Well, finally, our top-line goal is to "build core strength". To me that
means that fixing tests and quality *now have the greatest impact*, and
everyone's deliverables should reflect this new reality.

-Andrew

On 22/12/15 11:40 AM, Bobby Holley wrote:

Do we actually need to have everybody working on oranges / quality at once?
Shouldn't we just prioritize it (probably permanently) in a way such that
_some_ members of every team are working on it?

Saying "everybody needs to work on this together" is a socially expedient
way of getting volunteers to do things, but we have a large paid staff and
management/planning structure that is designed to direct some of our
resources to walking and some of them to chewing gum.

Last spring I worked on media quality while other people worked on
features. It went great, and there was far less coordination overhead than
if everyone was trying to work on the same thing.

bholley

On Tue, Dec 22, 2015 at 8:32 AM, Mike Conley  wrote:


You had me at "quality". :)

On 22/12/2015 11:16 AM, Douglas Turner wrote:

Mike -- totally supportive of this. I would *love* to see a release
cycle completely dedicated to quality.  We branch again on January 26.
We could use that cycle to focus on nothing but quality (fixing tests,
bug triaging, no feature development at all).

Thoughts?

On Tue, Dec 22, 2015 at 7:41 AM Mike Conley > wrote:

 I would support scheduled time[1] to do maintenance[2] and help
 improve our
 developer tooling and documentation. I'm less sure how to integrate
 such a
 thing in practice.

 [1]: A day, a week, heck maybe even a release cycle
 [2]: Where maintenance is fixing oranges, closing out papercuts,
 refactoring, etc.

 On 21 December 2015 at 17:35, > wrote:

 > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
 wrote:
 > > So, I propose that we create an orangefactor threshold above
 which the
 > > tree should just be closed until people start fixing intermittent
 > > oranges. Thoughts?
 > >
 > > kats
 >
 > How about regularly scheduled test fix days where everyone drops
 what they
 > are doing and spends a day fixing tests? mc could be closed to
 everything
 > except critical work and test fixes. Managers would be able to opt
 > individuals out of this as needed but generally everyone would be
 expected
 > to take part.
 >
 > Jim
 > ___
 > dev-platform mailing list
 > dev-platform@lists.mozilla.org 
dev-platform@lists.mozilla.org>

 > https://lists.mozilla.org/listinfo/dev-platform
 >
 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org 
dev-platform@lists.mozilla.org>

 https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[L. David Baron]

On Monday 2015-12-21 14:16 -0500, Kartikaya Gupta wrote:
> Is it just me, or has the number of intermittent oranges gone up quite
> a lot in the last couple of months? It seems like every try push I do
> has a lot of oranges which are completely unrelated to may patch.
> Clearly everbody has more urgent things to do than fix intermittent
> failures (myself included) but the net result is a lot of wasted time
> starring bugs and digging through failures to see if they are
> legitimate or not.
> 
> So, I propose that we create an orangefactor threshold above which the
> tree should just be closed until people start fixing intermittent
> oranges. Thoughts?

I agree it's definitely gone up recently, and agree that it causes a
lot of wasted time.  I'm not convinced about closing the tree,
though; keeping the tree closed for extended periods just leads to
big backups.

How about everybody reading this message takes a look at the list on
http://brasstacks.mozilla.com/orangefactor/ and takes one of them to
fix?  (Or, better, redoes the search filtered on the last 3 days
instead of last 7.)

-David

-- 
턞   L. David Baron http://dbaron.org/   턂
턢   Mozilla  https://www.mozilla.org/   턂
 Before I built a wall I'd ask to know
 What I was walling in or walling out,
 And to whom I was like to give offense.
   - Robert Frost, Mending Wall (1914)


signature.asc
Description: Digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[jmathies]

On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta wrote:
> So, I propose that we create an orangefactor threshold above which the
> tree should just be closed until people start fixing intermittent
> oranges. Thoughts?
> 
> kats

How about regularly scheduled test fix days where everyone drops what they are 
doing and spends a day fixing tests? mc could be closed to everything except 
critical work and test fixes. Managers would be able to opt individuals out of 
this as needed but generally everyone would be expected to take part.

Jim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Xidorn Quan]

On Tue, Dec 22, 2015 at 6:16 AM, Kartikaya Gupta  wrote:
> So, I propose that we create an orangefactor threshold above which the
> tree should just be closed until people start fixing intermittent
> oranges. Thoughts?

I think oranges on try is somehow different than oranges on
integration trees, because we currently do not run complete test set
for most of pushes on integration trees. This means you may not be
able to catch frequency of intermittents from integration trees to
reflect those you would meet in try pushes.

Also given we currently have auto-retrigger on try by default, I think
the most emergent issue for try push is some too frequent
intermittents which could usually happen even in continuous
retriggers. They waste most of the time. For example, I found bug
1113930 could happen in up to 9/10 retriggers on Linux 32bit.

- Xidorn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform