On 08/31/2012 12:33 AM, Rafael Ávila de Espíndola wrote:
> I was recently hit by most of the shortcomings you mentioned while
> trying to upgrade clang. Fortunately, I found the issue on try, but I
> will admit that comparing talos on try is something I only do when I
> expect a problem.
I'm a bit
On 12-09-02 12:33 AM, Justin Wood (Callek) wrote:
Taras Glek wrote:
* Joel will revisit maintaining Talos within mozilla-central to reduce
developer barriers to understanding what a particular Talos test result
means. This should also make Talos easier to run
To call out this point explicitly.
On 8/31/2012 8:46 AM, Justin Lebar wrote:
There are extremely non-stable Talos tests, and relatively stable ones.
Let's focus on the relatively stable ones.
It's not exclusively a question of noise in the tests. Even
regressions in stable tests are sometimes hard to track down. I spent
two mo
Thank you for posting this - I've become increasingly worried by the number of
real regressions that we are likely missing due to the current situation.
Like yourself I used to read every single dev.tree-management mail & try to act
on those that looked real, however after many months of not fee
On 8/29/2012 5:00 PM, Matt Brubeck wrote:
* I don't like our reactive approach that focuses on trying to identify
regressions, and then decide whether to fix them in place, back them
out, or ignore them. Instead we should proactively set goals for what
our performance should be, and focus on th
Taras Glek wrote:
* Joel will revisit maintaining Talos within mozilla-central to reduce
developer barriers to understanding what a particular Talos test result
means. This should also make Talos easier to run
To call out this point explicitly.
I'm not convinced that folding it into m-c is the
On Saturday, September 1, 2012 10:08:53 AM UTC-4, Ehsan Akhgari wrote:
> On 12-08-31 4:03 PM, Chris AtLee wrote:
>
> > On 31/08/12 03:59 PM, Ehsan Akhgari wrote:
>
> >> On 12-08-31 11:45 AM, Chris AtLee wrote:
>
> >>> On 31/08/12 11:32 AM, Ehsan Akhgari wrote:> There are extremely
>
> >>> non-s
On 12-08-31 4:03 PM, Chris AtLee wrote:
On 31/08/12 03:59 PM, Ehsan Akhgari wrote:
On 12-08-31 11:45 AM, Chris AtLee wrote:
On 31/08/12 11:32 AM, Ehsan Akhgari wrote:> There are extremely
non-stable Talos tests, and relatively stable ones.
> Let's focus on the relatively stable ones. There
On 31/08/12 03:59 PM, Ehsan Akhgari wrote:
On 12-08-31 11:45 AM, Chris AtLee wrote:
On 31/08/12 11:32 AM, Ehsan Akhgari wrote:> There are extremely
non-stable Talos tests, and relatively stable ones.
> Let's focus on the relatively stable ones. There are extremely hard
> to diagnose perform
On 12-08-31 11:45 AM, Chris AtLee wrote:
On 31/08/12 11:32 AM, Ehsan Akhgari wrote:> There are extremely
non-stable Talos tests, and relatively stable ones.
> Let's focus on the relatively stable ones. There are extremely hard
> to diagnose performance regressions, and extremely easy ones (i
There are a few issues here which should be easy to address and a few other
issues which are not so easy to address.
First off everybody who is interested in talos should read the wiki page:
https://wiki.mozilla.org/Buildbot/Talos
This explains where the code lives, what tests we run and on whic
On 31/08/12 11:32 AM, Ehsan Akhgari wrote:> There are extremely
non-stable Talos tests, and relatively stable ones.
> Let's focus on the relatively stable ones. There are extremely hard
> to diagnose performance regressions, and extremely easy ones (i.e.,
> let's not wait on this lock, do this
> There are extremely non-stable Talos tests, and relatively stable ones.
> Let's focus on the relatively stable ones.
It's not exclusively a question of noise in the tests. Even
regressions in stable tests are sometimes hard to track down. I spent
two months trying to figure out why I could not
On 12-08-31 5:37 AM, Justin Lebar wrote:
Sorry to continue beating this horse, but I don't think it's quite dead yet:
One of the best things we could do to make finding these regressions
easier is to never coalesce Talos on mozilla-inbound. It's crazy to
waste developer time bisecting Talos loc
On 12-08-31 6:01 AM, Justin Lebar wrote:
I'm not saying it should be OK to regress our performance tests, as a
rule. But I think we need to acknowledge that hunting regressions can
be time-consuming, and that a policy requiring that all regressions be
understood may hamstring our ability to get
> IMHO we *can* regress on synthetic ones as long as we know what is going on.
It's the requirement that we know what is going on that I think is unreasonable.
Indeed, we /have/ a no not-understood regresisons policy, IIRC. The
extent to which it's being ignored is at least partially indicative
Sorry to continue beating this horse, but I don't think it's quite dead yet:
One of the best things we could do to make finding these regressions
easier is to never coalesce Talos on mozilla-inbound. It's crazy to
waste developer time bisecting Talos locally when we don't run it on
every push.
A
On Thursday, August 30, 2012 9:05:40 PM UTC-4, Ben Hearsum wrote:
> On 08/30/12 07:22 PM, L. David Baron wrote:
>
> > On Thursday 2012-08-30 14:42 -0700, Taras Glek wrote:
>
> >> * Joel will revisit maintaining Talos within mozilla-central to
>
> >> reduce developer barriers to understanding wha
ollaborative dynamic.
- Original Message -
From: "Dave Mandelin"
To: dev-platform@lists.mozilla.org
Cc: "Dave Mandelin" , dev-platform@lists.mozilla.org
Sent: Thursday, August 30, 2012 6:13:33 PM
Subject: Re: The current state of Talos benchmarks
On Thursday, August 30, 2012 2:5
On 31/08/12 13:13, Dave Mandelin wrote:
Otherwise, it seems we just have to share the pain. Bisecting changesets is not
necessarily an enjoyable job but it is a necessary one. I would suggest that
sheriffs pick one of the 5 committers and ask that person to bisect the change
and try not to pic
On Thursday, August 30, 2012 2:54:55 PM UTC-7, Ehsan Akhgari wrote:
> Oh, sorry, I needed to ask my question better. I'm specifically
> wondering who needs to track and investigate the regression if it
> happened on a range of, let's say, 5 committers...
Ah. I believe that's a job for a bugmast
On 08/30/12 07:22 PM, L. David Baron wrote:
> On Thursday 2012-08-30 14:42 -0700, Taras Glek wrote:
>> * Joel will revisit maintaining Talos within mozilla-central to
>> reduce developer barriers to understanding what a particular Talos
>> test result means. This should also make Talos easier to ru
Some people have noted in the past that some Talos measurements are not
representative of something that the users would see, the Talos numbers
are noisy, and we don't have good tools to deal with these types of
regressions. There might be some truth to all of these, but I believe
that the bigger
On Thursday 2012-08-30 14:42 -0700, Taras Glek wrote:
> * Joel will revisit maintaining Talos within mozilla-central to
> reduce developer barriers to understanding what a particular Talos
> test result means. This should also make Talos easier to run
This will also solve one of the other problems
On 12-08-30 5:42 PM, Taras Glek wrote:
* Joel will revisit maintaining Talos within mozilla-central to reduce
developer barriers to understanding what a particular Talos test result
means. This should also make Talos easier to run
I have filed bug 787200 for this discussion.
Ehsan
On 12-08-30 5:28 PM, Dave Mandelin wrote:
On Thursday, August 30, 2012 9:11:25 AM UTC-7, Ehsan Akhgari wrote:
On 12-08-29 9:20 PM, Dave Mandelin wrote:
On Wednesday, August 29, 2012 4:03:24 PM UTC-7, Ehsan Akhgari wrote:
In my opinion, one of the reasons why Talos is disliked is because many
Hi,
We had a quick meeting focused on how to not regress Talos.
Attendance: Taras Glek, Ehsan Akhgari, Clint Talbert, Nathan Froyd, Dave
Mandelin, Christina Choi, Joel Maher
Notes:
* Clint's Automation&Tools team is improving Talos reporting abilities.
We should have much better tools for c
On Thursday, August 30, 2012 9:11:25 AM UTC-7, Ehsan Akhgari wrote:
> On 12-08-29 9:20 PM, Dave Mandelin wrote:
>
> > On Wednesday, August 29, 2012 4:03:24 PM UTC-7, Ehsan Akhgari wrote:
>
> In my opinion, one of the reasons why Talos is disliked is because many
> people don't know where its cod
On 12-08-29 9:46 PM, Joshua Cranmer wrote:
On 8/29/2012 6:03 PM, Ehsan Akhgari wrote:
Hi everyone,
The Talos regression detection emails caught a number of regressions
during the Monday uplift (see [1] for Aurora and [2] for Beta
regressions). To put things into perspective, I prepared a
sprea
On 12-08-29 9:20 PM, Dave Mandelin wrote:
On Wednesday, August 29, 2012 4:03:24 PM UTC-7, Ehsan Akhgari wrote:
Hi everyone,
The way the current situation happens is that many of the developers
ignore the Talos regression emails that go to dev-tree-management,
Talos is widely disliked and dist
On 12-08-30 2:34 AM, Mike Hommey wrote:
On Wed, Aug 29, 2012 at 07:03:17PM -0400, Ehsan Akhgari wrote:
I don't believe that the current situation is acceptable, especially
with the recent focus on performance (through the Snappy project),
and I would like to ask people if they have any ideas on
On 08/29/2012 07:32 PM, Nicholas Nethercote wrote:
> In my experience, a lot of those emails say "there was a regression
> caused by one of the following 100 patches", and I will have written 1
> of those patches. I usually ignore those ones (though it depends on
> the nature of the patch).
>
>
On 08/29/2012 09:54 PM, Robert O'Callahan wrote:
> On Thu, Aug 30, 2012 at 1:00 PM, Ehsan Akhgari wrote:
>
>> I agree with that if we talk about performance in general. But this
>> thread is about specific regressions in performance as a result of
>> changeset going into our tree. I don't think
I've just filed bug 786978 for this.
- Matt
- Original Message -
From: "Ehsan Akhgari"
To: rob...@ocallahan.org
Cc: dev-platform@lists.mozilla.org
Sent: Thursday, August 30, 2012 1:02:43 PM
Subject: Re: The current state of Talos benchmarks
On 12-08-29 8:41 PM, Robert O
On Thu, Aug 30, 2012 at 3:34 AM, Mike Hommey wrote:
> Ideally, we should make talos regressions visible on tbpl as oranges,
> and star them as other oranges.
FWIW, making this possible is an explicit goal of the SfN effort.
-Justin
___
dev-platform mai
On Wed, Aug 29, 2012 at 07:03:17PM -0400, Ehsan Akhgari wrote:
> I don't believe that the current situation is acceptable, especially
> with the recent focus on performance (through the Snappy project),
> and I would like to ask people if they have any ideas on what we can
> do to fix this. The fi
On Thu, Aug 30, 2012 at 1:00 PM, Ehsan Akhgari wrote:
> I agree with that if we talk about performance in general. But this
> thread is about specific regressions in performance as a result of
> changeset going into our tree. I don't think the same argument applies
> here, unless we decide that
On 8/29/2012 6:03 PM, Ehsan Akhgari wrote:
Hi everyone,
The Talos regression detection emails caught a number of regressions
during the Monday uplift (see [1] for Aurora and [2] for Beta
regressions). To put things into perspective, I prepared a
spreadsheet of the most notable performance re
On Wednesday, August 29, 2012 4:03:24 PM UTC-7, Ehsan Akhgari wrote:
> Hi everyone,
>
> The way the current situation happens is that many of the developers
> ignore the Talos regression emails that go to dev-tree-management,
Talos is widely disliked and distrusted by developers, because it's h
On Wed, Aug 29, 2012 at 8:00 PM, Matt Brubeck wrote:
> On 08/29/2012 04:03 PM, Ehsan Akhgari wrote:
>
>> I don't believe that the current situation is acceptable, especially
>> with the recent focus on performance (through the Snappy project), and I
>> would like to ask people if they have any id
On 12-08-29 8:41 PM, Robert O'Callahan wrote:
Some of the 16->17 regressions are known and due to DLBI patches (bug
539356). Since we don't have full DLBI on trunk yet, those changes should
just be preffed off on Aurora for 17. We should do that and see how that
affects the numbers. Matt Woodrow
On 12-08-29 8:56 PM, Anthony Jones wrote:
On 30/08/12 12:10, Justin Lebar wrote:
More on topic: I think the essential problem is, you can spend a week
chasing down a perf regression when there's a good chance it's not
your fault (and also a good chance it's not a regression). So people
are maki
On 12-08-29 8:10 PM, Justin Lebar wrote:
After getting an e-mail every single time m-c was merged for a day or
two, I filtered the e-mails and completely forgot about them. I
imagine most other people did the same. If we fix bug 752002, we'd
also need to change the e-mails so as to get around e
On 30/08/12 12:10, Justin Lebar wrote:
More on topic: I think the essential problem is, you can spend a week
chasing down a perf regression when there's a good chance it's not
your fault (and also a good chance it's not a regression). So people
are making a reasonable trade-off here when they ig
Some of the 16->17 regressions are known and due to DLBI patches (bug
539356). Since we don't have full DLBI on trunk yet, those changes should
just be preffed off on Aurora for 17. We should do that and see how that
affects the numbers. Matt Woodrow will take care of that :-).
Rob
--
“You have h
After getting an e-mail every single time m-c was merged for a day or
two, I filtered the e-mails and completely forgot about them. I
imagine most other people did the same. If we fix bug 752002, we'd
also need to change the e-mails so as to get around everyone's
existing filters.
More on topic:
On 08/29/2012 04:03 PM, Ehsan Akhgari wrote:
I don't believe that the current situation is acceptable, especially
with the recent focus on performance (through the Snappy project), and I
would like to ask people if they have any ideas on what we can do to fix
this. The fix might be turning off s
On 12-08-29 7:32 PM, Nicholas Nethercote wrote:
On Wed, Aug 29, 2012 at 4:03 PM, Ehsan Akhgari wrote:
Some people have noted in the past that some Talos measurements are not
representative of something that the users would see, the Talos numbers are
noisy, and we don't have good tools to deal
On 08/29/2012 04:32 PM, Nicholas Nethercote wrote:
In my experience, a lot of those emails say "there was a regression
caused by one of the following 100 patches", and I will have written 1
of those patches. I usually ignore those ones (though it depends on
the nature of the patch).
But if I ge
On Wed, Aug 29, 2012 at 4:03 PM, Ehsan Akhgari wrote:
>
> Some people have noted in the past that some Talos measurements are not
> representative of something that the users would see, the Talos numbers are
> noisy, and we don't have good tools to deal with these types of regressions.
> There mig
Hi everyone,
The Talos regression detection emails caught a number of regressions
during the Monday uplift (see [1] for Aurora and [2] for Beta
regressions). To put things into perspective, I prepared a spreadsheet
of the most notable performance regressions [3] (and please do take a
look at
51 matches
Mail list logo
