Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Armen Zambrano G.]

On 16-02-10 04:58 AM, Marco Bonardo wrote:
> On Wed, Feb 10, 2016 at 10:30 AM, James Graham 
> wrote:
> 
>> FWIW I think it's closer to the truth to say that these tests are not set
>> up to be performance regression tests
>>
> 
> Right, but this was just one of the aspects that was pointed out. I think
> the performance analysis is not much about catching regressions (even if
> that can effectively happen), rather figuring why a test takes 90 seconds
> instead of 10, is that a problem that may affect end users too?
> The larger we set the threshold, the less people are likely to investigate
> the reasons the test takes so much, cause there will be no bugs filed about
> the problem. It will just go unnoticed.
> Other reasons are bound to costs imo. Developers time is a cost, if
> everyone starts writing tests taking more time, cause now the timeout is
> much larger, we may double the time to run tests locally or to get results
> out of Try.
> Finally, bumping timeouts isn't a final solution, that means in a few years
> we may be back rediscussing another bump.
> 
> I'm not saying bumping the timeout is wrong, I'm saying we should also
> evaluate a long term strategy to avoid the downsides. Maybe we should have
> something like the orange factor tracking tests runtime (in all the
> harnesses) and automatically filing bugs in the appropriate component when
> some test goes over a threshold?
> 

In this case it would be better to have a tool that keeps track of test
runtimes and report regressions.
ActiveData has the data to do this.


-- 
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Armen Zambrano G.]

On 16-02-09 01:32 PM, Daniel Holbert wrote:
> Just to clarify, you're *only* talking about browser-chrome mochitests
> here, correct?  (not other mochitest suites like mochitest-plain)
> 
> (It looks like this is the case, based on the bug, but your dev.platform
> post here made it sound like this change affected all mochitests.)
> 
> Thanks,
> ~Daniel
I'm interested in browser-chrome. Is there a different variable for
those tests?

> 
> On 02/08/2016 02:51 PM, Armen Zambrano G. wrote:
>> Hello,
>> In order to help us have less timeouts when running mochitests under
>> docker, we've decided to double mochitests' gTimeoutSeconds and reduce
>> large multipliers in half.
>>
>> Here's the patch if you're curious:
>> https://bugzilla.mozilla.org/page.cgi?id=splinter.html=1246152=8717111
>>
>> If you have any comments or concerns please raise them in the bug.
>>
>> regards,
>> Armen
>>


-- 
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Daniel Holbert]

On 02/12/2016 11:02 AM, Armen Zambrano G. wrote:
> On 16-02-09 01:32 PM, Daniel Holbert wrote:
>> Just to clarify, you're *only* talking about browser-chrome mochitests
>> here, correct?  (not other mochitest suites like mochitest-plain)
>>
>> (It looks like this is the case, based on the bug, but your dev.platform
>> post here made it sound like this change affected all mochitests.)
>>
>> Thanks,
>> ~Daniel
> I'm interested in browser-chrome. Is there a different variable for
> those tests?

I don't know anything about browser-chrome tests. But I do know that
mochitest-plain at least has a 5 minute (300 second) timeout, by
default. (That's what SimpleTest.requestLongerTimeout(...) operates on
for those tests.)

I believe that's defined here:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/TestRunner.js?rev=debc5269aeac#91

~Daniel
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[James Graham]

On 09/02/16 19:51, Marco Bonardo wrote:

On Tue, Feb 9, 2016 at 6:54 PM, Ryan VanderMeulen  wrote:


I'd have a much easier time accepting that argument if my experience
didn't tell me that nearly every single "Test took longer than expected" or
"Test timed out" intermittent ends with a RequestLongerTimeout as the fix



this sounds equivalent to saying "Since we don't have enough resources (or
a plan) to investigate why some tests take so long, let's give up"... But
then maybe we should have that explicit discussion, rather than assuming
it's a truth.


FWIW I think it's closer to the truth to say that these tests are not 
set up to be performance regression tests and as such they are difficult 
to use as incidential tests for that use case. For example we don't run 
them on machines with well-defined performance characteristics, don't 
make any effort to keep the tests themselves unchanged over time, and 
don't track the test runtime carefully in order to notice regressions. 
Using the test timeout as a threshold is a poor substitute because we 
will miss large regressions that nevertheless allow the test to finish 
inside the timeout, but be indirectly alerted (via intermittency) to 
smaller regressions, or test changes, for tests that were already close 
to the limit.


This isn't to say that we should never care if tests get slower, but 
that isn't a thing that we can reliably determine in our current setup, 
except in very coarse ways that are ineffective at directing engineering 
time onto the most important issues.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Marco Bonardo]

On Wed, Feb 10, 2016 at 10:30 AM, James Graham 
wrote:

> FWIW I think it's closer to the truth to say that these tests are not set
> up to be performance regression tests
>

Right, but this was just one of the aspects that was pointed out. I think
the performance analysis is not much about catching regressions (even if
that can effectively happen), rather figuring why a test takes 90 seconds
instead of 10, is that a problem that may affect end users too?
The larger we set the threshold, the less people are likely to investigate
the reasons the test takes so much, cause there will be no bugs filed about
the problem. It will just go unnoticed.
Other reasons are bound to costs imo. Developers time is a cost, if
everyone starts writing tests taking more time, cause now the timeout is
much larger, we may double the time to run tests locally or to get results
out of Try.
Finally, bumping timeouts isn't a final solution, that means in a few years
we may be back rediscussing another bump.

I'm not saying bumping the timeout is wrong, I'm saying we should also
evaluate a long term strategy to avoid the downsides. Maybe we should have
something like the orange factor tracking tests runtime (in all the
harnesses) and automatically filing bugs in the appropriate component when
some test goes over a threshold?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Gijs Kruitbosch]

I concur with Ryan here, and I'd add that IME 90% if not more of these 
timeouts (where they are really timeouts because the test is long, 
rather than just brokenness in the test that leaves it hanging until the 
timeout) happen on debug/asan builds, where "perf regressions" isn't 
really a meaningful concept to regression-analyze for, compared to the 
debug and asan overhead.


~ Gijs

On 09/02/2016 17:54, Ryan VanderMeulen wrote:

I'd have a much easier time accepting that argument if my experience
didn't tell me that nearly every single "Test took longer than expected"
or "Test timed out" intermittent ends with a RequestLongerTimeout as the
fix.

-Ryan

On 2/9/2016 12:50 PM, Haik Aftandilian wrote:

On Tue, Feb 9, 2016 at 2:47 AM, Marco Bonardo 
wrote:


Based on that, bumping the timeout may have 2 downsides, long term:
- slower tests for everyone
- sooner or later 90 seconds won't be enough again. Are we going to
bump to
180 then?



Essentially restating Marco's concern, increasing timeouts has the side
effect where performance regressions are not noticed. i.e., a new bug
that
causes a test to take longer, but still pass, is not detected. With the
original lower timeouts, the test would fail with a timeout. So a little
bit of the value of the tests is lost, and it's difficult to address
later.

Haik




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Marco Bonardo]

On Tue, Feb 9, 2016 at 8:09 PM, Gijs Kruitbosch 
wrote:

> I concur with Ryan here, and I'd add that IME 90% if not more of these
> timeouts (where they are really timeouts because the test is long, rather
> than just brokenness in the test that leaves it hanging until the timeout)
> happen on debug/asan builds


The current timeout was already setup when we had debug tests, so it is
already accounting for that. We should at a maximum reduce the timeout on
opt if that matters.

I think it would be fine to have much bigger timeouts on asan or other
builds that are not our primary target. debug and opt are close enough to
what we ship and what devs use everyday that performance matters too.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Marco Bonardo]

On Tue, Feb 9, 2016 at 6:54 PM, Ryan VanderMeulen  wrote:

> I'd have a much easier time accepting that argument if my experience
> didn't tell me that nearly every single "Test took longer than expected" or
> "Test timed out" intermittent ends with a RequestLongerTimeout as the fix


this sounds equivalent to saying "Since we don't have enough resources (or
a plan) to investigate why some tests take so long, let's give up"... But
then maybe we should have that explicit discussion, rather than assuming
it's a truth.
Since we are focused on quality I don't think it's acceptable to say we are
fine if a test takes an unexpected amount of time to run. The fact those
bugs end up being resolved by bumping the timeout without any kind of
investigation (and it happens, I know) is worrisome.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Marco Bonardo]

90 seconds for a simple test sounds like a lot of time and a huge bump from
the current situation (45).
The risk is people will start writing much bigger tests instead of
splitting them into smaller an more manageable tests. Plus when a test
depends on a long timeout in the product, developers are used to figure out
ways to reduce those (through hidden prefs or such) so that test can finish
sooner and not timeout.
Based on that, bumping the timeout may have 2 downsides, long term:
- slower tests for everyone
- sooner or later 90 seconds won't be enough again. Are we going to bump to
180 then?

I think that's the main reason the default timeout was set to a low value,
while still allowing the multipliers as a special case for tests that
really require bigger times, cause there's no other way out.

Is docker doubling the time for every test? From the bug looks like it may
add 20-30% of overhead, so why are we not bumping the timeout of 30% (let's
say 60s) and investigating the original cause (the bug that takes 80s to
run) to figure if something can be done to make it finish sooner?

-m


On Mon, Feb 8, 2016 at 11:51 PM, Armen Zambrano G. 
wrote:

> Hello,
> In order to help us have less timeouts when running mochitests under
> docker, we've decided to double mochitests' gTimeoutSeconds and reduce
> large multipliers in half.
>
> Here's the patch if you're curious:
>
> https://bugzilla.mozilla.org/page.cgi?id=splinter.html=1246152=8717111
>
> If you have any comments or concerns please raise them in the bug.
>
> regards,
> Armen
>
> --
> Zambrano Gasparnian, Armen
> Automation & Tools Engineer
> http://armenzg.blogspot.ca
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Armen Zambrano G.]

I will try 60 seconds and see how it goes.

On 16-02-09 05:47 AM, Marco Bonardo wrote:
> 90 seconds for a simple test sounds like a lot of time and a huge bump from
> the current situation (45).
> The risk is people will start writing much bigger tests instead of
> splitting them into smaller an more manageable tests. Plus when a test
> depends on a long timeout in the product, developers are used to figure out
> ways to reduce those (through hidden prefs or such) so that test can finish
> sooner and not timeout.
> Based on that, bumping the timeout may have 2 downsides, long term:
> - slower tests for everyone
> - sooner or later 90 seconds won't be enough again. Are we going to bump to
> 180 then?
> 
> I think that's the main reason the default timeout was set to a low value,
> while still allowing the multipliers as a special case for tests that
> really require bigger times, cause there's no other way out.
> 
> Is docker doubling the time for every test? From the bug looks like it may
> add 20-30% of overhead, so why are we not bumping the timeout of 30% (let's
> say 60s) and investigating the original cause (the bug that takes 80s to
> run) to figure if something can be done to make it finish sooner?
> 
> -m
> 
> 
> On Mon, Feb 8, 2016 at 11:51 PM, Armen Zambrano G. 
> wrote:
> 
>> Hello,
>> In order to help us have less timeouts when running mochitests under
>> docker, we've decided to double mochitests' gTimeoutSeconds and reduce
>> large multipliers in half.
>>
>> Here's the patch if you're curious:
>>
>> https://bugzilla.mozilla.org/page.cgi?id=splinter.html=1246152=8717111
>>
>> If you have any comments or concerns please raise them in the bug.
>>
>> regards,
>> Armen
>>
>> --
>> Zambrano Gasparnian, Armen
>> Automation & Tools Engineer
>> http://armenzg.blogspot.ca
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>


-- 
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Chris AtLee]

On 9 February 2016 at 14:51, Marco Bonardo  wrote:

> On Tue, Feb 9, 2016 at 6:54 PM, Ryan VanderMeulen 
> wrote:
>
> > I'd have a much easier time accepting that argument if my experience
> > didn't tell me that nearly every single "Test took longer than expected"
> or
> > "Test timed out" intermittent ends with a RequestLongerTimeout as the fix
>
>
> this sounds equivalent to saying "Since we don't have enough resources (or
> a plan) to investigate why some tests take so long, let's give up"... But
> then maybe we should have that explicit discussion, rather than assuming
> it's a truth.
> Since we are focused on quality I don't think it's acceptable to say we are
> fine if a test takes an unexpected amount of time to run. The fact those
> bugs end up being resolved by bumping the timeout without any kind of
> investigation (and it happens, I know) is worrisome.
>

I agree. However, this has traditionally been a very difficult area for
Release Engineering and Engineering Productivity to make progress in.

Who can we work with to understand these timing characteristics in more
depth?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Haik Aftandilian]

On Tue, Feb 9, 2016 at 2:47 AM, Marco Bonardo  wrote:

> Based on that, bumping the timeout may have 2 downsides, long term:
> - slower tests for everyone
> - sooner or later 90 seconds won't be enough again. Are we going to bump to
> 180 then?
>

Essentially restating Marco's concern, increasing timeouts has the side
effect where performance regressions are not noticed. i.e., a new bug that
causes a test to take longer, but still pass, is not detected. With the
original lower timeouts, the test would fail with a timeout. So a little
bit of the value of the tests is lost, and it's difficult to address later.

Haik


>
> I think that's the main reason the default timeout was set to a low value,
> while still allowing the multipliers as a special case for tests that
> really require bigger times, cause there's no other way out.
>
> Is docker doubling the time for every test? From the bug looks like it may
> add 20-30% of overhead, so why are we not bumping the timeout of 30% (let's
> say 60s) and investigating the original cause (the bug that takes 80s to
> run) to figure if something can be done to make it finish sooner?
>
> -m
>
>
> On Mon, Feb 8, 2016 at 11:51 PM, Armen Zambrano G. 
> wrote:
>
> > Hello,
> > In order to help us have less timeouts when running mochitests under
> > docker, we've decided to double mochitests' gTimeoutSeconds and reduce
> > large multipliers in half.
> >
> > Here's the patch if you're curious:
> >
> >
> https://bugzilla.mozilla.org/page.cgi?id=splinter.html=1246152=8717111
> >
> > If you have any comments or concerns please raise them in the bug.
> >
> > regards,
> > Armen
> >
> > --
> > Zambrano Gasparnian, Armen
> > Automation & Tools Engineer
> > http://armenzg.blogspot.ca
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To bump mochitest's timeout from 45 seconds to 90 seconds

[Ryan VanderMeulen]

I'd have a much easier time accepting that argument if my experience 
didn't tell me that nearly every single "Test took longer than expected" 
or "Test timed out" intermittent ends with a RequestLongerTimeout as the 
fix.


-Ryan

On 2/9/2016 12:50 PM, Haik Aftandilian wrote:

On Tue, Feb 9, 2016 at 2:47 AM, Marco Bonardo  wrote:


Based on that, bumping the timeout may have 2 downsides, long term:
- slower tests for everyone
- sooner or later 90 seconds won't be enough again. Are we going to bump to
180 then?



Essentially restating Marco's concern, increasing timeouts has the side
effect where performance regressions are not noticed. i.e., a new bug that
causes a test to take longer, but still pass, is not detected. With the
original lower timeouts, the test would fail with a timeout. So a little
bit of the value of the tests is lost, and it's difficult to address later.

Haik


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

To bump mochitest's timeout from 45 seconds to 90 seconds

[Armen Zambrano G.]

Hello,
In order to help us have less timeouts when running mochitests under
docker, we've decided to double mochitests' gTimeoutSeconds and reduce
large multipliers in half.

Here's the patch if you're curious:
https://bugzilla.mozilla.org/page.cgi?id=splinter.html=1246152=8717111

If you have any comments or concerns please raise them in the bug.

regards,
Armen

-- 
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform