On 09/02/16 19:51, Marco Bonardo wrote:
On Tue, Feb 9, 2016 at 6:54 PM, Ryan VanderMeulen <rya...@gmail.com> wrote:
I'd have a much easier time accepting that argument if my experience
didn't tell me that nearly every single "Test took longer than expected" or
"Test timed out" intermittent ends with a RequestLongerTimeout as the fix
this sounds equivalent to saying "Since we don't have enough resources (or
a plan) to investigate why some tests take so long, let's give up"... But
then maybe we should have that explicit discussion, rather than assuming
it's a truth.
FWIW I think it's closer to the truth to say that these tests are not
set up to be performance regression tests and as such they are difficult
to use as incidential tests for that use case. For example we don't run
them on machines with well-defined performance characteristics, don't
make any effort to keep the tests themselves unchanged over time, and
don't track the test runtime carefully in order to notice regressions.
Using the test timeout as a threshold is a poor substitute because we
will miss large regressions that nevertheless allow the test to finish
inside the timeout, but be indirectly alerted (via intermittency) to
smaller regressions, or test changes, for tests that were already close
to the limit.
This isn't to say that we should never care if tests get slower, but
that isn't a thing that we can reliably determine in our current setup,
except in very coarse ways that are ineffective at directing engineering
time onto the most important issues.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
