Some tests are more likely than others to fail in response to a given
change, but our automation does little to optimize based on this. As a
result, test automation frequently runs more than necessary for a given
push, leading to excess load on automation and time spent waiting for the
results of tests unlikely to be impacted by a given push.
Work is ongoing on an approach to select/prioritize tests based on the
files that a push changes. The idea is to use build system metadata to map
changed files from source control to impacted tests. This will be based on
the test files included in manifests mentioned in moz.build files along the
path to a given file, and a syntax to specify other relationships between
source files and tests. The syntax isn’t finalized, but the proposal is
that an annotation such as:
with Files(‘testing/mozbase/**’):
IMPACTED_TESTS += [
‘testing/mochitest/tests/**’,
]
in a moz.build file will suggest that any file change under
‘testing/mozbase’ is potentially relevant to the mochitest sanity tests.
Specifying tests by tag or test flavor (so you can run all reftests when
parts of layout change, for instance) will be supported in addition to
wildcard file matching.
Tools will use this data to run a subset of tests either before or in place
of a full set of tests. Initially this will be an option for |./mach try|
and |./mach test|, and upcoming work will make it possible to take
advantage of this on integration branches and in coordination with
autoland. There’s also potential to refine and automate this process
further with auxiliary sources of data (such as code coverage), and longer
term I’d like to see annotations become a supplement to a more automatic
prioritization algorithm.
This isn’t intended to be a complete solution, or to replace the current
pipeline: there will always be interactions between code and tests that
can’t reasonably be anticipated. It is, however, an optimization that has
the potential to shorten feedback cycles on Try, help improve integration
tree availability by shortening time to detect regressions, and reduce
overall infrastructure load.
If you’d like to follow this or give feedback on the proposed syntax and
heuristics, initial work is tracked in bug 1184405.
Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
