Hi all,
There have been a couple of threads related to test automation in B2G,
asking why we haven't caught some especially egregious regressions; the
kind that basically "break the phone".
To answer that, I'd like to describe how our on-device automation
currently works, and what we're doing to expand it so we can more
effectively address these concerns.
We currently have a smallish number of real devices, managed by WebQA,
hooked up to on-device automation. They run a bank of tests against a
number of branches several times a day. The devices are time-consuming
to manage, since they occasionally get wedged during flashing,
rebooting, or other operations, and require manual intervention to fix.
For this reason, and because it's been very difficult to obtain
significant numbers of devices for automation, we haven't been able to
run any tests frequently enough to provide per-commit coverage.
When tests fail, WebQA engages in a fairly time-consuming process of
investigation and bisection. In the case of the homescreen breakage
(caused by https://bugzilla.mozilla.org/show_bug.cgi?id=957086), our
on-device tests did break, and the team was in the process of
investigating these failures, which has to be done in order to be able
to create specific, actionable bugs.
Clearly, what we really want is to be able to run at least a small set
of tests per-commit, so that when things break, we don't need to spend
lots of time investigating...we will already know which commit caused
the problem, and can back it out or address it otherwise promptly.
That's exactly what we are planning for Q3, thanks to the Flame device.
Jonathan Hylands has developed a power harness for this that allows us
to remotely restart the phone, which addresses some of the device
management concerns. The A*Team, WebQA, and jhylands are working
together to get 30 Flames in automation, and to reduce their management
costs. This is enough to allow us to run a small set of functional and
performance tests per-commit, which should be enough to catch most
"break the phone" problems.
Another issue we've had with device testing is test result visibility;
currently, test results are available on Jenkins, for which you need VPN
access. This is awkward for people not closely involved in maintaining
and running the tests.
Next quarter, we will be improving this as well. Jonathan Eads on the
A*Team is currently in the process of deploying Treeherder, a successor
to TBPL. Unlike TBPL, Treeherder is not tightly coupled with buildbot,
and is capable of displaying test results from arbitrary data sources.
As our bank of 30 Flames becomes available, we will start publishing
on-device test results to Treeherder, in the same UI that will be used
to display the per-commit tests being run in buildbot. This will give
people a "one-stop shop" for seeing test results for B2G, regardless of
whether they're run on devices or in VM's managed by buildbot.
Both of these pieces together will give us the ability to manage some
on-device tests in a manner similar to the way we currently handle
desktop and emulator tests in TBPL; especially bad commits should break
tests, the breakage should be visible in Treeherder, and the sheriffs
will back out the offending commits.
We won't have enough device capacity to run all device tests per-commit,
at least at first. We'll have to carefully select a small set of tests
that guard against the worst kinds of breakage. Whether we can scale
beyond 30 devices will depend on how stable the devices are and what
their management costs are, which is something we'll be looking at over
the next few months.
Regards,
Jonathan
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
