TL;DR - The a-team and rel-eng are currently preparing to get gaia unit
tests running in TBPL
(https://bugzilla.mozilla.org/show_bug.cgi?id=833666), on both trunk
gecko branches and gaia branches (e.g.,
https://tbpl.mozilla.org/?tree=Gaia-Master). We have a plan to
integrate these tests into the "main" TBPL pages for mozilla-inbound,
etc, and to provide try access to gaia developers. We'd like feedback to
ensure that this plan will satisfy developers before we begin
implementation.
== What is already in progress ==
All the harness work has been done to get gaia unit tests into
buildbot/TBPL (https://bugzilla.mozilla.org/show_bug.cgi?id=833666),
running on b2g desktop builds. There is some additional buildbot work
that is being done to enable these to be turned on for the "cedar"
project branch (https://bugzilla.mozilla.org/show_bug.cgi?id=865379).
Once we shake out any remaining bugs in production there, we will turn
these tests on for both gecko trunk branches, like mozilla-inbound, and
gaia branches, like gaia-master.
We are additionally endeavoring to get gaia ui tests (in Python) stable
on b2g desktop builds
(https://bugzilla.mozilla.org/show_bug.cgi?id=857622). We have been
running them on pandas for some time in buildbot, but the panda build is
not well-maintained enough for large-scale automation, and it is hard
for developers to debug panda failures, since most of them do not have
panda boards, and working with panda boards is not as easy as working
with other b2g builds. Is there any objection to turning these panda
tests off?
WebQA is also currently running gaia ui tests on unagi's in a Jenkins
cluster. We can't run these in buildbot, nor have them report to TBPL
in the near term, but we do have a longer-term plan to allow them to
report to TBPL
(https://wiki.mozilla.org/Auto-tools/Projects/TreeHerder_2013Q2).
After work for gaia unit and ui tests is completed, we will turn our
attention to the gaia integration tests (in JS)
(https://bugzilla.mozilla.org/show_bug.cgi?id=866909), but the gaia
developers will first have to get these working reliably.
== What we are planning to do ==
The current state of gaia tests in TBPL is not ideal. Among other
things, sheriffs can't always easily identify gaia regressions, and gaia
developers have no access to try. Having gaia results on separate pages
from gecko activity makes them harder (or sometimes impossible) to
sheriff effectively, and reduces their visibility and relevance. There
have been multiple plans floated to deal with this, but at the recent
b2g work week, a few of the B2G leads proposed an alternative.
There are two primary pieces to this proposal:
1) - couple gecko and gaia commits in buildbot so that there is no
uncertainty about which commits are used to create a build, and so that
gaia test results appear on "main" TBPL pages like mozilla-inbound and
mozilla-central
2) - create a gaia-try repo that would allow gaia developers to push
patches to try
This proposal initially will only target b2g desktop builds running
against mozilla-central and gaia's master branch.
== Coupling gecko and gaia commits ==
This would be accomplished by adding a file to mozilla-central that
contains a pointer to a gaia commit. Whenever buildbot produces a b2g
desktop build, it would use the corresponding gaia commit in the build.
When a gaia developer commits a patch to gaia's master branch, a
corresponding commit would be automatically submitted to gecko
(mozilla-inbound, or possibly birch) that would change gecko's gaia
pointer. This auto-commit would automatically trigger a full set of
test runs, as happens with any other gecko commit.
The gecko auto-commit would use the same commit message as the original
gaia commit. For example, if the original gaia commit was "Bug 865091 -
Enable systemXhr so youtube bypasses CORS. r=daleharvey", the gecko
auto-commit might be something like "Bug 865091 - Enable systemXhr so
youtube bypasses CORS. r=daleharvey, [gaia auto-commit]".
If an auto-commit caused a test to go orange, it would be easily
identifiable by the sheriffs, and they would back out the original gaia
commit.
== Gaia and tryserver ==
We will provide a gaia-try repo, and gaia developers may submit a patch
to this if they would like to kick off a tryserver run.
We realize that gaia developers would prefer to commit to a github repo
for this purpose, but that would require significant changes to our
build/test infrastructure such that it isn't feasible this quarter.
What we propose as a first iteration for try support is as follows:
* provide a gaia-try repo in hg
* when a gaia developer wishes to push to try, he would commit his gaia
patch to this hg repo (we would provide very easy-to-follow
instructions), and then commit another patch to the "regular" hg gecko
try repo (http://hg.mozilla.org/try/) that updates the gaia pointer;
this gecko commit would use normal trychooser syntax
(http://trychooser.pub.build.mozilla.org/).
In subsequent iterations of gaia-try (to be done post Q2), we could
improve the process by moving to a git-based gaia-try repo, and
providing a mechanism that would allow gaia developers to omit the gecko
try push if they were only interested in testing gaia changes using
mozilla-central's head revision. Developers that have coupled gecko/gaia
patches would still need to perform two commits, one to gaia-try and one
to gecko-try.
We believe this plan (sans future gaia-try improvements) is achievable
in a reasonable amount of time. Does this plan meet the needs of b2g
developers, and does it provide enough benefit beyond our current
ability to display separate pages for gecko and gaia in TBPL that it
merits the time it will take to implement?
We'd like to arrive at a decision by the end of the week, so that we may
begin implementation if there is general agreement.
Jonathan, on behalf of rel-eng and the a-team
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
