On 5/1/2013 4:35 PM, Jonas Sicking wrote:
On Mon, Apr 29, 2013 at 2:51 PM, Jonathan Griffin <[email protected]> wrote:
TL;DR - The a-team and rel-eng are currently preparing to get gaia unit
tests running in TBPL (https://bugzilla.mozilla.org/show_bug.cgi?id=833666),
on both trunk gecko branches and gaia branches (e.g.,
https://tbpl.mozilla.org/?tree=Gaia-Master). We have a plan to integrate
these tests into the "main" TBPL pages for mozilla-inbound, etc, and to
provide try access to gaia developers. We'd like feedback to ensure that
this plan will satisfy developers before we begin implementation.
== What is already in progress ==
All the harness work has been done to get gaia unit tests into buildbot/TBPL
(https://bugzilla.mozilla.org/show_bug.cgi?id=833666), running on b2g
desktop builds. There is some additional buildbot work that is being done
to enable these to be turned on for the "cedar" project branch
(https://bugzilla.mozilla.org/show_bug.cgi?id=865379). Once we shake out
any remaining bugs in production there, we will turn these tests on for both
gecko trunk branches, like mozilla-inbound, and gaia branches, like
gaia-master.
Does this mean that we would enable these tests to show up on
mozilla-inbound before we have the "coupling" discussed below? That
worries me a bit since it might further fan the bad reputation that
gaia issues on TBPL are hard to understand.
I'd almost say that I'd prefer to only surface these results on cedar
and gaia until we have the coupling that allows us to have more
certainty in the data.
Or are you saying that we wouldn't turn this on in mozilla-inbound
until we have the coupling?
I'm even thinking that we might not want to turn things on in
mozilla-inbound until we have try support for gaia. Otherwise we are
putting gaia developers that got backed out in a difficult position of
not being able to test their fixes.
You're right; I was thinking we might want to hide these tests until
this coupling was in place. We could also decide to leave them hidden
until gaia-try was operational.
But, for gaia-unit-tests, the developers can easily run them locally, so
they may not need try access to debug failures there, unless we begin to
see machine-specific failures.
We are additionally endeavoring to get gaia ui tests (in Python) stable on
b2g desktop builds (https://bugzilla.mozilla.org/show_bug.cgi?id=857622).
We have been running them on pandas for some time in buildbot, but the panda
build is not well-maintained enough for large-scale automation, and it is
hard for developers to debug panda failures, since most of them do not have
panda boards, and working with panda boards is not as easy as working with
other b2g builds. Is there any objection to turning these panda tests off?
WebQA is also currently running gaia ui tests on unagi's in a Jenkins
cluster. We can't run these in buildbot, nor have them report to TBPL in
the near term, but we do have a longer-term plan to allow them to report to
TBPL (https://wiki.mozilla.org/Auto-tools/Projects/TreeHerder_2013Q2).
After work for gaia unit and ui tests is completed, we will turn our
attention to the gaia integration tests (in JS)
(https://bugzilla.mozilla.org/show_bug.cgi?id=866909), but the gaia
developers will first have to get these working reliably.
== What we are planning to do ==
The current state of gaia tests in TBPL is not ideal. Among other things,
sheriffs can't always easily identify gaia regressions, and gaia developers
have no access to try. Having gaia results on separate pages from gecko
activity makes them harder (or sometimes impossible) to sheriff effectively,
and reduces their visibility and relevance. There have been multiple plans
floated to deal with this, but at the recent b2g work week, a few of the B2G
leads proposed an alternative.
There are two primary pieces to this proposal:
1) - couple gecko and gaia commits in buildbot so that there is no
uncertainty about which commits are used to create a build, and so that gaia
test results appear on "main" TBPL pages like mozilla-inbound and
mozilla-central
2) - create a gaia-try repo that would allow gaia developers to push patches
to try
This proposal initially will only target b2g desktop builds running against
mozilla-central and gaia's master branch.
== Coupling gecko and gaia commits ==
This would be accomplished by adding a file to mozilla-central that contains
a pointer to a gaia commit. Whenever buildbot produces a b2g desktop build,
it would use the corresponding gaia commit in the build.
When a gaia developer commits a patch to gaia's master branch, a
corresponding commit would be automatically submitted to gecko
(mozilla-inbound, or possibly birch) that would change gecko's gaia pointer.
This auto-commit would automatically trigger a full set of test runs, as
happens with any other gecko commit.
The gecko auto-commit would use the same commit message as the original gaia
commit. For example, if the original gaia commit was "Bug 865091 - Enable
systemXhr so youtube bypasses CORS. r=daleharvey", the gecko auto-commit
might be something like "Bug 865091 - Enable systemXhr so youtube bypasses
CORS. r=daleharvey, [gaia auto-commit]".
If an auto-commit caused a test to go orange, it would be easily
identifiable by the sheriffs, and they would back out the original gaia
commit.
== Gaia and tryserver ==
We will provide a gaia-try repo, and gaia developers may submit a patch to
this if they would like to kick off a tryserver run.
We realize that gaia developers would prefer to commit to a github repo for
this purpose, but that would require significant changes to our build/test
infrastructure such that it isn't feasible this quarter. What we propose as
a first iteration for try support is as follows:
* provide a gaia-try repo in hg
* when a gaia developer wishes to push to try, he would commit his gaia
patch to this hg repo (we would provide very easy-to-follow instructions),
and then commit another patch to the "regular" hg gecko try repo
(http://hg.mozilla.org/try/) that updates the gaia pointer; this gecko
commit would use normal trychooser syntax
(http://trychooser.pub.build.mozilla.org/).
In subsequent iterations of gaia-try (to be done post Q2), we could improve
the process by moving to a git-based gaia-try repo, and providing a
mechanism that would allow gaia developers to omit the gecko try push if
they were only interested in testing gaia changes using mozilla-central's
head revision. Developers that have coupled gecko/gaia patches would still
need to perform two commits, one to gaia-try and one to gecko-try.
We believe this plan (sans future gaia-try improvements) is achievable in a
reasonable amount of time. Does this plan meet the needs of b2g developers,
and does it provide enough benefit beyond our current ability to display
separate pages for gecko and gaia in TBPL that it merits the time it will
take to implement?
We'd like to arrive at a decision by the end of the week, so that we may
begin implementation if there is general agreement.
Jonathan, on behalf of rel-eng and the a-team
This all sounds awesome to me!
But definitely would like to hear input from gaia developers.
/ Jonas
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
