Re: [b2g] Gaia and TBPL

[Jonathan Griffin]

We actually can't use that script to run tests in buildbot.  There are some particular requirements for interfacing with buildbot and for setting up test environments on the test slaves, which means we normally have to write a buildbot-friendly script for every test we run.  The script we use for gaia-unit-tests is here: http://hg.mozilla.org/build/mozharness/file/00942b436ae9/scripts/gaia_unit.py Jonathan On 4/29/2013 11:57 PM, Yuren Ju wrote: That is great! There is a script travis.sh[1] in GAIA_DIR/tools/ci/unit which is used for running gaia unit tests on travis-ci.org [2] (we still have some problems which I'm fixing... like 864795 & 863691). I think you can also use this script for TBPL. [1] https://github.com/mozilla-b2g/gaia/blob/master/tools/ci/unit/travis.sh [2] https://travis-ci.org/mozilla-b2g/gaia/builds 2013/4/30 Jonathan Griffin <jgrif...@mozilla.com> TL;DR - The a-team and rel-eng are currently preparing to get gaia unit tests running in TBPL (https://bugzilla.mozilla.org/show_bug.cgi?id=833666), on both trunk gecko branches and gaia branches (e.g., https://tbpl.mozilla.org/?tree=Gaia-Master).  We have a plan to integrate these tests into the "main" TBPL pages for mozilla-inbound, etc, and to provide try access to gaia developers. We'd like feedback to ensure that this plan will satisfy developers before we begin implementation. == What is already in progress == All the harness work has been done to get gaia unit tests into buildbot/TBPL (https://bugzilla.mozilla.org/show_bug.cgi?id=833666), running on b2g desktop builds.  There is some additional buildbot work that is being done to enable these to be turned on for the "cedar" project branch (https://bugzilla.mozilla.org/show_bug.cgi?id=865379).  Once we shake out any remaining bugs in production there, we will turn these tests on for both gecko trunk branches, like mozilla-inbound, and gaia branches, like gaia-master. We are additionally endeavoring to get gaia ui tests (in Python) stable on b2g desktop builds (https://bugzilla.mozilla.org/show_bug.cgi?id=857622).  We have been running them on pandas for some time in buildbot, but the panda build is not well-maintained enough for large-scale automation, and it is hard for developers to debug panda failures, since most of them do not have panda boards, and working with panda boards is not as easy as working with other b2g builds.  Is there any objection to turning these panda tests off? WebQA is also currently running gaia ui tests on unagi's in a Jenkins cluster.  We can't run these in buildbot, nor have them report to TBPL in the near term, but we do have a longer-term plan to allow them to report to TBPL (https://wiki.mozilla.org/Auto-tools/Projects/TreeHerder_2013Q2). After work for gaia unit and ui tests is completed, we will turn our attention to the gaia integration tests (in JS) (https://bugzilla.mozilla.org/show_bug.cgi?id=866909), but the gaia developers will first have to get these working reliably. == What we are planning to do == The current state of gaia tests in TBPL is not ideal.  Among other things, sheriffs can't always easily identify gaia regressions, and gaia developers have no access to try.  Having gaia results on separate pages from gecko activity makes them harder (or sometimes impossible) to sheriff effectively, and reduces their visibility and relevance.  There have been multiple plans floated to deal with this, but at the recent b2g work week, a few of the B2G leads proposed an alternative. There are two primary pieces to this proposal: 1) - couple gecko and gaia commits in buildbot so that there is no uncertainty about which commits are used to create a build, and so that gaia test results appear on "main" TBPL pages like mozilla-inbound and mozilla-central 2) - create a gaia-try repo that would allow gaia developers to push patches to try This proposal initially will only target b2g desktop builds running against mozilla-central and gaia's master branch. == Coupling gecko and gaia commits == This would be accomplished by adding a file to mozilla-central that contains a pointer to a gaia commit.  Whenever buildbot produces a b2g desktop build, it would use the corresponding gaia commit in the build. When a gaia developer commits a patch to gaia's master branch, a corresponding commit would be automatically submitted to gecko (mozilla-inbound, or possibly birch) that would change gecko's gaia pointer.  This auto-commit would automatically trigger a full set of test runs, as happens with any other gecko commit. The gecko auto-commit would use the same commit message as the original gaia commit.  For example, if the original gaia commit was "Bug 865091 - Enable systemXhr so youtube bypasses CORS. r=daleharvey", the gecko auto-commit might be something like "Bug 865091 - Enable systemXhr so youtube bypasses CORS. r=daleharvey, [gaia auto-commit]". If an auto-commit caused a test to go orange, it would be easily identifiable by the sheriffs, and they would back out the original gaia commit. == Gaia and tryserver == We will provide a gaia-try repo, and gaia developers may submit a patch to this if they would like to kick off a tryserver run. We realize that gaia developers would prefer to commit to a github repo for this purpose, but that would require significant changes to our build/test infrastructure such that it isn't feasible this quarter.  What we propose as a first iteration for try support is as follows: * provide a gaia-try repo in hg * when a gaia developer wishes to push to try, he would commit his gaia patch to this hg repo (we would provide very easy-to-follow instructions), and then commit another patch to the "regular" hg gecko try repo (http://hg.mozilla.org/try/) that updates the gaia pointer; this gecko commit would use normal trychooser syntax (http://trychooser.pub.build.mozilla.org/). In subsequent iterations of gaia-try (to be done post Q2), we could improve the process by moving to a git-based gaia-try repo, and providing a mechanism that would allow gaia developers to omit the gecko try push if they were only interested in testing gaia changes using mozilla-central's head revision. Developers that have coupled gecko/gaia patches would still need to perform two commits, one to gaia-try and one to gecko-try. We believe this plan (sans future gaia-try improvements) is achievable in a reasonable amount of time.  Does this plan meet the needs of b2g developers, and does it provide enough benefit beyond our current ability to display separate pages for gecko and gaia in TBPL that it merits the time it will take to implement? We'd like to arrive at a decision by the end of the week, so that we may begin implementation if there is general agreement. Jonathan, on behalf of rel-eng and the a-team _______________________________________________ dev-gaia mailing list dev-g...@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-gaia

_______________________________________________
dev-b2g mailing list
dev-b2g@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-b2g