To add to jason's points, It's absolutely too much overhead for my team to smoketest each checkin. The only feasible way is having automated tests that run per checkin. This is worth discussing with the A-team / Gaia UI team if that is feasible, and in what time frame.
our current QA team does write automated tests, but it currently supports 1.01 smoketests and we havent gotten around to testing new features. We're still one build behind and can't catch up current release fast enough for this. We would request the help of gaia dev in this effort. (device level testing is better, but maybe running on desktop build is good enough) The other condition you need is parallelizing gecko changes as gaia changes land. I know some checkins are independent of gecko, but some aren't and require parallel landings. To test end to end infra per checkin, you'd need to sync up landings with gecko. At least within 24 hours i suppose. This means if we sync master, we'll have to sync mozilla-central as well, to truly get integration testing going. Tony On Jun 11, 2013, at 3:22 PM, Jason Smith <[email protected]> wrote: > The one problem I have with this proposal is that this sounds like this is > proposing to do smoke test runs on every single pull request. That's likely > going to be too much overhead for our smoke test management. > > For patch-specific testing, usually the reviewer handles doing the code > review of the patch and manually testing the patch to ensure that the patch > does what it intends to do and that the patch does not cause severe > regressions. However, there are times when there is value to get QA involved > on the specific patch (e.g. complex patch that has high regression risk). In > those cases, I think there's value to following a process such as what's > you've specified below to generate a try build with the pull request and ask > for QA support to do testing around the patch to check for regressions, > ensure the patch does it was says it should do, etc. > > I think there's a bug on file to add support for Gaia try builds to implement > this. Note that QA can manually generate a custom Gaia build to follow the > try build testing process right now, although having the automated try build > process would definitely make this process easier to execute. > Sincerely, > Jason Smith > > Firefox OS QA Engineer > Mozilla Corporation > https://quality.mozilla.com > On 6/11/2013 3:06 PM, Gareth Aye wrote: >> Hi Everyone! >> >> I wanted to propose an idea for how we can make the RIL builds more relevant >> for and accessible to developers. Let me begin by saying that I truly >> appreciate the work that QA does and that I do personally read through all >> of the smoketest build results to see whether there are any regressions my >> work has introduced. I think it's a great resource! This is simply one idea >> for how we could make it an even better one. >> >> I believe (and I think others would agree) that tests are most useful when a >> patch is submitted for review. When I do a code review, I like to see the >> linter and test results alongside the patch to get a full perspective. >> Travis is actually *much* more useful to me than TBPL because Travis tells >> me how patches affect things when it matters most: when I'm reading code and >> deciding whether a patch is ready to land. Imagine how many things we >> wouldn't have broken if reviewers knew (when they accepted patches) whether >> or not they were breaking the smoketests! >> >> So, without further adieu, I propose that we do just that! Let's run the >> smoketests on pull requests! >> >> There's a catch though -- we have a very small QA team and there are *lots* >> of pull requests. It's totally reasonable to question whether this is >> possible. It's my opinion that with some clever automation we could make the >> volume of QA work totally manageable. In broad strokes... >> >> 1. GitHub sends a notification to our service-x to tell us that a pull >> request has been opened against mozilla-b2g/gaia. >> 2. service-x parses from the pull request information which bugzilla issue >> the pull request is trying to fix. >> 3. service-x parses from the pull request which smoketests would be affected >> by the patch. >> 4. service-x fetches the bug from bugzilla and parses out the environment >> (ie gaia version and gecko version) that the patch is intended to be applied >> to. >> 5. service-x cherry picks the patch onto the appropriate gaia branch and >> builds b2g. >> 6. service-x sends a notification (maybe an email?) to qa with a link to >> download the build, the smoketests that need to be run, and a link to the >> pull request to comment on with the result of the smoketests. >> >> I think if we built a service-x, it would make the amount of work that qa >> needed to do in order to provide smoketest results to all of the pull >> requests manageable. The manual work would be minimized to flashing a build >> to a device, running only the affected smoketests, and commenting on a >> GitHub issue. >> >> Any thoughts :) ? >> >> -- >> Best, >> Gareth >> >> >> >> _______________________________________________ >> Qa-b2g mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/qa-b2g > > _______________________________________________ > B2g-release-drivers mailing list > [email protected] > https://mail.mozilla.org/listinfo/b2g-release-drivers _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
