The point is to identify significant functional regressions or failures, so this new approach sounds just fine to me.
On Tue, Apr 30, 2013 at 4:55 PM, Marcia Knous <[email protected]> wrote: > Dietrich: Sorry for failing to include this in the email: In our > smoketest meeting the other day, we starting the process of clarifying > the smoketests. In addition, we decided not to report many of the partial > fails that are in some cases tangential to the actual smoketest as written > in MozTrap. We discussed trying to emphasize functional blockers rather > than some of the partial fails that in some cases were cropping up every > day. > > https://bugzilla.mozilla.org/show_bug.cgi?id=837219 - [B2G][Settings > Apps] Privacy policy – Firefox OS privacy policies directs to “404: Page > not found “ and https://bugzilla.mozilla.org/show_bug.cgi?id=849001 - > Pinterest > crashes when attempting to load > are too good examples of bugs that were continually reported as partial > failures that are now being shown as an overall "Pass." > > If you would like is to revert to the previous reporting methodology we > can do that, but rest assured we will be vigilant in raising significant > functional issues in the report, and the other bugs found during testing > will continue to be reported in Bugzilla. Feedback is more than welcome. > > marcia > > > > > Dietrich Ayala <[email protected]> > April 30, 2013 4:36 PM > This is fantastic. What happened that we went from high-30s to only > failing a single test in one day? > Marcia Knous <[email protected]> > April 30, 2013 4:27 PM > 44 out of 45 tests passed for the2013-04-30 Inari v1.0.1 Commercial > RIL Build. Here are the major issues that prevented the smoke tests > from passing. > > Smoketest Results > Daily Smoketest Logs: > > https://docs.google.com/a/qanalydocs.com/spreadsheet/ccc?key=0AqDJxnTd7VDLdDRKOE1uanE3M3FtMDBOVjRIV253S1E > Moztrap link:https://moztrap.mozilla.org/runtests/run/1187/env/316/ > > Tests Were Performed With: > Build ID: 20130430070201 > Gecko: 010498e599a7 > Gaia: de0f1fc6c58616b8a33fca482f1f8684d4e74e9e > 0 reboots > 0 crashes > > Build Updated From: > - No OTA performed. > > New Bugs Breaking Smoke Tests > - None to report. > > Existing Bugs Breaking Smoke Tests > - None to report. > > New Issues Not Breaking Smoke Tests > - None to report > > Sincerely, > > Mozilla QA team > > > > > > -- > Call or text me on my B2G phone: 650-282-0970. > >
_______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
