>From what I'm getting from Randell and Ekr's feedback, it sounds like getting >the smoke tests for the full stack of webrtc is the priority, not the >crashtests. I think that makes logical sense as the primary focus area, >although I should note that given the high bug rates for crashes, I do think >we need to continuously burn through expanding the crashtest automation to >protect ourselves carefully from crash regressions. From the QA side, what I'd >like to see is a combination of both ideas proposed to achieve the joint >balance of effective sunny day automation + protection against crash >regressions to achieve the benefits of both areas.
My suggestion I would go for is something along the lines of this: 1. Any work Henrik already started for certain crashtests (since I know he's got a few in queue) - finish those off 2. Henrik shifts focus towards smoke test automation for the full stack, I'll finish off the gUM smoke test automation (which is 90% done). 3. We establish a review policy that any patches that can definitely have a crashtest associated with them has to be included on check-in by the developer building the patch. If the patch comes in without a crashtest that can indeed possibly have one, the patch gets a review- until the test is included. That will balance both validating that the patch actually does work at check-in, but also starts to burn through progressively building out a crashtest regression automation suite. Does that work? I do understand the concerns Ekr and Randell raise, but I also don't want to take risks on crash regressions too much given the high crash bug rate, even with the feature currently prefed off. _______________________________________________ dev-media mailing list [email protected] https://lists.mozilla.org/listinfo/dev-media
