On Wed, Nov 21, 2012 at 12:28 AM, <[email protected]> wrote: > From what I'm getting from Randell and Ekr's feedback, it sounds like > getting the smoke tests for the full stack of webrtc is the priority, not > the crashtests. I think that makes logical sense as the primary focus area, > although I should note that given the high bug rates for crashes, I do > think we need to continuously burn through expanding the crashtest > automation to protect ourselves carefully from crash regressions. From the > QA side, what I'd like to see is a combination of both ideas proposed to > achieve the joint balance of effective sunny day automation + protection > against crash regressions to achieve the benefits of both areas. > > My suggestion I would go for is something along the lines of this: > > 1. Any work Henrik already started for certain crashtests (since I know > he's got a few in queue) - finish those off >
I think it depends on how long this takes. We are already suffering from this now, so if it means that we won't see a start of functional testing till next week (I'm assuming that Henrik isn't taking TG off) then I would prefer to just suspend the crashtests. 2. Henrik shifts focus towards smoke test automation for the full stack, > I'll finish off the gUM smoke test automation (which is 90% done). > 3. We establish a review policy that any patches that can definitely have > a crashtest associated with them has to be included on check-in by the > developer building the patch. If the patch comes in without a crashtest > that can indeed possibly have one, the patch gets a review- until the test > is included. That will balance both validating that the patch actually does > work at check-in, but also starts to burn through progressively building > out a crashtest regression automation suite. > > I don't really understand what this means. I agree that if a crashtest exists, it should pass before we submit the patch. Do you mean something else? -Ekr > Does that work? I do understand the concerns Ekr and Randell raise, but I > also don't want to take risks on crash regressions too much given the high > crash bug rate, even with the feature currently prefed off. > _______________________________________________ > dev-media mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-media > _______________________________________________ dev-media mailing list [email protected] https://lists.mozilla.org/listinfo/dev-media
