On Mon, Jun 15, 2015 at 6:20 AM, Johan Lorenzo <[email protected]> wrote: > +qa-b2g > > TL;DR: Graphics smoketests are not well covered because of our tools. Also > automate end-to-end tests are costly to run and flaky. We probably need to > focus more on unit tests, and to know where we need to add these tests, we > need coverage reports.
You raise some excellent points. I agree that we need way more unit tests, definitely. And getting more code coverage reports is also definitely useful so we know which parts we should focus on first. However keep in mind that a good chunk of the mozilla codebase doesn't easily lend itself to being unit tested. There are dependencies all over the place and it's not easy to test components in isolation. For many of the graphics bugs we run into, it's actually impossible to write unit tests for without basically duplicating functionality from other components into "mock objects". The blog post you linked to is interesting, but doesn't apply to every single software project out there; we have to take into consideration realities like API surface area between different components, modularity of the codebase, etc. In general there is no one-size-fits-all approach to testing. I would love to spend time into refactoring our codebase so that it's more unit-testable but it's hard to justify spending time on something with such a nebulous return on investment. I do agree with the blog post that end-to-end tests are harder to debug and slower to fix, and maybe that's not the right option for us. Maybe we need to work more on integration testing across just a few components that have high interdependencies. I'm not sure. Regardless, if it's easy to get more code coverage reports then we should definitely do that. At the very least it will give us a better idea of where to focus our efforts. Cheers, kats _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
