On Thu, Jul 25, 2013 at 8:10 AM, Fabrice Desre <[email protected]> wrote: > Hi All, > > Development for 1.1 on the b2g18 branch is approaching its term (yeah!) > which means that we will soon rely on current trunk for 1.2. > Unfortunately the builds from m-c + gaia-master have been less than > stellar lately: gfx regressions on some devices, various crashes, etc. > to the point of not being dogfoodable at all. See > https://bugzilla.mozilla.org/show_bug.cgi?id=884399 to feel the pain. > > Tests are not in great shape either, and sheriffs are considering to > hide the B2G Marionnette tests later today due to the ongoing high > failure rate > (https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure%2C%20&keywords_type=allwords&short_desc=socket.timeout&resolution=---&query_format=advanced&short_desc_type=allwordssubstr&list_id=7335976) > > We need to improve a lot there, but it's not clear to me why we are in > this situation: are we rushing patches in? is the review-roundrip > pressure too high to ensure a decent quality level? are we just > regressing because of a lack of tests (we still have a huge technical > debt to repay there)? > > I would also like QA to run daily smoketests on these builds if possible. > > Thoughts?
I would say that lack of tests is probably the biggest problem here. It's the only way we can prevent tests from happening in the first place. Unfortunately this won't be a quick fix. We have created a huge amount of code with very little tests. All teams need to make it a priority to spend time on writing tests and working on testing infrastructure. Allocating tasks in the back log for the various teams is probably the way to do this. So far I haven't heard anyone opposing doing this, so I hope people feel free to do this. This is only going to become more important as time passes. As we get more partners and ship on more types of hardware, both gaia and gecko code is going to run in environments that see a lot less testing. However we can always ensure that automated tests pass in all shipped products. So the best way to make sure that a feature doesn't break when running on new hardware or on top of new drivers, is to have automated tests for it. When we find types of code that is hard to test, please raise it on this list so that we can come up with strategies for how to fix that. It's only ok to skip testing something due to it being too hard to test, if we instead work on making it easier to test. And likewise, it needs to be ok to spend time on fixing test failures for tests that already exist. Any time we have a test that is intermittently failing, it means that there's a good chance that we actually have a bug. One that will affect end users and/or developers. But I would also be interested to hear if people feel that we are rushing patches and/or reviews. If that's the case then we simply need to start saying "no" more often. It is more important that we do a good job on the features that we do land, than that we land all features on the targetted schedule. If a feature has to be rushed in order to land before feature complete, then we should simply slip that feature into the next release. Or reduce the scope of the feature such that it can comfortably land on schedule. We need to feel comfortable telling drivers/management/product that a feature is at risk. That's better than landing rushed code. Rushed code just tends to lead to bugs that bite us later in the release cycle. / Jonas _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
