I was recently asked to look at regression bugs found late in the beta cycle or post-release to see what discernible trends there might be and how it might inform our testing strategy going forward.
I looked at all bugs that met those criteria that were filed against Firefox 46-50 up through early January (since it's a moving target over time) - nearly 300 in total. A report of the findings with some areas of potential improvement identified is available for reading below: http://bit.ly/2ltlXXz To address some feedback that was already received, the conclusions I've drawn both with respect to individual bugs and overall findings are my own subjective views on the bugs and I certainly recognize that opinions can and will differ about them. For that reason, I've generally tried to avoid dwelling too much on finer-grained details and focus more on the broader issues brought to light. I also recognize that there are dimensions that I chose not to look into (such as dot-release drivers) that could also yield some useful insights. It is my hope that this analysis will spur productive discussion about what changes to our practices we can reasonably achieve to ship fewer regressions in the first place and catch them earlier when we do, especially as we turn more focus this year into increasing the speed at which we ship updates to our users. Thanks! -Ryan _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
