While the other thread about fuzzing friendly Gecko is an interesting option I would like to go back to the original topic, and start another thread to collect other ideas too, that might help getting better on the performance front. Here are some of my thoughts after spending some time with the profiler and Talos tests in the past couple of weeks.
Probably most regression happens where we don't detect them because of the lack of perf. test coverage. It should be easy and straightforward to add a new Talos test (it isn't right now). There is an ongoing work on this I think but don't know where is that work being tracked. We clearly need more tests. A lot more. Especially if we want to ship features with huge impact like multi-process Firefox or removing XUL. I don't think we have all the metrics we need yet to make the best decisions. We do have some explanations about each Talos tests at https://wiki.mozilla.org/Buildbot/Talos/Tests and I'm thankful for that but some of the tests need more explanation, and some of them does not have any. We could further improve that, it will save a lot of engineering time (this wiki rocks by the way). The great thing about Talos tests that they can be profiled. What would be even better if I could just compare two runs on the profiler level as easily as I can compare the results now on threeherder compare. It would be simpler to assign perf bugs, also less time consuming to fix them if I knew instantly where that test used to spend the time and where is it spending it right now. And most importantly where do I have to tune my module to make the biggest impact on performance even if there is no regression. (Backing out all the features that causing regression is not always the option or the best way to gain back performance.) I don't think the goal is the optimize Gecko. We need to optimize our end products. So performance tests that go through the entire browser and reproducing common user stories are the most important I think (we need more). Gecko and Firefox is often deeply intertwined to a level that joint effort between Platform and Firefox team folks has the best chance to tackle the more complex cases. I don't want to end up with a super fast engine and slow browser because we put our focus on the wrong goal. Add-ons. Last number I heard is that 40% of our users using some Add-ons, we have access to these Add-ons code yet we don't have any performance tests using them. It should be our responsibility to make sure if we regress the user experience of our users with some of the most popular Add-ons, we at least give a heads up to the authors and help them to address the problem. I know resources are limited but maybe there are some low hanging fruit here that would make a huge impact. These are my two cents off the top of my head, I hope others have even better ideas to share. - Gabor On Wed, Mar 9, 2016 at 12:09 AM, David Bryant <dbry...@mozilla.com> wrote: > Platform Peeps, > > Improving release quality is one of the three fundamental goals Platform > Engineering committed to this year. To this end, lmandel built a Bugzilla > dashboard that allows us to track regressions found in any given release > cycle. This dashboard is up on monitors in many of the offices and can also > be found at: http://mozilla.github.io/releasehealth/ > > While this metric might not be perfect, it does expose the number of > newly-discovered regressions we would ship in a release. As of Monday*, we > had *58* new regressions in Firefox 45 Beta -- this is the version that was > released today. Of these bugs, 43 of them are unassigned**. Both of these > things are unacceptable, and we will not continue to operate this way. > > Starting in release 46, we will *not* ship unless all new regressions are > triaged and are either fixed or explicitly deferred by release management > (working with Firefox engineering and Platform Engineering leads). We will > hold a triage meeting every Monday at 2pm PT in the ReleaseCoordination > Vidyo room, open to all of engineering, to stay on top of the overall > regression list, and our first such meeting was yesterday. Bugs will be > assigned by engineering managers and treated as release blockers. > > Engineering managers own the triage of their team's components. Please > work with them and also let Johnny, Doug, or me know if you need help. > > All of us, together, are accountable for adopting this fundamental change > in how we work. This is one of several changes that we’ll be making, so > more to come. > > > Thanks, > David > > > * Yes, I am aware that dougt, dveditz, and jst triaged on Monday, so this > number may be slightly lower now. Still it isn’t zarro. > > ** Yes, I am aware that some of the teams don’t always assign bugs, but I > am asking everyone to start doing this. Unowned bugs will signal to us that > they need help. Basically, we want the assignee field to be someone that is > directly responsible for moving the bug to the next state. That might be > the engineering manager or it might be an engineer. > -- > ------------------------------ > <http://www.mozilla.org> *David Bryant* > Vice President of Platform Engineering and CTO > Mozilla > 331 E. Evelyn Avenue, Mountain View CA 94041 > Phone: +1 408 728 9311 | Email: <dbry...@mozilla.com>dbry...@mozilla.com > ------------------------------ > > -- > You received this message because you are subscribed to the Google Groups > "gecko-all" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to gecko-all+unsubscr...@mozilla.com. > To post to this group, send email to gecko-...@mozilla.com. > To view this discussion on the web visit > https://groups.google.com/a/mozilla.com/d/msgid/gecko-all/56DF5BC2.7070205%40mozilla.com > <https://groups.google.com/a/mozilla.com/d/msgid/gecko-all/56DF5BC2.7070205%40mozilla.com?utm_medium=email&utm_source=footer> > . > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
