Thanks for the detailed comments, Erik.

As someone who spent several volunteer hours reviewing the current Annual
Plan, I would appreciate getting an understanding of how the change of
emphasis to quarterly reviews affects budgets, hiring plans, and
fundraising goals. Is that something that you can address or should I ask

On Oct 9, 2014 4:44 AM, "Erik Moeller" <> wrote:

> On Thu, Oct 9, 2014 at 10:26 AM, Pine W <> wrote:
> > I'm sure a Board member, Lila, or Erik will correct me if I am mistaken,
> > but my understanding is that there is internal agreement at Board level
> > that the Product side of the org needs some systemic changes, that Lila
> was
> > chosen with the goal of making those changes, and that some changes are
> > already happening.
> There's agreement at all levels that we want to continue down the path
> set by Sue back in 2012 [1] for WMF to truly understand itself as a
> technology and grantmaking organization. That path led to where we are
> today:
> 1) As part of the ED transition, Sue recommended (and the Board
> accepted the recommendation) to seek an ED with a strong
> technology/product background, and we hired Lila Tretikov as Sue's
> successor who matches those requirements.
> 2) In November 2012, I recommended that we prepare for building out
> new functions for UX and Analytics, and prepare for dedicated
> leadership for Engineering and Product. Sue accepted this
> recommendation. I hired Directors for UX and Analytics in 2013,
> followed by Community Engagement in 2014, and finally we hired a VP
> Engineering last week to complete the process.
> 3) To better account for the need to learn quickly and adjust course
> as appropriate, we introduced quarterly reviews in December 2012 [3]
> and increasingly reduced the specificity of Annual Plan level
> commitments while increasing the focus on metrics and accountability
> in the reviews.
> 4) On the technology and product front, many improvements to process
> and support infrastructure have been implemented in the last couple of
> years, including but not limited to:
> - Development of MediaWiki Vagrant as a standardized dev environment,
> to reduce failure cases due to developer environment inconsistencies
> - Improvements to continuous integration infrastructure for PHP unit
> tests and QUnit JavaScript unit tests, and increased focus (but not
> nearly enough yet) on automated tests, especially for newly developed
> features
> - Introduction and continued improvement of BetaLabs as a staging
> environment for all commits, increased use of automated end-to-end
> browser tests and QA testing by humans to catch bugs and regressions
> prior to production rollouts
> - Introduction and use of various tools for measuring the impact of
> features, including EventLogging as a standard instrumentation
> framework for measuring feature usage, dashboards for visualizing
> usage, WikiMetrics for analyzing editor cohort behavior, Editor
> Engagement Vital Signs for understanding system-wide user behavior,
> analysis of pageview data using Hadoop (just rolled out), etc.
> - Highly specialized automated testing frameworks for specific
> projects, e.g. Parsoid round-trip testing and visual diffing (!) to
> detect dirty diffs or output problems
> - Introduction of design research as a discipline in the UX team
> (through hiring of Abbey Ripstra as User Research Lead)  and
> incorporation of user studies in a much more systematic way across
> products
> - Community liaisons dedicated to key products, responding to user
> feedback and helping Product Managers understand more complex
> community needs
> - Continued shortening of release/deployment cycles; significant
> improvements to deployment tooling, rewriting our legacy "scap" tools
> to increase the ability to monitor and reason about deployments;
> introduction of daily "SWAT" deploys to quickly release fixes, etc.
> - Introduction of various infrastructure tools that help us better
> analyze/profile issues, including logstash for log analysis, increased
> use of graphite for performance metrics collection and various
> front-ends for visualizing those metrics
> - Shift towards loosely coupled services, addressing the difficulty of
> maintaining and improving our highly monolithic codebase (examples
> include Parsoid, Citoid, Mathoid, and the new Content API in
> development)
> - Introduction of Beta Features framework to stage features for early
> adopters
> 5) The changes Lila has pushed for since we started include:
> - Greater focus on quarterly prioritization and a "rolling roadmap"
> rather than a fiscal year view of the world
> - Increased emphasis on understanding the needs of different user
> personas at all cycles of software development, including through use
> of qualitative and quantitative methods
> - Reducing velocity of user-facing changes (esp. on desktop) to
> increase focus on foundations (platform/process improvements) that
> ultimately will enable us to move faster and more effectively
> - Documenting product development methodology on-wiki and establishing
> a clearer social contract (to reduce the reliance on RFCs/votes
> regarding feature configurations)
> - Surveying the needs of current users to more systematically balance
> projects that serve future/new users vs. projects that serve the users
> we have today
> - Improved communication channels for community engagement to make it
> easier to understand what major projects are currently in development
> and how to provide feedback
> This already means, effectively, that the commitments in the Annual
> Plan developed during Sue's time should be taken with a big block of
> salt at this point in time -- we're slowing down the deployment (not
> development) of big user-facing features like Flow and VE as much as
> needed to ensure that we incorporate user feedback, data and
> qualitative research into the product development process
> appropriately and spend sufficient time on the technical foundations
> for these projects.
> The quarterly prioritization alone has been, IMO, a huge improvement
> that's already paying off. In the "Annual Plan" view of the world,
> it's unlikely that we would have prioritized a project like HHVM the
> way we did, because we were generally stuck on the priorities set for
> the whole year. But it was very clear that this project would provide
> huge benefits to our users, and I'm glad we were able to call it out
> as _the_ top priority for Q1 and give the team the space to really
> focus on getting it done (almost there now, starting to serve reader
> traffic [4]).
> Our draft Q2 top priorities (not yet posted on-wiki, but discussed in
> the metrics meeting last week) are consistent with the above, with the
> main user-facing push being on mobile web/apps and editing
> performance, while the other priorities are more
> platform/process-related. Once again, we're continuing to work on VE /
> Flow, but focusing more on fundamentals (performance, architecture,
> testing, use case analysis, etc.) than accelerating deployments.
> My focus over the coming days is to flesh out the details for the Q2
> priorities, and then shift to putting more effort in documenting and
> refining product development methodologies and processes on-wiki.  On
> the engineering side, there's plenty of process/infrastructure
> improvement to do as well. From my point of view, continued
> improvement to test coverage and CI/testing infrastructure, developer
> tools, profiling/instrumentation, staged roll-out support and
> strengthening of architectural leadership are the big pieces for
> coming months, but I'll let Damon speak to his focus areas as he gets
> the lay of the land.
> Erik
> [1]
> [2]
> [3]
> [4]
> --
> Erik Möller
> VP of Product & Strategy, Wikimedia Foundation
> _______________________________________________
> Wikimedia-l mailing list, guidelines at:
> Unsubscribe:,
> <>
Wikimedia-l mailing list, guidelines at:

Reply via email to