We are planning to do a review of alignment/divergence with the plan mid
year. That would be the right time for this discussion as we will have a
good overview/gap analysis at that time.

On Thu, Oct 9, 2014 at 11:37 AM, Pine W <wiki.p...@gmail.com> wrote:

> Thanks for the detailed comments, Erik.
>
> As someone who spent several volunteer hours reviewing the current Annual
> Plan, I would appreciate getting an understanding of how the change of
> emphasis to quarterly reviews affects budgets, hiring plans, and
> fundraising goals. Is that something that you can address or should I ask
> Garfield?
>
> Thanks,
> Pine
> On Oct 9, 2014 4:44 AM, "Erik Moeller" <e...@wikimedia.org> wrote:
>
> > On Thu, Oct 9, 2014 at 10:26 AM, Pine W <wiki.p...@gmail.com> wrote:
> >
> > > I'm sure a Board member, Lila, or Erik will correct me if I am
> mistaken,
> > > but my understanding is that there is internal agreement at Board level
> > > that the Product side of the org needs some systemic changes, that Lila
> > was
> > > chosen with the goal of making those changes, and that some changes are
> > > already happening.
> >
> > There's agreement at all levels that we want to continue down the path
> > set by Sue back in 2012 [1] for WMF to truly understand itself as a
> > technology and grantmaking organization. That path led to where we are
> > today:
> >
> > 1) As part of the ED transition, Sue recommended (and the Board
> > accepted the recommendation) to seek an ED with a strong
> > technology/product background, and we hired Lila Tretikov as Sue's
> > successor who matches those requirements.
> >
> > 2) In November 2012, I recommended that we prepare for building out
> > new functions for UX and Analytics, and prepare for dedicated
> > leadership for Engineering and Product. Sue accepted this
> > recommendation. I hired Directors for UX and Analytics in 2013,
> > followed by Community Engagement in 2014, and finally we hired a VP
> > Engineering last week to complete the process.
> >
> > 3) To better account for the need to learn quickly and adjust course
> > as appropriate, we introduced quarterly reviews in December 2012 [3]
> > and increasingly reduced the specificity of Annual Plan level
> > commitments while increasing the focus on metrics and accountability
> > in the reviews.
> >
> > 4) On the technology and product front, many improvements to process
> > and support infrastructure have been implemented in the last couple of
> > years, including but not limited to:
> >
> > - Development of MediaWiki Vagrant as a standardized dev environment,
> > to reduce failure cases due to developer environment inconsistencies
> > - Improvements to continuous integration infrastructure for PHP unit
> > tests and QUnit JavaScript unit tests, and increased focus (but not
> > nearly enough yet) on automated tests, especially for newly developed
> > features
> > - Introduction and continued improvement of BetaLabs as a staging
> > environment for all commits, increased use of automated end-to-end
> > browser tests and QA testing by humans to catch bugs and regressions
> > prior to production rollouts
> > - Introduction and use of various tools for measuring the impact of
> > features, including EventLogging as a standard instrumentation
> > framework for measuring feature usage, dashboards for visualizing
> > usage, WikiMetrics for analyzing editor cohort behavior, Editor
> > Engagement Vital Signs for understanding system-wide user behavior,
> > analysis of pageview data using Hadoop (just rolled out), etc.
> > - Highly specialized automated testing frameworks for specific
> > projects, e.g. Parsoid round-trip testing and visual diffing (!) to
> > detect dirty diffs or output problems
> > - Introduction of design research as a discipline in the UX team
> > (through hiring of Abbey Ripstra as User Research Lead)  and
> > incorporation of user studies in a much more systematic way across
> > products
> > - Community liaisons dedicated to key products, responding to user
> > feedback and helping Product Managers understand more complex
> > community needs
> > - Continued shortening of release/deployment cycles; significant
> > improvements to deployment tooling, rewriting our legacy "scap" tools
> > to increase the ability to monitor and reason about deployments;
> > introduction of daily "SWAT" deploys to quickly release fixes, etc.
> > - Introduction of various infrastructure tools that help us better
> > analyze/profile issues, including logstash for log analysis, increased
> > use of graphite for performance metrics collection and various
> > front-ends for visualizing those metrics
> > - Shift towards loosely coupled services, addressing the difficulty of
> > maintaining and improving our highly monolithic codebase (examples
> > include Parsoid, Citoid, Mathoid, and the new Content API in
> > development)
> > - Introduction of Beta Features framework to stage features for early
> > adopters
> >
> > 5) The changes Lila has pushed for since we started include:
> >
> > - Greater focus on quarterly prioritization and a "rolling roadmap"
> > rather than a fiscal year view of the world
> > - Increased emphasis on understanding the needs of different user
> > personas at all cycles of software development, including through use
> > of qualitative and quantitative methods
> > - Reducing velocity of user-facing changes (esp. on desktop) to
> > increase focus on foundations (platform/process improvements) that
> > ultimately will enable us to move faster and more effectively
> > - Documenting product development methodology on-wiki and establishing
> > a clearer social contract (to reduce the reliance on RFCs/votes
> > regarding feature configurations)
> > - Surveying the needs of current users to more systematically balance
> > projects that serve future/new users vs. projects that serve the users
> > we have today
> > - Improved communication channels for community engagement to make it
> > easier to understand what major projects are currently in development
> > and how to provide feedback
> >
> > This already means, effectively, that the commitments in the Annual
> > Plan developed during Sue's time should be taken with a big block of
> > salt at this point in time -- we're slowing down the deployment (not
> > development) of big user-facing features like Flow and VE as much as
> > needed to ensure that we incorporate user feedback, data and
> > qualitative research into the product development process
> > appropriately and spend sufficient time on the technical foundations
> > for these projects.
> >
> > The quarterly prioritization alone has been, IMO, a huge improvement
> > that's already paying off. In the "Annual Plan" view of the world,
> > it's unlikely that we would have prioritized a project like HHVM the
> > way we did, because we were generally stuck on the priorities set for
> > the whole year. But it was very clear that this project would provide
> > huge benefits to our users, and I'm glad we were able to call it out
> > as _the_ top priority for Q1 and give the team the space to really
> > focus on getting it done (almost there now, starting to serve reader
> > traffic [4]).
> >
> > Our draft Q2 top priorities (not yet posted on-wiki, but discussed in
> > the metrics meeting last week) are consistent with the above, with the
> > main user-facing push being on mobile web/apps and editing
> > performance, while the other priorities are more
> > platform/process-related. Once again, we're continuing to work on VE /
> > Flow, but focusing more on fundamentals (performance, architecture,
> > testing, use case analysis, etc.) than accelerating deployments.
> >
> > My focus over the coming days is to flesh out the details for the Q2
> > priorities, and then shift to putting more effort in documenting and
> > refining product development methodologies and processes on-wiki.  On
> > the engineering side, there's plenty of process/infrastructure
> > improvement to do as well. From my point of view, continued
> > improvement to test coverage and CI/testing infrastructure, developer
> > tools, profiling/instrumentation, staged roll-out support and
> > strengthening of architectural leadership are the big pieces for
> > coming months, but I'll let Damon speak to his focus areas as he gets
> > the lay of the land.
> >
> > Erik
> >
> > [1] https://meta.wikimedia.org/wiki/User:Sue_Gardner/Narrowing_focus
> > [2]
> >
> https://lists.wikimedia.org/pipermail/wikimedia-l/2012-November/122663.html
> > [3]
> >
> https://lists.wikimedia.org/pipermail/wikimedia-l/2012-December/123088.html
> > [4] https://gerrit.wikimedia.org/r/#/c/165004/
> > --
> > Erik Möller
> > VP of Product & Strategy, Wikimedia Foundation
> >
> > _______________________________________________
> > Wikimedia-l mailing list, guidelines at:
> > https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> > Wikimedia-l@lists.wikimedia.org
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> > <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe>
> _______________________________________________
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe>
>
_______________________________________________
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
<mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe>

Reply via email to