Re: [Wikimedia-l] Update from the Wikimedia Performance Team

[rupert THURNER]

I appreciate a ton what you guys achieve, many thanks!!

Rupert
On Dec 7, 2015 23:28, "Gilles Dubuc"  wrote:

> Hello,
>
> This is the monthly report from the Wikimedia Performance team.
>
> ## Our progress ##
>
> * Availability. We've done a major overhaul of the ObjectCache interfaces.
> Many
> factory methods were deprecated or removed, reducing it to just four simple
> entry points. New docs at
>
> https://doc.wikimedia.org/mediawiki-core/master/php/classObjectCache.html#details
>
> We've written a new IExpiringStore interface for convenient TTL constants,
> e.g. $cache::TTL_WEEK. See
>
> https://doc.wikimedia.org/mediawiki-core/master/php/interfaceIExpiringStore.html
>
> We've migrated most use of wfGetMainCache() to WANObjectCache. Work
> continued on the librarization of BagOStuff, Memcached, and other object
> cache classes.
>
> * Performance testing infrastructure. We've created dedicated dashboards
> for portals:
>
> https://grafana.wikimedia.org/dashboard/db/webpagetest-portals
>
> And for mobile:
>
> https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest
>
> We now test one page using real 3G connections (from San Francisco and
> Bangalore) and test other pages using the following physical devices:
> iPhone 6, iPad mini 2 and Moto G.
>
> * Media stack. We've extended Thumbor with 12 small plugins to meet our
> needs and match our existing thumbnailing feature set. This includes
> support for all the file formats in use on Commons. The Thumbor Vagrant
> stack is now very close to having all the moving parts needed in
> production, with basic Vagrant roles for Varnish and Swift having been
> written to that end. Our objective is to finish the work on VM by the
> holidays and have it ready to be showcased and discussed collectively at
> the developer summit in a breakout session.
>
> * ResourceLoader. We've written a new mw.requestIdleCallback API for
> scheduling deferred tasks. We've removed usage of the  msg_resource_links
> DB table. We now use message config from the module registry directly.
> We've migrated MessageBlobStore msg_resource DB table to an object cache
> (to be deployed in January 2016):
> https://phabricator.wikimedia.org/T113092
>
> ## How are we doing? ##
>
> Client-side performance has remained stable over the past month. Save
> Timing
> has also remained stable, around the 1s median mark.
>
> The job queue's health improved greatly after adding a new server to the
> pool, with the job queue size dropping drastically and the 99th percentile
> job processing time going from one day to one hour:
>
> * https://grafana.wikimedia.org/dashboard/db/job-queue-health
>
> There was a small scare about a sudden increase of the SpeedIndex value
> across the board:
>
> https://grafana.wikimedia.org/dashboard/db/webpagetest
>
> But it was entirely explained by the fundraising banner, which doesn't
> appear immediately on pageload. SpeedIndex measures the time it takes for
> the above-the-fold area to "settle" visually. The banner appears late and
> pushes the content down, which delays the time when visual changes stop
> happening for the above-the-fold area.
>
> Until next time,
> Aaron, Gilles, Peter, Timo, and Ori.
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

[Wikimedia-l] Update from the Wikimedia Performance Team

[Gilles Dubuc]

Hello,

This is the monthly report from the Wikimedia Performance team.

## Our progress ##

* Availability. We've done a major overhaul of the ObjectCache interfaces. Many
factory methods were deprecated or removed, reducing it to just four simple
entry points. New docs at
https://doc.wikimedia.org/mediawiki-core/master/php/classObjectCache.html#details

We've written a new IExpiringStore interface for convenient TTL constants,
e.g. $cache::TTL_WEEK. See
https://doc.wikimedia.org/mediawiki-core/master/php/interfaceIExpiringStore.html

We've migrated most use of wfGetMainCache() to WANObjectCache. Work
continued on the librarization of BagOStuff, Memcached, and other object
cache classes.

* Performance testing infrastructure. We've created dedicated dashboards
for portals:

https://grafana.wikimedia.org/dashboard/db/webpagetest-portals

And for mobile:

https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest

We now test one page using real 3G connections (from San Francisco and
Bangalore) and test other pages using the following physical devices:
iPhone 6, iPad mini 2 and Moto G.

* Media stack. We've extended Thumbor with 12 small plugins to meet our
needs and match our existing thumbnailing feature set. This includes
support for all the file formats in use on Commons. The Thumbor Vagrant
stack is now very close to having all the moving parts needed in
production, with basic Vagrant roles for Varnish and Swift having been
written to that end. Our objective is to finish the work on VM by the
holidays and have it ready to be showcased and discussed collectively at
the developer summit in a breakout session.

* ResourceLoader. We've written a new mw.requestIdleCallback API for
scheduling deferred tasks. We've removed usage of the  msg_resource_links
DB table. We now use message config from the module registry directly.
We've migrated MessageBlobStore msg_resource DB table to an object cache
(to be deployed in January 2016): https://phabricator.wikimedia.org/T113092

## How are we doing? ##

Client-side performance has remained stable over the past month. Save Timing
has also remained stable, around the 1s median mark.

The job queue's health improved greatly after adding a new server to the
pool, with the job queue size dropping drastically and the 99th percentile
job processing time going from one day to one hour:

* https://grafana.wikimedia.org/dashboard/db/job-queue-health

There was a small scare about a sudden increase of the SpeedIndex value
across the board:

https://grafana.wikimedia.org/dashboard/db/webpagetest

But it was entirely explained by the fundraising banner, which doesn't
appear immediately on pageload. SpeedIndex measures the time it takes for
the above-the-fold area to "settle" visually. The banner appears late and
pushes the content down, which delays the time when visual changes stop
happening for the above-the-fold area.

Until next time,
Aaron, Gilles, Peter, Timo, and Ori.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

[Wikimedia-l] Update from the Wikimedia Performance Team

[Ori Livneh]

Hello,

This is the first monthly report from the Wikimedia Performance Team.
The purpose
of these reports is to showcase our work, track our progress, and spark
conversation.

## Who we are ##

As the Wikimedia Foundation’s Performance Team, we want to create
value for readers
and editors by making it possible to retrieve and render content at the
speed of thought, from anywhere in the world, on the broadest range of
devices and connection profiles.

We are five engineers: Aaron Schulz, Gilles Dubuc, Peter Hedenskog,
Timo Tijhof,
and me. Each of us was born in a different country and has a different mother
tongue. We work out of San Francisco, London, Stockholm, and Montpellier.

## What we are working on ##

* Availability. Although Wikimedia Foundation currently operates six data
centers, MediaWiki is only running on one (in Ashburn, Virginia). If you
are an editor in Jakarta, Indonesia, content has to travel over 15,000
kilometers to get from our servers to you (or vice versa). To run MediaWiki
concurrently from multiple places across the globe, our code needs to be
more resilient to failure modes that can occur when different subsystems
are geographically remote from one another.
https://phabricator.wikimedia.org/T88445

* Performance testing infrastructure. WebPageTest provides a stable
reference for a set of browsers, devices, and connection types from
different points in the world. It collects very detailed telemetry that we
use to find regressions and pinpoint where problems are coming from. This
is addition to the more basic Navigation Timing metrics we gather from real
users in production.
https://phabricator.wikimedia.org/T109666

* Media stack. We're currently working on overhauling our thumbnail
infrastructure to achieve multiple goals. Improving future-proof
maintainability by taking thumbnail code out of MediaWiki-Core and using a
service instead to perform thumbnail operations. Saving on expensive
storage by no longer storing multiple copies of all thumbnails on disk
forever. Enabling far-future expires for images, to greatly improve their
client cacheability. And finally switching to the best performing
underlying software to generate thumbnails faster.
https://phabricator.wikimedia.org/T111718

* ResourceLoader. ResourceLoader is the MediaWiki subsystem that is
responsible for loading JavaScript and CSS. Whereas much of MediaWiki's
code executes only sparingly (in reaction to editors modifying content)
ResourceLoader code runs over half a billion times a day on hundreds of
millions of devices. Its contribution to how users experience our sites is
very large. Our current focus is on improving ResourceLoader's cache
efficiency by packaging and delivering JavaScript and CSS code in a way
that allows it to be reused across page views without needing to be
repeatedly downloaded.
https://phabricator.wikimedia.org/T102578

## How are we doing? ##

In future reports, we will spend less time on introductions and more time
reporting on how particular performance metrics have trended since the
previous report and why. In the meantime, we invite you to check out our
dashboards:

* https://performance.wikimedia.org/
* https://grafana.wikimedia.org/dashboard/db/navigation-timing
* https://grafana.wikimedia.org/dashboard/db/time-to-first-byte
* https://grafana.wikimedia.org/dashboard/db/job-queue-health

## Feedback? ##

Please let us know what you think about these reports and whether we have
picked the right lists to send it to. (We're going to make sure this
information is available on mediawiki.org too.)

Until next time,
Aaron, Gilles, Peter, Timo & Ori
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Update from the Wikimedia Performance Team

[Brad Jorsch (Anomie)]

On Thu, Nov 5, 2015 at 2:13 PM, Ori Livneh  wrote:

> Improving future-proof maintainability by taking thumbnail code out of
> MediaWiki-Core and using a service instead to perform thumbnail operations.


How does that affect third-party use of MediaWiki where people aren't able
to (or just aren't wanting to) install bespoke services?


-- 
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Update from the Wikimedia Performance Team

[Ori Livneh]

On Thu, Nov 5, 2015 at 12:17 PM, Brad Jorsch (Anomie)  wrote:

> On Thu, Nov 5, 2015 at 2:13 PM, Ori Livneh  wrote:
>
> > Improving future-proof maintainability by taking thumbnail code out of
> > MediaWiki-Core and using a service instead to perform thumbnail
> operations.
>
> How does that affect third-party use of MediaWiki where people aren't able
> to (or just aren't wanting to) install bespoke services?
>

It doesn't. There is no plan to strip MediaWiki of its existing
capabilities or to make it depend on this service. And since the intention
is to improve existing quality of service (as opposed to providing new
functionality), I don't expect the feature-set to diverge.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,