The report covering Wikimedia engineering activities in January 2014 is now

Wiki version:
Blog version:

We're also proposing a shorter, simpler and translatable version of this
report that does not assume specialized technical knowledge:

Below is the HTML text of the report.

As always, feedback is appreciated on the usefulness of the report and its
summary, and on how to improve them.


Major news in January include:

   - the transition of our search
Wikimedia sites to ElasticSearch;
   - a presentation of how the Tech newsletter
   including a historical perspective;
   - an invitation to comment on the 2-year
the multimedia engineering team;
   - a request for
whether the MP4 video format should be supported on Wikimedia sites.

*Note: We’re also providing a shorter, simpler and translatable version of
this report
that does not assume specialized technical knowledge.*

Engineering metrics in January:

   - 159 unique committers contributed patchsets of code to MediaWiki.
   - The total number of unresolved
from around 1386 to about 1320.
   - About 22 shell requests
<https://www.mediawiki.org/wiki/Shell_requests>were processed.


      - Work with
   - Technical 
   - Features 
      - Editor retention: Editing
      - Core 
   - Language 
   - Platform 
      - MediaWiki
      - Quality
      - Engineering Community

 Personnel Work with us <https://wikimediafoundation.org/wiki/Work_with_us>

Are you looking to work for Wikimedia? We have a lot of hiring coming up,
and we really love talking to active community members about these roles.

   - VP of Engineering <http://hire.jobvite.com/Jobvite/Job.aspx?j=ods8Xfwu>
   - Software Engineer –
   - Software Engineer – VisualEditor
   - Software Engineer – Language
   - Software Engineer <http://hire.jobvite.com/Jobvite/Job.aspx?j=o09WXfwM>
   - Software Engineer – Mobile (Android
   - QA Automation Engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=oe09Yfw5>
   - Test Infrastructure
   - Analytics – Product
   - Director of Community Engagement
   - Sr. Operations
   - Operations Security
   - User Experience Research Lead <http://grnh.se/3vmjby>


   - Shahyar Ghobadpour joined the Wikimedia Core features team as Software
   - David Chan joined the Language Engineering team as Software Engineer (
   - Gilles Dubuc joined the Multimedia team as Senior Software Engineer
   and Tech Lead
   - Charles Salvia joined the Analytics team
   - Sam Smith joined the Features Engineering team as Software Engineer,
   working with the Growth team
   - Preteek Saxena joined the UX Design team as UX Prototyping contractor (

 Technical Operations

*Datacenter RFP <https://wikimediafoundation.org/wiki/RFP/2013_Datacenter>*
The Wikimedia Operations team is in the final stages of the selection
process. A short list of 4 bids has been created and final negotiations are
underway. The winner of the bid will be selected in February based on the
technical criteria listed in the RFP, and pricing.

Labs metrics in January:

   - Number of projects: 131
   - Number of instances: 441
   - Amount of RAM in use (in MBs): 1,734,144
   - Amount of allocated storage (in GBs): 23,505
   - Number of virtual CPUs in use: 867
   - Number of users: 2,595

 *Wikimedia Labs <https://www.mediawiki.org/wiki/Wikimedia_Labs>*
The Labs Migration team, consisting of Andrew Bogott and Marc-Andre
Pelletier, have made good progress with testing the newest version of
Openstack (called Havana <https://www.openstack.org/software/havana/>) and
with Neutron <https://wiki.openstack.org/wiki/Neutron>, an OpenStack
project to provide “networking as a service”. The plan is to upgrade the
Openstack software when we migrate the Labs infrastructure out of the Tampa
data center.
retention: Editing tools

*VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>*
In January, the VisualEditor team continued their work on improving the
stability and performance of the system, and added some new features. Most
of the team’s focus was on major new features and fixing bugs. You can now
edit some page settings like whether to display a table of contents or
whether to show section edit labels, set the size of a media file manually,
see a keyboard shortcuts help screen, and create and edit media galleries
using a very basic stand-in editor whilst the final form is being designed.
Work also continued on a dialog for quickly adding “citation” references
based on templates, more media and page settings, setting content language
and right-to-left flags, and equation editing. The deployed version of the
code was updated four times

*Parsoid <https://www.mediawiki.org/wiki/Parsoid>*

In January, the Parsoid team did a lot of bug fixing around images, links,
references and various other areas. See the deployment
a summary.

Part of the team has been mentoring two Outreach Program for
interns. Others are mentoring a group of students in a Facebook
Open Academy <https://www.mediawiki.org/wiki/Facebook_Open_Academy> project
to build a Cassandra storage back-end for the Parsoid round-trip test

We also participated in the architecture summit, where our RFCs about embracing
a service 
PHP bindings for
a general-purpose storage
on our Rashomon revision store, and a
public content API based on
well received.

Following up on this, we started Debian packaging for Parsoid, which will
soon make the installation of Parsoid as easy as apt-get install parsoid.
 Core Features

*Flow <https://www.mediawiki.org/wiki/Flow/Project_information>*
This month, the Core Features team worked on integrating MediaWiki tools
for dealing with spam and vandalism (AbuseFilter and Spam Blacklist) into
Flow. We also launched an updated visual design and UI, based on the first
round of experienced user feedback last month, as well as ongoing user
testing with new users. Lastly, we created a script to disable Flow and
return Flow discussions back into unstructured wikitext, so that we can
begin trialing Flow in production in an extremely safe-to-fail manner. We
are set to deploy our first trial on February 3, 2014 to two WikiProjects
that volunteered on the English Wikipedia.

*Growth <https://www.mediawiki.org/wiki/Growth>*
In the first month of the year, the Growth team focused on two projects.
First, we enhanced and refactored the
in part to support local configuration for different Wikipedias.
The latest version of GettingStarted and GuidedTour will be released in
English and 23 other languages in early February. Second, the team wrapped
up several iterations of design and data analysis in support of upcoming
work on Wikipedia article
We presented new designs for the Draft namespace, and completed a series of
remote usability
We also finalized and published extensive quantitative
trends in article creation across the largest Wikipedias. Last but not
least, the Growth team welcomed its newest member in January, Software
Engineer Sam Smith.

*Wikipedia Education Program
This month, once again we divided our time between the existing Education
Program extension and work towards a new version of the software. We
thoroughly analyzed database transactions in the current extension and
fixed a slew of long-standing database-related bugs. Also on the current
extension, we finished adding a notification type and notifications
infrastructure, and worked on an improved course editing UX. For the new
version, we studied workflow systems and considered how software for the
Education Program and other outreach activities might use such a system.
Adam Wight started on prototype workflow code. He also went through our
code review backlog, bringing a multitude of new features and improvements
to production.
 Mobile <https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering>

*Wikipedia Zero <https://www.mediawiki.org/wiki/Wikipedia_Zero>*
During the last month, the team added forward compatibility to Varnish
scripting for Wikipedia Zero, and resubmitted a Varnish script patch to
support HTTPS for select Wikipedia Zero partners under the new IP
address-based zero-rating scheme, after analysis with the Operations team.
We also continued proof of concept work on an HTML5 web app for Firefox OS,
fixed bugs in the legacy Firefox OS Wikipedia app, and prepared alpha
functionality for the integration of Wikipedia Zero with the rebooted
Android Wikipedia app. The team also continued work toward a generic JSON
configuration extension for use by extensions like ZeroRatedMobileAcces,
submitted code for the core MediaWiki API, submitted a ResourceLoader (RL)
enhancement and cooperated on alternatives for performance enhancement of
RL on non-WMF Redis-backed ResourceLoaders, and submitted a small UX
enhancement for the Android rebooted Wikipedia app. January 2014 was also a
month of planning: the partners engineering team met for two days with the
business development team to plan for partners and Wikipedia Zero-related
work at large. The partners engineering team also applied itself to two
days of product planning for the Partner Portal. Finally, the team
conducted normal tech facilitation to enable partner launches and align
approaches with current and future partners.

*Mobile web projects <https://www.mediawiki.org/wiki/Mobile_web_projects>*
We have been directing much of our attention over the last month at
delivering a tablet-friendly MobileFrontend experience. We’ve added support
for tables of contents in MobileFrontend for tablets, made some design
improvements for tablets, and have worked towards making VisualEditor work
with MobileFrontend for tablets (in alpha for now). We’ve hit some
roadblocks and are hoping to collaborate more with the VE team in the near
future to keep moving forward on the project. Following up from last month,
we have also released our overlay UI improvements as well as an improved
inline diff view for MobileFrontend into stable. Finally, we have also been
working to expand our coverage of browser tests to facilitate quality
assurance and help prevent the introduction of bugs and regressions.

*Language tools <https://www.mediawiki.org/wiki/Language_tools>*
disabled on January 21 2014 in production for all language Wikimedia
sites (other than wikidata.org) due to font delivery performance issues.
Users can still enable ULS for their language needs by going to their user
profile preferences and enabling ULS from the internationalization
settings. Development is in progress for a solution to enable ULS when a
user logs in and selects their language preferences explicitly to enable
webfonts. David Chan continued his work on language support integration for
VisualEditor <https://www.mediawiki.org/wiki/VisualEditor> for phase 5
languages <https://www.mediawiki.org/wiki/VisualEditor#Timeline>. Niklas
Laxstrom and Santhosh Thottingal participated in the architecture summit in
San Francisco in January in RFC discussions and JSONification of i18n

*Language Engineering Communications and Outreach
The team continued their collaborative projects with Google, Twitter,
Microsoft internationalization and MT teams on webfonts, input tools and
machine translation.

*Content translation <https://www.mediawiki.org/wiki/Content_translation>*
The language engineering team kicked off development of a prototype version
of context translation workflow. This functionality aims to create a
workspace for helping editors bootstrap new articles in non-Latin language
Wikipedias. In the prototype, Russian and Welsh are being used for initial
concept verification.

*Site performance and architecture
The team worked on performance dashboards for
VisualEditor<http://gdash.wikimedia.org/dashboards/ve/>and page
load time <http://gdash.wikimedia.org/dashboards/frontend/>, the
ProfilerMwprof profiler class for MediaWiki, and draft performance

*Admin tools development
This project is on hold, so there have been no significant developments,
although some patches have been contributed by volunteer developers.

*Search <https://www.mediawiki.org/wiki/Search>*
As of February 3, CirrusSearch is available as a Beta Feature on wikis
representing about three quarters of all pages, and serves about 7.5% of
our search traffic. Next month, we hope to get the hardware that we need to
be a Beta Feature on the remaining wikis. We also hope to be the primary
search back-end for more wikis. To that end, we’re working through
performance and recall issues as well as trying to save space in the

*Auth systems <https://www.mediawiki.org/wiki/Auth_systems>*
The team focused on minor updates to close some of the high priority OAuth

*Deployment Tooling <https://www.mediawiki.org/wiki/Deployment_Tooling>*
The Logstash <https://wikitech.wikimedia.org/wiki/Logstash> service was
deployed to production at https://logstash.wikimedia.org/. Work has started
on an analysis of the current scap process which will be used to draft
requirements for further deployment scripting work in the current quarter.

*Security auditing and response
We announced the MediaWiki 1.22.1 and 1.22.2 security releases, and
continued to respond to reported vulnerabilities.
Quality assurance

*Quality Assurance <https://www.mediawiki.org/wiki/Quality_Assurance>*
January saw the QA team working closely with the Mobile team in particular
to enhance the existing suite of test for MobileFrontend. We also
participated in the discussion of the Release Engineering deployment
process at the architecture summit. Hiring is underway for two open
positions, QA Automation Engineer and Test Infrastructure Engineer.

*Beta cluster <https://www.mediawiki.org/wiki/Beta_cluster>*
Beta is being used to test the Math extension rewrite. The Parsoid
extension is now deploying continuously via a Jenkins job, status can be
found on the CI dashboard job “Parsoid
bug 57233 <https://bugzilla.wikimedia.org/show_bug.cgi?id=57233>. The wikis
now send updates to the irc.wikimedia.org server bug

*Continuous integration
Zuul has been upgraded and uses a Gearman bus to communicate with Jenkins,
the l10n-bot is no more triggering change and we enabled a proper gating
system to test changes in parallel. The workflow is smoother and faster to
provide feedback in Gerrit. Jenkins slave lanthanum does not offer direct
access to internet, we configured the jobs to use a web proxy in MediaWiki
(web proxy.eqiad.wmnet or webproxy.pmtpa.wmnet. Finally, the Zuul status
page <https://integration.wikimedia.org/zuul/> now shows the progress of
jobs being run.

*Browser testing
In January, we had a number of contributions from the students of Google
Code-in <https://www.mediawiki.org/wiki/Google_Code-in>, from tests to
Jenkins configuration to documentation. We released two entirely new
features: one test that monitors the file upload API interface on both
production Commons and beta labs Commons, and another test that monitors
fatal errors in Beta Labs. We are very close to announcing general
availability for two other new features: the ability to run tests headless
using Firefox under Xvfb, and the ability to create test data like wiki
pages in the target wiki at run time.

*Multimedia <https://www.mediawiki.org/wiki/Multimedia>*
In January, the multimedia team focused on developing the Media Viewer and
planning our next projects for the year. Gilles Dubuc, Mark Holmquist,
Gergo Tisza and volunteer Aaron Arcos implemented a number of improvements
to the beta version of the Media
Some of the features we created or improved include: faster image load, a
full-screen mode, better navigation between files, an expanded meta-data
panel with location, categories, permissions and assessments. We invite you
to test the new UI features on this beta
faster image load can be tested on this MediaWiki.org
page<https://www.mediawiki.org/wiki/Lightbox_demo>(In both cases, you
need to create an account, then click on ‘Beta’ in your
personal menu and enable Media Viewer.) Pau Giner also designed a new user
interface for displaying slides, video and audio
the upcoming v0.3 version of Media Viewer, based on team
recommendations. Fabrice Florin started a community discussion of our
team’s Multimedia Vision for
which proposes a range of improvements to help engage users and support
productive collaborations in coming years (more comments
We also planned our work for this quarter’s
which focuses on Media Viewer through the end of March, and started
planning our next big priorities for the rest of the year: UploadWizard and
Structured Data on Commons. Lastly, we started a Request for
possible support for the MP4 video standard: we invite you to
participate in this discussion, which is due to end in mid-February; we
will plan our next steps for video based on community feedback for this
RfC. To discuss these projects and keep up with our work, we invite you to
join the multimedia mailing
Engineering Community

*Bug management <https://www.mediawiki.org/wiki/Bug_management>*
Valhallasw wrote a script to import tickets from
by Toolserver) to Wikimedia Bugzilla, added a “Browse
projects” link <https://gerrit.wikimedia.org/r/#/c/104342/> to Bugzilla’s
sidebar, and added an “Upload to Gerrit”
button<https://gerrit.wikimedia.org/r/#/c/110702/>for Bugzilla
attachments to the Greasemonkey
Inline displaying of image
Bugzilla was re-enabled and the default bug assignee was renamed from
“Nobody” to “Nobody – You can work on
this!”<https://bugzilla.wikimedia.org/show_bug.cgi?id=57648>to be more
descriptive. The link to the guided bug entry form at the top of
the standard bug entry form now sets the already chosen product
saving you two clicks when you switch to the guided form. Work continued on
preparing the Bugzilla 4.4 upgrade: Andre Klapper’s patches for porting
custom changes from 4.2 to 4.4 were deployed on the Bugzilla test instance
on Zirconium and tested, and Daniel Zahn fixed a problem with Bugzilla’s
collectstats.pl <https://bugzilla.wikimedia.org/show_bug.cgi?id=29203>, so
the 4.4 upgrade and server move will take place in February. Bug management
documentation related, Andre added a “Situation specific information”
section to the Triage
about purging and profiling.

*Project management tools review
Andre Klapper reached out to the
list as well as individual stakeholders, asking users to share
their workflow and
project management and tracking tools. Guillaume Paumier
summarized all that content into consolidated
those are now in the process of being compared to features offered by
available tools, in order to assemble a shortlist of candidates for
community discussion.

*Mentorship programs <https://www.mediawiki.org/wiki/Mentorship_programs>*

[image: Google Code-in 2013, the Wikimedia


Wikimedia’s first participation in the Google
Code-In<https://www.mediawiki.org/wiki/Google_Code-In>program ended up
with great success: 273 tasks completed by 46 students
with the help of about 30 mentors. Theo Patt and Mateusz Maćkowski
were selected
winners for 
and we sent a special mention to Mayank Madan.

Round 7 of the FOSS Outreach Program for
and all projects are on track so far:

   - Compacting interlanguage
   - MediaWiki Homepage
   - Complete the MediaWiki API development course on
   - Clean up Parsoid round-trip testing
   - Clean up tracing/debugging/logging inside
   - UploadWizard: OSM

Facebook Open Academy <https://www.mediawiki.org/wiki/Facebook_Open_Academy>‘s
warm-up period saw a slow progress in the beginning of the projects. At the
end it seemed that everybody was waiting for the official start at the
kick-off in Facebook headquarters on February 7−9.

*Technical communications
In January, Guillaume Paumier
<https://www.mediawiki.org/wiki/User:Guillom>wrapped up work on
mentoring Google
Code-in <https://www.mediawiki.org/wiki/Google_Code-in> students and
continued to provide ongoing communications
the engineering staff. He contributed to writing, simplifying,
publishing and distributing the weekly technical
and published an in-depth
the process by which the newsletter is put together every week.

*Volunteer coordination and outreach
We helped organizing the Architecture Summit
2014<https://www.mediawiki.org/wiki/Architecture_Summit_2014>in San
Francisco (January 23−24) and we got everything ready for
FOSDEM <https://www.mediawiki.org/wiki/Events/FOSDEM> in Brussels (February
1−2). We continued working with the tech community
metrics<https://www.mediawiki.org/wiki/Community_metrics>around two
key performance indicators: who
contributes code<http://korma.wmflabs.org/browser/who_contributes_code.html>,
and the Gerrit review
 Analytics <https://www.mediawiki.org/wiki/Analytics>

*Kraken <https://www.mediawiki.org/wiki/Analytics/Kraken>*
The team has been monitoring the mobile stream and adding additional load
to Kafka which has exposed some scaling issues. These have been resolved.
In addition, work has been done with the Operations team on designing and
implementing a Java deployment system for use with Hadoop and other
systems. Finally, work has been initiated to use the data in the warehouse
on mobile browser distribution and session length.

*Limn <https://www.mediawiki.org/wiki/Analytics/Limn>*
Usability issues continue to be addressed while the team explores options
around other visualization frameworks. This month we implemented a feature
that simplifies the creation of dashboard by automatically inferring
metadata from the data source.

*Wikimetrics <https://www.mediawiki.org/wiki/Analytics/Wikimetrics>*
We are adding features to Wikimetrics to support scheduled jobs and data
access via evergreen URLs. This will support dashboarding and other
services that are will be built on top of the service. In addition, we are
preparing a Wikimetrics-Vagrant image to help getting started with
Wikimetrics development.

*Kafka <https://www.mediawiki.org/wiki/Analytics/Logging_infrastructure>*
We’ve increased the throughput on Kafka from 6K/RPS to 50K/RPS to test
stability under higher loads.

*Data Quality <https://www.mediawiki.org/wiki/Analytics/Data_Quality>*


Review of 2013 traffic trends by the Wikimedia Analytics Team.

The team has spent an intense month analyzing data to explain the page view
issues identified in December. The team’s
shared at the February metrics meeting.

*Research and Data

We conducted a thorough review of traffic data and
confirmed a downward trend in desktop pageviews in 2013. This trend is
not reflected in desktop unique visitors or mobile traffic. We are working
on complementing pageviews with other traffic metrics that will help us
better monitor readership trends. We engaged with external parties (Google
and comScore) to obtain data about referral and mobile traffic respectively.

We completed research on article creation
the largest Wikipedias and found substantial differences between
different language Wikipedias; specifically, where anonymous editors are
allowed to create articles, their success rate (% of articles kept) is
substantially higher than that of newly registered editors. We also found
that articles that started as Articles for Creation (AfC) and userspace
drafts have a near 100% success rate, but the transition that English
Wikipedia made toward directing newcomers to start AfC drafts appears to
have substantially reduced the amount of successful articles created by
newcomers, presumably due to the large review backlog.

We published an
Visual Editor usage on Wikipedia projects where the editor is enabled

We continued work on metrics standardization for the editor engagement
and published supportive analysis on definitions and parameter
exploration for two proposed standardized user classes: new
new editor <https://meta.wikimedia.org/wiki/Research:Productive_new_editor>.

We worked with the Analytics Development and Legal teams to articulate use
cases and the retention and anonymization
data subject to the retention guidelines, in particular with respect
to user
agents <https://www.mediawiki.org/wiki/EventLogging/UserAgentSanitization>.

We welcomed Sahar Massachi as a research contractor supporting the team
with data analysis for fundraising tests and iterated on new modeling
strategies for estimating test success (such as the number of dollars per
banner impression). Before he joined us, Sahar worked with the fundraising
team, where most recently he focused on writing tools to help the team
easily and quickly understand the results of each test.

*Kiwix <http://www.kiwix.org>*
*The Kiwix project is funded and executed by Wikimedia CH
<https://meta.wikimedia.org/wiki/Wikimedia_CH>.* Much time this month was
spent planning for 2014. We mainly worked on
almost managed to create a full English Wikipedia ZIM file with
thumbnails. The upgrade of our main storage platform allowed us to start
our automatic ZIM file generation system.
 Wikidata <https://meta.wikimedia.org/wiki/Wikidata>

*The Wikidata project is funded and executed by Wikimedia Deutschland
In January, the team worked mainly on performance improvements around
Wikidata. The Quantities datatype was deployed so it is now possible to
enter data like the number of inhabitants of a country. Wikisource can now
manage its language links via Wikidata as well just like Wikipedia,
Wikivoyage and Commons could already. Two new front-end developers, Adrian
and Thiemo joined the team to help improve Wikidata’s user interface. Last
but not least, the team released their plan for the development of Wikidata
in 2014 and beyond <https://www.wikidata.org/wiki/Wikidata:Development_plan>
 Future The engineering management team continues to update the *Deployments
<https://wikitech.wikimedia.org/wiki/Deployments>* page weekly, providing
up-to-date information on the upcoming deployments to Wikimedia sites, as
well as the *annual goals
listing ongoing and future Wikimedia engineering efforts.

*This article was written collaboratively by Wikimedia engineers and
managers. See revision history
and associated status pages. A wiki version
is also available.*

Guillaume Paumier
Technical Communications Manager — Wikimedia Foundation
Please note: all replies sent to this mailing list will be immediately directed 
to Wikimedia-l, the public mailing list of the Wikimedia community. For more 
information about Wikimedia-l:
WikimediaAnnounce-l mailing list
Wikimedia-l mailing list
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Reply via email to