[Wikitech-l] Production Excellence #45: June 2022

[Krinkle]

How are we doing in our strive for operational excellence? Read on to find out!

Incidents
There were 6 incidents in June this year. That's double the median of three per 
month, over the past two years (Incident graphs 
).

2022-06-01 cloudelastic 

Impact: For 41 days, Cloudelastic was missing search results about files from 
commons.wikimedia.org.

2022-06-10 overload varnish haproxy 

Impact: For 3 minutes, wiki traffic was disrupted in multiple regions for 
cached and logged-in responses.

2022-06-12 appserver latency 

Impact: For 30 minutes, wiki backends were intermittently slow or unresponsive, 
affecting a portion of logged-in requests and uncached page views.

2022-06-16 MariaDB password 

Impact: For 2 hours, a current production database password was publicly known. 
Other measures ensured that no data could be compromised (e.g. firewalls and 
selective IP grants).

2022-06-21 asw-a2-codfw power 

Impact: For 11 minutes, one of the Codfw server racks lost network 
connectivity. Among the affected servers was an LVS host. Another LVS host in 
Codfw automatically took over its load balancing responsibility for wiki 
traffic. During the transition, there was a brief increase in latency for 
regions served by Codfw (Mexico, and parts of US/Canada).

2022-06-30 asw-a4-codfw power 

Impact: For 18 minutes, servers in the A4-codfw rack lost network connectivity. 
Little to no external impact.

Incident follow-up
Recently completed incident follow-up:


Audit database usage of GlobalBlocking extension 

Filed by Amir (Ladsgroup) in May following an outage due to db load from 
GlobalBlocking. Amir reduced the extensions' DB load by 10%, through avoiding 
checks for edit traffic from WMCS and Toolforge. And he implemented stats for 
monitoring GlobalBlocking DB queries going forward.


Reduce Lilypond shellouts from VisualEditor 

Filed by Reuven (RLazarus) and Kunal (Legoktm) after a shellbox incident. Ed 
(Esanders) and Sammy (TheresNoTime) improved the Score extension's VisualEditor 
plugin to increase its debounce duration.

Remember to review and schedule Incident Follow-up work 
 in Phabricator! These 
are preventive measures and tech debt mitigations written down after an 
incident is concluded. Read more about past incidents at Incident status 
 on Wikitech.

Trends

In June and July (which is almost over), we reported 27 new production errors 
 and 25 
production errors 
 
respectively. Of these 52 new issues, 27 were closed in weeks since then, and 
25 remain unresolved and will carry over to August.

We also addressed 25 stagnant problems that we carried over from previous 
months, thus the workboard overall remains at exactly 299 unresolved production 
errors.

Take a look at the Wikimedia-production-error 
 workboard 
and look for tasks that could use your help.

 *Did you know?* To zoom in and find your team's error reports, use the 
appropriate "Filter" link in the sidebar of the workboard .
For the month-over-month numbers, refer to the spreadsheet data 
.

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving 
problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof




 Share or read later via 
https://phabricator.wikimedia.org/phame/post/view/292/ 

___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] MediaWiki 1.39-alpha will be branched as a beta on 5 September 2022

[Tyler Cipriani]

Hey all,

This is a quick note to highlight that in six weeks' time, the REL1_39
branch will be created for MediaWiki core and each of the extensions and
skins in Wikimedia git, with some (the 'tarball') included as sub-modules
of MediaWiki itself[0]. This is the first step in the release process for
MediaWiki 1.39, which should be out in November 2022, approximately
six months after MediaWiki 1.38.

The branches will reflect the code as of the last 'alpha' branch for the
release, 1.39.0-wmf.28, which will be deployed to Wikimedia wikis in the
week beginning 12 September 2022 for MediaWiki itself and those extensions
and skins available there.

After that point, patches that land in the main development branch of
MediaWiki and its bundled extensions and skins will be slated for the
MediaWiki 1.40 release unless specifically backported[1].

If you are working on a new feature that you wish to land for the release,
you now have a few days to finish your work and land it in the development
branch; feature changes should not be backported except in an urgent case.
If your work might not be complete in time, and yet should block release
for everyone else, please file a task against the `mw-1.39-release` project
on Phabricator.[2]

If you have tickets that are already tagged for `mw-1.39-release`, please
finish them, untag them, or reach out to get them resolved in the next few
weeks.

We hope to issue the first release candidate, 1.39.0-rc.0, two weeks after
the branch point, and if all goes well, to release MediaWiki 1.39.0 a few
weeks after that.

Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation

[0]: 
[1]: 
[2]: 
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] July 2022 Edition of the Technical Community Newsletter

[Melinda Seckington]

Hi everyone,

We’re happy to share the July 2022 edition of the Technical Community
Newsletter:
https://www.mediawiki.org/wiki/Technical_Community_Newsletter/2022/July

The newsletter is compiled by the Wikimedia Developer Advocacy Team. It
aims to share highlights, news, and information of interest from and about
the Wikimedia technical community.

The Wikimedia Technical Community is large and diverse, and we know we
can't capture everything perfectly. We would love to hear your ideas for
future newsletters. Got something you would like to see or something you
want to highlight in the next quarterly newsletter? Add your suggestion to
the talk page:
https://www.mediawiki.org/wiki/Talk:Technical_Community_Newsletter

If you'd like to keep up with updates and information, subscribe to the
Technical Community Newsletter:
https://www.mediawiki.org/wiki/Newsletter:Technical_Community_Newsletter

Thanks,

Melinda

-- 
Melinda Seckington
Developer Advocacy Manager
Wikimedia Foundation 
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Talk to the Search Platform / Query Service Team—August 3rd, 2022

[Trey Jones]

The Search Platform Team
 usually holds an
open meeting on the first Wednesday of each month. Come talk to us about
anything related to Wikimedia search, Wikidata Query Service (WDQS),
Wikimedia Commons Query Service (WCQS), etc.!


Feel free to add your items to the Etherpad Agenda for the next meeting.


Details for our next meeting:

Date: Wednesday, August 3rd, 2022

Time: 15:00-16:00 UTC / 08:00-09:00 PDT / 11:00-12:00 EDT / 16:00-17:00 WAT
/ 17:00-18:00 CEST

Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours

Google Meet link: https://meet.google.com/vgj-bbeb-uyi

Join by phone: https://tel.meet/vgj-bbeb-uyi?pin=8118110806927


Hope to talk to you next week!

—Trey


Trey Jones
Staff Computational Linguist, Search Platform
Wikimedia Foundation
UTC–4 / EDT
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/