[Wikitech-l] Production Excellence #37: October 2021

2021-11-04 Thread Krinkle 
How’d we do in our strive for operational excellence last month? Read on to 
find out!

Incidents
There were 4 documented incidents last month. This is currently on average, 
compared to the past five years (per Incident graphs 
).

2021-10-08 network provider 
;
 Impact: For upto an hour, some regions experienced a partial connectivity 
outage. This primarily affected the US East Coast for 13 minutes, and Russia 
for 1 hour. It was caused by a routing problem with one among several network 
providers.

2021-10-22 eqiad networking 
;
 Impact: For 40 minutes clients that are normally geographically routed to 
Eqiad experienced connection or timeout errors. We lost about 7K req/s during 
this time. After initial recovery, Eqiad was ready and repooled in 10 minutes.

2021-10-25 s3 db replica 
;
 Impact: For 30min MediaWiki backends were slower than usual. For 12 hours, 
many wiki replicas were stale for Wikimedia Cloud Services such as Toolforge.
2021-10-29 graphite 
;
 Impact: During a server upgrade, historical data was lost for a subset of 
Graphite metrics. Some were recovered via the redundant server, but others were 
lost as the redundant was also upgraded since then and lost some in a similar 
fashion.

Remember to review and schedule Incident Follow-up work 
 in Phabricator, which 
are preventive measures and tech debt mitigations written down after an 
incident is concluded. Read about past incidents at Incident status 
 on Wikitech.


Trends


**Norwegian blue** 🐦
*298 bugs were up on the board.
We solved 20 of those over the past thirty days.*

*How many might now be left unexplored?
We also added new bugs to our database.*

*Half those bugs are pining for their fjord.
The other 23 carry on, with their dossiers.*

*All in all, 301 bugs up on the board.*



In October, 49 new tasks 
 were 
reported as production errors. Of these, we resolved 26, and 23 remain 
unresolved and carry forward to the next month.
Previously, the production error workboard held an accumulated total of 298 
still-open error reports. We resolved 20 of those. Together with the 23 new 
errors carried over from October, this brings us to 301 unresolved errors on 
the board.

Figure 1: Unresolved error reports by month. 


For the month-over-month numbers, refer to the spreadsheet data 
.

Outstanding errors

Take a look at the workboard and look for tasks that could use your help:
View Workboard 


Issues carried over from recent months:
Apr 2021: 
9 of 42 issues left.
May 2021: 
16 of 54 issues left.
Jun 2021:
9 of 26 issues left.
Jul 2021:
12 of 31 issues left.
Aug 2021:
12 of 46 issues left.
Sep 2021:
11 of 24 issues left.
Oct 2021:
23 of 49 new issues 
 are carried 
forward.

Thanks

Thank you to everyone who helped by reporting, investigating, or resolving 
problems in Wikimedia production. Thanks!
Until next time,

– Timo Tijhof



🔗  Share or read later via 
https://phabricator.wikimedia.org/phame/post/view/260/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] MediaWiki 1.37.0-rc.1 is ready for testing

2021-11-04 Thread Sam Reed 
I'm pleased to announce the immediate availability of MediaWiki
1.37.0-rc.1, the second release candidate for 1.37.x. Download links are at
the end of the e-mail. The tag has been signed and pushed to Git.

This is not a final release, and should not be used for production
websites. Known issues are tracked in Phabricator on the release workboard
[1]. As with every release of MediaWiki, a large number of changes have
landed in the last six months (over 1700 commits since 1.36.0 was cut), and
you should read over the preliminary release notes as part of assuring
yourself of areas that may have issues with your configuration, your skins,
and/or your extensions.

As always, please try out the release candidate in a test environment and
do report any issues that you discover. Please use the #MW-1.37-Release [2]
tag in Phabricator when reporting issues specific to this release, to make
sure that we find them as quickly as possible.

It is expected that MediaWiki 1.37 will become final in late November 2021,
though the date may slip if blockers are identified.

Preliminary release notes:
https://gerrit.wikimedia.org/g/mediawiki/core/+/REL1_37/RELEASE-NOTES-1.37
https://www.mediawiki.org/wiki/Release_notes/1.37

Changes since 1.37.0-rc.0:
* (T294043) checkStorage: pass no parameters to WikiRevision::getContent().
* (T292763) Do not cache private wiki completion results.
* (T293783) ApiQueryImageInfo: don't show empty comments as deleted.
* (T294316) Revert "Mark ApiClientLogin/ApiLogin as requiring write mode".
* (T294796) JobQueueRedis: Replace deprecated zSize with zCard.
* Remove duplicate settings from DefaultSettings.
* (T278037) NoLocalSettings: Pass an EmptyBagOStuff to TemplateParser.

Open Bugs:
[1] https://phabricator.wikimedia.org/project/board/4928/

Bug report form:
[2]
https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?tags=MW-1.37-Release

**
Download:
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.tar.gz
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.zip

Download without bundled extensions:
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-core-1.37.0-rc.1.tar.gz
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-core-1.37.0-rc.1.zip

Patch to previous version (1.37.0-rc.0):
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.patch.gz
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.patch.zip

GPG signatures:
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-core-1.37.0-rc.1.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-core-1.37.0-rc.1.zip.sig
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.zip.sig
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.patch.gz.sig
https://releases.wikimedia.org/mediawiki/1.37/mediawiki-1.37.0-rc.1.patch.zip.sig

Public keys:
https://www.mediawiki.org/keys/keys.html
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/