[Wikitech-l] [Train] 1.42.0-wmf.19 / wmf.20 status: blocked on T336504 (Transcluding Special:Prefixindex forces default skin), may require Monday rollback

[Brennen Bearnes]

This week's 1.42.0-wmf.20 version of MediaWiki is blocked[0], and 
1.42.0-wmf.19 may require a Monday rollback, which is a situation we 
hope to avoid.


We can't proceed until the following issue is resolved:

* T336504 - Transcluding Special:Prefixindex can force the default skin
- https://phabricator.wikimedia.org/T336504

For context, see the Village pump discussion linked here:

* https://phabricator.wikimedia.org/T336504#9572477

Once this issue is resolved, the train can proceed normally for the week.

Thank you for any help!

-- Your designated sacrificial train victims

[0]. https://phabricator.wikimedia.org/T354438
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.42.0-wmf.18 status update: blocked at group1 on T356368, T357668; urgent assistance requested

[Brennen Bearnes]

The 1.42.0-wmf.18 version of MediaWiki is blocked[0].

The new version is currently deployed to group1[1], and can proceed no 
further until these issues are resolved:


* Revision endpoint: InvalidArgumentException: ParserOutput does not 
have a render ID

- https://phabricator.wikimedia.org/T356368

* TypeError: Argument 1 passed to 
MediaWiki\Parser\Sanitizer::encodeAttribute() must be of the type 
string, null given, called in 
/srv/mediawiki/php-1.42.0-wmf.18/includes/xml/Xml.php on line 81

- https://phabricator.wikimedia.org/T357668

The second issue blew up after a rollback for the first issue, so we're 
currently in a much-less-than-optimal state.


Once these issues are resolved train can resume.

Thank you for any help!

-- Your not entirely placid train operator

[0]. https://phabricator.wikimedia.org/T354436
[1]. https://versions.toolforge.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.42.0-wmf.17 status update: blocked at group0 on T356895

[Brennen Bearnes]

The 1.42.0-wmf.17 version of MediaWiki is blocked[0].

The new version is currently deployed to group0[1], but can proceed no 
further until this issue is resolved:


* T356895 - [wmf.17 - testwiki] Impact module: Temporary delay in 
getting your information

- https://phabricator.wikimedia.org/T356895

Once this issue is resolved train can resume.

Thank you for any help!

-- Your downright placid train operator

[0]. https://phabricator.wikimedia.org/T354435
[1]. https://versions.toolforge.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] phabricator.wikimedia.org downtime, Saturday 2024-01-20 @ 12:00 PST / 20:00 UTC

[Brennen Bearnes]

Hi all,

We'll be upgrading[0] phabricator.wikimedia.org this coming Saturday, 
starting at noon PST.  This involves re-imaging a machine, so some 
downtime is expected, although we'll do our best to keep it at a minimum.


As usual, operations will be conducted on libera.chat's 
#wikimedia-operations.


[0]. https://phabricator.wikimedia.org/T334519

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.41.0-wmf.27 status update: blocked at testwikis on T346800

[Brennen Bearnes]

The 1.41.0-wmf.27 version of MediaWiki is blocked[0].

The new version is currently deployed to testwikis, but can proceed no 
further until this issue is resolved:


* T346800 - startupregistrystats-testwiki periodic job fails
- https://phabricator.wikimedia.org/T346800

Once this issue is resolved train can resume.

Thank you for any help resolving these issues!

-- Your relatively calm train operator

[0]. https://phabricator.wikimedia.org/T345888
[1]. https://versions.toolforge.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Brief Phabricator downtime incoming for update to Phorge @ 15:00 UTC

[Brennen Bearnes]

Background: https://phabricator.wikimedia.org/T333885

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.41.0-wmf.9 status update: blocked at group1 on T336962, T336964

[Brennen Bearnes]

The 1.41.0-wmf.9 version of MediaWiki is blocked[0].

The new version is currently deployed to group1, but can proceed no 
further until these issues are resolved:


* T336962 - UnexpectedValueException: Unknown image suggestions API 
kind: istype-depicts

- https://phabricator.wikimedia.org/T336962

* T336964 - InvalidArgumentException: Data for lt_namespace and lt_title 
must be non-empty

- https://phabricator.wikimedia.org/T336964

Once these issues are resolved train can resume.

Thank you for any help resolving these issues!

-- Your disgruntled train drivers

[0]. https://phabricator.wikimedia.org/T330215
[1]. https://versions.toolforge.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Permission model for /repos/mediawiki on GitLab

[Brennen Bearnes]

Hey all,

RelEng are working through some of the nitty-gritty details of migrating 
MediaWiki repositories to GitLab, and we could use some input on a 
permission model for the planned /repos/mediawiki namespace.


Draft permission model policy:

* https://www.mediawiki.org/wiki/GitLab/Policy#MediaWiki_permission_model

Task:

* T336807 - Define a permissions model for the /repos/mediawiki/ 
namespace on GitLab

- https://phabricator.wikimedia.org/T336807

Thanks!

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.41.0-wmf.8 status update: blocked at group1 on T336504 (Vector 2022 force-deploying on arbitrary pages)

[Brennen Bearnes]

The 1.41.0-wmf.7 version of MediaWiki is blocked[0].

The new version is currently deployed to group1, but can proceed no 
further until this issue is resolved:


* T336504 - Vector 2022 force-deploying on arbitrary pages
- https://phabricator.wikimedia.org/T336504

Once these issues are resolved train can resume.

Thank you for any help resolving these issues!

-- Your humble train toilers

[0]. https://phabricator.wikimedia.org/T330214
[1]. https://versions.toolforge.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.41.0-wmf.7 status update: blocked at group1 on T336008, T336022

[Brennen Bearnes]

The 1.41.0-wmf.7 version of MediaWiki is blocked[0].

The new version is currently deployed to group1, but can proceed no 
further until these issues are resolved:


* T336022 - 1.41.0-wmf.7 increases Memcached call rate by +100%
- https://phabricator.wikimedia.org/T336022

* T336008 - MWException: Internal error in 
ApiQueryRevisionsBase::getRevisionRecords: RevisionStore does not return 
record for [n]

- https://phabricator.wikimedia.org/T336008

Once these issues are resolved train can resume.  If they are not 
resolved before the end of working day today, the train can resume on 
Monday, May 8th.


Thank you for any help resolving these issues!

-- Your humble train toilers

[0]. https://phabricator.wikimedia.org/T330213
[1]. https://versions.toolforge.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Phabricator update window Thurs Mar 30 15:30 - 16:00 UTC (today)

[Brennen Bearnes]

We're planning a brief window for Phabricator updates this (US) morning. 
 I do not expect significant noticeable downtime, but a handful of 
requests are likely to error out during the restart.


This update includes fixes for last week's UI changes, and one lingering 
patch from upstream.


Phab task:

  https://phabricator.wikimedia.org/T333516

As usual, we'll track work on #wikimedia-operations.

Thanks,

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.40.0-wmf.27 status update: blocked at group1 on T321160

[Brennen Bearnes]

The 1.40.0-wmf.27 version of MediaWiki is blocked[0].

The new version is currently deployed to group1, but can proceed no 
further until this issue is resolved or triaged as irrelevant:


* Lcobucci\JWT\Signer\InvalidKeyProvided: Key cannot be empty
- https://phabricator.wikimedia.org/T321160

Once these issues are resolved train can resume.

Thank you for any help resolving these issues!

-- Your humble train toilers

[0]. https://phabricator.wikimedia.org/T330205
[1]. https://versions.toolforge.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Phabricator update window Tues Mar 14 21:00 - 23:00 UTC - Moving to Weds 15th

[Brennen Bearnes]

On 3/13/23 12:39, Brennen Bearnes wrote:
This Tuesday, Mar 14, at 21:00 UTC (14:00 PDT), we're planning a window 
for updates to the production Phabricator instance.


Update: Due to a scheduling conflict, we've moved this one day to Weds 
15th @ 21:00 UTC.


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Phabricator update window Tues Mar 14 21:00 - 23:00 UTC

[Brennen Bearnes]

This Tuesday, Mar 14, at 21:00 UTC (14:00 PDT), we're planning a window 
for updates to the production Phabricator instance.  While we've 
reserved 2 hours out of an abundance of caution, we intend for the 
actual downtime to be much shorter.


This window is contingent on the status of the weekly deployment train. 
If something important is broken in production, we'll push it to a later 
date.


This update should include several minor UI improvements, but does not 
otherwise change the code.


There's a Phab task for this release; as usual we'll track the work on 
Etherpad and #wikimedia-operations once underway:


  https://phabricator.wikimedia.org/T331915

Thanks,

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Phabricator migration on Mon Nov 22 - 22:00 UTC (14:00 PST)

[Brennen Bearnes]

On 11/23/22 15:11, Daniel Zahn wrote:

We believe we have fixed the issues and have scheduled a new window
for this for:

this coming Monday, Nov 28th, at 22:00 UTC (14:00 PST)


Phabricator migration to phab1004 is finished.  Please reply on the 
migration ticket if you notice errors:


https://phabricator.wikimedia.org/T280597

If the ticket is unreachable for any reason, feel free to ping on IRC in 
#wikimedia-releng or, if using WMF Slack, hit up @brennen.


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Summary of 1.40.0-wmf.3 train deployment

[Brennen Bearnes]

This email is a summary of the Wikimedia production deployment of 
1.40.0-wmf.3


 * Conductor: Brennen Bearnes
 * Backup Conductor: Jaime Nuche
 * Blocker Task: T314192 <https://phabricator.wikimedia.org/T314192>
 * Current Status <https://versions.toolforge.org>


    Numbers

Sparklines comparing with the last 5 trains.

 * 208 Patches ██▇▁▃
 * 1 Rollbacks ▁▄█▁▄
 * 1 Days of delay ▁██▁█
 * 1 Blockers ▄▄█▄▁


    Traintastic Folks ✨

Thanks to folks who reported or resolved blockers:

 * revi | 레비
 * Amir Sarabadani

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: No non-emergency deploys until further notice

[Brennen Bearnes]

On 7/25/22 19:08, Brennen Bearnes wrote:

Out of an abundance of caution, please hold all non-emergency deploys 
until we have a fix for this one:


...and just to be super clear, this should only affect WMF production 
MediaWiki deployment.


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] No non-emergency deploys until further notice

[Brennen Bearnes]

Hey all,

Out of an abundance of caution, please hold all non-emergency deploys 
until we have a fix for this one:


* scap no longer restarts php-fpm on canary servers
- https://phabricator.wikimedia.org/T313770

For urgent situations, be aware that:

1. Canary servers don't currently offer any assurances, since they 
aren't currently being restarted after code is synced.


2. You'll need to restart PHP on:

{mw1414,mw1447,mw1415,mw1417,mw1418,mw1449,mw1450,mw1416,mw1448}.eqiad.wmnet

Thanks, and sorry for the disruption.  We expect to return you to your 
regularly scheduled deployments shortly!


(CC'd to listed deployers for upcoming windows.)

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Summary of 1.39.0-wmf.16 train deployment

[Brennen Bearnes]

This email is a summary of the Wikimedia production deployment of 
1.39.0-wmf.16


 * Conductor: Brennen Bearnes
 * Backup Conductor: Dan Duvall
 * Blocker Task: T308069 <https://phabricator.wikimedia.org/T308069>
 * Current Status <https://versions.toolforge.org>


    Numbers

Sparklines comparing with the last 5 trains.

 * 212 Patches █▆▃▅▁
 * 0 Rollbacks ▁▁▁█▁
 * 0 Days of delay ▁▁▁█▁
 * 4 Blockers █▁▁█▁


   ✨ Trainbow Love 殺

Thanks to folks who helped report or resolve blockers:

 * Seddon
 * Tim Starling
 * PleaseStand
 * Timo Tijhof
 * Aaron Schulz
 * Tyler Cipriani
 * Jon Robson
 * James D. Forrester

Worth highlighting in particular: Krinkle's breakdown of increased 
Mcrouter requests <https://phabricator.wikimedia.org/T310532#8004500>.


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [engineering-all] Phabricator maintenance tomorrow 15:00 UTC (08:00 PDT)

[Brennen Bearnes]

On 6/14/22 18:21, Tyler Cipriani wrote:

Phabricator (phabricator.wikimedia.org) will be down tomorrow sometime
during the 15:00–17:00 UTC[0] maintenance window[1].

We're deploying updates during that time, and we'll stop Phabricator
briefly to run database migrations.


This is completed - please let us know in #wikimedia-releng or file a 
task if you encounter any bugs.


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.39.0-wmf.16 status update: blocked at testwikis on T310532

[Brennen Bearnes]

The 1.39.0-wmf.16 version of MediaWiki is blocked[0].

The new version is currently being deployed to testwikis, but can 
proceed no further until this issue is resolved or triaged as irrelevant:


* Investigate McRouter GET request spike from wmf.15
- https://phabricator.wikimedia.org/T310532

Once these issues are resolved train can resume.

Thank you for your help resolving these issues!

-- Your humble train toilers

[0]. https://phabricator.wikimedia.org/T308069
[1]. https://versions.toolforge.org/___
Ops mailing list -- o...@lists.wikimedia.org
To unsubscribe send an email to ops-le...@lists.wikimedia.org

___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] GitLab: Planned downtime for migration to physical hardware: Thursday, 2022-06-02 @ 15:00 UTC

[Brennen Bearnes]

Hey all,

Just a brief notice that we're planning to take gitlab.wikimedia.org 
down for around 2 hours this Thursday, June 2nd, at 15:00 UTC for a 
migration to new physical hardware.


You can follow this work in Phabricator:

https://phabricator.wikimedia.org/T307142

We're also available in #wikimedia-gitlab on libera.chat for any questions.

Thanks,

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] RelEng GitLab-a-thon sprint next week; skipping 1.39.0-wmf.11 train

[Brennen Bearnes]

Hey all,

A note that WMF Release Engineering are spending a sprint focusing as a 
team on GitLab migration efforts. *As a consequence, we're pausing the 
deployment train for the week of **M**ay 9th*so we can work on the 
problem as a group without losing 1-2 people to train duties for most of 
the week.


Unlike our recent experiment at running multiple trains in a week, this 
/shouldn't/have much immediate impact on your work, but if you're 
interested in GitLab adoption, you might want to follow along on the 
sprint workboard:


https://phabricator.wikimedia.org/project/board/5873/

Goal for this sprint:***Migration of blubber / blubberoid to GitLab*

 * https://phabricator.wikimedia.org/T301168
 * What that means;
 * Project exists on GitLab
 * CI runs on commits
 * We can publish an image from GitLab

<3

-- Brennen Bearnes, forRelease Engineering___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.39.0-wmf.1 status update: blocked at group1 on T304331

[Brennen Bearnes]

The 1.39.0-wmf.1 version of MediaWiki is blocked[0].

The new version is deployed to group1, but can proceed no
further until this issue is resolved or triaged as irrelevant:

* PageSplitterHooks: Error: Call to a member function exists() on null
- https://phabricator.wikimedia.org/T304331

Once these issues are resolved train can resume. If these issues are
resolved on a Friday the train will resume Monday.

Thank you for your help resolving these issues!

-- Your humble train toilers

[0].https://phabricator.wikimedia.org/T300203 


[1]. >___
Ops mailing list -- o...@lists.wikimedia.org
To unsubscribe send an email to ops-le...@lists.wikimedia.org

___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l]  Summary of 1.38.0-wmf.24 train deployment

[Brennen Bearnes]

This email is a summary of the Wikimedia production deployment of 
1.38.0-wmf.24


 * Conductor: Brennen Bearnes
 * Backup Conductor: Dan Duvall
 * Blocker Task: T300200 <https://phabricator.wikimedia.org/T300200>
 * Current Status <https://versions.toolforge.org>


    By the Numbers

Sparklines comparing with the last 5 trains.

 * 306 Patches █▄▆▁▅
 * 0 Rollbacks ▄
 * 0 Days of delay ▄
 * 5 Blockers ▁▆▆▆█


   殺 Trainlicious Shoutouts 

Thanks to folks who reported or resolved blockers, as well as those who 
assisted with /potential/ blockers:


 * Peter Pelberg
 * Jon Robson
 * Zabe
 * Legoktm
 * Bartosz Dziewoński
 * Krinkle
 * ssastry
 * Arlolra
 * DLynch

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.38.0-wmf.24 status: blocked before deploy by T298638, T302643

[Brennen Bearnes]

Hi all,

The 1.38.0-wmf.24[0] train is currently blocked by the following:

* Make modifications to Pager HTML to add heading separators to support 
Minerva skinning

- https://phabricator.wikimedia.org/T298638

* Beta Meta-Wiki throws an error on Special:Preferences: 
DomainException: HTMLForm::getField: no field named globalwatchlist-prefs

- https://phabricator.wikimedia.org/T302643

Thanks in advance for any assistance with these tasks!

As ever, you can follow train progress on libera.chat's
#wikimedia-operations as well as on the blocker task[0].

Regards,

-- Your stoic slingers of stuff to production

[0]. https://phabricator.wikimedia.org/T300200
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Summary of 1.38.0-wmf.19 train deployment

[Brennen Bearnes]

This email is a summary of the Wikimedia production deployment of 
1.38.0-wmf.19


 * Conductor: Brennen Bearnes
 * Backup Conductor: Jeena Huneidi
 * Blocker Task: T293960 <https://phabricator.wikimedia.org/T293960>
 * Current Status <https://versions.toolforge.org>: Stable on all wikis.


    By the Numbers

Sparklines comparing with the last 5 trains.

 * 319 Patches ▆▁▇▇█
 * 1 Rollback ▄▁█▁▂
 * 1 Day of delay ▂▂█▁▂
 * 18 Blockers ▂▁▂▆█


    Trainbow Love 

Thanks to folks who reported or resolved blockers:

 * Amir Sarabadani
 * Taavi Väänänen
 * Zabe
 * Legoktm
 * Umherirrender
 * Bartosz Dziewoński
 * Jon Robson
 * Urbanecm
 * Tyler Cipriani
 * Ammarpad
 * Lucas Werkmeister (WMDE)

Others no doubt contributed, but there'd be a rather long list of 
blockers to parse to figure out who I'm missing.


Additionally, a special thanks to everyone who helped debug T300214 - 
'No such file or directory' CI failures in multiple repos 
<https://phabricator.wikimedia.org/T300214> in #wikimedia-releng:


 * Zabe
 * hashar
 * bd8080
 * James_F
 * dancy
 * thcipriani
 * Amir1
 * andrewbogott
 * TimStarling

...and anyone else I'm more than likely missing in scrollback.

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.38.0-wmf.19 status update (blocked at group0 on T300194)

[Brennen Bearnes]

Hi all,

The 1.38.0-wmf.19[0] train is currently blocked at group0 by the
following regression:

* Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent [n]s in 
writes, exceeding the 3s limit

- https://phabricator.wikimedia.org/T300194

Thanks in advance for any assistance with this!

As ever, you can follow train progress on libera.chat's
#wikimedia-operations as well as on the blocker task[0].

Regards,

-- Your plodding train purveyors

[0]. https://phabricator.wikimedia.org/T293960
[1]. 


___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l]  Summary of 1.38.0-wmf.4 train deployment

[Brennen Bearnes]

This email is a summary of last week's Wikimedia production deployment 
of 1.38.0-wmf.4:


  Conductor: Ahmon Dancy
  Backup Conductor: Brennen Bearnes
  Blocker Task: https://phabricator.wikimedia.org/T281168
  Current Status: https://versions.toolforge.org/

 Numbers

Sparklines comparing with the last 5 trains.

  273 Patches ▇██▁▂
  0 Rollbacks ▃▁▃█▁
  0 Days of delay ▃▁▁█▁
  8 Blockers ▃▁█▆█

✨ Traintastic Folks 

Thanks to folks who reported/resolved blockers, and otherwise assisted 
with this train:


  Jon Robson
  Bartosz Dziewoński
  Zabe
  xSavitar
  Tim Starling
  Majavah
  James D. Forrester
  DannyS712
  ppelberg
  EAkinloose
  Krinkle

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] GitLab update: You can log in now

[Brennen Bearnes]

= tl;dr =

Users with Wikimedia developer accounts can now log in to 
https://gitlab.wikimedia.org/ and create projects under their user 
namespaces, regardless of institutional affiliation.


= The longer version =

Hey all,

This is an update on the Wikimedia GitLab project[0].

GitLab access was formerly restricted to WMF/WMDE/NDA users.  We've 
lifted that restriction.[1]


With that out of the way, we're working to sort out open questions about 
use of shared CI runners[2] and what Docker images they're allowed to 
run[3], along with shoring up various aspects of our infrastructure, 
cleaning up some lingering authentication bugs, and integrating with 
other development services and communication channels.


In general, you can follow this project as a whole on Phabricator, on 
the GitLab workboard.[4]  As ever, questions are welcome on Phab, in 
#wikimedia-releng on libera.chat, or directly in e-mail.


[0]. https://www.mediawiki.org/wiki/GitLab/Roadmap
[1]. https://phabricator.wikimedia.org/T288162
[2]. https://phabricator.wikimedia.org/T292094
[3]. https://phabricator.wikimedia.org/T291978
[4]. https://phabricator.wikimedia.org/tag/gitlab/

Best,

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] GitLab update: You can log in now

[Brennen Bearnes]

= tl;dr =

Users with Wikimedia developer accounts can now log in to 
https://gitlab.wikimedia.org/ and create projects under their user 
namespaces, regardless of institutional affiliation.


= The longer version =

Hey all,

This is an update on the Wikimedia GitLab project[0].

GitLab access was formerly restricted to WMF/WMDE/NDA users.  We've 
lifted that restriction.[1]


With that out of the way, we're working to sort out open questions about 
use of shared CI runners[2] and what Docker images they're allowed to 
run[3], along with shoring up various aspects of our infrastructure, 
cleaning up some lingering authentication bugs, and integrating with 
other development services and communication channels.


In general, you can follow this project as a whole on Phabricator, on 
the GitLab workboard.[4]  As ever, questions are welcome on Phab, in 
#wikimedia-releng on libera.chat, or directly in e-mail.


[0]. https://www.mediawiki.org/wiki/GitLab/Roadmap
[1]. https://phabricator.wikimedia.org/T288162
[2]. https://phabricator.wikimedia.org/T292094
[3]. https://phabricator.wikimedia.org/T291978
[4]. https://phabricator.wikimedia.org/tag/gitlab/

Best,

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l]  Summary of 1.38.0-wmf.3 train deployment (now with more automation)

[Brennen Bearnes]

Courtesy of Tyler's new https://trainbow.toolforge.org/ - this email is 
a belated summary of last week's Wikimedia production deployment of 
1.38.0-wmf.3:


Conductor: Brennen Bearnes
Backup Conductor: Jeena Huneidi
Blocker Task: https://phabricator.wikimedia.org/T281167
Current Status: https://versions.toolforge.org/

 By the Numbers

Sparklines comparing with the last 5 trains.

261 Patches ▄▇██▁
4 Rollbacks ▁▃▁▃█
3 Days of delay ▁▃▁▁█
7 Blockers ▃▃▁█▆

 Traintastic Folks 

Thanks to folks who reported, resolved, or triaged blockers, and 
otherwise assisted mightily with this train:


Timo Tijhof
Umherirrender
Zabe
DannyS712
Taavi Väänänen
James Forrester
Legoktm
Daniel Kinzler
Tim Starling
Pchelolo
GeoffreyT2000
AntiCompositeNumber
Pigsonthewing
Nikerabbit

(We only automated generating the e-mail, not the train.  Automating the 
train out of existence is a longer-term goal.)


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: GitLab: Users renamed to match shell names

[Brennen Bearnes]

On 9/27/21 9:20 PM, Jay prakash wrote:

Is there any tentative date for an individual volunteer to use Wikimedia 
GitLab?
The migration to shell usernames was the last item blocking that, so I'm 
hopeful we can open it up yet this week.  That's tracked on Phabricator 
here:


https://phabricator.wikimedia.org/T288162

We've also got a high-level roadmap for GitLab adoption:

https://www.mediawiki.org/wiki/GitLab/Roadmap

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] GitLab: Users renamed to match shell names

[Brennen Bearnes]

Hey all,

A quick note that, after discussion on T288392[0], we've reconfigured 
gitlab.wikimedia.org to use shell names for users instead of the LDAP 
CN, and renamed all existing logins to match. This seems like a better 
match for most users' expectations, and better ergonomics overall.


As a quick example, I was BrennenBearnes before, and now am at:

https://gitlab.wikimedia.org/brennen/

GitLab handles redirects for renamed users both in the web interface and 
for git remotes, but if you want to reconfigure the remote URL for an 
existing clone of a repo, you can do something like the following, 
substituting your new username:


$ git remote set-url origin g...@gitlab.wikimedia.org:brennen/test.git

[0]. https://phabricator.wikimedia.org/T288392

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l]  Summary of the 1.38.0-wmf.1 train deployment

[Brennen Bearnes]

This is a summary of the 1.38.0-wmf.1 train deployment for the week of 
2021-09-20.


Conductors: dduvall and hashar

Phabricator Task: 

Deployment Status: Live on all wikis: 

 Stats:

  - 352 patches
  - 3 risky patch notifications 
  - 5 blockers (0 remain open)
  - 0 rollbacks

 Notes:

The initial branch cut was for a 1.37.0 branch, and had to be re-run. 
See: 


Several user-facing regressions that could have been train blockers 
didn't get surfaced to deployers, which has prompted some thinking about 
the mechanisms by which the train actually gets blocked, and additional 
places we could steer people to the train blocker task.


Thanks to everyone who helped see this train through to completion, in 
particular (but in no particular order):


  - Zabe
  - DannyS712
  - Legoktm
  - matmarex
  - Majavah
  - kostajh
  - James_F

As ever, we couldn't do it without you! 
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] GitLab: Implementation & migration status update

[Brennen Bearnes]

tl;dr: https://gitlab.wikimedia.org/ exists now and individuals can host 
projects there; at the team level we're moving some RelEng projects over 
and will be in touch with early adopters soon; you can find a roadmap 
for this work at https://www.mediawiki.org/wiki/GitLab/Roadmap - read on 
for more details.




Hi all,

It's been a while since my last update on WMF's implementation of 
GitLab, so here's another one.


When I last wrote, we were hoping to have a minimum-viable installation 
of GitLab by the end of June.  That mostly worked out. 
gitlab.wikimedia.org is now live, and seeing (limited) real usage.


So can you use it? Yes, with caveats.

* Once T288162 - https://phabricator.wikimedia.org/T288162 - is resolved 
in the near future, everyone with a Wikimedia developer account should 
be able to sign in.  Until then, use is limited to members of the WMF, 
WMDE, and NDA groups.


* No shared CI job runners are provided yet

* No teams just yet, only individual projects, getting ACL right is hard :)

* Your data will not be obliterated from here forward, *probably*

* We have backups even!

Can teams use it? Soon!  RelEng is porting a few projects over, and then 
we'll get in touch with teams on the early adopter list.


Things we're thinking about:

* Shared CI runners: https://phabricator.wikimedia.org/T287279

* ACL groups and membership: https://phabricator.wikimedia.org/T282842

* Early adopters: https://phabricator.wikimedia.org/T282842

The WMF has also hired two full-time serviceops folks with a GitLab focus.

Lastly, I'll note that we have a roadmap for this work, with rough 
timelines, here:


https://www.mediawiki.org/wiki/GitLab/Roadmap

As ever, you can follow work on this project on Phabricator:

https://phabricator.wikimedia.org/project/board/5057/

Feel free to reach out with any questions via mail, Phabricator, or 
#wikimedia-releng on Libera Chat IRC.


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [Ops] Re: Gerrit patch upgrade July 19 16:00 UTC

[Brennen Bearnes]

On 7/19/21 11:05 AM, Antoine Musso wrote:

Both Gerrit have been upgraded to 3.2.11.  There is a configuration 
settings that is off which we manually fixed while doing the update.  A 
patch will follow up to address it.


Meanwhile puppet is still disabled, but the Gerrit service is up and 
running.


Further (and hopefully final) update:  Configuration patch has been 
merged; puppet is re-enabled on gerrit1001 and the service has been 
successfully restarted.


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [Ops] [Train] 1.37.0-wmf.12 status update (blocked at group1 on T285951)

[Brennen Bearnes]

On 7/1/21 9:54 AM, Martin Urbanec wrote:

Hello Brennen and team,

no longer on Freenode -- we're on Libera chat those days. Didn't manage 
to find the template on wiki, so I inform via mail instead.


Yeah, thanks for the heads up - I typically just re-edit the last of 
these messages I sent, and missed this.


For future reference, our rough template is here:

https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#If_the_train_is_blocked

(Sending this to all lists since several folks have already responded 
with the same correction.  Glad to know people are reading these. :) )


--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.37.0-wmf.12 status update (blocked at group1 on T285951)

[Brennen Bearnes]

Hi all,

The 1.37.0-wmf.12[0] train is currently blocked at group1 by the
following regression:

* Some section links in search results are redlinks
- https://phabricator.wikimedia.org/T285951

Thanks in advance for any assistance with this!

As ever, you can follow train progress on Freenode's
#wikimedia-operations as well as on the blocker task[0].

Regards,

-- Your irascible train crew

[0]. 
[1]. 


___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Summary of this week's deployment of 1.37.0-wmf.9: All good, no train next week.

[Brennen Bearnes]

This is a summary of this week's deployment of the 1.37.0-wmf.9 branch 
of MediaWiki and its extensions (also known as "the train"). The primary 
person in charge this week is Jeena Huneidi, with Mukunda Modell as 
backup, both from the Wikimedia Foundation Release Engineering team.


The summary/blocker task for this week is 
https://phabricator.wikimedia.org/T281150


The new version is running on all sites: https://versions.toolforge.org/

Please note that the next train will be 1.37.0-wmf.11, since the week of 
June 14th is taken up by the WMF's annual All Hands.


== Blockers ==

There were a total of four blockers for this train:

* Regression: Echo new talk page message banner has lost its orange 
background

** https://phabricator.wikimedia.org/T284496
* regression: Reply tool padding and height broken
** https://phabricator.wikimedia.org/T284567
* TypeError: Argument 1 passed to LanguageConverter::__construct() must 
be an instance of Language, instance of StubUserLang given, called in 
/srv/mediawiki/php-1.37.0-wmf.9/includes/language/LanguageConverterFactory.php 
on line 132

** https://phabricator.wikimedia.org/T284685
* Code editor failing to load completely on English Wiktionary
** https://phabricator.wikimedia.org/T284716

== Deprecation notices ==

Release Engineering has recently adopted the practice of filing 
deprecation notices in production as blockers for the following train. 
For 1.37.0-wmf.11, we've filed:


* PHP Deprecated: Passing extra fields (quality) to 
LocalFile::loadFromRow was deprecated in MediaWiki 1.37. Property 
assignment will be removed in a later version. [Called from 
OldLocalFile::loadFromRow]

** https://phabricator.wikimedia.org/T284777

==  ==

As usual, lots of people helped find, triage, and solve problems with 
this train.  Thanks to all involved (listed by Phabricator username):


* DannyS712
* matmarex
* Zabe
* Ladsgroup
* ovasileva
* Jdlrobson
* Edtadros
* Esanders
* ppelberg
* Erutuon
* Xiplus
* Urbanecm

...and anyone else we've missed.  Without your help we wouldn't be able 
to deploy the train.


Thanks!

-- Your train conductor collective
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] GitLab: Implementation status update and a call for early adopters

[Brennen Bearnes]

tl;dr: We're still moving to GitLab.  WMF Release Engineering & SRE are 
working with contractors on an MVP installation of GitLab CE, which we 
hope to have running by July, and in use by early-adopter projects soon 
thereafter. Read on for more details.




Hi all,

It's been over six months since we wrapped up the GitLab consultation 
process[0] with a decision to migrate from Gerrit to GitLab's Community 
Edition.  It seemed like a good idea to update the community with our 
progress since then.


We currently have contractors from Speed & Function working on a 
minimum-viable installation of GitLab CE.  We hope to be fairly 
confident installing this by the end of June (end Q4 of WMF's fiscal 
year). You can follow this work on the "GitLab (Initialization)" 
workboard in Phabricator.[1]


Additionally, we're hiring for two Service Operations SRE positions to 
build and support a more robust production installation[3], and planning 
for new job runner capacity to handle CI on GitLab.


Tentatively, we in RelEng hope to begin using GitLab for some of our own 
projects around July (early Q1 of WMF fiscal year) and begin migrating 
other projects thereafter.  (Keeping in mind, of course, that everything 
takes longer than you expect.[4])


We've already heard from some people that their project or team is 
interested in being an early adopter of GitLab.  If that describes you, 
please add a comment on T282842:


  https://phabricator.wikimedia.org/T282842

Feel free to reach out with any questions!

[0]. https://www.mediawiki.org/wiki/GitLab_consultation#Outcome
[1]. https://phabricator.wikimedia.org/project/view/5212/
[3]. https://boards.greenhouse.io/wikimedia/jobs/2659677
[4]. https://en.wikipedia.org/wiki/Hofstadter%27s_law

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Train] 1.37.0-wmf.5 status update (blocked again at group1)

[Brennen Bearnes]

Hi all,

A patch was provided for T282735 earlier today and the 1.37.0-wmf.5[0] 
train rolled to all wikis at the scheduled time.  Thanks to Zabe for 
speedy assistance there.


Unfortunately, the train is now blocked at group1 by the
following issues:

* MapSourcesPage.php: PHP Notice: Undefined offset: 13
- https://phabricator.wikimedia.org/T282833

* LogEventsList.php: PHP Notice: Undefined variable: pageName
- https://phabricator.wikimedia.org/T282834

Thanks in advance for any assistance with these!

As ever, you can follow train progress on Freenode's
#wikimedia-operations as well as on the blocker task[0].

Regards,

-- Your irascible train crew

[0]. 
[1]. 


___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

[Wikitech-l] [Train] 1.37.0-wmf.4 status update (blocked at group0 again)

[Brennen Bearnes]

Hi all,

The 1.37.0-wmf.4[0] train is currently blocked at group0 by the 
following issue:


* Query time out in ApiQueryLogEvents query
- https://phabricator.wikimedia.org/T282193

As ever, you can follow train progress on Freenode's 
#wikimedia-operations as well as on the blocker task[0].


Regards,

-- Your bedraggled train crew

[0]. 
[1]. 



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.37.0-wmf.4 status update (blocked at group0)

[Brennen Bearnes]

Hi all,

The 1.37.0-wmf.4[0] train is currently blocked at group0 by the 
following issue:


* Special:RecentChanges with userExpLevel=newcomer causes Fatal 
exception of type "Wikimedia\Rdbms\DBQueryError": Unknown column 
'actor_user'

- https://phabricator.wikimedia.org/T281981

If this issue is resolved before the 19:00 UTC train deployment window, 
the train will proceed as normal to group1.


As ever, you can follow train progress on Freenode's 
#wikimedia-operations as well as on the blocker task[0].


Regards,

-- Your bedraggled train crew

[0]. 
[1]. 



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.34 status update (blocked at group0)

[Brennen Bearnes]

Hi all,

The 1.36.0-wmf.29[0] train is currently blocked at group0 by the 
following issue:


* Bug in client error logging stops any errors from being logged in 
group 0 wikis

- https://phabricator.wikimedia.org/T277094

Note that we have recently updated the train policy[1] to use 
client-side errors as a signal of deployment health.


Jon Robson is working on T277094, and hopes to have a fix yet this 
afternoon, but if this is an area you're familiar with, help is likely 
welcome.


If all issues are resolved before the cutoff at 15:00 PST, the train 
will roll forward today.  Otherwise it will resume tomorrow during the 
US workday.


As ever, you can follow train progress on Freenode's 
#wikimedia-operations as well as on the blocker task[0].


Regards,

-- Your over-communicative train operator

[0]. 
[1]. 



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.29 status update (still blocked, but only one blocker remaining!)

[Brennen Bearnes]

Hi all,

After considerable effort this week, we're down to one blocker for the 
1.36.0-wmf.29[0] train:


* MemcachedPeclBagOStuff: Serialization of 'Closure' is not allowed
- https://phabricator.wikimedia.org/T273242

Thanks in advance for any help resolving this issue!  Once resolved, the 
train can resume Monday (European) morning.


-- Brennen, on behalf of your faithful train crew

[0]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.27 - 29 status update (abandoning .28, proceeding with .29 when unblocked)

[Brennen Bearnes]

Hi all,

A note that after a difficult end of last week and some further 
complications today, we're abandoning the 1.36.0-wmf.28 train[0], 
rolling all wikis back to wmf.27, and proceeding with wmf.29[1] once 
unblocked during this week's usual train windows.


This is a somewhat unusual step, but hopefully it will simplify the 
process this week.


We're treating the following outstanding issues as blockers for wmf.29:

* ApiEchoUnreadNotificationPages.php PHP Notice: Undefined index: query
- https://phabricator.wikimedia.org/T273479

* MemcachedPeclBagOStuff: Serialization of 'Closure' is not allowed
- https://phabricator.wikimedia.org/T273242

* Accessing WikiPage that cannot exist as a page: w:Help:Books/Book 
creator text. [Called from WikiPage::exists]

- https://phabricator.wikimedia.org/T273101

* MimeAnalyzer::improveTypeFromExtension() must be of the type string, 
null given

- https://phabricator.wikimedia.org/T273249

Thanks in advance for any help resolving these issues.  Also thanks to 
Urbanecm and Pchelolo for work over the weekend clearing some obstacles 
for the wmf.28 version, and all who helped out late last week.


-- Brennen, on behalf of your faithful train crew

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [MediaWiki-Docker] Splitting PHP-FPM and httpd containers

[Brennen Bearnes]

On 1/12/21 5:20 PM, Brennen Bearnes wrote:

tl;dr: We're splitting PHP & HTTP containers in the MediaWiki-Docker[0] 
development environment.  If you're not currently using 
MediaWiki-Docker, you can safely ignore this message.


[0]. https://www.mediawiki.org/wiki/MediaWiki-Docker
[1]. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/630988


After a round of feedback (thanks to Kosta, Željko, Timo, Bryan Davis, 
and probably others I'm forgetting), we've gone ahead and merged this one.


A working list of related documentation updates:

https://etherpad.wikimedia.org/p/mw-docker-docs

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.27 status update (blocked at group0 by T272508)

[Brennen Bearnes]

The 1.36.0-wmf.27 version of MediaWiki[0] is at group0[1], and is 
currently blocked.  Thanks to James Forrester, RhinosF1, abi_, 
Nikerabbit, Urbanecm, Daimona, Ladsgroup, Marostegui, and probably 
others I'm forgetting for getting us this far.


The new version cannot be deployed further until this issue is resolved:

* PropertyInfoSnakUrlExpander: Bad value for parameter 
$snak->getDataValue(): must be a DataValues\StringValue

- https://phabricator.wikimedia.org/T272508

Thanks for any help resolving this issue.  If a fix is merged with 
reasonable time to deploy before the 15:00 Pacific deployment cutoff, 
the train will continue to group1 today.[2]  Otherwise it will resume in 
the European morning window at earliest.


-- Your temporary train tender

[0]. 
[1]. 
[2]. 
https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#What_happens_next?


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.27 status update (rolled back from group0, blocked by T272428)

[Brennen Bearnes]

The 1.36.0-wmf.27 version of MediaWiki[0] has been rolled back from 
group0[1], and is currently blocked.


The new version cannot be deployed until this issue is resolved:

* Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist
- https://phabricator.wikimedia.org/T272428

Thanks for any help resolving this issue.  Since we're near the 3pm 
Pacific cutoff point for the day, at earliest, the train can roll 
forward in the European morning.[2]


-- Your loquacious locomotive laggard

[0]. 
[1]. 
[2]. 
https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#What_happens_next?


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [MediaWiki-Docker] Splitting PHP-FPM and httpd containers

[Brennen Bearnes]

Hey folks,

tl;dr: We're splitting PHP & HTTP containers in the MediaWiki-Docker[0] 
development environment.  If you're not currently using 
MediaWiki-Docker, you can safely ignore this message.


---

We're planning to merge a change to the MediaWiki-Docker environment to 
split PHP-FPM into a separate container from Apache.[1]


This should improve build efficiency for these images, and remove the 
need to duplicate so much stuff (Apache, etc.) between versions of PHP. 
 It also unblocks support for PHP 7.3/7.4, and upgrades XDebug to the 
3.x series.


What you'll need to change in existing setups:

In docker-compose-override.yaml:

* Linux users should specify a MW_DOCKER_UID & MW_DOCKER_GID for all 
containers.


In .env:

* Set XDEBUG_ENABLE=true and XHPROF_ENABLE=true if you want the 
corresponding extensions turned on.


* If you have an XDEBUG_CONFIG set, it may need updated to reflect
new configuration value names.[2]

Finally, I'd like to get it out the door this week, since it unblocks a 
number of requested improvements, but feedback is of course welcome in 
Gerrit.[1]


[0]. https://www.mediawiki.org/wiki/MediaWiki-Docker
[1]. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/630988
[2]. https://xdebug.org/docs/upgrade_guide

Thanks!

--
Brennen Bearnes
Release Engineering
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.16 status update (not yet deployed, 4 blockers)

[Brennen Bearnes]

The 1.36.0-wmf.16 version of MediaWiki[0] has not yet been deployed to 
any wikis[1] due to this week's US Election Day holiday, and is 
currently blocked.


The new version cannot be deployed until these issues are resolved or 
appropriately triaged as less urgent:


* Use of FormatMetadata::formatNum with non-numeric value was deprecated 
in MediaWiki 1.36. [Called from FormatMetadata::makeFormattedData]

- https://phabricator.wikimedia.org/T266677

* CommonsMetadata bad wfTimestamp call
- https://phabricator.wikimedia.org/T267033

* Undefined index: . at Language.php:3348
- https://phabricator.wikimedia.org/T267091

* Live preview shows  unparsed, with raw wikitext
- https://phabricator.wikimedia.org/T266311

Thanks for any help resolving these issues.  If they're handled in time 
to safely deploy before 15:00 PDT / 23:00 UTC today, the train can roll 
forward.  Otherwise it can resume during US working hours on Thursday.[2]


-- Your torpid train trundler

[0]. 
[1]. 
[2]. 
https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#What_happens_next?


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l]  Wikimedia production errors help

[Brennen Bearnes]

On 9/15/20 9:43 AM, Alex Ezell wrote:


Do we use levels for any of these error log outputs? That is, are they
classified on output as High, Medium, Low, Info, or something like that?


To an extent, yes.  We have separate channels for PHP errors and 
exceptions, for example, and although I don't think we currently 
differentiate in logstash, maybe we could plausibly draw a further 
distinction between PHP error levels.  Intuitively, a low number of PHP 
notices probably indicates something of lower severity than a high 
number of fatals, and so forth.


Teasing out more detail about reported error severity could be a useful 
exercise, but I'm not sure it would result in much more meaningful 
signals than we currently have about production health.  Serious 
problems can manifest as trivial-seeming notices, some issues start out 
that way and cascade over time, and generally any form of recurring 
logspam needs human evaluation before we can easily say much more than 
"this is a problem".



Or do we have to triage each of them as we examine them?


Yeah.  There are doubtless a lot of ways to improve the tooling we use 
for that process, but right now I think it would be most helpful if we 
just had more eyes _routinely_ on the logs and the workboard.  (See 
Tyler's earlier and much more detailed/thoughtful response to this thread.)


--
Brennen Bearnes
Release Engineering

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.3 status update (currently at group0, blocked)

[Brennen Bearnes]

The 1.36.0-wmf.3 version of MediaWiki  is currently blocked at group0.[0]

The new version can proceed no further until these issues are resolved 
or appropriately triaged:


* Uncaught ArgumentCountError: Too few arguments to function 
OOUI\Tag::appendContent(), 0 passed

- https://phabricator.wikimedia.org/T259745

* Argument 3 passed to 
CachingFallbackLabelDescriptionLookup::buildCacheKey() must be of the 
type string, null given

- https://phabricator.wikimedia.org/T259744

Thanks for any help resolving these issues.  If they're handled by 
roughly 14:30 PDT / 21:30 UTC today, the train can roll forward. 
Otherwise it can resume during US working hours on Thursday.


-- Your typical train temporizer

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.2 status update (currently at group1, blocked)

[Brennen Bearnes]

The 1.36.0-wmf.2 version of MediaWiki  is currently blocked at group1.[0]

The new version can proceed no further until this issue is resolved:

* Argument 1 passed to Wikimedia\Parsoid\Utils\DOMDataUtils::getDataMw() 
must be an instance of DOMElement

- https://phabricator.wikimedia.org/T259311

Thanks for any help resolving this issue.  Since we're past today's 
cutoff and there are no deploys on Friday, the train can roll forward on 
Monday morning, assuming a fix.


Thanks to Mholloway, Msantos, Legoktm, tgr, and RoanKattouw for their 
assistance with earlier blockers this week.


-- Your temporary train trundler

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.36.0-wmf.2 status update (currently at group0, blocked)

[Brennen Bearnes]

The 1.36.0-wmf.2 version of MediaWiki is blocked at group0.[0]

The new version can proceed no further until this issue is resolved:

* Warning: Locally stored wiki page has unsupported content model (from 
JsonConfig)

- https://phabricator.wikimedia.org/T259126

Thank you for your help resolving this issue!

-- Your temporary train toiler

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.38 status update (currently at group0, blocked)

[Brennen Bearnes]

The 1.35.0-wmf.38 version of MediaWiki is blocked at group0.

The new version can proceed no further until this issue is resolved:

* Fatal Error: Class MediaWiki\HookContainer\HookRunner contains 1 
abstract method and must therefore be declared abstract

- https://phabricator.wikimedia.org/T256305

Thank you for your help resolving this issue!

-- Your stoic train supplicant

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.37 status update (currently at group0, blocked)

[Brennen Bearnes]

The 1.35.0-wmf.37 version of MediaWiki was briefly unblocked and on 
group1 (thanks to Demian, Jdlrobson, and James_F for that fix), but is 
blocked[0] and reverted to group0.


The new version can proceed no further until these issues are resolved:

* DBUnexpectedError when moving a page with a StructuredDiscussion talkpage
- https://phabricator.wikimedia.org/T255608

* PHP Notice: Object of class MediaWiki\Debug\DeprecatablePropertyArray 
could not be converted to int

- https://phabricator.wikimedia.org/T255614

The train is currently stalled for the remainder of the US-local 
workday, and can resume during the European morning at the earliest, 
pending fixes.


Please note that, as Friday is a holiday, the week is shorter, and 
Release Engineering would like to finish the train on Wednesday instead 
of Thursday. If the issues are resolved by tomorrow, we can resume the 
train. Otherwise, train can continue on Monday.


Thank you for your help resolving these issues!

-- Your humble train trundler

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.32 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.32 version of MediaWiki is currently blocked[0].

The new version can't proceed to group0 until this issue is resolved:

* T247028: Database 'INSERT' query rate doubled (module_deps regression?)
- https://phabricator.wikimedia.org/T247028

Thanks in advance for any assistance!

-- Your train conductor from last week, making a guest appearance to 
send this mail


[0]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.31 status update (blocked again)

[Brennen Bearnes]

The 1.35.0-wmf.31 version of MediaWiki is currently blocked[0], and has 
been reverted to group1.


The new version can't proceed to all wikis until this issue is resolved:

* Increase in "Main slot of revision [number] not found in database!" 
after deploy of 1.35.0-wmf.31 to all wikis

- https://phabricator.wikimedia.org/T252156

If all issues are resolved before 15:00 Pacific / 22:00 UTC today, the 
train can proceed to all wikis.  Otherwise it will resume Monday, May 11th.


Thanks in advance for any assistance.

-- Your basic train brakeperson (North American edition)

[0]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.31 status update (unblocked)

[Brennen Bearnes]

The 1.35.0-wmf.31 version of MediaWiki is currently unblocked[0], has 
been released to group1, and (barring further blockers) will be rolled 
to all wikis at 19:00 UTC / 12:00 PDT, as scheduled.


Thanks to all who assisted in resolving T252079.

-- Your torpid train trundler (North American edition)

[0]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.31 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.31 version of MediaWiki is blocked[0], and has been 
reverted to group0.


The new version can't proceed beyond group0 until this issue is resolved:

* T252079: mw.wikibase.getLabelByLang('Q1','en') returning nil today
- https://phabricator.wikimedia.org/T252079

Assuming a fix, the train will resume, at the earliest, during the 
European backup conductor's workday.


Thanks for any help in resolving this issue.

-- Your torpid train trundler (North American edition)

[0]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.30 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.30 version of MediaWiki is blocked[0], and has been 
reverted to group0.


The new version can't proceed to all wikis until this issue is resolved:

* LoadBalancer: Transaction spent [n] second(s) in writes, exceeding the 
limit of [n]

- https://phabricator.wikimedia.org/T251457

Per policy, if all issues are resolved before 15:00 Pacific, the train 
can proceed to all wikis.  Otherwise, the train is likely to resume 
Monday, May 3rd at the earliest.


Thanks for any help in resolving this issue.

-- Your anxious train adjunct

[0]. 
[1]. 



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.30 status update (unblocked)

[Brennen Bearnes]

The 1.35.0-wmf.30 version of MediaWiki is no longer blocked[0], and is 
expected to proceed to group2 during the European train deploy window[1] 
on Thursday.


Thanks to cscott, ssastry, and James_F for assistance in unblocking the 
train.


-- Your scattered train secondary

[0]. 
[1]. 



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.30 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.30 version of MediaWiki is blocked[0].

The new version can't proceed to group2 [1] until this issue is resolved:

* T251409: Argument 1 passed to Parser::fetchTemplateAndTitle() must be 
an instance of Title, null given, called in 
/srv/mediawiki/php-1.35.0-wmf.30/extensions/Quiz/includes/Quiz.php on 
line 258

- https://phabricator.wikimedia.org/T251409

If all issues are resolved before Thursday's train window, the train can 
proceed to group2 at the usual time.


Thank you for your assistance!

-- Your scattered train secondary

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.23 and wmf.24 status update (both blocked)

[Brennen Bearnes]

The 1.35.0-wmf.23 version of MediaWiki is still blocked[0].

Neither this version nor the subsequent 1.35.0-wmf.24 can proceed until 
this issue is resolved:


* T247562: Warning: Memcached::setMulti(): failed to set key 
global:segment:...

- https://phabricator.wikimedia.org/T247562

Thanks to the many folks who have contributed debugging on T247562 so 
far.  If anyone else has any insight, further input would be certainly 
be appreciated.


-- Your erratic train enabler

[0]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.23 status update (still still blocked)

[Brennen Bearnes]

The 1.35.0-wmf.23 version of MediaWiki is still blocked[0].

The new version can't proceed to all wikis [1] until this issue is resolved:

* T247562: Warning: Memcached::setMulti(): failed to set key 
global:segment:...

- https://phabricator.wikimedia.org/T247562

This was generating a large spike of warnings of the following form:

ErrorException from line 340 of 
/srv/mediawiki/php-1.35.0-wmf.23/includes/libs/objectcache/MemcachedPeclBagOStuff.php: 
PHP Warning: Memcached::setMulti(): failed to set key 
global:segment:enwiki%3Apcache%3Aidhash%3A23309859-0!canonical:ce9eb2174b45be4c0a2966f5bfbf5f045e3b6388


Thanks for any help on this issue.

-- Your discombobulated train trundler

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.23 status update (still blocked)

[Brennen Bearnes]

The 1.35.0-wmf.23 version of MediaWiki is blocked[0].

The new version can't proceed to group2 [1] until this issue is resolved:

* T247562: Warning: Memcached::setMulti(): failed to set key 
global:segment:...

- https://phabricator.wikimedia.org/T247562

Assuming a fix, the train will likely resume Monday, March 16th.

Thank you for your help resolving this issue!

-- Your harried train functionary

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.23 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.23 version of MediaWiki is blocked[0].

The new version can't proceed to group2 [1] until these issues are 
resolved, and may be rolled back to group0 if necessary:


* T247458 PHP Notice: Undefined index: wgKartographerLiveData
- https://phabricator.wikimedia.org/T247458

* T247466 SimpleCacheWithBagOStuff: Cache key contains characters that 
are not allowed

- https://phabricator.wikimedia.org/T247466

* T247484 Lots of "EventBus: Unable to deliver all events"
- https://phabricator.wikimedia.org/T247484

If all issues are resolved before 15:00 PST, the train can resume today, 
otherwise the train will resume Monday, March 16th at the earliest.


Thank you for your help resolving these issues!

-- Your harried train functionary

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] New, simple Docker development environment for MediaWiki core

[Brennen Bearnes]

Hey all,

TL;DR: `docker-compose up` gets you a Docker environment with which
to develop.

The Engineering Productivity group is happy to announce the
availability of a new, official Docker environment for MediaWiki
core. [0] This is a component of our work on improving developer
productivity, as part of the Wikimedia Foundation's "Platform
Evolution" [1] multi-year priority, looking to support faster, more
reliable technical change for our communities. We've been exploring
options for a year now, and we had a great deal of input,
particularly at the TechConf 2019, where Kosta Harlan worked
closely with us to move this forward. [2]

This new environment has been built for simple experimentation,
development, and testing of proposed changes to MediaWiki core. It
is designed to be particularly simple and easy to use, and intended
particularly to be a good option for newbies, be they testers,
designers, developers, or others who have not yet invested a great
deal of their time in setting up local environments.

We intend for this environment to become the official, supported,
and advertised entry point for small-scale development. If you find
issues, or have suggestions for improvements, we'd love to hear
from you. [3] We will be adjusting various bits of documentation
over time to encourage futher use, but if you find some out-of-date
instructions, please do fix them, or flag for us to do.

We know that there are number of people who need a more complex,
configurable, and powerful development and testing environment,
even up to being a "Wikimedia production-like" state. This is not
that environment; we plan to provide a more configurable and thus
more complex, "heavy-weight" alternative for that use case in the
future. You may wish to follow our work in Phabricator. [4]

Our huge thanks to pioneering volunteer and staff colleagues who
have provided support, testing, and advice, and who have explored
different uses of Vagrant, Docker, and other techniques for
providing better forms of MediaWiki testing, development, and
hosting, which have inspired us on how to best provide this.  We
all owe you a great debt. Thank you.

Again, if you have questions, comments, or concerns, please do file
a task so that we can help you! [3]

[0] - https://www.mediawiki.org/wiki/Docker
[1] - 
https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Medium-term_plan_2019/Platform_evolution

[2] - https://phabricator.wikimedia.org/T238224
[3] - https://phabricator.wikimedia.org/tag/mediawiki-docker/
[4] - https://phabricator.wikimedia.org/tag/local-charts/

Yours,

--
Brennen Bearnes (he/him)
WMF Release Engineering

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.16 status update: Remains blocked

[Brennen Bearnes]

The 1.35.0-wmf.16 version of MediaWiki is blocked[0] from moving to
group2.  (A previous iteration of this mail that I sent on the 23rd of
January indicated that the train had only rolled as far as group0 - that
was an error on my part.)

The new version can't proceed to group2 [1] until this issue is resolved:

* T243548: Elevated response times and CPU usage after deploy of
1.35.0-wmf.16 to all wikis
- https://phabricator.wikimedia.org/T243548

If all issues are resolved before 15:00 PST, the train can resume today.

Thanks for any assistance!

-- Your bemused train enabler

[0]. 
[1]. 
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.16 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.16 version of MediaWiki is blocked[0].

The new version can't proceed to group1 and group2 [1] until this issue is
resolved:

* T243548: Elevated response times and CPU usage after deploy of
1.35.0-wmf.16 to all wikis
- https://phabricator.wikimedia.org/T243548

If all issues are resolved before 15:00 PST, the train can resume today,
otherwise the train will resume Monday, February 3rd at the earliest.

Please note that there are no deploys during the week of the Wikimedia
Foundation's annual All Hands gathering.

Thank you for your help resolving these issues!

-- Your bemused train enabler

[0]. 
[1]. 
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.8 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.8 version of MediaWiki is blocked[0].

The train was tentatively unblocked on Friday.  Details can be read
here:

* After deploy of 1.35.0-wmf.8 to group1, surge of "Cannot access the
  database: Unknown error"
  - https://phabricator.wikimedia.org/T239877

The new version can't proceed to group2 [1] until this (new) issue is
resolved:

* "PHP Notice: Undefined index: key" and similar in Cite.php and
  ReferenceStack.php
  - https://phabricator.wikimedia.org/T240248

If this issues is resolved, we will attempt to roll wmf.8 forward to all
wikis today, in order that wmf.10 can proceed as usual on Tuesday.

Thank you for your help with this issue!

-- Your frazzled train drudge

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.8 status update (blocked)

[Brennen Bearnes]

The 1.35.0-wmf.8 version of MediaWiki is blocked[0].

The new version can't proceed to group1 and group2 [1] until these
issues are resolved:

* After deploy of 1.35.0-wmf.8 to group1, surge of "Cannot access the
database: Unknown error"
- https://phabricator.wikimedia.org/T239877

Additionally, an incident report has been started for the outage caused
by T239877:

*
https://wikitech.wikimedia.org/wiki/Incident_documentation/20191204-MediaWiki

If these issues are resolved, the train can resume on Thursday
2019-12-05. If these issues are resolved on a Friday the train will
resume Monday.

Thank you for your help resolving these issues!

-- Your frazzled train drudge

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] This Week in Logspam (Thanksgiving Edition)

[Brennen Bearnes]

Hi all,

A high volume of error messages in production logs can make
it hard to glance at error logs after a deployment and
reason about the deployment's impact.[0]  This recurring mail
is part of an ongoing effort to reduce log noise.

At present I'm aware of three unresolved-but-tracked issues
which are causing a fair amount of noise:

* https://phabricator.wikimedia.org/T143756
** UnresolvedRedirectException in EntityAccessor::getEntity
   (scribunto)
** ~8000 occurrences today

* https://phabricator.wikimedia.org/T226751
** PHP error "non well formed numeric value encountered" from
   FormatMetadata->formatCoords
** ~600 occurrences today

* https://phabricator.wikimedia.org/T239165
** PHP Warning: count(): Parameter must be an array or an
   object that implements Countable in WikibaseMediaInfo
** ~120 occurrences today

As ever, help in eliminating these errors from production
logs is greatly appreciated.

As a procedural note, I'm sending this early in the week due
to the Thanksgiving holiday.  We'll be releasing 1.35.0-wmf.8 to
group 0 tomorrow (Tuesday) and then pausing there until
2019-12-04.

[0].
https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#Logspam

-- 
Brennen Bearnes (he/him)
Release Engineering
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] This Week in Logspam

[Brennen Bearnes]

Hi all,

From recent discussion with colleagues while deploying the train, it's
come up that there used to be a recurring e-mail along these lines, and
it seems like it may be useful to resume on a regular basis.

A high volume of error messages in production logs makes it hard to
glance at error logs after a deployment and reason about the
deployment's impact.[0]  This can be especially problematic at times
when our visibility into logs is impaired for unrelated reasons, as it
has been in recent weeks.[1]

It's my good fortune to be resuming this tradition during a week in
which the logspam situation seems to have improved.  I'm aware of two
currently tracked issues which are producing a noticeable amount of noise:

* T226751: PHP error "non well formed numeric value encountered" from
FormatMetadata->formatCoords[2]

* T233349: A data update callback triggered a fail-safe exception in
DataUpdateAdapter[3]

Help in eliminating these errors from production logs is greatly
appreciated, and I want to extend my thanks to everyone who has spent
time this week fixing similar issues.  I'd also like to thank everyone
who's been working on getting logstash back into usable shape.

[0].
https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#Logspam
[1]. https://phabricator.wikimedia.org/T234283
[2]. https://phabricator.wikimedia.org/T226751
[3]. https://phabricator.wikimedia.org/T233349

-- 
Brennen Bearnes (he/him)
Release Engineering
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.35.0-wmf.3 status update

[Brennen Bearnes]

The 1.35.0-wmf.3 version of MediaWiki is no longer blocked[0], and was
deployed to group 0 earlier today at 19:27 UTC, using the American
deploy window.

Thanks to andrewbogott for supplying helpful context in unblocking the
train, as well as hashar, thcipriani, mutante, and paladox for dealing
speedily with Gerrit migration issues.

The train should proceed to group 1 as regularly scheduled, during the
European deploy window.[1]

-- Your unassuming train minion

[0]. 
[1].


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] MediaWiki development images now published from deployment pipeline

[Brennen Bearnes]

Hey all,

Just a quick announcement that we (Release Engineering) are now
publishing Docker images[0] for mediawiki/core from Zuul's postmerge
pipeline.  At present, the image built from the most recent merge to
master is always tagged "dev".

These images are intended for use with local-charts[1], and make some
assumptions specific to that environment.  I hope to make them more
generally usable in the long run.

Images are built from specifications in .pipeline/config.yaml[2] and
.pipeline/blubber.yaml[3].

Please let me know if you have any concerns or questions!

[0]. https://tools.wmflabs.org/dockerregistry/wikimedia/mediawiki-core/tags/
[1]. https://gerrit.wikimedia.org/r/plugins/gitiles/releng/local-charts/
[2].
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/.pipeline/config.yaml
[3].
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/.pipeline/blubber.yaml

Thanks,

-- Brennen



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.34.0-wmf.16 status update

[Brennen Bearnes]

The 1.34.0-wmf.16 version of MediaWiki is no longer blocked[0], and has
been deployed to group1 and group2[1] as of 21:23 UTC.

Thanks to andre__, Krinkle, aaron, thcipriani, everyone else who worked
on unblocking the train, and everyone who has been so helpful during my
first week of train duty.

[0]. <https://phabricator.wikimedia.org/T220741>
[1]. <https://tools.wmflabs.org/versions/>

-- Your unassuming train minion

Brennen Bearnes

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.34.0-wmf.16 status update

[Brennen Bearnes]

The 1.34.0-wmf.16 version of MediaWiki is blocked[0].

The new version can't proceed to group1 and group2 [1] until these
issues are resolved:

* File pages are not created: Fatal exception of type
"Wikimedia\Rdbms\DBQueryError" - https://phabricator.wikimedia.org/T229605

* PHP Notice: Undefined property:
MediaWiki\Revision\RevisionRenderer::$wikiId -
https://phabricator.wikimedia.org/T229589

Once these issues are resolved, the train can resume. If these issues
are resolved on a Friday the train will resume Monday.

Thank you for your help resolving these issues!

-- Your unassuming train minion

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.34.0-wmf.16 status update

[Brennen Bearnes]

The 1.34.0-wmf.16 version of MediaWiki is no longer blocked[0]!

Thanks to mdholloway, Amir1, egardner, alaa_wmde, tstarling, and
everyone else who helped unblock the train.

We plan to deploy the new version to both group1 and group2[1] today.

[0]. <https://phabricator.wikimedia.org/T220741>
[1]. <https://tools.wmflabs.org/versions/>

-- Your unassuming train minion

Brennen Bearnes

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.34.0-wmf.16 status update

[Brennen Bearnes]

The 1.34.0-wmf.16 version of MediaWiki is blocked[0], after a brief
period of being unblocked.  Thanks to Jdlrobson and everyone else who
worked on unblocking the earlier issues.

1.34.0-wmf.16 cannot proceed to group1[1] until these issues are resolved:

* PHP Warning:
Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup::getEntityRevision:
Entity not loaded - https://phabricator.wikimedia.org/T229482

Once these issues are resolved, the train can resume. If these issues
are resolved on a Friday the train will resume Monday.

Thank you for your help resolving these issues!

-- Your unassuming train minion

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.34.0-wmf.16 status update

[Brennen Bearnes]

The 1.34.0-wmf.16 version of MediaWiki is blocked[0].

The new version can't proceed to group1[1] until these issues are resolved:

* Disable grouped results on RecentChanges page on mobile -
https://phabricator.wikimedia.org/T228280

* Inform AMC users that grouped results are not available on the recent
changes page - https://phabricator.wikimedia.org/T228516

Once these issues are resolved, the train can resume. If these issues
are resolved on a Friday the train will resume Monday.

Thank you for your help resolving these issues!

-- Your unassuming train minion

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.34.0-wmf.16 status update

[Brennen Bearnes]

The 1.34.0-wmf.16 version of MediaWiki is no longer blocked[0]!

Thanks to Krinkle, tstarling, and everyone else who worked on unblocking
the train.

We plan to deploy the new version to both group0 and group1[1] today.

[0]. <https://phabricator.wikimedia.org/T220741>
[1]. <https://tools.wmflabs.org/versions/>

-- Your unassuming train minion

Brennen Bearnes

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Train] 1.34.0-wmf.16 status update

[Brennen Bearnes]

The 1.34.0-wmf.16 version of MediaWiki is blocked[0].

The new version can't proceed to group0[1] until this issue is resolved:

* serialize(): "" returned as member variable from __sleep() but does
not exist - https://phabricator.wikimedia.org/T229366

Once this issue is resolved train can resume. If this issue is
resolved on a Friday the train will resume Monday.

Thank you for your help resolving these issues!

-- Your unassuming train minion

[0]. 
[1]. 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l