Final update: Anomie's theory was spot on. 1.29.0-wmf.9 has been deployed to all wikis for ~7 hours and appears to be stable.
I've filed a follow-up task for the root cause. It will very likely bite us in the future if not properly addressed. Further discussion is on phabricator: https://phabricator.wikimedia.org/T156541 Thanks to everyone who pitched in on this and especially to Anomie for getting to the bottom of it. Have a great weekend everyone! - Mukunda --- Mukunda Modell Wikimedia Release Engineering On Fri, Jan 27, 2017 at 12:47 PM, Mukunda Modell <[email protected]> wrote: > Status update: > > Anomie has a solid theory about the cause of > https://phabricator.wikimedia.org/T156364 and we are going ahead with > deploying wmf.9 today ASAP. Assuming everything goes as planned then this > will be resolved shortly. > > On Fri, Jan 27, 2017 at 11:02 AM, Mukunda Modell <[email protected]> > wrote: > >> Correction: It looks like I only reverted group2 to wmf.8 >> >> I'm not sure whether to go ahead and revert all wikis or leave it like it >> is for now. >> >> On Fri, Jan 27, 2017 at 10:24 AM, Mukunda Modell <[email protected]> >> wrote: >> >>> The deployment of MediaWiki 1.29.0-wmf.9 is currently blocked and all >>> wikis have been reverted to wmf.8. This latest revert is due to excessive >>> warnings which appeared in logs immediately after deploying to group2. The >>> warnings all read as follows: >>> >>> Warning: Empty regular expression in /srv/mediawiki/php-1.29.0-wmf. >>> 9/includes/parser/DateFormatter.php on line 200 >>> >>> This issue is tracked in https://phabricator.wikimedia.org/T156364 and >>> currently blocks further deployments. >>> >>> More background: >>> >>> A similar flood of log spam occurred after deploying the branch to >>> group1, however, the problem subsided after a while and did not reappear >>> until Thursday's deployment. >>> >>> There have been recent changes to DateFormatter.php, however, the >>> changes do not appear to be the direct culprit. In fact, after rolling back >>> the changes, a different message appears in the log, however, with less >>> frequency than the line 200 error. >>> >>> Several engineers have looked at this issue but so far we have not >>> isolated the root cause. Additional eyes on this issue would be welcomed >>> and further discussion should be directed to Phabricator. In the interest >>> of keeping everyone in the loop, I will follow up in this thread if/when >>> there is a resolution. >>> >>> Regards, >>> Mukunda >>> >>> --- >>> Mukunda Modell >>> Wikimedia Release Engineering >>> >> >> > _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
