Joe triaged this task as "Unbreak Now!" priority. Joe added projects: Platform Engineering, Wikidata. Joe added a comment.
I'm not 100% sure that slabs are the problem here, but I'll try to followup later. In the meantime, the servers we've rebooted yesterday are definitely showing the same behaviour again. This means that in a week we'll have to reboot them all again. I am therefore: 1 - raising this task to UBN! priority, as this has the potential to disrupt the work of all SREs constantly 2 - focusing on looking at what was changed on August 4th in the morning, mostly in code, and try to rule out stuff. This might result in some reverts. 3 - Trying to build better automation around the task of rebooting a large fleet of servers. Also adding tags for platform engineering and wikidata as the two potentially worrisome code changes regard Score's enablement and a change to Wikibase. TASK DETAIL https://phabricator.wikimedia.org/T260281 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
