Addshore added a comment.
So there is one more aspect that plays into this issue, which means the affects will not entirely disappear for another ~5 hours for all lookups, even with page purges. There is a layer of memcached data that now sits between wikidata and the renderings of client pages for LUA calls. This cache has a TTL of 24 hours and holds the data that is displayed by LUA that has been affected. As a result of the issue this cache now holds nil values for all terms that were used in page renderings during the hours that group1 had buggy code (that were not already in the cache with correct values.0 In T249963#6115102 <https://phabricator.wikimedia.org/T249963#6115102>, @brennen wrote: > Rolled back to group0 at 02:56 UTC for T252079 <https://phabricator.wikimedia.org/T252079>; writing a "blocked" status update mail. So around 02:56 on 8th May this cache will no longer have bad data in it and page purges would correctly fix the page renderings. We have no way to actively purge the cache keys that hold bad data currently. We will look into better ways to recover from an incident like this in the future. Just off the top of my head, this could be: - a way to force this cache to be updated when performing a page purge (similar to force links update?) - a way to ditch all of the cache keys in this cache (probably just incrementing some value in the cache key) Of course we have also identified an area that somehow misses all existing testing and we will dig into this too. TASK DETAIL https://phabricator.wikimedia.org/T252079 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Jdforrester-WMF, VIGNERON, Lyokoi, Fralambert, RhinosF1, kaldari, Hsarrazin, Vriullop, Husky, Lucas_Werkmeister_WMDE, Addshore, Mike_Peel, WMDE-leszek, Ladsgroup, brennen, Dominicbm, Mholloway, WilliamGraham, Lydia_Pintscher, Liuxinyu970226, Aklapper, matej_suchanek, Jarekt, Blissjay007, Oblanco79, Alter-paule, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, darthmon_wmde, Kent7301, alaa_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, lucamauri, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
