Joe closed this task as "Resolved". Joe added a comment.
Reporting here in brief: - We confirmed the problem had to do with activating firejail for all executions of external programs. That triggered a kernel bug - This kernel bug can be bypassed by disabling kernel memory accounting in cgroups, or by moving to a newer kernel like 4.19 - For now we've gone through a full rolling reboot of our mediawiki servers to disable kernel memory accounting in cgroups - We might move to 4.19 at a later stage. The issue is resolved and we're seeing ~ stable memory usage across all clusters, so the issue can be considered resolved. TLDR for third parties: if you run MediaWiki with `$wgShellRestrictionMethod = 'firejail';` you should do so with a relatively recent kernel, 4.19+ or 5.3+ IIRC. TASK DETAIL https://phabricator.wikimedia.org/T260281 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: eprodromou, Michael, NullPointer, Platonides, hashar, Addshore, Majavah, Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
