| hashar changed the title from "Wikidata at 1.28.0-wmf.19 no more replicate to wikis (replag raise / dispatch stop)" to "Jobs invoking SiteConfiguration::getConfig cause HHVM to fail updating the bytecode cache due to being filesize limited to 512MBytes". hashar removed a project: Patch-For-Review. hashar edited the task description. (Show Details) |
EDIT DETAILS
Incident report for some summary https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWiki
Summary
=======
Some jobs such as **account creation**, **account rename** or **Wikidata dispatcher** invokes `SiteConfiguration::getConfig()` since they have to act on several wikis. That method shells out to `mwscript maintenance/getConfiguration.php` with ulimits being applied, most notably a file size limit of 512MBytes.
However, when HHVM runs the command, it tries to update the byte code cache (either `/var/cache/hhvm/fcgi.sq3` or `/var/cache/hhvm/cli.sq3`). That causes a system error `EFBIG (File too large)` and the job fail.
See [[https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWiki incident report]] for actionables.
See below comments for debugging / technical details.
Original task
=========
After pushing 1.28.0-wmf.19 to group1 (which includes wikidatawiki) the dispatch seems to have stopped entirely.
Summary
=======
Some jobs such as **account creation**, **account rename** or **Wikidata dispatcher** invokes `SiteConfiguration::getConfig()` since they have to act on several wikis. That method shells out to `mwscript maintenance/getConfiguration.php` with ulimits being applied, most notably a file size limit of 512MBytes.
However, when HHVM runs the command, it tries to update the byte code cache (either `/var/cache/hhvm/fcgi.sq3` or `/var/cache/hhvm/cli.sq3`). That causes a system error `EFBIG (File too large)` and the job fail.
See [[https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWiki incident report]] for actionables.
See below comments for debugging / technical details.
Original task
=========
After pushing 1.28.0-wmf.19 to group1 (which includes wikidatawiki) the dispatch seems to have stopped entirely.
...
TASK DETAIL
EMAIL PREFERENCES
To: hashar
Cc: MZMcBride, Tobi_WMDE_SW, FastLizard4, JJMC89, zeljkofilipin, Lydia_Pintscher, daniel, aude, Addshore, Aklapper, greg, Legoktm, demon, gerritbot, Stashbot, hashar, D3r1ck01, Liudvikas, Izno, Luke081515, Wikidata-bugs, ArielGlenn, JanZerebecki, Mbch331, Jay8g, Joe, jeremyb, mmodell
Cc: MZMcBride, Tobi_WMDE_SW, FastLizard4, JJMC89, zeljkofilipin, Lydia_Pintscher, daniel, aude, Addshore, Aklapper, greg, Legoktm, demon, gerritbot, Stashbot, hashar, D3r1ck01, Liudvikas, Izno, Luke081515, Wikidata-bugs, ArielGlenn, JanZerebecki, Mbch331, Jay8g, Joe, jeremyb, mmodell
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
