[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Lucas_Werkmeister_WMDE added a comment. Well, test.wikidata.org seems to be on .24 now and no breakage to be seen so far…TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: thcipriani, WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Lucas_Werkmeister_WMDE added a comment. Would you be willing to proceed with group0 without knowing whether this bug will occur there or not, as @Addshore suggested, or does this already block the train?TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: thcipriani, WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
thcipriani added a comment. In T188924#4027930, @Addshore wrote: This is starting to sound like some sort of odd beta issue, it feels like something is cached somewhere? I also wonder if the version of hvvm is the same on beta on production. I guess we can wait and see if this hurts testwikidata in group0 deployment tonight. I'm starting branch cut for this Shortly™, FYI. Also the version of hhvm in beta and production do seem different: 3.18.5+dfsg-1+wmf5+icu57 on beta vs 3.18.5+dfsg-1+wmf4 in prod.TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: thciprianiCc: thcipriani, WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Lucas_Werkmeister_WMDE added a comment. I also wonder if the version of hvvm is the same on beta on production. Good question. Special:Version reports the same string (3.18.6-dev (srv)) on mediawiki.org and en.wikipedia.beta.wmlabs.org. dpkg -s hhvm on deployment-mediawiki04.… reports 3.18.5+dfsg-1+wmf5+icu57, which has a different minor version number than what’s reported on Special:Version (by the same system, according to the server header). I don’t have production shell access, so I can’t check which dpkg version is installed there.TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Addshore added a comment. This is starting to sound like some sort of odd beta issue, it feels like something is cached somewhere? I also wonder if the version of hvvm is the same on beta on production. I guess we can wait and see if this hurts testwikidata in group0 deployment tonight.TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Lucas_Werkmeister_WMDE added a comment. So I temporarily added a few debug statements (only long enough for me to make one web request, then removed again immediately), and it seems that adding this right after the wbsearchentities definition in Wikibase.php (L251) file_put_contents( '/tmp/wbsearchentities-factory', var_export( $wgAPIModules['wbsearchentities']['factory'], true ) ); and this after WikibaseRepo.entitytypes.php is loaded in WikibaseRepo::getDefaultEntityTypes (L560) file_put_contents( '/tmp/item-content-handler-factory-callback', var_export( $repoEntityTypes['item']['content-handler-factory-callback'], true ) ); makes the wiki work again. Somehow. (It seems to be reliable – I tried adding and removing them about ten times, and each time the wiki went from broken to working back to broken.) The reason I added those debug statements was that I suspected the interpreter was mixing up closures, and I wanted to see if these two were dumped with the same ID (e. g. Closure$#5;2604). But it looks like those statements are actually enough to fix the bug, somehow. If my suspicion is correct (which could also explain the stack overflow in the initialization – there’s a closure in that stack trace too), I suppose it’s possible that printing a closure is enough to stop the interpreter from mixing them up, perhaps because it causes some kind of deoptimization or something… does this sound at all plausible? Counterpoint: as far as I can tell from /var/log/dpkg.log*, HHVM hasn’t been upgraded on that host since February 8th, so if this really is an interpreter bug you’d think it should have started to happen sooner. So this theory definitely doesn’t provide a complete explanation yet as to how this problem came to be.TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Lucas_Werkmeister_WMDE added a comment. it looks to me like $wgApiModules['wbsearchentities']['factory'] (Wikibase.php L250) was somehow registered as a content handler factory‽ Yup, the wbsearchentities factory for one entity type and the wbremoveclaims for another one. No idea how. Investigation continues…TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Lucas_Werkmeister_WMDE added a comment. #0 /srv/mediawiki/php-master/extensions/Wikibase/repo/Wikibase.php(250): MWExceptionHandler::handleError(integer, string, string, integer, array, array) #1 [internal function]: Closure$#5() #2 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContentFactory.php(204): call_user_func(Closure$#5;2604) #3 /srv/mediawiki/php-master/extensions/Wikibase/repo/RepoHooks.php(116): Wikibase\Repo\Content\EntityContentFactory->getContentHandlerForType(string) I’m trying to understand what’s happening here, and it looks to me like $wgApiModules['wbsearchentities']['factory'] (Wikibase.php L250) was somehow registered as a content handler factory‽ So the EntityContentFactory tries to call that factory to get a content handler (with no arguments, because those factories aren’t supposed to take arguments) and it crashes because the wbsearchentities factory (a completely unrelated kind of factory!) expects an ApiMain. But how does that happen? I’m bewildered.TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Lucas_Werkmeister_WMDE added a comment. The first instance of that “must be an instance of ApiMain” error in EntityContentFactory is at 13:46 UTC, so a bit before the first stack overflow. No idea if they’re related.TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Addshore added a comment. I see the following in the logs after hitting https://wikidata.beta.wmflabs.org/wiki/Wikidata:Main_Page a few times Fatal error: Stack overflow in /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/WikibaseRepo.php on line 622 [Mon Mar 5 20:05:17 2018] [hphp] [20682:7fa8713ff700:22210:01] [] \nFatal error: Stack overflow in /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/WikibaseRepo.php on line 622 [Mon Mar 5 20:05:02 2018] [hphp] [20682:7fa7d2fff700:22063:01] [] \nCatchable fatal error: Argument 1 passed to __invoke() must be an instance of ApiMain, undefined variable given in /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content Catchable fatal error: Argument 1 passed to __invoke() must be an instance of ApiMain, undefined variable given in /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContentFactory.php on line 204TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188924: Stack overflow in WikibaseRepo initialization on Wikidata Beta
Addshore added a comment. Marked as high so that we investigate this, in theory the cause would be in this weeks train.TASK DETAILhttps://phabricator.wikimedia.org/T188924EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
