| Addshore lowered the priority of this task from "High" to "Normal". Addshore added a subscriber: MoritzMuehlenhoff. Addshore added a comment. |
So i poked around a little bit and did the following on mediawiki04 while reading https://wikitech.wikimedia.org/wiki/HHVM/Troubleshooting between 21:48 and 21:53 UTC:
which hhvm-needs-restart cat /usr/local/bin/hhvm-needs-restart hhvmadm check-health hhvm-dump-debug links -dump http://127.0.0.1/server-status -width 80 curl localhost:9002/dump-apc # The apc dump is visible @ /home/addshore/apc_dump cd /var/log/hhvm ls sudo tail -f error.log sudo tail -f *
And once I had done all of that https://wikidata.beta.wmflabs.org/wiki/Wikidata:Main_Page started responding.
During that time it looks like hhvm crashed and restarted https://logstash-beta.wmflabs.org/goto/a476120d0c2416111790af1e7e907e52
Apparently there is a stacktrace @ /var/log/hhvm/stacktrace.20682.logMar 6 21:51:37 deployment-mediawiki04 hhvm[20682]: Assertion failure: /srv/hhvm-3.18.5+dfsg/hphp/runtime/base/program-functions.cpp:2331: void HPHP::hphp_memory_cleanup(): assertion `g_context.isNull() || !g_context->hasRequestEventHandlers()' failed. Mar 6 21:51:37 deployment-mediawiki04 hhvm: Core dumped: Aborted Mar 6 21:51:37 deployment-mediawiki04 hhvm: Stack trace in /var/log/hhvm/stacktrace.20682.log Mar 6 21:51:37 deployment-mediawiki04 hhvm[20682]: [Tue Mar 6 21:51:37 2018] [hphp] [20682:7fa83bbff700:0:000001] [7fa8b0df8890:7fa8ab25f067:7fa8ab260448:d53718:21362a6:11724ad:1172b99:d4f2ef:d4f4bb:f31306:7fa8b0df1064:7fa8ab31262d] Core dumped: Aborted Mar 6 21:51:37 deployment-mediawiki04 hhvm[20682]: [Tue Mar 6 21:51:37 2018] [hphp] [20682:7fa83bbff700:0:000002] [7fa8b0df8890:7fa8ab25f067:7fa8ab260448:d53718:21362a6:11724ad:1172b99:d4f2ef:d4f4bb:f31306:7fa8b0df1064:7fa8ab31262d] Stack trace in /var/log/hhvm/stacktrace.20682.logLooking back at the commands I executed in .bash_history with timestamps one lines up perfectly with the crash:
#1520373097 curl localhost:9002/dump-apchttps://www.epochconverter.com/timezones?q=1520373097 = Mar 06 2018 21:51:37
I'm not sure if @MoritzMuehlenhoff might be able to speculate a bit about what might have happened here?
Setting to normal now as this doesn't seem so important, and also removing the train blocker.
Cc: MoritzMuehlenhoff, thcipriani, WMDE-leszek, daniel, hoo, aude, Ladsgroup, Addshore, Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
