Dzahn added a comment.
I was looking at Icinga for other reasons and noticed: wdqs1004 - "..Query Service HTTP Port on wdqs1004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable ". (unhandled CRIT since about 18 hours, does it have notifications?) I did a `systemctl restart wdqs-blazegraph` and that caused: RECOVERY - Query Service HTTP Port on wdqs1004 is OK: HTTP OK: HTTP/1.1 200 OK but in turn also a new: <+icinga-wm> PROBLEM - WDQS high update lag on wdqs1004 is CRITICAL: 1.224e+05 ge.... https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag told me to also restart the `wdqs-updater` service, so I did that. When that did not seem to immediately resolve it I also depooled the server as the docs above say to do until it catches up. Reusing this ticket. TASK DETAIL https://phabricator.wikimedia.org/T290832 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: So9q, Dzahn Cc: Dzahn, So9q, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org