Dzahn added a comment.
[20:17:19] <mutante> ryankemper: I guess it makes sense that
"job_wikidata-updateQueryServiceLag" could not run during current work
[20:19:46] <ryankemper> mutante: yeah, I'm not familiar with how the
job works specifically but that would make sense
[20:20:42] <ryankemper> if it's supposed to alert at <50% availability
then that might be a bit unexpected because only one node in each `dc x
[internal, external]` is being re-imaged at a time
[20:21:12] <ryankemper> So for codfw for example there's one codfw
wdqs-internal host that would be unable to report and one codfw external wdqs
host
[20:21:48] <mutante> ryankemper: no, this case is not about
availability, it's "one of the mediawiki maintenance 'crons' (that are now
systemd timers) failed to run on the maintenance servers
[20:26:03] <mutante> ryankemper: what happens is: "maintenance job
tries to update what the current lag is like.. tries to get lag data from
prometheus and that fails. now since it's a systemd timer and not a cron it
means it's a failed service which then turns into an Icinga alert about
"systemd state is bad on a mwmaint server". and nothing clears it.. but this
job runs every minute.. so it's more alert than
[20:26:09] <mutante> is appropriate
[20:26:35] <mutante> let me just clear that failed service and wait
a minute
[20:27:14] <mutante> !log mwmaint1002 - systemctl reset-failed to
clear icinga alert
TASK DETAIL
https://phabricator.wikimedia.org/T269693
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Dzahn
Cc: Dzahn, RLazarus, RKemper, Aklapper, Ramtin0071, Devnull, lmata, Muchiri124,
CBogen, Akuckartz, Legado_Shulgin, Nandana, Namenlos314, Davinaclare77,
Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic,
Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen,
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs,
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
