Lucas_Werkmeister_WMDE created this task. Lucas_Werkmeister_WMDE added projects: Wikidata, Wikidata-Query-Service, SRE, Wikidata.org. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: wdwb-tech.
TASK DESCRIPTION As a Wikidata editor, I want the API maxlag calculation to only take into account the update lag of fully pooled WDQS servers, so that maintenance on depooled servers does not slow down bot edits. **Problem:** It seems that query service backend servers can be depooled in (at least?) two different ways, LVS and DNS. During the upgrade T331126 <https://phabricator.wikimedia.org/T331126>, query service servers were reportedly depooled via DNS, but mostly still pooled in LVS: lucaswerkmeister-wmde@mwmaint2002:~$ curl -sH 'Accept: application/json' 'http://lvs1019:9090/pools/wdqs_80' | jq . { "wdqs1013.eqiad.wmnet": { "pooled": true, "enabled": true, "up": true, "weight": 10 }, "wdqs1012.eqiad.wmnet": { "pooled": true, "enabled": true, "up": true, "weight": 10 }, "wdqs1015.eqiad.wmnet": { "pooled": true, "enabled": true, "up": true, "weight": 10 }, "wdqs1006.eqiad.wmnet": { "pooled": false, "enabled": false, "up": true, "weight": 10 }, "wdqs1004.eqiad.wmnet": { "pooled": false, "enabled": false, "up": true, "weight": 10 }, "wdqs1014.eqiad.wmnet": { "pooled": true, "enabled": true, "up": true, "weight": 10 }, "wdqs1016.eqiad.wmnet": { "pooled": true, "enabled": true, "up": true, "weight": 10 }, "wdqs1007.eqiad.wmnet": { "pooled": true, "enabled": true, "up": true, "weight": 10 }, "wdqs1005.eqiad.wmnet": { "pooled": true, "enabled": true, "up": true, "weight": 10 } } Because the `updateQueryServiceLag.php` maintenance script in the Wikidata.org extension (specifically, `WikimediaLoadBalancerQueryServicePoolStatusProvider`) only takes into account LVS, but not DNS, these servers were still considered pooled for maxlag purposes, but as they were not receiving updates, the lag skyrocketed. **Example:** Grafana <https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m&from=1678146711574&to=1678189911575&viewPanel=8> F36896429: image.png <https://phabricator.wikimedia.org/F36896429> **Screenshots/mockups:** **BDD** GIVEN AND WHEN AND THEN AND **Acceptance criteria:** - **Open questions:** - //Should// the maintenance script consult multiple sources of truth for pooled/depooled status, or can we perhaps consolidate them? TASK DETAIL https://phabricator.wikimedia.org/T331405 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE Cc: Aklapper, dcausse, Lucas_Werkmeister_WMDE, Astuthiodit_1, AWesterinen, karapayneWMDE, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, Muchiri124, CBogen, ItamarWMDE, Akuckartz, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi
_______________________________________________ Wikidata-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected]
