[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

bking closed subtask T361114: Alert Search Platform and/or DPE SRE when Wikidata is lagged as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, bking Cc: dr0ptp4kt, bking,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Gehel Cc: dr0ptp4kt, bking, Aklapper, dcausse, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

bking moved this task from In Progress to Done on the Data-Platform-SRE (2024.05.06 - 2024.05.26) board. bking added a comment. As far as I know, this one is done...moving to "Done" status on Data Platform SRE workboard. TASK DETAIL https://phabricator.wikimedia.org/T360993 WORKBOARD

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Gehel edited projects, added Data-Platform-SRE (2024.05.06 - 2024.05.26); removed Data-Platform-SRE (2024.04.15 - 2024.05.05). TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Gehel Cc:

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Maintenance_bot Cc: dr0ptp4kt, bking, Aklapper, dcausse, Danny_Benjafield_WMDE,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014584 **merged** by Bking: [operations/puppet@production] updateQueryServiceLag: tune the min query rate of a pooled server https://gerrit.wikimedia.org/r/1014584 TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board. Gehel removed a project: Wikidata-Query-Service. TASK DETAIL https://phabricator.wikimedia.org/T360993 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Gehel edited projects, added Data-Platform-SRE (2024.04.15 - 2024.05.05); removed Data-Platform-SRE. TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Gehel Cc: dr0ptp4kt, bking, Aklapper,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Gehel added a project: Data-Platform-SRE. TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Gehel Cc: dr0ptp4kt, bking, Aklapper, dcausse, Danny_Benjafield_WMDE, Isabelladantes1983,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse closed subtask T361106: Restore wdqs1013 with a data transfer as Declined. TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: bking, Aklapper, dcausse, Danny_Benjafield_WMDE,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-29T08:36:32Z] repooling wdqs1013 (T360993 ) TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse added a comment. I could re-enable puppet on wdqs1013 and restart the updater to catchup on updates. But apparently this machine was repooled yesterday (as part of the wdqs scap deploy I suppose) and thus started to serve stale data without triggering any maxlag. It's when

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse added a comment. depooling the node we can see that the query rate actually going down to 0, request rate is generally very low on codfw so we might have to tune the threshold at around 0.2. F43663858: image.png TASK DETAIL

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-28T13:17:10Z] repooling wdqs2009 (test query rate when depooled T360993 ) TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-28T13:07:34Z] temporarily depooling wdqs2009 (test query rate when depooled T360993 ) TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.25; 2024-04-02). TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, ReleaseTaggerBot Cc: bking, Aklapper, dcausse,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014580 **merged** by jenkins-bot: [mediawiki/extensions/Wikidata.org@master] updateQueryServiceLag: add an option to tune the query rate https://gerrit.wikimedia.org/r/1014580 TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-27T23:58:24Z] T360993 [WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-27T22:30:28Z] T360993 [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test'

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-27T22:30:23Z] T360993 [WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'`

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-27T22:30:19Z] T360993 [WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` TASK

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-27T22:17:55Z] T360993 [WDQS Deploy] Tests passing following deploy of `0.3.138` on canary `wdqs1003`; proceeding to rest of fleet TASK DETAIL

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-27T22:16:57Z] T360993 [WDQS Deploy] Gearing up for deploy of wdqs `0.3.138`. Pre-deploy tests passing on canary `wdqs1003` TASK DETAIL

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:55:13Z] bking@cumin2002 running puppet against A:wdqs-main to apply nginx changes T360993 TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014551 **merged** by Bking: [operations/puppet@production] wdqs: add x-monitoring-query https://gerrit.wikimedia.org/r/1014551 TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014566 **merged** by jenkins-bot: [wikidata/query/rdf@master] Add support for x-monitoring-query header https://gerrit.wikimedia.org/r/1014566 TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

bking added a comment. Per `sudo cumin A:prometheus 'w'` from a cumin host, there are 8 active prometheus hosts. We also have 3 load balancer pools for each wdqs host : - wdqs - wdqs-ssl - wdqs-heavy-queries Each one of

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse added a comment. The approach taken is: - from nginx control a new header named 'x-monitoring-query' set to true if a list of criteria is met (currently using user-agent strings but could be extended to using source IPs as well I suppose) - from blazegraph, do not log query

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board. dcausse claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T360993 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014584 had a related patch set uploaded (by DCausse; author: DCausse): [operations/puppet@production] updateQueryServiceLag: tune the min query rate on a pooled server https://gerrit.wikimedia.org/r/1014584 TASK DETAIL

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014580 had a related patch set uploaded (by DCausse; author: DCausse): [mediawiki/extensions/Wikidata.org@master] updateQueryServiceLag: add an option to tune the query rate https://gerrit.wikimedia.org/r/1014580 TASK DETAIL

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014566 had a related patch set uploaded (by DCausse; author: DCausse): [wikidata/query/rdf@master] Add support for x-monitoring-query header https://gerrit.wikimedia.org/r/1014566 TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: bking, Aklapper, dcausse, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

gerritbot added a comment. Change #1014551 had a related patch set uploaded (by DCausse; author: DCausse): [operations/puppet@production] wdqs: add x-monitoring-query https://gerrit.wikimedia.org/r/1014551 TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse added a comment. Here are the UAs seen in hour of a depooled server: +--+-+ |UA|count|

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

bking added a comment. > It is possible that this metric is polluted with monitoring queries that do not relate to serving user traffic I did a little checking around this. Prometheus blackbox checks are defined here

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

Lucas_Werkmeister_WMDE added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE Cc: Aklapper, dcausse, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse triaged this task as "High" priority. dcausse added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Aklapper, dcausse, AWesterinen,

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse added a comment. Mitigation: - blazegraph stopped - updater stopped with the `/srv/wdqs/data_loaded` flag removed - puppet disabled TASK DETAIL https://phabricator.wikimedia.org/T360993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T360993: WDQS lag propagation to wikidata not working as intended

dcausse created this task. dcausse added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Propagating the lag of a wdqs host should only be done if this host is ''pooled'' (actually serving user traffic). Determining the ''pooling''