[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel closed subtask T324811: Create WDQS Lag SLO dashboard with Grizzly documentation as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: bking, RLazarus, Gehel,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: bking, RLazarus, Gehel, MPhamWMF, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. With https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/917938, we now have the grizzly dashboard where we want it. That was the last blocker for

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel reopened subtask T324811: Create WDQS Lag SLO dashboard with Grizzly documentation as Open. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: bking, RLazarus, Gehel,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel added a project: Data-Platform-SRE. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: bking, RLazarus, Gehel, MPhamWMF, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel merged a task: T305951: Create SLI for Blazegraph uptime. Gehel added a subscriber: bking. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: bking, RLazarus, Gehel,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel closed subtask T323064: Create WDQS Uptime SLO dashboard in Grizzly as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: RLazarus, Gehel, MPhamWMF, Aklapper,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel closed subtask T324811: Create WDQS Lag SLO dashboard with Grizzly documentation as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: RLazarus, Gehel, MPhamWMF,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel closed subtask T325324: Evaluate options to soften wdqs paging as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: RLazarus, Gehel, MPhamWMF, Aklapper,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a subtask: T325324: Evaluate options to soften wdqs paging. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RLazarus, Gehel, MPhamWMF, Aklapper, Astuthiodit_1,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RLazarus, Gehel, MPhamWMF, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel closed subtask T323066: Understand meaning of trafficserver wdqs request data vs turnilo webrequest data as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc:

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. In T313751#8388946 , @Gehel wrote: > A few comments on the current dashboard : > > - a very quick

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Gehel added a comment. A few comments on the current dashboard : - a very quick look at Turnilo : the graph look different enough that I'd like to know why the

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. With respect to recording nginx request responses: **Getting direct logs**: One idea is to add `/var/log/nginx/access.log` to `RollingFileAppender` in `modules/query_service/templates/logback.xml.erb`

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. The current approach we're trying to work towards is recording the nginx response codes for requests. That will give us insight into the number of failures we're seeing. At a high level, these are the various response codes we expect for different scenarios:

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Maintenance_bot Cc: MPhamWMF, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

gerritbot added a comment. Change 841518 **merged** by Ryan Kemper: [operations/puppet@production] Revert "wdqs-test: try installing nginx w extras" https://gerrit.wikimedia.org/r/841518 TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

gerritbot added a comment. Change 841518 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] Revert "wdqs-test: try installing nginx w extras" https://gerrit.wikimedia.org/r/841518 TASK DETAIL

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

gerritbot added a comment. Change 841582 **merged** by Ryan Kemper: [operations/puppet@production] wdqs-test: try installing nginx w extras https://gerrit.wikimedia.org/r/841582 TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, gerritbot Cc: MPhamWMF, Aklapper, Jersione, Hellket777, LisafBia6531, Astuthiodit_1,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

gerritbot added a comment. Change 841582 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] [wip] query_service: try installing nginx w extras https://gerrit.wikimedia.org/r/841582 TASK DETAIL

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. With respect to the SLO itself, our goal is an SLO that captures the promise we make about service availability: namely, that WDQS is available on a **best-effort** basis. In practice, this means that if an issue arises out of "business hours", it's acceptable to

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. Gehel and I met with bblack today. Some highlights: - Best to use real user traffic if possible, rather than artificial. However this might be difficult for our use case (given that some subset of queries we consider invalid/failing) - If going

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. Intro (some context for traffic team) - Search team is working on creating an SLI to measure uptime of WDQS. We want our SLI to map as well to the actual user experience as possible, so to that end we're trying to come up

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. Some quick pros/cons of two possible approaches to getting the SLI metrics: approach #1 is to run a query or set of queries per-dc at a certain frequency, approach #2 is just to run a query on each host at a certain frequency * Approach #1: Hit

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper added a comment. Aisha has written some Jupyter notebooks to pull together a random selection from groupings of query by time-to-completion and query structure (which operators are used, basically). On the ops side of things we'll need to decide between whether we want to just

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

MPhamWMF set the point value for this task to "5". TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, MPhamWMF Cc: MPhamWMF, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

MPhamWMF moved this task from Incoming to Current work on the Wikidata-Query-Service board. MPhamWMF added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T313751 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MPhamWMF, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, CBogen,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T313751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Maintenance_bot Cc: MPhamWMF, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T313751: Create WDQS uptime SLO

MPhamWMF created this task. MPhamWMF added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION As a user and a maintainer of WDQS, I want an expectation of service availability so that I know when issues can/should be resolved. The WDQS