[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

Gehel moved this task from Incoming to In Progress on the Discovery-Search (Current work) board. Gehel added a comment. Changing priority from UBN to High as the outage is over. TASK DETAIL https://phabricator.wikimedia.org/T324346 WORKBOARD

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

Gehel lowered the priority of this task from "Unbreak Now!" to "High". Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T324346 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: BCornwall,

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

bking added a comment. Adding a reminder to myself to capture dashboard links, log-diving commands and other troubleshooting info from today's IRC log and add to the WDQS runbook . TASK DETAIL

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

BCornwall added a comment. Incident doc: https://wikitech.wikimedia.org/wiki/Incidents/2022-12-02_wdqs_outage Status doc: https://docs.google.com/document/d/1yLpK5yB9moJi1srCdgoEOBUn-9sUe1-ONJ4OGUgV4pE/ TASK DETAIL https://phabricator.wikimedia.org/T324346 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

bking added a comment. Follow-up email sent to Wikidata Users list: At this point, we believe we have mitigated the stability issues, and legitimate users should be able to access the WDQS service once again. If you are still having issues, please respond here or on the Phabricator

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

bking added a comment. Email sent to Wikidata Users list: We are aware of an ongoing issue with WDQS that begun around 1500 UTC today. To stabilize the service, we have introduced drastic measures including banning cloud provider IPs and certain user agents. Unfortunately, this will

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

Gehel added a comment. Just a short comment to acknowledge the issue. There are a number of people working on this at the moment. We'll post an update and an incident report once we know more. TASK DETAIL https://phabricator.wikimedia.org/T324346 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

Bugreporter added a project: Wikimedia-Incident. TASK DETAIL https://phabricator.wikimedia.org/T324346 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Bugreporter Cc: Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, Zabe,

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T324346 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

Bugreporter triaged this task as "Unbreak Now!" priority. TASK DETAIL https://phabricator.wikimedia.org/T324346 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Bugreporter Cc: Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

Bugreporter created this task. Bugreporter added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION See https://grafana.wikimedia.org/d/00489/wikidata-query-service?orgId=1=1m Many requests such as this one