[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-05 Thread Gehel 
Gehel moved this task from Incoming to In Progress on the Discovery-Search 
(Current work) board.
Gehel added a comment.


  Changing priority from UBN to High as the outage is over.

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: BCornwall, WMDE-leszek, RKemper, GeneralNotability, Gehel, jhathaway, 
bking, Michael, Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, 
karapayneWMDE, Invadibot, Zabe, MPhamWMF, maantietaja, lmata, CBogen, 
ItamarWMDE, Akuckartz, PatsagornY, DannyS712, Nandana, Namenlos314, jijiki, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Bawolff, 
Manybubbles, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-05 Thread Gehel 
Gehel lowered the priority of this task from "Unbreak Now!" to "High".
Gehel added a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: BCornwall, WMDE-leszek, RKemper, GeneralNotability, Gehel, jhathaway, 
bking, Michael, Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, 
karapayneWMDE, Invadibot, Zabe, MPhamWMF, maantietaja, lmata, CBogen, 
ItamarWMDE, Akuckartz, PatsagornY, DannyS712, Nandana, Namenlos314, jijiki, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Bawolff, 
Manybubbles, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread bking 
bking added a comment.


  Adding a reminder to myself to capture dashboard links, log-diving commands 
and other troubleshooting info from today's IRC log and add to the WDQS runbook 
  .

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: BCornwall, WMDE-leszek, RKemper, GeneralNotability, Gehel, jhathaway, 
bking, Michael, Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, 
karapayneWMDE, Invadibot, Zabe, MPhamWMF, maantietaja, lmata, CBogen, 
ItamarWMDE, Akuckartz, PatsagornY, DannyS712, Nandana, Namenlos314, jijiki, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Bawolff, 
Manybubbles, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread BCornwall 
BCornwall added a comment.


  Incident doc: 
https://wikitech.wikimedia.org/wiki/Incidents/2022-12-02_wdqs_outage
  
  Status doc: 
https://docs.google.com/document/d/1yLpK5yB9moJi1srCdgoEOBUn-9sUe1-ONJ4OGUgV4pE/

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: BCornwall
Cc: BCornwall, WMDE-leszek, RKemper, GeneralNotability, Gehel, jhathaway, 
bking, Michael, Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, 
karapayneWMDE, Invadibot, Zabe, MPhamWMF, maantietaja, lmata, CBogen, 
ItamarWMDE, Akuckartz, PatsagornY, DannyS712, Nandana, Namenlos314, jijiki, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Bawolff, 
Manybubbles, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread bking 
bking added a comment.


  Follow-up email sent to Wikidata Users list:
  
At this point, we believe we have mitigated the stability issues, and 
legitimate users should be able to access the WDQS service once again. If you 
are still having issues, please respond here or on the Phabricator ticket 
linked below.

Thanks once again for your patience and have a great weekend!

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: WMDE-leszek, RKemper, GeneralNotability, Gehel, jhathaway, bking, Michael, 
Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, 
Zabe, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, PatsagornY, 
DannyS712, Nandana, Namenlos314, jijiki, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Bawolff, Manybubbles, ArielGlenn, He7d3r, 
Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread bking 
bking added a comment.


  Email sent to Wikidata Users list:
  
We are aware of an ongoing issue with WDQS that begun around 1500 UTC 
today. To stabilize the service, we have introduced drastic measures including 
banning cloud provider IPs and certain user agents. Unfortunately, this will 
likely affect legitimate users of the service. We are currently working on less 
disruptive measures and will update https://phabricator.wikimedia.org/T324346 
when we have more information to share. Thanks for your patience, and we 
sincerely apologize for the inconvenience.

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bking
Cc: Gehel, jhathaway, bking, Michael, Aklapper, Bugreporter, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, Zabe, MPhamWMF, maantietaja, lmata, 
CBogen, ItamarWMDE, Akuckartz, PatsagornY, DannyS712, Nandana, Namenlos314, 
jijiki, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Bawolff, 
Manybubbles, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread Gehel 
Gehel added a comment.


  Just a short comment to acknowledge the issue. There are a number of people 
working on this at the moment. We'll post an update and an incident report once 
we know more.

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Gehel, jhathaway, bking, Michael, Aklapper, Bugreporter, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, Zabe, MPhamWMF, maantietaja, lmata, 
CBogen, ItamarWMDE, Akuckartz, PatsagornY, DannyS712, Nandana, Namenlos314, 
jijiki, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Bawolff, 
Manybubbles, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread Bugreporter 
Bugreporter added a project: Wikimedia-Incident.

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, Zabe, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, 
PatsagornY, DannyS712, Nandana, Namenlos314, jijiki, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Bawolff, 
Manybubbles, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread Maintenance_bot 
Maintenance_bot added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread Bugreporter 
Bugreporter triaged this task as "Unbreak Now!" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Aklapper, Bugreporter, Astuthiodit_1, AWesterinen, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, 
RhinosF1
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T324346: High error rate in WDQS since 15:00 UTC

2022-12-02 Thread Bugreporter 
Bugreporter created this task.
Bugreporter added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  See 
https://grafana.wikimedia.org/d/00489/wikidata-query-service?orgId=1&refresh=1m
  
  Many requests such as this one 
https://query.wikidata.org/sparql?query=SELECT%20%3Fid%7B%3Fid%20wdt%3AP2600%20%22xxx%22%7D%20LIMIT%201%231337
 times out or returns 429 (tried in PAWS shell).

TASK DETAIL
  https://phabricator.wikimedia.org/T324346

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Aklapper, Bugreporter, AWesterinen, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org