[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

amy_rc closed subtask T255426: Create Grafana tracking `wikibase.view.TermboxRemoteRenderer.requestError` and maybe add alert as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Addshore closed subtask T255450: Understand Termbox SSR timouts better as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael, Addshore Cc: toan, Lucas_Werkmeister_WMDE, Sakretsu,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

noarave closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael, noarave Cc: toan, Lucas_Werkmeister_WMDE, Sakretsu, akosiaris, JMeybohm, WMDE-leszek, Pablo-WMDE,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Pablo-WMDE added a comment. Thanks for updating the documentation on wikitech! TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael, Pablo-WMDE Cc: toan, Lucas_Werkmeister_WMDE, Sakretsu,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Pablo-WMDE added a comment. > We are starting to work on some preliminary/draft SLOs for mediawiki. There is some work to be done on getting numbers, but when we come up with those, it would be prudent to adjust the SLO of termbox to those as termbox is dependent on mediawiki and for it to

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

akosiaris added a comment. In T255410#6550492 , @Michael wrote: > > That seems very strange. I would have expected the //error rate// to be calculated by `(number of errors / number of total requests)` for the given

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

toan added a comment. Documentation looks great in my opinion. TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael, toan Cc: toan, Lucas_Werkmeister_WMDE, Sakretsu, akosiaris, JMeybohm,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Michael added a comment. In T255410#6549439 , @akosiaris wrote: > In T255410#6543118 , @Michael wrote: > >> @akosiaris Thank you a lot for your detailed response. I did look into

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

akosiaris added a comment. In T255410#6543118 , @Michael wrote: > @akosiaris Thank you a lot for your detailed response. I did look into those errors a tiny bit more to properly document them as can be now seen on wikitech

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Michael added a comment. @akosiaris Thank you a lot for your detailed response. I did look into those errors a tiny bit more to properly document them as can be now seen on wikitech

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

toan added a comment. First off, thank you @akosiaris for your time investigating this. At the start of this task we were presented with the title "Termbox SSR connection terminated very often" and the logstash screenshot of the events that took place starting at 2020-06-10. Since then

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

toan reassigned this task from toan to Michael. Restricted Application added a project: User-Michael. TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael, toan Cc: toan,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

akosiaris added a project: serviceops-radar. akosiaris added a comment. Sorry for not answering earlier. In T255410#6494077 , @Pablo-WMDE wrote: > > I unfortunately don't know how to do this for single documents. The

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

toan added a comment. @akosiaris I did some tinkering in the kibana ui and came up with this (hopefully) shareable link

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Pablo-WMDE added a comment. Hi, > be sure we are looking into the same thing before we delve deeper into this? That makes absolute sense. > Could you please update with a logstash shared url I unfortunately don't know how to do this for single documents. The links show the

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

akosiaris added a comment. In T255410#6491711 , @Pablo-WMDE wrote: > We are seeing our termbox service, when reaching out to the mediawiki API, occasionally receiving a 503 (e.g.

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Lucas_Werkmeister_WMDE moved this task from Peer Review to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board. Lucas_Werkmeister_WMDE added a comment. It seems like we’ve gone as far as we can in investigating this; now waiting for WMF input in the subtasks. TASK

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Pablo-WMDE added a comment. We (@Michael + @toan + @Tarrow + myself) looked at the last 12 hours of those events. We could not immediately see a connection between the items of interest, but what stood out was that there seem to be bursts of those (`"Request failed with status 0. Usually

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

toan claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: toan Cc: Sakretsu, akosiaris, JMeybohm, WMDE-leszek, Pablo-WMDE, Tarrow, Jakob_WMDE, Addshore, Aklapper, Michael,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Michael added a comment. Note that while we are still seeing this issue, the events are still much lower than what we used to see : F32361068: image.png Further, looking

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

WMDE-leszek moved this task from Prioritized Tech to Wikidata-Campsite-Iteration-∞ on the Wikidata-Campsite board. WMDE-leszek edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞); removed Wikidata-Campsite. TASK DETAIL https://phabricator.wikimedia.org/T255410 WORKBOARD

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Sakretsu, akosiaris, JMeybohm, WMDE-leszek, Pablo-WMDE, Tarrow, Jakob_WMDE, Addshore,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

WMDE-leszek updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Sakretsu, akosiaris, JMeybohm, WMDE-leszek, Pablo-WMDE, Tarrow, Jakob_WMDE, Addshore,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Addshore edited projects, added Wikidata-Campsite; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞). TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Sakretsu, akosiaris,

[Wikidata-bugs] [Maniphest] T255410: Termbox SSR connection terminated very often

Addshore added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞). TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Sakretsu, akosiaris, JMeybohm, WMDE-leszek, Pablo-WMDE,