Michael added a comment.
The patch above (Ib7bfacf88 <https://gerrit.wikimedia.org/r/#/q/Ib7bfacf88eef436ed0f34a16e9d39eb0c053a44f>/02109a44 <https://gerrit.wikimedia.org/g/operations/puppet/+/02109a444a6ece2f480aee183dc02074d0f3734d>) seemingly resolved the original issue, but some new errors started appearing (though much less common than before). Those new errors seem to come from a timeout in Termbox/Kubernetes which result in a timeout in the php/mediawiki code. The following two links illustrate that error for the rendering process of a single item: 庞家村 (Q14836111) <https://www.wikidata.org/wiki/Q14836111> Termbox/k8s: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-syslog-2020.06.15/syslog?id=AXK3_osLZmYAikdJbyT-&_g=h@e3739c2 Mediawiki: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-mediawiki-2020.06.15/mediawiki?id=AXK3_omyK6TSR36GrxZd&_g=h@e1c60c6 This was judged to be not pressing (@Tarrow .@Addshore please correct me if I'm talking rubbish here). Currently, the internal timeouts seem to be 3 seconds both for the Mediawiki side (ssrServerTimeout <https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/57e673d69ee8357a65a4746ed5dd289fc9f6b61b/docs/topics/options.md#ssrservertimeout>?) and for the Termbox side. It is not clear what we can do about those errors. While the internal requests **should** not take longer than 3 seconds, they sometimes do. **Follow-ups:** - It would be good if a timeout noticed and reported by Termbox would be treated differently by the Mediawiki error logging infrastructure than a timeout of Termbox/k8s itself: T255436: Termbox Error Logging Should Differentiate between RemoteRenderer and Service timeouts. <https://phabricator.wikimedia.org/T255436>. - there is an metric counting such request errors, but currently it is not tracked: T255426: Create Grafana tracking `wikibase.view.TermboxRemoteRenderer.requestError` and maybe add alert <https://phabricator.wikimedia.org/T255426> - we currently do not have dashboard tracking the errors in Termbox SSR-service/Kibana: T255437: Create Logstash Dashboard Tracking Termbox Errors <https://phabricator.wikimedia.org/T255437> TASK DETAIL https://phabricator.wikimedia.org/T255410 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Michael Cc: JMeybohm, WMDE-leszek, Pablo-WMDE, Tarrow, Jakob_WMDE, Addshore, Aklapper, Michael, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
