ArielGlenn added a comment.
at around 6:50 UTC this morning we began seeing this:
icinga-wm: PROBLEM - MediaWiki eqiad exceptions and fatals per minute on
icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR
site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers
https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops
Probably the interaction between a script and a bot. The candidates for fast
editing bots at the time were KrBot, LargeDatasetBot, ԱշբոտՏՆՂ At the time I
looked, I saw about 35 edits a minute from KrBot, and about that many
exceptions a minute. The bot has slowed down to about 10 edits a minute some
time later and the number of fatals decreased as well:
https://logstash.wikimedia.org/goto/8fa30aac1e6166f08bdfb0f906b2c14f
TASK DETAIL
https://phabricator.wikimedia.org/T225056
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup, ArielGlenn
Cc: jijiki, ArielGlenn, Marostegui, Ladsgroup, Aklapper, alaa_wmde,
darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden,
LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher,
Mbch331
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
