QChris has uploaded a new change for review.
https://gerrit.wikimedia.org/r/186091
Change subject: Bump alert thresholds for EventLogging's overall events/s
......................................................................
Bump alert thresholds for EventLogging's overall events/s
Since EventLogging volume recently outgrew the 350 events/s and
EventLogging is known to be able to handle more events/s, we bump the
threshold to avoid getting unneeded Icinga warnings for EventLogging.
The 450 events/s threshold is arbitrary, but EventLogging is known to
currently handle 450 events/s amount of traffic, and that threshold
will silence the false alarms for now.
The Analytics team has to come up with more realistic thresholds
(T86244).
Change-Id: I2312d62fff1ad851640c2f2fced646478833b7a4
---
M modules/eventlogging/manifests/monitoring.pp
1 file changed, 6 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/91/186091/1
diff --git a/modules/eventlogging/manifests/monitoring.pp
b/modules/eventlogging/manifests/monitoring.pp
index 97c064b..3695f70 100644
--- a/modules/eventlogging/manifests/monitoring.pp
+++ b/modules/eventlogging/manifests/monitoring.pp
@@ -41,13 +41,16 @@
# of incoming events.
class eventlogging::monitoring::graphite {
- # Alarms if 1% of overall event throughput goes beyond 350 req/sec
+ # Warn if 1% of overall event throughput goes beyond 450 events/s
# in a 15 min period
+ # These thresholds are somewhat arbtirary at this point, but it
+ # was seen that the current setup can handle 450 events/s.
+ # Better thresholds are pending (see T86244).
monitoring::graphite_threshold { 'eventlogging_throughput':
description => 'Throughput of event logging events',
metric => 'eventlogging.overall.raw.rate',
- warning => 350,
- critical => 500,
+ warning => 450,
+ critical => 600,
from => '15min',
contact_group => 'analytics'
}
--
To view, visit https://gerrit.wikimedia.org/r/186091
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2312d62fff1ad851640c2f2fced646478833b7a4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: QChris <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
