coren has uploaded a new change for review.
https://gerrit.wikimedia.org/r/228180
Change subject: labstore: tweak alerting thresholds
......................................................................
labstore: tweak alerting thresholds
With the new more efficient system, it is normal for iowait
to increase to as much as 40-45% for extended periods of time
when more than one backup is taking place in parallel.
Since that level of activity clearly has no noticable
negative impact on performance overall[1] it is wiser to raise
the threshold to above that number to avoid false positives.
[1] Indeed, the backups run at ionice level Idle and therefore
wait on clients - with the net result that they spend most
of the time in iowait themselves and raise the number.
Change-Id: I4d36f46e845d404bb528f084db9cb4a740b59a53
---
M modules/labstore/manifests/monitoring.pp
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/80/228180/1
diff --git a/modules/labstore/manifests/monitoring.pp
b/modules/labstore/manifests/monitoring.pp
index 98a5e38..f12b16e 100644
--- a/modules/labstore/manifests/monitoring.pp
+++ b/modules/labstore/manifests/monitoring.pp
@@ -33,8 +33,8 @@
description => 'Persistent high iowait',
metric => "servers.${::hostname}.cpu.total.iowait",
from => '10min',
- warning => '25', # Based off looking at history of metric
- critical => '35',
+ warning => '40', # Based off looking at history of metric
+ critical => '60',
percentage => '50', # Ignore small spikes
}
--
To view, visit https://gerrit.wikimedia.org/r/228180
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4d36f46e845d404bb528f084db9cb4a740b59a53
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: coren <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
