Giuseppe Lavagetto has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/401714 )
Change subject: mediawiki::appserver::api: add load monitoring
......................................................................
mediawiki::appserver::api: add load monitoring
We've had quite a few cases of HHVM API appservers with high cpu usage
causing noitceable latencies to users; the easiest way to detect such
deadlocks is quite simply checking the machine CPU usage/load - at least
that's what I do manually.
This change won't solve the issue per-se, but it will make ops aware of
what is going on proactively.
Bug: T182568, T184048
Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
---
M modules/role/manifests/mediawiki/appserver/api.pp
1 file changed, 11 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/14/401714/1
diff --git a/modules/role/manifests/mediawiki/appserver/api.pp
b/modules/role/manifests/mediawiki/appserver/api.pp
index 83494bb..39d2ffb 100644
--- a/modules/role/manifests/mediawiki/appserver/api.pp
+++ b/modules/role/manifests/mediawiki/appserver/api.pp
@@ -13,4 +13,15 @@
},
priority => 90,
}
+
+ # Check the load to detect clearly hosts hanging (see T184048, T182568)
+ $nproc = $facts['processorcount']
+ $warning = join([ $nproc * 0.95, $nproc * 0.8, $nproc * 0.75], ',')
+ $critical = join([ $nproc * 1.5, $nproc * 1.1, $nproc * 1], ',')
+ # Since we're checking the load, that is already a moving average, we can
+ # alarm at the first occurrence
+ nrpe::monitor_service { 'cpu_load':
+ command => "check_load -w ${warning} -c ${critical}",
+ retries => 1,
+ }
}
--
To view, visit https://gerrit.wikimedia.org/r/401714
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
