Giuseppe Lavagetto has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/401714 )
Change subject: mediawiki::appserver::api: add load monitoring
......................................................................
mediawiki::appserver::api: add load monitoring
We've had quite a few cases of HHVM API appservers with high cpu usage
causing noitceable latencies to users; the easiest way to detect such
deadlocks is quite simply checking the machine CPU usage/load - at least
that's what I do manually.
This change won't solve the issue per-se, but it will make ops aware of
what is going on proactively.
Bug: T182568
Bug: T184048
Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
---
A modules/profile/manifests/mediawiki/api.pp
M modules/role/manifests/mediawiki/appserver/api.pp
2 files changed, 23 insertions(+), 8 deletions(-)
Approvals:
Giuseppe Lavagetto: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/profile/manifests/mediawiki/api.pp
b/modules/profile/manifests/mediawiki/api.pp
new file mode 100644
index 0000000..0f0e9d6
--- /dev/null
+++ b/modules/profile/manifests/mediawiki/api.pp
@@ -0,0 +1,22 @@
+# == Class profile::mediawiki::api
+#
+# Specific settings for the mediawiki API servers
+class profile::mediawiki::api {
+ # Using fastcgi we need more local ports
+ sysctl::parameters { 'raise_port_range':
+ values => { 'net.ipv4.local_port_range' => '22500 65535', },
+ priority => 90,
+ }
+
+ # Check the load to detect clearly hosts hanging (see T184048, T182568)
+ $nproc = $facts['processorcount']
+ $warning = join([ $nproc * 0.95, $nproc * 0.8, $nproc * 0.75], ',')
+ $critical = join([ $nproc * 1.5, $nproc * 1.1, $nproc * 1], ',')
+ # Since we're checking the load, that is already a moving average, we can
+ # alarm at the first occurrence
+ nrpe::monitor_service { 'cpu_load':
+ description => 'High CPU load on API appserver',
+ nrpe_command => "/usr/lib/nagios/plugins/check_load -w ${warning} -c
${critical}",
+ retries => 1,
+ }
+}
diff --git a/modules/role/manifests/mediawiki/appserver/api.pp
b/modules/role/manifests/mediawiki/appserver/api.pp
index 83494bb..c36d4b7 100644
--- a/modules/role/manifests/mediawiki/appserver/api.pp
+++ b/modules/role/manifests/mediawiki/appserver/api.pp
@@ -5,12 +5,5 @@
include ::profile::base::firewall
include ::profile::prometheus::apache_exporter
include ::profile::prometheus::hhvm_exporter
-
- # Using fastcgi we need more local ports
- sysctl::parameters { 'raise_port_range':
- values => {
- 'net.ipv4.local_port_range' => '22500 65535',
- },
- priority => 90,
- }
+ include ::profile::mediawiki::api
}
--
To view, visit https://gerrit.wikimedia.org/r/401714
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I06af45cbf8f42ade5753dc7397c6e1aa2b32c4ea
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto <[email protected]>
Gerrit-Reviewer: Elukey <[email protected]>
Gerrit-Reviewer: Giuseppe Lavagetto <[email protected]>
Gerrit-Reviewer: Muehlenhoff <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
