Giuseppe Lavagetto has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/315938

Change subject: role::mediawiki::webserver: restart hhvm routinely
......................................................................

role::mediawiki::webserver: restart hhvm routinely

This cron will restart HHVM if it's running since more than 3 days.

Bug: T147773
Change-Id: I204331607ba80169fafdd205ba9bffeeabf9a443
---
M modules/role/manifests/mediawiki/webserver.pp
1 file changed, 13 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/38/315938/1

diff --git a/modules/role/manifests/mediawiki/webserver.pp 
b/modules/role/manifests/mediawiki/webserver.pp
index bc270e2..774c9ae 100644
--- a/modules/role/manifests/mediawiki/webserver.pp
+++ b/modules/role/manifests/mediawiki/webserver.pp
@@ -26,6 +26,19 @@
             lvs_class_hosts     => $lvs::configuration::lvs_class_hosts,
             lvs_services_config => $lvs::configuration::lvs_services
         }
+
+        # Restart HHVM if it is running since more than 3 days.
+        # This should prevent a series of cpu usage surges we've been seeing
+        # on long-running HHVM processes. T147773
+        $conftool_config = $lvs::configuration::lvs_services['conftool']
+        $site_nodes = loadyaml("conftool-data/nodes/${::site}.yaml")
+        $pool_nodes = keys($site_nodes[$conftool_config['cluster']])
+        $times = cron_splay($pool_nodes, 'daily', 'hhvm-conditional-restarts')
+        cron { 'hhvm-conditional-restart':
+            command => '(( $(ps -C hhvm -o etimes= | head -n 1 ) > 259200 )) 
&& /usr/local/bin/run-no-puppet /usr/local/bin/restart-hhvm > /dev/null',
+            hour    => $times['hour'],
+            minute  => $times['minute'],
+        }
     }
 
     ferm::service { 'mediawiki-http':

-- 
To view, visit https://gerrit.wikimedia.org/r/315938
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I204331607ba80169fafdd205ba9bffeeabf9a443
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Giuseppe Lavagetto <glavage...@wikimedia.org>

_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

Reply via email to