Hashar has uploaded a new change for review. https://gerrit.wikimedia.org/r/108039
Change subject: beta: monitor fatal errors on beta labs ...................................................................... beta: monitor fatal errors on beta labs We discussed doing this at the quarterly review: https://www.mediawiki.org/wiki/Wikimedia_Release_and_QA_Team/Quarterly_review,_November_2013 This script is intended to run each hour. In its current form it could be started as a cron job. Alternately, we could have it loop and sleep for seconds_since_last_run while running in a screen() session. I tested this by running it on deployment-bastion and sending messages to myself, and it works nicely. Script has been imported by Antoine from the qa/browsertests repository so it can be puppetized. Originally submitted as: https://gerrit.wikimedia.org/r/#/c/105715/ bug: 60046 Change-Id: I0498bb87dd3cb239ea08d521ddf285faea9cffee --- A files/misc/beta/monitor_fatals.rb 1 file changed, 26 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/39/108039/1 diff --git a/files/misc/beta/monitor_fatals.rb b/files/misc/beta/monitor_fatals.rb new file mode 100644 index 0000000..0ba041e --- /dev/null +++ b/files/misc/beta/monitor_fatals.rb @@ -0,0 +1,26 @@ +require 'net/smtp' + +fatals_file="/data/project/logs/fatals.log" +seconds_since_last_run=3600 + +def send_email + message = <<MESSAGE_END +From: Beta fatals <[email protected]> +To: Software quality <[email protected]> +Subject: New fatal errors on beta labs + + The file at /data/project/logs/fatal.log on the deployment cluster has a new entry within the last hour. You should check it out. +MESSAGE_END + + Net::SMTP.start('mchenry.wikimedia.org') do |smtp| + smtp.send_message message, '[email protected]', + '[email protected]' + end +end + +if File.exist?(fatals_file) + last_modified_time=File.mtime(fatals_file) + if Time.now.to_i - last_modified_time.to_i < seconds_since_last_run + send_email + end +end -- To view, visit https://gerrit.wikimedia.org/r/108039 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0498bb87dd3cb239ea08d521ddf285faea9cffee Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Hashar <[email protected]> Gerrit-Reviewer: Cmcmahon <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
