[Betacluster-alerts] beta-scap-eqiad - Build # 197382 - Still Failing!

beta-scap-eqiad - Build # 197382 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197382/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197383 - Still Failing!

beta-scap-eqiad - Build # 197383 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197383/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197384 - Still Failing!

beta-scap-eqiad - Build # 197384 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197384/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197385 - Still Failing!

beta-scap-eqiad - Build # 197385 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197385/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197386 - Still Failing!

beta-scap-eqiad - Build # 197386 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197386/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197387 - Still Failing!

beta-scap-eqiad - Build # 197387 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197387/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197388 - Still Failing!

beta-scap-eqiad - Build # 197388 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197388/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Tue 27 Feb 08:53:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Bet

[Betacluster-alerts] beta-scap-eqiad - Build # 197389 - Still Failing!

beta-scap-eqiad - Build # 197389 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197389/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197390 - Still Failing!

beta-scap-eqiad - Build # 197390 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197390/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197391 - Still Failing!

beta-scap-eqiad - Build # 197391 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197391/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197392 - Still Failing!

beta-scap-eqiad - Build # 197392 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197392/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikime

[Betacluster-alerts] beta-scap-eqiad - Build # 197393 - Fixed!

beta-scap-eqiad - Build # 197393 - Fixed: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/197393/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Tue 27 Feb 09:45:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<33.33%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Tue 27 Feb 09:50:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Beta

[Betacluster-alerts] ** PROBLEM alert - deployment-eventlog02/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-eventlog02 Address: 10.68.18.138 State: CRITICAL Date/Time: Tue 27 Feb 10:11:48 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<100.00%) __

[Betacluster-alerts] ** PROBLEM alert - deployment-eventlogging05/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-eventlogging05 Address: 10.68.23.51 State: CRITICAL Date/Time: Tue 27 Feb 10:18:23 UTC 2018 Notes URLs: Additional Info: CRITICAL: 55.56% of data above the critical threshold [0.0] _

[Betacluster-alerts] ** RECOVERY alert - deployment-eventlogging05/Puppet staleness is OK **

Notification Type: RECOVERY Service: Puppet staleness Host: deployment-eventlogging05 Address: 10.68.23.51 State: OK Date/Time: Tue 27 Feb 10:23:51 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [3600.0] ___ Betacluste

[Betacluster-alerts] ** RECOVERY alert - deployment-eventlogging05/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-eventlogging05 Address: 10.68.23.51 State: OK Date/Time: Tue 27 Feb 10:33:26 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-aler

[Betacluster-alerts] Host DOWN alert for deployment-videoscaler01!

Notification Type: PROBLEM Host: deployment-videoscaler01 State: DOWN Address: 10.68.19.130 Info: CRITICAL - Host Unreachable (10.68.19.130) Date/Time: Tue 27 Feb 10:54:06 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.or

[Betacluster-alerts] Host DOWN alert for deployment-tmh01!

Notification Type: PROBLEM Host: deployment-tmh01 State: DOWN Address: 10.68.16.211 Info: CRITICAL - Host Unreachable (10.68.16.211) Date/Time: Tue 27 Feb 10:54:51 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https:

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-ores01 Address: 10.68.23.165 State: CRITICAL Date/Time: Tue 27 Feb 12:30:52 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___ Bet

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Tue 27 Feb 13:01:46 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Tue 27 Feb 13:11:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Beta

[Betacluster-alerts] Host DOWN alert for deployment-puppetdb01!

Notification Type: PROBLEM Host: deployment-puppetdb01 State: DOWN Address: 10.68.23.76 Info: CRITICAL - Host Unreachable (10.68.23.76) Date/Time: Tue 27 Feb 14:53:14 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org htt

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Tue 27 Feb 16:33:16 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - Generic Beta Cluster/English Wikipedia Mobile Main page is CRITICAL **

Notification Type: PROBLEM Service: English Wikipedia Mobile Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: CRITICAL Date/Time: Tue 27 Feb 17:18:52 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Tue 27 Feb 17:21:30 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** PROBLEM alert - Generic Beta Cluster/English Wikipedia Main page is CRITICAL **

Notification Type: PROBLEM Service: English Wikipedia Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: CRITICAL Date/Time: Tue 27 Feb 17:21:16 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** RECOVERY alert - Generic Beta Cluster/English Wikipedia Mobile Main page is OK **

Notification Type: RECOVERY Service: English Wikipedia Mobile Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: OK Date/Time: Tue 27 Feb 17:23:49 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 36196 bytes in 3.966 second response time ___

[Betacluster-alerts] ** RECOVERY alert - Generic Beta Cluster/English Wikipedia Main page is OK **

Notification Type: RECOVERY Service: English Wikipedia Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: OK Date/Time: Tue 27 Feb 17:26:13 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 47743 bytes in 3.642 second response time __

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Tue 27 Feb 17:26:25 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 47134 bytes in 3.827 second response time ___

[Betacluster-alerts] ** PROBLEM alert - Generic Beta Cluster/English Wikipedia Mobile Main page is CRITICAL **

Notification Type: PROBLEM Service: English Wikipedia Mobile Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: CRITICAL Date/Time: Tue 27 Feb 17:39:54 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds

[Betacluster-alerts] ** PROBLEM alert - Generic Beta Cluster/English Wikipedia Main page is CRITICAL **

Notification Type: PROBLEM Service: English Wikipedia Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: CRITICAL Date/Time: Tue 27 Feb 17:42:18 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Tue 27 Feb 17:42:31 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** RECOVERY alert - Generic Beta Cluster/English Wikipedia Mobile Main page is OK **

Notification Type: RECOVERY Service: English Wikipedia Mobile Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: OK Date/Time: Tue 27 Feb 17:54:46 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 36213 bytes in 3.408 second response time ___

[Betacluster-alerts] ** PROBLEM alert - deployment-prometheus01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-prometheus01 Address: 10.68.20.247 State: CRITICAL Date/Time: Tue 27 Feb 18:00:55 UTC 2018 Notes URLs: Additional Info: CRITICAL: 40.00% of data above the critical threshold [0.0] __

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-upload04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-cache-upload04 Address: 10.68.18.109 State: CRITICAL Date/Time: Tue 27 Feb 18:32:41 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki07/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki07 Address: 10.68.17.40 State: CRITICAL Date/Time: Tue 27 Feb 18:34:35 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipe

[Betacluster-alerts] ** PROBLEM alert - deployment-secureredirexperiment/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-secureredirexperiment Address: 10.68.17.132 State: CRITICAL Date/Time: Tue 27 Feb 18:54:27 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-prometheus01/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-prometheus01 Address: 10.68.20.247 State: OK Date/Time: Tue 27 Feb 19:35:55 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alert

[Betacluster-alerts] ** PROBLEM alert - deployment-mx02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-mx02 Address: 10.68.23.220 State: CRITICAL Date/Time: Tue 27 Feb 21:08:21 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___ Betac

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Tue 27 Feb 21:32:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Tue 27 Feb 21:37:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Beta

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Tue 27 Feb 22:33:48 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Tue 27 Feb 22:38:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Beta

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: WARNING Date/Time: Tue 27 Feb 23:44:38 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) _

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Wed 28 Feb 01:42:28 UTC 2018 Notes URLs: Additional Info: CRITICAL: 20.00% of data above the critical threshold [10.0] _

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Wed 28 Feb 01:47:28 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___ Betac

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Wed 28 Feb 01:57:46 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) _

[Betacluster-alerts] ** PROBLEM alert - deployment-snapshot01/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-snapshot01 Address: 10.68.19.94 State: WARNING Date/Time: Wed 28 Feb 01:59:32 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoint

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Wed 28 Feb 02:04:40 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deplo

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: WARNING Date/Time: Wed 28 Feb 02:07:46 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<100.00%) __

[Betacluster-alerts] ** RECOVERY alert - deployment-snapshot01/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-snapshot01 Address: 10.68.19.94 State: OK Date/Time: Wed 28 Feb 02:09:32 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: CRITICAL Date/Time: Wed 28 Feb 02:19:38 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) ___

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Wed 28 Feb 02:19:41 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) __

[Betacluster-alerts] ** RECOVERY alert - deployment-fluorine02/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: OK Date/Time: Wed 28 Feb 02:49:39 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Beta