[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: CRITICAL Date/Time: Thu 17 Nov 03:02:34 UTC 2016 Additional Info: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-pdfrender02/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-pdfrender02 Address: 10.68.21.240 State: OK Date/Time: Wed 16 Nov 21:30:32 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-apertium02/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-apertium02 Address: 10.68.22.254 State: OK Date/Time: Wed 16 Nov 21:30:05 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-urldownloader/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-urldownloader Address: 10.68.16.135 State: OK Date/Time: Wed 16 Nov 21:29:28 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-jobrunner02/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-jobrunner02 Address: 10.68.19.42 State: OK Date/Time: Wed 16 Nov 21:29:08 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-elastic06/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-elastic06 Address: 10.68.17.186 State: OK Date/Time: Wed 16 Nov 21:28:34 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-logstash2/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-logstash2 Address: 10.68.16.147 State: OK Date/Time: Wed 16 Nov 21:27:28 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-zookeeper01/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-zookeeper01 Address: 10.68.17.157 State: OK Date/Time: Wed 16 Nov 21:27:27 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-sca03/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-sca03 Address: 10.68.21.183 State: OK Date/Time: Wed 16 Nov 21:26:45 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-parsoid09/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-parsoid09 Address: 10.68.20.142 State: OK Date/Time: Wed 16 Nov 21:25:34 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Wed 16 Nov 21:24:28 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-cache-text04/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-cache-text04 Address: 10.68.18.103 State: OK Date/Time: Wed 16 Nov 21:23:47 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-restbase01/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-restbase01 Address: 10.68.16.128 State: OK Date/Time: Wed 16 Nov 21:18:46 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-memc05/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-memc05 Address: 10.68.23.49 State: OK Date/Time: Wed 16 Nov 21:16:41 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-ms-be02/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-ms-be02 Address: 10.68.21.135 State: OK Date/Time: Wed 16 Nov 21:15:40 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-cache-upload04/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-cache-upload04 Address: 10.68.18.109 State: OK Date/Time: Wed 16 Nov 21:14:29 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-elastic07/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-elastic07 Address: 10.68.17.187 State: OK Date/Time: Wed 16 Nov 21:14:14 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-elastic05/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-elastic05 Address: 10.68.17.182 State: OK Date/Time: Wed 16 Nov 21:13:18 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-puppetmaster02/Long lived cherry-picks on puppetmaster is CRITICAL **

Notification Type: PROBLEM Service: Long lived cherry-picks on puppetmaster Host: deployment-puppetmaster02 Address: 10.68.21.200 State: CRITICAL Date/Time: Wed 16 Nov 21:13:25 UTC 2016 Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-db04/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-db04 Address: 10.68.18.35 State: OK Date/Time: Wed 16 Nov 21:13:14 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-sca03/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-sca03 Address: 10.68.21.183 State: CRITICAL Date/Time: Wed 16 Nov 20:51:44 UTC 2016 Additional Info: CRITICAL: 50.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-pdfrender02/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-pdfrender02 Address: 10.68.21.240 State: CRITICAL Date/Time: Wed 16 Nov 20:50:29 UTC 2016 Additional Info: CRITICAL: 55.56% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-zotero01/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-zotero01 Address: 10.68.17.102 State: CRITICAL Date/Time: Wed 16 Nov 20:50:13 UTC 2016 Additional Info: CRITICAL: 44.44% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-apertium02/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-apertium02 Address: 10.68.22.254 State: CRITICAL Date/Time: Wed 16 Nov 20:50:03 UTC 2016 Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-urldownloader/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-urldownloader Address: 10.68.16.135 State: CRITICAL Date/Time: Wed 16 Nov 20:49:29 UTC 2016 Additional Info: CRITICAL: 55.56% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-jobrunner02/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-jobrunner02 Address: 10.68.19.42 State: CRITICAL Date/Time: Wed 16 Nov 20:49:07 UTC 2016 Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-fluorine02 Address: 10.68.23.106 State: CRITICAL Date/Time: Wed 16 Nov 20:48:47 UTC 2016 Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-elastic06/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-elastic06 Address: 10.68.17.186 State: CRITICAL Date/Time: Wed 16 Nov 20:48:34 UTC 2016 Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-redis02/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-redis02 Address: 10.68.16.231 State: CRITICAL Date/Time: Wed 16 Nov 20:47:40 UTC 2016 Additional Info: CRITICAL: 60.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-zookeeper01/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-zookeeper01 Address: 10.68.17.157 State: CRITICAL Date/Time: Wed 16 Nov 20:47:28 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-parsoid09/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-parsoid09 Address: 10.68.20.142 State: CRITICAL Date/Time: Wed 16 Nov 20:45:36 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-conf03/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-conf03 Address: 10.68.20.134 State: CRITICAL Date/Time: Wed 16 Nov 20:44:26 UTC 2016 Additional Info: CRITICAL: 20.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-text04/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-cache-text04 Address: 10.68.18.103 State: CRITICAL Date/Time: Wed 16 Nov 20:43:46 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-ms-fe01/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-ms-fe01 Address: 10.68.16.96 State: CRITICAL Date/Time: Wed 16 Nov 20:35:02 UTC 2016 Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-upload04/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-cache-upload04 Address: 10.68.18.109 State: CRITICAL Date/Time: Wed 16 Nov 20:39:29 UTC 2016 Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-restbase01/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-restbase01 Address: 10.68.16.128 State: CRITICAL Date/Time: Wed 16 Nov 20:38:46 UTC 2016 Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-elastic05/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-elastic05 Address: 10.68.17.182 State: CRITICAL Date/Time: Wed 16 Nov 20:38:19 UTC 2016 Additional Info: CRITICAL: 44.44% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-memc05/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-memc05 Address: 10.68.23.49 State: CRITICAL Date/Time: Wed 16 Nov 20:36:39 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-elastic07/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-elastic07 Address: 10.68.17.187 State: CRITICAL Date/Time: Wed 16 Nov 20:34:15 UTC 2016 Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-sca02/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-sca02 Address: 10.68.20.153 State: OK Date/Time: Wed 16 Nov 16:09:28 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-sca01/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-sca01 Address: 10.68.20.183 State: OK Date/Time: Wed 16 Nov 16:07:14 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-sca03/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-sca03 Address: 10.68.21.183 State: CRITICAL Date/Time: Wed 16 Nov 15:35:44 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-sca02/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-sca02 Address: 10.68.20.153 State: CRITICAL Date/Time: Wed 16 Nov 15:34:29 UTC 2016 Additional Info: CRITICAL: 20.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-sca01/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-sca01 Address: 10.68.20.183 State: CRITICAL Date/Time: Wed 16 Nov 15:32:14 UTC 2016 Additional Info: CRITICAL: 55.56% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Wed 16 Nov 13:03:19 UTC 2016 Additional Info: HTTP OK: HTTP/1.1 200 OK - 44262 bytes in 3.517 second response time

[Betacluster-alerts] beta-scap-eqiad - Build # 129114 - Still Failing!

beta-scap-eqiad - Build # 129114 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/129114/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 129113 - Still Failing!

beta-scap-eqiad - Build # 129113 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/129113/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 129112 - Failure!

beta-scap-eqiad - Build # 129112 - Failure: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/129112/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 129111 - Aborted!

beta-scap-eqiad - Build # 129111 - Aborted: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/129111/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-code-update-eqiad - Build # 130375 - Aborted!

beta-code-update-eqiad - Build # 130375 - Aborted: Check console output at https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/130375/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Wed 16 Nov 10:35:20 UTC 2016 Additional Info: HTTP OK: HTTP/1.1 200 OK - 44274 bytes in 3.499 second response time

[Betacluster-alerts] ** RECOVERY alert - Generic Beta Cluster/English Wikipedia Main page is OK **

Notification Type: RECOVERY Service: English Wikipedia Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: OK Date/Time: Wed 16 Nov 10:33:20 UTC 2016 Additional Info: HTTP OK: HTTP/1.1 200 OK - 44706 bytes in 2.460 second response time

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Wed 16 Nov 10:32:19 UTC 2016 Additional Info: HTTP OK: HTTP/1.1 200 OK - 44264 bytes in 5.213 second response time

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Wed 16 Nov 09:45:25 UTC 2016 Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Puppet run is OK **

Notification Type: RECOVERY Service: Puppet run Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Wed 16 Nov 09:43:27 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Wed 16 Nov 09:39:18 UTC 2016 Additional Info: HTTP OK: HTTP/1.1 200 OK - 44265 bytes in 2.913 second response time

[Betacluster-alerts] Host UP alert for deployment-mediawiki04!

Notification Type: RECOVERY Host: deployment-mediawiki04 State: UP Address: 10.68.19.128 Info: PING OK - Packet loss = 0%, RTA = 0.83 ms Date/Time: Wed 16 Nov 09:34:24 UTC 2016 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] Host DOWN alert for deployment-mediawiki04!

Notification Type: PROBLEM Host: deployment-mediawiki04 State: DOWN Address: 10.68.19.128 Info: CRITICAL - Host Unreachable (10.68.19.128) Date/Time: Wed 16 Nov 09:31:23 UTC 2016 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/SSH is CRITICAL **

Notification Type: PROBLEM Service: SSH Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Wed 16 Nov 09:30:54 UTC 2016 Additional Info: Connection refused ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Puppet run is CRITICAL **

Notification Type: PROBLEM Service: Puppet run Host: deployment-mediawiki05 Address: 10.68.22.21 State: CRITICAL Date/Time: Wed 16 Nov 09:18:26 UTC 2016 Additional Info: CRITICAL: 40.00% of data above the critical threshold [0.0] ___