[Betacluster-alerts] Host DOWN alert for deployment-phab01!

Notification Type: PROBLEM Host: deployment-phab01 State: DOWN Address: 10.68.18.216 Info: CRITICAL - Host Unreachable (10.68.18.216) Date/Time: Tue 13 Jun 21:36:04 UTC 2017 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka01/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-kafka01 Address: 10.68.21.219 State: CRITICAL Date/Time: Tue 13 Jun 21:23:33 UTC 2017 Additional Info: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<10.00%)

[Betacluster-alerts] ** RECOVERY alert - deployment-ms-be04/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-ms-be04 Address: 10.68.16.139 State: OK Date/Time: Tue 13 Jun 20:32:47 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-ms-be04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-ms-be04 Address: 10.68.16.139 State: CRITICAL Date/Time: Tue 13 Jun 19:57:46 UTC 2017 Additional Info: CRITICAL: 60.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-imagescaler01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-imagescaler01 Address: 10.68.19.158 State: CRITICAL Date/Time: Tue 13 Jun 19:31:55 UTC 2017 Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-db04/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-db04 Address: 10.68.18.35 State: OK Date/Time: Tue 13 Jun 19:21:01 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-etcd-01/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-etcd-01 Address: 10.68.19.227 State: OK Date/Time: Tue 13 Jun 19:00:42 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-cache-upload04/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-cache-upload04 Address: 10.68.18.109 State: OK Date/Time: Tue 13 Jun 18:57:19 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing

[Betacluster-alerts] ** RECOVERY alert - deployment-cache-text04/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-cache-text04 Address: 10.68.18.103 State: OK Date/Time: Tue 13 Jun 18:57:31 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 13 Jun 18:52:57 UTC 2017 Additional Info: WARNING: 20.00% of data above the warning threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-db04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-db04 Address: 10.68.18.35 State: CRITICAL Date/Time: Tue 13 Jun 18:41:00 UTC 2017 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-text04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-cache-text04 Address: 10.68.18.103 State: CRITICAL Date/Time: Tue 13 Jun 18:22:33 UTC 2017 Additional Info: CRITICAL: 55.56% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Puppet staleness is OK **

Notification Type: RECOVERY Service: Puppet staleness Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Tue 13 Jun 18:13:10 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [3600.0] ___ Betacluster-alerts mailing

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-upload04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-cache-upload04 Address: 10.68.18.109 State: CRITICAL Date/Time: Tue 13 Jun 18:12:18 UTC 2017 Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-etcd-01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-etcd-01 Address: 10.68.19.227 State: CRITICAL Date/Time: Tue 13 Jun 18:10:42 UTC 2017 Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Puppet staleness is WARNING **

Notification Type: PROBLEM Service: Puppet staleness Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Tue 13 Jun 17:58:11 UTC 2017 Additional Info: WARNING: 33.33% of data above the warning threshold [3600.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 13 Jun 14:28:57 UTC 2017 Additional Info: WARNING: 20.00% of data above the warning threshold [1.0] ___

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 13 Jun 14:22:57 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [1.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 13 Jun 14:12:57 UTC 2017 Additional Info: WARNING: 20.00% of data above the warning threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-phab01/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-phab01 Address: 10.68.18.216 State: CRITICAL Date/Time: Tue 13 Jun 14:03:27 UTC 2017 Additional Info: CRITICAL: deployment-prep.deployment-phab01.diskspace.root.byte_percentfree (<100.00%)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 13 Jun 13:36:58 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [1.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 13 Jun 13:31:58 UTC 2017 Additional Info: WARNING: 20.00% of data above the warning threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-conf03/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-conf03 Address: 10.68.20.134 State: CRITICAL Date/Time: Tue 13 Jun 13:04:47 UTC 2017 Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 13 Jun 11:50:57 UTC 2017 Additional Info: WARNING: 20.00% of data above the warning threshold [1.0] ___

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 13 Jun 11:44:56 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [1.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-pdf01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-pdf01 Address: 10.68.16.73 State: CRITICAL Date/Time: Tue 13 Jun 10:59:31 UTC 2017 Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 13 Jun 10:48:58 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [1.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 13 Jun 10:43:56 UTC 2017 Additional Info: WARNING: 20.00% of data above the warning threshold [1.0] ___

[Betacluster-alerts] Host DOWN alert for deployment-zookeeper01!

Notification Type: PROBLEM Host: deployment-zookeeper01 State: DOWN Address: 10.68.17.157 Info: CRITICAL - Host Unreachable (10.68.17.157) Date/Time: Tue 13 Jun 10:43:40 UTC 2017 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - deployment-zookeeper02/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-zookeeper02 Address: 10.68.18.75 State: OK Date/Time: Tue 13 Jun 10:42:45 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-puppetmaster02/Long lived cherry-picks on puppetmaster is CRITICAL **

Notification Type: PROBLEM Service: Long lived cherry-picks on puppetmaster Host: deployment-puppetmaster02 Address: 10.68.21.200 State: CRITICAL Date/Time: Tue 13 Jun 10:22:25 UTC 2017 Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 13 Jun 06:12:57 UTC 2017 Additional Info: OK: Less than 1.00% above the threshold [1.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 13 Jun 06:02:58 UTC 2017 Additional Info: WARNING: 20.00% of data above the warning threshold [1.0] ___