[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Thu 10 May 05:28:43 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Thu 10 May 04:51:44 UTC 2018 Notes URLs: Additional Info: CRITICAL: 60.00% of data above the critical threshold [10.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Thu 10 May 04:46:44 UTC 2018 Notes URLs: Additional Info: WARNING: 100.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-ores01/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-ores01 Address: 10.68.16.235 State: OK Date/Time: Thu 10 May 02:08:26 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: WARNING Date/Time: Thu 10 May 01:08:44 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** PROBLEM alert - deployment-puppetmaster02/Long lived cherry-picks on puppetmaster is CRITICAL **

Notification Type: PROBLEM Service: Long lived cherry-picks on puppetmaster Host: deployment-puppetmaster02 Address: 10.68.21.200 State: CRITICAL Date/Time: Thu 10 May 00:31:15 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 09 May 23:25:46 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-ores01 Address: 10.68.16.235 State: CRITICAL Date/Time: Wed 09 May 23:23:25 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] Host UP alert for deployment-ores01!

Notification Type: RECOVERY Host: deployment-ores01 State: UP Address: 10.68.16.235 Info: PING OK - Packet loss = 0%, RTA = 0.65 ms Date/Time: Wed 09 May 23:23:22 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] Host DOWN alert for deployment-ores01!

Notification Type: PROBLEM Host: deployment-ores01 State: DOWN Address: 10.68.20.26 Info: CRITICAL - Host Unreachable (10.68.20.26) Date/Time: Wed 09 May 23:04:20 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 09 May 23:04:46 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-ores01 Address: 10.68.20.26 State: CRITICAL Date/Time: Wed 09 May 22:56:45 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Wed 09 May 22:53:46 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 09 May 22:38:44 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-ores01/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-ores01 Address: 10.68.20.26 State: OK Date/Time: Wed 09 May 21:45:45 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-ores01 Address: 10.68.20.26 State: CRITICAL Date/Time: Wed 09 May 21:35:45 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-ores01/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-ores01 Address: 10.68.20.26 State: OK Date/Time: Wed 09 May 21:19:48 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-ores01 Address: 10.68.20.26 State: CRITICAL Date/Time: Wed 09 May 21:09:46 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-kafka-main-2/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-kafka-main-2 Address: 10.68.23.182 State: OK Date/Time: Wed 09 May 19:57:02 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka-main-2/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-kafka-main-2 Address: 10.68.23.182 State: CRITICAL Date/Time: Wed 09 May 19:47:03 UTC 2018 Notes URLs: Additional Info: CRITICAL: 11.11% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Wed 09 May 19:43:44 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: WARNING Date/Time: Wed 09 May 19:28:44 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-kafka-main-1/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-kafka-main-1 Address: 10.68.18.219 State: OK Date/Time: Wed 09 May 19:15:30 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-kafka-main-2/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-kafka-main-2 Address: 10.68.23.182 State: OK Date/Time: Wed 09 May 19:16:04 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Wed 09 May 18:57:45 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 09 May 18:52:43 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Wed 09 May 18:48:45 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid

[Betacluster-alerts] ** PROBLEM alert - deployment-mx/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mx Address: 10.68.17.78 State: CRITICAL Date/Time: Wed 09 May 18:20:37 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<40.00%)

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka-main-1/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-kafka-main-1 Address: 10.68.18.219 State: CRITICAL Date/Time: Wed 09 May 18:05:33 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka-main-2/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-kafka-main-2 Address: 10.68.23.182 State: CRITICAL Date/Time: Wed 09 May 18:01:05 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: WARNING Date/Time: Wed 09 May 17:58:44 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka-jumbo-1/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-kafka-jumbo-1 Address: 10.68.23.243 State: CRITICAL Date/Time: Wed 09 May 17:43:33 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka-jumbo-2/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-kafka-jumbo-2 Address: 10.68.16.87 State: CRITICAL Date/Time: Wed 09 May 17:42:12 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Wed 09 May 17:23:43 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: WARNING Date/Time: Wed 09 May 16:28:46 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] beta-scap-eqiad - Build # 207061 - Fixed!

beta-scap-eqiad - Build # 207061 - Fixed: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207061/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Wed 09 May 15:58:27 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Wed 09 May 15:46:44 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 09 May 15:41:46 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] beta-scap-eqiad - Build # 207060 - Still Failing!

beta-scap-eqiad - Build # 207060 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207060/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 207059 - Failure!

beta-scap-eqiad - Build # 207059 - Failure: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207059/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-redis02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-redis02 Address: 10.68.16.231 State: CRITICAL Date/Time: Wed 09 May 15:33:41 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-redis01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-redis01 Address: 10.68.16.177 State: CRITICAL Date/Time: Wed 09 May 15:27:43 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Wed 09 May 15:12:05 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki06/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki06 Address: 10.68.19.241 State: WARNING Date/Time: Wed 09 May 15:02:04 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki06.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Wed 09 May 14:33:44 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Wed 09 May 14:23:27 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** PROBLEM alert - deployment-eventlog05/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-eventlog05 Address: 10.68.18.180 State: CRITICAL Date/Time: Wed 09 May 13:36:32 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-ms-be03/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-ms-be03 Address: 10.68.22.125 State: CRITICAL Date/Time: Wed 09 May 12:54:26 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-ms-be04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-ms-be04 Address: 10.68.16.139 State: CRITICAL Date/Time: Wed 09 May 12:38:39 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Wed 09 May 10:32:26 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** RECOVERY alert - deployment-memc06/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-memc06 Address: 10.68.22.239 State: OK Date/Time: Wed 09 May 10:17:28 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-memc06/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-memc06 Address: 10.68.22.239 State: CRITICAL Date/Time: Wed 09 May 09:42:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: 55.56% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Wed 09 May 08:02:29 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Wed 09 May 07:56:27 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-mx/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mx Address: 10.68.17.78 State: WARNING Date/Time: Wed 09 May 06:35:36 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<100.00%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki-07/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki-07 Address: 10.68.18.62 State: OK Date/Time: Wed 09 May 06:27:38 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki-07/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki-07 Address: 10.68.18.62 State: WARNING Date/Time: Wed 09 May 06:17:38 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki-07.diskspace.root.byte_percentfree (<11.11%)