[Betacluster-alerts] ** PROBLEM alert - deployment-secureredirexperiment/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-secureredirexperiment Address: 10.68.17.132 State: CRITICAL Date/Time: Fri 09 Mar 18:54:27 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Fri 09 Mar 19:35:41 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Fri 09 Mar 20:06:48 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: WARNING Date/Time: Fri 09 Mar 22:02:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Fri 09 Mar 19:59:48 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mx02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-mx02 Address: 10.68.23.220 State: CRITICAL Date/Time: Fri 09 Mar 21:08:21 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-maps02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-maps02 Address: 10.68.21.90 State: CRITICAL Date/Time: Fri 09 Mar 21:54:35 UTC 2018 Notes URLs: Additional Info: CRITICAL: 80.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: WARNING Date/Time: Fri 09 Mar 19:56:46 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Fri 09 Mar 19:33:30 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Fri 09 Mar 20:09:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Fri 09 Mar 19:13:29 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Fri 09 Mar 19:07:28 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Fri 09 Mar 18:52:29 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Fri 09 Mar 22:12:48 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Fri 09 Mar 18:03:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Fri 09 Mar 18:04:39 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Fri 09 Mar 18:08:46 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Fri 09 Mar 18:10:46 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Fri 09 Mar 18:14:38 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki07/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki07 Address: 10.68.17.40 State: CRITICAL Date/Time: Fri 09 Mar 18:34:35 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Sat 10 Mar 04:42:31 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: CRITICAL Date/Time: Fri 09 Mar 23:20:38 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Sat 10 Mar 04:52:28 UTC 2018 Notes URLs: Additional Info: CRITICAL: 60.00% of data above the critical threshold [10.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Fri 09 Mar 23:18:46 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: CRITICAL Date/Time: Sat 10 Mar 03:41:37 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%)

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Sat 10 Mar 05:22:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: 60.00% of data above the critical threshold [10.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: WARNING Date/Time: Sat 10 Mar 00:41:39 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Fri 09 Mar 23:13:48 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Sat 10 Mar 05:52:31 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Sat 10 Mar 05:17:29 UTC 2018 Notes URLs: Additional Info: WARNING: 60.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-fluorine02/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: OK Date/Time: Sat 10 Mar 07:06:39 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Sat 10 Mar 06:50:41 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Fri 09 Mar 11:02:48 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Fri 09 Mar 11:07:48 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Fri 09 Mar 11:08:38 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-ores01 Address: 10.68.23.165 State: CRITICAL Date/Time: Fri 09 Mar 12:30:52 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] Host DOWN alert for deployment-tmh01!

Notification Type: PROBLEM Host: deployment-tmh01 State: DOWN Address: 10.68.16.211 Info: CRITICAL - Host Unreachable (10.68.16.211) Date/Time: Fri 09 Mar 10:54:51 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] Host DOWN alert for deployment-videoscaler01!

Notification Type: PROBLEM Host: deployment-videoscaler01 State: DOWN Address: 10.68.19.130 Info: CRITICAL - Host Unreachable (10.68.19.130) Date/Time: Fri 09 Mar 10:54:06 UTC 2018 ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Fri 09 Mar 11:18:40 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Fri 09 Mar 13:57:31 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Fri 09 Mar 13:47:30 UTC 2018 Notes URLs: Additional Info: WARNING: 60.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Fri 09 Mar 14:19:30 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Fri 09 Mar 14:03:28 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Fri 09 Mar 14:13:28 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] Host DOWN alert for deployment-puppetdb01!

Notification Type: PROBLEM Host: deployment-puppetdb01 State: DOWN Address: 10.68.23.76 Info: CRITICAL - Host Unreachable (10.68.23.76) Date/Time: Fri 09 Mar 14:53:14 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: WARNING Date/Time: Fri 09 Mar 18:00:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Fri 09 Mar 16:33:16 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___