[Betacluster-alerts] ** RECOVERY alert - deployment-fluorine02/Free space - all mounts is OK **
Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: OK Date/Time: Thu 08 Mar 07:09:38 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Thu 08 Mar 05:18:30 UTC 2018 Notes URLs: Additional Info: WARNING: 60.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Thu 08 Mar 05:13:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: 20.00% of data above the critical threshold [10.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Thu 08 Mar 05:08:28 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Thu 08 Mar 04:53:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: 40.00% of data above the critical threshold [10.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Thu 08 Mar 04:43:28 UTC 2018 Notes URLs: Additional Info: WARNING: 60.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Thu 08 Mar 04:28:10 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mathoid/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-mathoid Address: 10.68.23.236 State: OK Date/Time: Thu 08 Mar 04:27:14 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-jobrunner02/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-jobrunner02 Address: 10.68.19.42 State: OK Date/Time: Thu 08 Mar 04:27:38 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-aqs02/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-aqs02 Address: 10.68.17.90 State: OK Date/Time: Thu 08 Mar 04:24:49 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-cassandra3-02/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-cassandra3-02 Address: 10.68.21.237 State: OK Date/Time: Thu 08 Mar 04:25:31 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mira/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-mira Address: 10.68.20.135 State: OK Date/Time: Thu 08 Mar 04:23:02 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-imagescaler01/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-imagescaler01 Address: 10.68.19.158 State: OK Date/Time: Thu 08 Mar 04:22:39 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-sca01/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-sca01 Address: 10.68.20.183 State: OK Date/Time: Thu 08 Mar 04:18:50 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-changeprop/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-changeprop Address: 10.68.16.88 State: OK Date/Time: Thu 08 Mar 04:18:44 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-aqs01/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-aqs01 Address: 10.68.18.237 State: OK Date/Time: Thu 08 Mar 04:17:54 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-cpjobqueue/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-cpjobqueue Address: 10.68.22.161 State: OK Date/Time: Thu 08 Mar 04:14:51 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-parsoid09/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-parsoid09 Address: 10.68.20.142 State: OK Date/Time: Thu 08 Mar 04:14:51 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-sca02/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-sca02 Address: 10.68.20.153 State: OK Date/Time: Thu 08 Mar 04:14:48 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-cassandra3-01/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-cassandra3-01 Address: 10.68.17.103 State: OK Date/Time: Thu 08 Mar 04:14:54 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-aqs03/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-aqs03 Address: 10.68.17.125 State: OK Date/Time: Thu 08 Mar 04:14:31 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-eventlogging04/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-eventlogging04 Address: 10.68.23.204 State: OK Date/Time: Thu 08 Mar 04:14:24 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-pdfrender02/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-pdfrender02 Address: 10.68.21.240 State: OK Date/Time: Thu 08 Mar 04:09:13 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mcs01/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-mcs01 Address: 10.68.17.18 State: OK Date/Time: Thu 08 Mar 04:07:51 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-restbase01/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-restbase01 Address: 10.68.16.128 State: OK Date/Time: Thu 08 Mar 04:05:55 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Thu 08 Mar 04:05:34 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-zotero01/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-zotero01 Address: 10.68.17.102 State: OK Date/Time: Thu 08 Mar 04:05:18 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-restbase02/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-restbase02 Address: 10.68.17.189 State: OK Date/Time: Thu 08 Mar 04:03:40 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Puppet errors is OK **
Notification Type: RECOVERY Service: Puppet errors Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Thu 08 Mar 04:02:43 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is CRITICAL **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: CRITICAL Date/Time: Thu 08 Mar 03:09:38 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-sca01/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-sca01 Address: 10.68.20.183 State: CRITICAL Date/Time: Thu 08 Mar 02:13:50 UTC 2018 Notes URLs: Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-aqs01/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-aqs01 Address: 10.68.18.237 State: CRITICAL Date/Time: Thu 08 Mar 02:12:54 UTC 2018 Notes URLs: Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-cpjobqueue/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-cpjobqueue Address: 10.68.22.161 State: CRITICAL Date/Time: Thu 08 Mar 02:09:50 UTC 2018 Notes URLs: Additional Info: CRITICAL: 60.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-changeprop/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-changeprop Address: 10.68.16.88 State: CRITICAL Date/Time: Thu 08 Mar 02:08:44 UTC 2018 Notes URLs: Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-eventlogging04/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-eventlogging04 Address: 10.68.23.204 State: CRITICAL Date/Time: Thu 08 Mar 02:04:24 UTC 2018 Notes URLs: Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-cassandra3-01/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-cassandra3-01 Address: 10.68.17.103 State: CRITICAL Date/Time: Thu 08 Mar 02:04:55 UTC 2018 Notes URLs: Additional Info: CRITICAL: 20.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-aqs03/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-aqs03 Address: 10.68.17.125 State: CRITICAL Date/Time: Thu 08 Mar 02:04:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-sca02/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-sca02 Address: 10.68.20.153 State: CRITICAL Date/Time: Thu 08 Mar 02:04:47 UTC 2018 Notes URLs: Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-parsoid09/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-parsoid09 Address: 10.68.20.142 State: CRITICAL Date/Time: Thu 08 Mar 02:04:52 UTC 2018 Notes URLs: Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-zotero01/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-zotero01 Address: 10.68.17.102 State: CRITICAL Date/Time: Thu 08 Mar 02:00:17 UTC 2018 Notes URLs: Additional Info: CRITICAL: 50.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-restbase01/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-restbase01 Address: 10.68.16.128 State: CRITICAL Date/Time: Thu 08 Mar 02:00:53 UTC 2018 Notes URLs: Additional Info: CRITICAL: 70.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-pdfrender02/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-pdfrender02 Address: 10.68.21.240 State: CRITICAL Date/Time: Thu 08 Mar 01:59:12 UTC 2018 Notes URLs: Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mcs01/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-mcs01 Address: 10.68.17.18 State: CRITICAL Date/Time: Thu 08 Mar 01:57:52 UTC 2018 Notes URLs: Additional Info: CRITICAL: 20.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-mediawiki05 Address: 10.68.22.21 State: CRITICAL Date/Time: Thu 08 Mar 01:57:43 UTC 2018 Notes URLs: Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Thu 08 Mar 01:55:33 UTC 2018 Notes URLs: Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki06/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-mediawiki06 Address: 10.68.19.241 State: CRITICAL Date/Time: Thu 08 Mar 01:53:09 UTC 2018 Notes URLs: Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-restbase02/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-restbase02 Address: 10.68.17.189 State: CRITICAL Date/Time: Thu 08 Mar 01:53:42 UTC 2018 Notes URLs: Additional Info: CRITICAL: 44.44% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-jobrunner02/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-jobrunner02 Address: 10.68.19.42 State: CRITICAL Date/Time: Thu 08 Mar 01:52:41 UTC 2018 Notes URLs: Additional Info: CRITICAL: 60.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-tin Address: 10.68.21.205 State: CRITICAL Date/Time: Thu 08 Mar 01:47:37 UTC 2018 Notes URLs: Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mathoid/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-mathoid Address: 10.68.23.236 State: CRITICAL Date/Time: Thu 08 Mar 01:47:14 UTC 2018 Notes URLs: Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-imagescaler01/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-imagescaler01 Address: 10.68.19.158 State: CRITICAL Date/Time: Thu 08 Mar 01:47:41 UTC 2018 Notes URLs: Additional Info: CRITICAL: 60.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-cassandra3-02/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-cassandra3-02 Address: 10.68.21.237 State: CRITICAL Date/Time: Thu 08 Mar 01:45:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: 40.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-aqs02/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-aqs02 Address: 10.68.17.90 State: CRITICAL Date/Time: Thu 08 Mar 01:44:49 UTC 2018 Notes URLs: Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mira/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-mira Address: 10.68.20.135 State: CRITICAL Date/Time: Thu 08 Mar 01:43:04 UTC 2018 Notes URLs: Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is WARNING **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: WARNING Date/Time: Thu 08 Mar 00:39:38 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-snapshot01/Free space - all mounts is OK **
Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-snapshot01 Address: 10.68.19.94 State: OK Date/Time: Wed 07 Mar 23:59:35 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **
Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Wed 07 Mar 23:58:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is CRITICAL **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: CRITICAL Date/Time: Wed 07 Mar 23:56:38 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<30.00%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-snapshot01/Free space - all mounts is WARNING **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-snapshot01 Address: 10.68.19.94 State: WARNING Date/Time: Wed 07 Mar 23:54:34 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<11.11%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is CRITICAL **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Wed 07 Mar 23:53:47 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **
Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Wed 07 Mar 22:14:49 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Wed 07 Mar 22:04:46 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mx02/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-mx02 Address: 10.68.23.220 State: CRITICAL Date/Time: Wed 07 Mar 21:08:21 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 07 Mar 19:21:29 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **
Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Wed 07 Mar 19:09:29 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 07 Mar 18:54:31 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-secureredirexperiment/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-secureredirexperiment Address: 10.68.17.132 State: CRITICAL Date/Time: Wed 07 Mar 18:54:27 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki07/App Server Main HTTP Response is CRITICAL **
Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki07 Address: 10.68.17.40 State: CRITICAL Date/Time: Wed 07 Mar 18:34:35 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.009 second response time ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Puppet errors is CRITICAL **
Notification Type: PROBLEM Service: Puppet errors Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Wed 07 Mar 16:33:16 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] Host DOWN alert for deployment-puppetdb01!
Notification Type: PROBLEM Host: deployment-puppetdb01 State: DOWN Address: 10.68.23.76 Info: CRITICAL - Host Unreachable (10.68.23.76) Date/Time: Wed 07 Mar 14:53:14 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Wed 07 Mar 14:41:39 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<100.00%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-snapshot01/Free space - all mounts is OK **
Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-snapshot01 Address: 10.68.19.94 State: OK Date/Time: Wed 07 Mar 14:28:33 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **
Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Wed 07 Mar 14:27:49 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is CRITICAL **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: CRITICAL Date/Time: Wed 07 Mar 14:26:38 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-snapshot01/Free space - all mounts is WARNING **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-snapshot01 Address: 10.68.19.94 State: WARNING Date/Time: Wed 07 Mar 14:23:33 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<11.11%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is CRITICAL **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Wed 07 Mar 14:22:47 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 07 Mar 12:48:28 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Wed 07 Mar 12:43:28 UTC 2018 Notes URLs: Additional Info: CRITICAL: 40.00% of data above the critical threshold [10.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 07 Mar 12:33:29 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Wed 07 Mar 12:13:28 UTC 2018 Notes URLs: Additional Info: CRITICAL: 20.00% of data above the critical threshold [10.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **
Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Wed 07 Mar 12:08:30 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] Host DOWN alert for deployment-videoscaler01!
Notification Type: PROBLEM Host: deployment-videoscaler01 State: DOWN Address: 10.68.19.130 Info: CRITICAL - Host Unreachable (10.68.19.130) Date/Time: Wed 07 Mar 10:54:06 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] Host DOWN alert for deployment-tmh01!
Notification Type: PROBLEM Host: deployment-tmh01 State: DOWN Address: 10.68.16.211 Info: CRITICAL - Host Unreachable (10.68.16.211) Date/Time: Wed 07 Mar 10:54:51 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
