[Betacluster-alerts] ** PROBLEM alert - deployment-db2/Puppet staleness is CRITICAL **
Notification Type: PROBLEM Service: Puppet staleness Host: deployment-db2 Address: 10.68.17.94 State: CRITICAL Date/Time: Sun 25 Sep 04:27:01 UTC 2016 Additional Info: CRITICAL: 100.00% of data above the critical threshold [43200.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-ms-fe01/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-ms-fe01 Address: 10.68.16.96 State: OK Date/Time: Sun 25 Sep 04:12:43 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-cache-text04/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-cache-text04 Address: 10.68.18.103 State: OK Date/Time: Sun 25 Sep 03:53:28 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-elastic08/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-elastic08 Address: 10.68.17.188 State: OK Date/Time: Sun 25 Sep 03:22:39 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-cache-text04/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-cache-text04 Address: 10.68.18.103 State: CRITICAL Date/Time: Sun 25 Sep 03:13:28 UTC 2016 Additional Info: CRITICAL: 40.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-elastic08/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-elastic08 Address: 10.68.17.188 State: CRITICAL Date/Time: Sun 25 Sep 02:42:35 UTC 2016 Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-kafka05/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-kafka05 Address: 10.68.21.106 State: OK Date/Time: Sun 25 Sep 01:01:35 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-kafka05/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-kafka05 Address: 10.68.21.106 State: CRITICAL Date/Time: Sun 25 Sep 00:26:36 UTC 2016 Additional Info: CRITICAL: 60.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-kafka05/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-kafka05 Address: 10.68.21.106 State: OK Date/Time: Sat 24 Sep 23:30:36 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-cache-upload04/Free space - all mounts is CRITICAL **
Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-cache-upload04 Address: 10.68.18.109 State: CRITICAL Date/Time: Sat 24 Sep 23:31:10 UTC 2016 Additional Info: CRITICAL: deployment-prep.deployment-cache-upload04.diskspace._srv_vdb.byte_percentfree (<44.44%) ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-kafka05/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-kafka05 Address: 10.68.21.106 State: CRITICAL Date/Time: Sat 24 Sep 22:55:35 UTC 2016 Additional Info: CRITICAL: 50.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-kafka05/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-kafka05 Address: 10.68.21.106 State: OK Date/Time: Sat 24 Sep 22:04:36 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-kafka05/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-kafka05 Address: 10.68.21.106 State: CRITICAL Date/Time: Sat 24 Sep 21:24:35 UTC 2016 Additional Info: CRITICAL: 40.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-sca01/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-sca01 Address: 10.68.20.183 State: OK Date/Time: Sat 24 Sep 21:06:14 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-tin02/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-tin02 Address: 10.68.18.163 State: OK Date/Time: Sat 24 Sep 20:47:57 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-sca01/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-sca01 Address: 10.68.20.183 State: CRITICAL Date/Time: Sat 24 Sep 20:31:12 UTC 2016 Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-mira/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-mira Address: 10.68.20.135 State: OK Date/Time: Sat 24 Sep 20:27:31 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-parsoid09/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-parsoid09 Address: 10.68.20.142 State: OK Date/Time: Sat 24 Sep 20:22:35 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] beta-scap-eqiad - Build # 121450 - Fixed!
beta-scap-eqiad - Build # 121450 - Fixed: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/121450/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] beta-scap-eqiad - Build # 121449 - Still Failing!
beta-scap-eqiad - Build # 121449 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/121449/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] Host DOWN alert for deployment-tin!
Notification Type: PROBLEM Host: deployment-tin State: DOWN Address: 10.68.17.240 Info: CRITICAL - Host Unreachable (10.68.17.240) Date/Time: Sat 24 Sep 20:09:32 UTC 2016 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tin/SSH is CRITICAL **
Notification Type: PROBLEM Service: SSH Host: deployment-tin Address: 10.68.17.240 State: CRITICAL Date/Time: Sat 24 Sep 20:08:09 UTC 2016 Additional Info: CRITICAL - Socket timeout after 10 seconds ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-cache-upload04/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-cache-upload04 Address: 10.68.18.109 State: CRITICAL Date/Time: Sat 24 Sep 20:07:41 UTC 2016 Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tin02/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-tin02 Address: 10.68.18.163 State: CRITICAL Date/Time: Sat 24 Sep 20:07:58 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-db04/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-db04 Address: 10.68.18.35 State: OK Date/Time: Sat 24 Sep 20:05:35 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-db03/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-db03 Address: 10.68.23.30 State: OK Date/Time: Sat 24 Sep 19:59:53 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-db04/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-db04 Address: 10.68.18.35 State: CRITICAL Date/Time: Sat 24 Sep 19:50:33 UTC 2016 Additional Info: CRITICAL: 40.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-db03/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-db03 Address: 10.68.23.30 State: CRITICAL Date/Time: Sat 24 Sep 19:49:54 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-parsoid09/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-parsoid09 Address: 10.68.20.142 State: CRITICAL Date/Time: Sat 24 Sep 19:47:34 UTC 2016 Additional Info: CRITICAL: 33.33% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-eventlogging03/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-eventlogging03 Address: 10.68.18.111 State: OK Date/Time: Sat 24 Sep 19:32:57 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-eventlogging03/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-eventlogging03 Address: 10.68.18.111 State: CRITICAL Date/Time: Sat 24 Sep 18:52:59 UTC 2016 Additional Info: CRITICAL: 20.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-db03/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-db03 Address: 10.68.23.30 State: OK Date/Time: Sat 24 Sep 18:43:53 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-db03/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-db03 Address: 10.68.23.30 State: CRITICAL Date/Time: Sat 24 Sep 18:03:54 UTC 2016 Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tin02/Keyholder status is CRITICAL **
Notification Type: PROBLEM Service: Keyholder status Host: deployment-tin02 Address: 10.68.18.163 State: CRITICAL Date/Time: Sat 24 Sep 16:32:56 UTC 2016 Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-db03/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-db03 Address: 10.68.23.30 State: OK Date/Time: Sat 24 Sep 16:12:52 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-mx/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-mx Address: 10.68.17.78 State: CRITICAL Date/Time: Sat 24 Sep 14:55:42 UTC 2016 Additional Info: CRITICAL: 50.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-ores-redis/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-ores-redis Address: 10.68.21.235 State: CRITICAL Date/Time: Sat 24 Sep 11:24:12 UTC 2016 Additional Info: CRITICAL: 40.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-eventlogging04/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-eventlogging04 Address: 10.68.23.204 State: OK Date/Time: Sat 24 Sep 10:13:18 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-eventlogging04/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-eventlogging04 Address: 10.68.23.204 State: CRITICAL Date/Time: Sat 24 Sep 09:38:20 UTC 2016 Additional Info: CRITICAL: 66.67% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-cache-text04/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-cache-text04 Address: 10.68.18.103 State: OK Date/Time: Sat 24 Sep 08:52:28 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-cache-text04/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-cache-text04 Address: 10.68.18.103 State: CRITICAL Date/Time: Sat 24 Sep 08:12:27 UTC 2016 Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-tmh01/Puppet run is OK **
Notification Type: RECOVERY Service: Puppet run Host: deployment-tmh01 Address: 10.68.16.211 State: OK Date/Time: Sat 24 Sep 07:10:28 UTC 2016 Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** RECOVERY alert - deployment-puppetmaster/Long lived cherry-picks on puppetmaster is OK **
Notification Type: RECOVERY Service: Long lived cherry-picks on puppetmaster Host: deployment-puppetmaster Address: 10.68.16.63 State: OK Date/Time: Sat 24 Sep 07:01:06 UTC 2016 Additional Info: OK: Less than 100.00% above the threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
[Betacluster-alerts] ** PROBLEM alert - deployment-tmh01/Puppet run is CRITICAL **
Notification Type: PROBLEM Service: Puppet run Host: deployment-tmh01 Address: 10.68.16.211 State: CRITICAL Date/Time: Sat 24 Sep 06:35:28 UTC 2016 Additional Info: CRITICAL: 50.00% of data above the critical threshold [0.0] ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts
