[Betacluster-alerts] ** RECOVERY alert - deployment-cache-text04/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-cache-text04 Address: 10.68.18.103 State: OK Date/Time: Wed 21 Mar 21:13:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-text04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-cache-text04 Address: 10.68.18.103 State: CRITICAL Date/Time: Wed 21 Mar 21:19:42 UTC 2018 Notes URLs: Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-mx02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-mx02 Address: 10.68.23.220 State: CRITICAL Date/Time: Wed 21 Mar 21:08:21 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki07/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki07 Address: 10.68.17.40 State: CRITICAL Date/Time: Wed 21 Mar 18:34:35 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Wed 21 Mar 19:21:41 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** RECOVERY alert - deployment-cache-text04/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-cache-text04 Address: 10.68.18.103 State: OK Date/Time: Wed 21 Mar 21:44:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-text04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-cache-text04 Address: 10.68.18.103 State: CRITICAL Date/Time: Wed 21 Mar 20:48:41 UTC 2018 Notes URLs: Additional Info: CRITICAL: 20.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Wed 21 Mar 18:06:39 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] beta-scap-eqiad - Build # 200419 - Fixed!

beta-scap-eqiad - Build # 200419 - Fixed: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/200419/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki06/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-mediawiki06 Address: 10.68.19.241 State: CRITICAL Date/Time: Wed 21 Mar 18:19:11 UTC 2018 Notes URLs: Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-jobrunner02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-jobrunner02 Address: 10.68.19.42 State: CRITICAL Date/Time: Wed 21 Mar 18:19:38 UTC 2018 Notes URLs: Additional Info: CRITICAL: 30.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Wed 21 Mar 18:59:11 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Wed 21 Mar 18:00:41 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-secureredirexperiment/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-secureredirexperiment Address: 10.68.17.132 State: CRITICAL Date/Time: Wed 21 Mar 18:54:27 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] beta-scap-eqiad - Build # 200418 - Failure!

beta-scap-eqiad - Build # 200418 - Failure: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/200418/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-etcd-01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-etcd-01 Address: 10.68.19.227 State: CRITICAL Date/Time: Wed 21 Mar 18:35:57 UTC 2018 Notes URLs: Additional Info: CRITICAL: 22.22% of data above the critical threshold [0.0] ___

[Betacluster-alerts] Host DOWN alert for deployment-tmh01!

Notification Type: PROBLEM Host: deployment-tmh01 State: DOWN Address: 10.68.16.211 Info: CRITICAL - Host Unreachable (10.68.16.211) Date/Time: Wed 21 Mar 10:54:52 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] Host DOWN alert for deployment-videoscaler01!

Notification Type: PROBLEM Host: deployment-videoscaler01 State: DOWN Address: 10.68.19.130 Info: CRITICAL - Host Unreachable (10.68.19.130) Date/Time: Wed 21 Mar 10:54:06 UTC 2018 ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Wed 21 Mar 12:28:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: WARNING Date/Time: Wed 21 Mar 11:37:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Wed 21 Mar 11:47:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Wed 21 Mar 12:18:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-maps01/Puppet staleness is OK **

Notification Type: RECOVERY Service: Puppet staleness Host: deployment-maps01 Address: 10.68.16.73 State: OK Date/Time: Wed 21 Mar 22:16:15 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [3600.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki06/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki06 Address: 10.68.19.241 State: CRITICAL Date/Time: Wed 21 Mar 22:24:21 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** PROBLEM alert - Generic Beta Cluster/English Wikipedia Mobile Main page is CRITICAL **

Notification Type: PROBLEM Service: English Wikipedia Mobile Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: CRITICAL Date/Time: Wed 21 Mar 22:23:52 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki05 Address: 10.68.22.21 State: CRITICAL Date/Time: Wed 21 Mar 22:23:12 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Wed 21 Mar 22:37:25 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 47503 bytes in 4.721 second response time

[Betacluster-alerts] ** RECOVERY alert - Generic Beta Cluster/English Wikipedia Main page is OK **

Notification Type: RECOVERY Service: English Wikipedia Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: OK Date/Time: Wed 21 Mar 22:36:12 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 48040 bytes in 5.628 second response time

[Betacluster-alerts] Host DOWN alert for deployment-maps01!

Notification Type: PROBLEM Host: deployment-maps01 State: DOWN Address: 10.68.16.73 Info: PING CRITICAL - Packet loss = 100% Date/Time: Wed 21 Mar 23:03:59 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Wed 21 Mar 22:52:38 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Wed 21 Mar 23:14:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: WARNING Date/Time: Wed 21 Mar 23:24:37 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<66.67%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Wed 21 Mar 22:22:30 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds ___

[Betacluster-alerts] ** PROBLEM alert - Generic Beta Cluster/English Wikipedia Main page is CRITICAL **

Notification Type: PROBLEM Service: English Wikipedia Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: CRITICAL Date/Time: Wed 21 Mar 22:21:18 UTC 2018 Notes URLs: Additional Info: CRITICAL - Socket timeout after 10 seconds

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Wed 21 Mar 23:17:39 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Wed 21 Mar 23:23:41 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] Host UP alert for deployment-maps01!

Notification Type: RECOVERY Host: deployment-maps01 State: UP Address: 10.68.16.73 Info: PING OK - Packet loss = 0%, RTA = 0.75 ms Date/Time: Wed 21 Mar 23:43:05 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Thu 22 Mar 04:41:40 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Thu 22 Mar 05:50:46 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Thu 22 Mar 04:11:40 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Thu 22 Mar 04:57:38 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] Host DOWN alert for deployment-puppetdb01!

Notification Type: PROBLEM Host: deployment-puppetdb01 State: DOWN Address: 10.68.23.76 Info: CRITICAL - Host Unreachable (10.68.23.76) Date/Time: Wed 21 Mar 14:53:14 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: CRITICAL Date/Time: Thu 22 Mar 01:39:39 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%)

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Thu 22 Mar 03:50:40 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Thu 22 Mar 02:49:39 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Thu 22 Mar 03:20:38 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - Generic Beta Cluster/English Wikipedia Mobile Main page is OK **

Notification Type: RECOVERY Service: English Wikipedia Mobile Main page Host: Generic Beta Cluster Address: en.wikipedia.beta.wmflabs.org State: OK Date/Time: Wed 21 Mar 22:33:50 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 36422 bytes in 7.398 second response time

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Wed 21 Mar 22:34:19 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 47503 bytes in 7.082 second response time

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Wed 21 Mar 23:19:46 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-elastic07/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-elastic07 Address: 10.68.19.180 State: OK Date/Time: Wed 21 Mar 16:10:49 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___ Betacluster-alerts

[Betacluster-alerts] ** PROBLEM alert - deployment-maps01/Puppet staleness is CRITICAL **

Notification Type: PROBLEM Service: Puppet staleness Host: deployment-maps01 Address: 10.68.16.73 State: CRITICAL Date/Time: Wed 21 Mar 15:31:16 UTC 2018 Notes URLs: Additional Info: CRITICAL: 11.11% of data above the critical threshold [43200.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-elastic07/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-elastic07 Address: 10.68.19.180 State: CRITICAL Date/Time: Wed 21 Mar 15:35:47 UTC 2018 Notes URLs: Additional Info: CRITICAL: 60.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-ores01 Address: 10.68.20.26 State: CRITICAL Date/Time: Wed 21 Mar 16:50:50 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints