[Betacluster-alerts] ** PROBLEM alert - deployment-mx02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-mx02 Address: 10.68.23.220 State: CRITICAL Date/Time: Fri 23 Feb 21:08:21 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: WARNING Date/Time: Fri 23 Feb 22:30:39 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Fri 23 Feb 22:56:32 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Fri 23 Feb 22:51:30 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Sat 24 Feb 00:33:28 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: CRITICAL Date/Time: Sat 24 Feb 01:20:37 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%)

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Sat 24 Feb 05:14:28 UTC 2018 Notes URLs: Additional Info: CRITICAL: 20.00% of data above the critical threshold [10.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Sat 24 Feb 05:09:28 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Sat 24 Feb 04:44:31 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is CRITICAL **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: CRITICAL Date/Time: Sat 24 Feb 04:54:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: 40.00% of data above the critical threshold [10.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Sat 24 Feb 05:24:29 UTC 2018 Notes URLs: Additional Info: WARNING: 80.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-fluorine02/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: OK Date/Time: Sat 24 Feb 07:00:37 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-eventlogging05/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-eventlogging05 Address: 10.68.23.51 State: CRITICAL Date/Time: Fri 23 Feb 17:02:23 UTC 2018 Notes URLs: Additional Info: CRITICAL: 75.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-logstash2/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-logstash2 Address: 10.68.16.147 State: CRITICAL Date/Time: Fri 23 Feb 16:33:16 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-eventlogging05/Puppet errors is OK **

Notification Type: RECOVERY Service: Puppet errors Host: deployment-eventlogging05 Address: 10.68.23.51 State: OK Date/Time: Fri 23 Feb 19:07:24 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki07/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki07 Address: 10.68.17.40 State: CRITICAL Date/Time: Fri 23 Feb 18:34:35 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki06/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki06 Address: 10.68.19.241 State: CRITICAL Date/Time: Fri 23 Feb 18:48:13 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on

[Betacluster-alerts] ** PROBLEM alert - deployment-secureredirexperiment/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-secureredirexperiment Address: 10.68.17.132 State: CRITICAL Date/Time: Fri 23 Feb 18:54:27 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-cache-upload04/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-cache-upload04 Address: 10.68.18.109 State: CRITICAL Date/Time: Fri 23 Feb 18:32:41 UTC 2018 Notes URLs: Additional Info: CRITICAL: 55.56% of data above the critical threshold [0.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Fri 23 Feb 18:53:17 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 47202 bytes in 6.956 second response time

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Fri 23 Feb 19:18:28 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] beta-scap-eqiad - Build # 196863 - Still Failing!

beta-scap-eqiad - Build # 196863 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196863/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196865 - Still Failing!

beta-scap-eqiad - Build # 196865 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196865/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196866 - Still Failing!

beta-scap-eqiad - Build # 196866 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196866/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196870 - Fixed!

beta-scap-eqiad - Build # 196870 - Fixed: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196870/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: WARNING Date/Time: Fri 23 Feb 12:00:49 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] beta-scap-eqiad - Build # 196861 - Still Failing!

beta-scap-eqiad - Build # 196861 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196861/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] Host DOWN alert for deployment-tmh01!

Notification Type: PROBLEM Host: deployment-tmh01 State: DOWN Address: 10.68.16.211 Info: CRITICAL - Host Unreachable (10.68.16.211) Date/Time: Fri 23 Feb 10:54:51 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196862 - Still Failing!

beta-scap-eqiad - Build # 196862 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196862/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196864 - Still Failing!

beta-scap-eqiad - Build # 196864 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196864/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196867 - Still Failing!

beta-scap-eqiad - Build # 196867 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196867/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196868 - Still Failing!

beta-scap-eqiad - Build # 196868 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196868/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196869 - Still Failing!

beta-scap-eqiad - Build # 196869 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196869/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Fri 23 Feb 12:05:49 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Fri 23 Feb 09:29:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: CRITICAL Date/Time: Fri 23 Feb 09:30:49 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<55.56%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Fri 23 Feb 09:38:48 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] beta-scap-eqiad - Build # 196854 - Still Failing!

beta-scap-eqiad - Build # 196854 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196854/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196851 - Still Failing!

beta-scap-eqiad - Build # 196851 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196851/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196852 - Still Failing!

beta-scap-eqiad - Build # 196852 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196852/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196858 - Still Failing!

beta-scap-eqiad - Build # 196858 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196858/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196859 - Still Failing!

beta-scap-eqiad - Build # 196859 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196859/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196850 - Failure!

beta-scap-eqiad - Build # 196850 - Failure: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196850/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki06/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki06 Address: 10.68.19.241 State: WARNING Date/Time: Fri 23 Feb 09:28:48 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki06.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Fri 23 Feb 09:39:47 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] beta-scap-eqiad - Build # 196853 - Still Failing!

beta-scap-eqiad - Build # 196853 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196853/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196855 - Still Failing!

beta-scap-eqiad - Build # 196855 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196855/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-eventlog02/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-eventlog02 Address: 10.68.18.138 State: CRITICAL Date/Time: Fri 23 Feb 10:11:48 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<100.00%)

[Betacluster-alerts] beta-scap-eqiad - Build # 196856 - Still Failing!

beta-scap-eqiad - Build # 196856 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196856/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196857 - Still Failing!

beta-scap-eqiad - Build # 196857 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196857/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] beta-scap-eqiad - Build # 196860 - Still Failing!

beta-scap-eqiad - Build # 196860 - Still Failing: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/196860/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] Host DOWN alert for deployment-videoscaler01!

Notification Type: PROBLEM Host: deployment-videoscaler01 State: DOWN Address: 10.68.19.130 Info: CRITICAL - Host Unreachable (10.68.19.130) Date/Time: Fri 23 Feb 10:54:06 UTC 2018 ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-ores01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-ores01 Address: 10.68.23.165 State: CRITICAL Date/Time: Fri 23 Feb 12:30:52 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki06/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki06 Address: 10.68.19.241 State: WARNING Date/Time: Fri 23 Feb 12:39:49 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki06.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Fri 23 Feb 12:40:47 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Fri 23 Feb 12:45:48 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki06/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki06 Address: 10.68.19.241 State: OK Date/Time: Fri 23 Feb 12:49:48 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] Host DOWN alert for deployment-puppetdb01!

Notification Type: PROBLEM Host: deployment-puppetdb01 State: DOWN Address: 10.68.23.76 Info: CRITICAL - Host Unreachable (10.68.23.76) Date/Time: Fri 23 Feb 14:53:14 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org