[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Tue 16 Jan 07:41:53 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Tue 16 Jan 07:31:53 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] ** RECOVERY alert - deployment-tin/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: OK Date/Time: Tue 16 Jan 07:25:54 UTC 2018 Notes URLs: Additional Info: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 16 Jan 07:21:43 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 16 Jan 07:16:41 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-tin/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-tin Address: 10.68.21.205 State: WARNING Date/Time: Tue 16 Jan 07:15:54 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints

[Betacluster-alerts] beta-scap-eqiad - Build # 191032 - Fixed!

beta-scap-eqiad - Build # 191032 - Fixed: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/191032/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - deployment-fluorine02/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: OK Date/Time: Tue 16 Jan 07:02:06 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka01/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-kafka01 Address: 10.68.21.219 State: WARNING Date/Time: Tue 16 Jan 06:54:30 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%)

[Betacluster-alerts] beta-scap-eqiad - Build # 191031 - Failure!

beta-scap-eqiad - Build # 191031 - Failure: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/191031/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Tue 16 Jan 06:38:19 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki05/App Server Main HTTP Response is OK **

Notification Type: RECOVERY Service: App Server Main HTTP Response Host: deployment-mediawiki05 Address: 10.68.22.21 State: OK Date/Time: Tue 16 Jan 06:35:43 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 46843 bytes in 5.920 second response time

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 16 Jan 06:35:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki05 Address: 10.68.22.21 State: WARNING Date/Time: Tue 16 Jan 06:33:19 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki05/App Server Main HTTP Response is CRITICAL **

Notification Type: PROBLEM Service: App Server Main HTTP Response Host: deployment-mediawiki05 Address: 10.68.22.21 State: CRITICAL Date/Time: Tue 16 Jan 06:30:36 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 16 Jan 06:30:41 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-fluorine02/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-fluorine02 Address: 10.68.23.106 State: WARNING Date/Time: Tue 16 Jan 05:07:06 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 16 Jan 04:38:43 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 16 Jan 04:33:42 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Tue 16 Jan 04:12:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Tue 16 Jan 04:07:40 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] beta-scap-eqiad - Build # 191005 - Failure!

beta-scap-eqiad - Build # 191005 - Failure: Check console output at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/191005/ to view the results.___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** PROBLEM alert - deployment-puppetmaster02/Long lived cherry-picks on puppetmaster is CRITICAL **

Notification Type: PROBLEM Service: Long lived cherry-picks on puppetmaster Host: deployment-puppetmaster02 Address: 10.68.21.200 State: CRITICAL Date/Time: Tue 16 Jan 00:31:15 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 23:06:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 23:01:41 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - deployment-mx/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mx Address: 10.68.17.78 State: OK Date/Time: Mon 15 Jan 22:11:17 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mx/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mx Address: 10.68.17.78 State: CRITICAL Date/Time: Mon 15 Jan 21:06:14 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<100.00%)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 20:34:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 20:14:42 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] Host DOWN alert for deployment-netbox!

Notification Type: PROBLEM Host: deployment-netbox State: DOWN Address: 10.68.19.203 Info: CRITICAL - Host Unreachable (10.68.19.203) Date/Time: Mon 15 Jan 20:01:06 UTC 2018 ___ Betacluster-alerts mailing list Betacluster-alerts@lists.wikimedia.org

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 19:58:43 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 19:53:43 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 19:07:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 19:02:41 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka03/Puppet staleness is CRITICAL **

Notification Type: PROBLEM Service: Puppet staleness Host: deployment-kafka03 Address: 10.68.16.138 State: CRITICAL Date/Time: Mon 15 Jan 18:58:54 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [43200.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-kafka03/Free space - all mounts is CRITICAL **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-kafka03 Address: 10.68.16.138 State: CRITICAL Date/Time: Mon 15 Jan 18:54:29 UTC 2018 Notes URLs: Additional Info: CRITICAL: deployment-prep.deployment-kafka03.diskspace.root.byte_percentfree (<100.00%)

[Betacluster-alerts] ** PROBLEM alert - deployment-mx/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-mx Address: 10.68.17.78 State: CRITICAL Date/Time: Mon 15 Jan 18:39:21 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - deployment-mediawiki04/Free space - all mounts is OK **

Notification Type: RECOVERY Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: OK Date/Time: Mon 15 Jan 18:00:58 UTC 2018 Notes URLs: Additional Info: OK: All targets OK ___ Betacluster-alerts mailing list

[Betacluster-alerts] ** PROBLEM alert - deployment-mediawiki04/Free space - all mounts is WARNING **

Notification Type: PROBLEM Service: Free space - all mounts Host: deployment-mediawiki04 Address: 10.68.19.128 State: WARNING Date/Time: Mon 15 Jan 17:50:59 UTC 2018 Notes URLs: Additional Info: WARNING: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%)

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 16:21:40 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 16:16:41 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - deployment-redis02/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-redis02 Address: 10.68.16.231 State: CRITICAL Date/Time: Mon 15 Jan 15:33:41 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 15:30:40 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - deployment-redis01/Puppet errors is CRITICAL **

Notification Type: PROBLEM Service: Puppet errors Host: deployment-redis01 Address: 10.68.16.177 State: CRITICAL Date/Time: Mon 15 Jan 15:27:43 UTC 2018 Notes URLs: Additional Info: CRITICAL: 100.00% of data above the critical threshold [0.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 15:25:41 UTC 2018 Notes URLs: Additional Info: WARNING: 40.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 13:49:43 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 13:43:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 13:38:44 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 13:27:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 13:22:40 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 11:51:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 11:46:42 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 10:05:41 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 09:55:41 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]

[Betacluster-alerts] ** RECOVERY alert - Graphite Labs/Mediawiki Error Rate is OK **

Notification Type: RECOVERY Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: OK Date/Time: Mon 15 Jan 08:48:40 UTC 2018 Notes URLs: Additional Info: OK: Less than 1.00% above the threshold [1.0] ___

[Betacluster-alerts] ** PROBLEM alert - Graphite Labs/Mediawiki Error Rate is WARNING **

Notification Type: PROBLEM Service: Mediawiki Error Rate Host: Graphite Labs Address: graphite-labs.wikimedia.org State: WARNING Date/Time: Mon 15 Jan 08:43:41 UTC 2018 Notes URLs: Additional Info: WARNING: 20.00% of data above the warning threshold [1.0]