Re: [openstack-dev] [qa] Plan for failing successful tempest jobs when new ERRORs appear in logs
On Mon, Nov 18, 2013 at 2:58 PM, David Kranz dkr...@redhat.com wrote: So we are close to being able to start doing this. The current whitelist is here https://github.com/openstack/tempest/blob/master/etc/ whitelist.yaml. I have a find-errors script that watches for successful builds and pulls out the non-whitelisted errors. For the past few weeks I have been doing the following: 1. Run find-errors 2. File bugs on any new errors 3. Add to whitelist 4. Repeat There are still some very flaky cases. I will do one more iteration of this. Right now this script https://github.com/openstack/ tempest/blob/master/tools/check_logs.py dumps non-whitelisted errors to the console log but always returns success. The question now is how long should all jobs run with no new errors showing, before changing check_logs.py to fail if there are any new errors? The sooner the better. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Plan for failing successful tempest jobs when new ERRORs appear in logs
On Mon, Nov 25, 2013 at 11:06 AM, Joe Gordon joe.gord...@gmail.com wrote: On Mon, Nov 18, 2013 at 2:58 PM, David Kranz dkr...@redhat.com wrote: So we are close to being able to start doing this. The current whitelist is here https://github.com/openstack/tempest/blob/master/etc/ whitelist.yaml. I have a find-errors script that watches for successful builds and pulls out the non-whitelisted errors. For the past few weeks I have been doing the following: 1. Run find-errors 2. File bugs on any new errors 3. Add to whitelist 4. Repeat There are still some very flaky cases. I will do one more iteration of this. Right now this script https://github.com/openstack/ tempest/blob/master/tools/check_logs.py dumps non-whitelisted errors to the console log but always returns success. The question now is how long should all jobs run with no new errors showing, before changing check_logs.py to fail if there are any new errors? The sooner the better. Also this is awesome. I expect this to significantly help with keeping the gate stable. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Plan for failing successful tempest jobs when new ERRORs appear in logs
On Mon, Nov 25, 2013 at 11:06:36AM -0800, Joe Gordon wrote: On Mon, Nov 18, 2013 at 2:58 PM, David Kranz dkr...@redhat.com wrote: So we are close to being able to start doing this. The current whitelist is here https://github.com/openstack/tempest/blob/master/etc/ whitelist.yaml. I have a find-errors script that watches for successful builds and pulls out the non-whitelisted errors. For the past few weeks I have been doing the following: 1. Run find-errors 2. File bugs on any new errors 3. Add to whitelist 4. Repeat There are still some very flaky cases. I will do one more iteration of this. Right now this script https://github.com/openstack/ tempest/blob/master/tools/check_logs.py dumps non-whitelisted errors to the console log but always returns success. The question now is how long should all jobs run with no new errors showing, before changing check_logs.py to fail if there are any new errors? The sooner the better. +1 I would just turn it on today. This is the week to do it because of the holiday. Based on my experience with flipping the switch for parallel the only way to iron out all of the kinks is to make it gating so people will notice when something fails. There will be some pain at first but the end result makes it worth it. -Matt Treinish ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] Plan for failing successful tempest jobs when new ERRORs appear in logs
So we are close to being able to start doing this. The current whitelist is here https://github.com/openstack/tempest/blob/master/etc/whitelist.yaml. I have a find-errors script that watches for successful builds and pulls out the non-whitelisted errors. For the past few weeks I have been doing the following: 1. Run find-errors 2. File bugs on any new errors 3. Add to whitelist 4. Repeat There are still some very flaky cases. I will do one more iteration of this. Right now this script https://github.com/openstack/tempest/blob/master/tools/check_logs.py dumps non-whitelisted errors to the console log but always returns success. The question now is how long should all jobs run with no new errors showing, before changing check_logs.py to fail if there are any new errors? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
