Re: [openstack-dev] [all project] Time to fail tempest gate jobs when new log errors appear

2013-11-27 Thread Joe Gordon
On Nov 27, 2013 11:06 AM, "Jay Pipes"  wrote:
>
> On 11/27/2013 01:53 PM, David Kranz wrote:
>>
>> tl;dr Soon, perhaps next week, tempest gate jobs will start failing if
>> there are any ERROR lines in the logs that are not matched by an entry
>> in https://github.com/openstack/tempest/blob/master/etc/whitelist.yaml.
>> There is an exception for neutron because
>> more work needs to be done there for this to be feasible.
>>
>> The whitelist file contains a lot of entries that look more like
>> substantial bugs than incorrectly logging an ERROR due to bad
>> client data. I have been tracking this for a while and new things show
>> up frequently. But since the tests pass no one looks at the logs and
>> notices the bug indicators. We need to stop these bugs from getting
>> merged. I have filed individual bugs for many of the items in the
>> whitelist, but by no means all of them. The neutron team is taking on
>> the task of getting rid of their errors and there are so many it is too
>> much work to keep an up-to-date whitelist for neutron. So for the time
>> being, neutron runs will dump all errors to the console but not fail.
>>
>> In addition to the fact that these log errors indicate bugs, they make
>> it more difficult to diagnose a problem when builds actually fail in the
>> tempest tests because it can be hard to tell which log errors are
>> "known" and which might be causing the failure. Hopefully some priority
>> will be given to fixing these bugs and removing entries from the
>> whitelist until it is driven to zero.
>>
>> If any one has any comments or suggestions to improve this process,
>> please speak up.
>
>
> Yay! \o/

+1

>
> -jay
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all project] Time to fail tempest gate jobs when new log errors appear

2013-11-27 Thread Jay Pipes

On 11/27/2013 01:53 PM, David Kranz wrote:

tl;dr Soon, perhaps next week, tempest gate jobs will start failing if
there are any ERROR lines in the logs that are not matched by an entry
in https://github.com/openstack/tempest/blob/master/etc/whitelist.yaml.
There is an exception for neutron because
more work needs to be done there for this to be feasible.

The whitelist file contains a lot of entries that look more like
substantial bugs than incorrectly logging an ERROR due to bad
client data. I have been tracking this for a while and new things show
up frequently. But since the tests pass no one looks at the logs and
notices the bug indicators. We need to stop these bugs from getting
merged. I have filed individual bugs for many of the items in the
whitelist, but by no means all of them. The neutron team is taking on
the task of getting rid of their errors and there are so many it is too
much work to keep an up-to-date whitelist for neutron. So for the time
being, neutron runs will dump all errors to the console but not fail.

In addition to the fact that these log errors indicate bugs, they make
it more difficult to diagnose a problem when builds actually fail in the
tempest tests because it can be hard to tell which log errors are
"known" and which might be causing the failure. Hopefully some priority
will be given to fixing these bugs and removing entries from the
whitelist until it is driven to zero.

If any one has any comments or suggestions to improve this process,
please speak up.


Yay! \o/

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev