On 12/03/2013 09:30 AM, Eoghan Glynn wrote:
----- Original Message -----
On 12/02/2013 10:24 AM, Julien Danjou wrote:
On Fri, Nov 29 2013, David Kranz wrote:
In preparing to fail builds with log errors I have been trying to make
things easier for projects by maintaining a whitelist. But these bugs in
ceilometer are coming in so fast that I can't keep up. So I am just
putting
".*" in the white list for any cases I find before gate failing is turned
on, hopefully early this week.
Following the chat on IRC and the bug reports, it seems this might come
From the tempest tests that are under reviews, as currently I don't
think Ceilometer generates any error as it's not tested.
So I'm not sure we want to whitelist anything?
So I tested this with https://review.openstack.org/#/c/59443/. There are
flaky log errors coming from ceilometer. You
can see that the build at 12:27 passed, but the last build failed twice,
each with a different set of errors. So the whitelist needs to remain
and the ceilometer team should remove each entry when it is believed to
be unnecessary.
Hi David,
Just looking into this issue.
So when you say the build failed, do you mean that errors were detected
in the ceilometer log files? (as opposed to a specific Tempest testcase
having reported a failure)
Yes, exactly. This patch removed the whitelist entries for ceilometer
and so those errors then "failed" the build.
If that interpretation of build failure is correct, I think there's a simple
explanation for the compute agent ERRORs seen in the log file for the CI
build related to your patch referenced above, specifically:
ERROR ceilometer.compute.pollsters.disk [-] Requested operation is not
valid: domain is not running
The problem I suspect is a side-effect of a nova test that suspends the
instance in question, followed by a race between the ceilometer logic that
discovers the local instances via the nova-api followed by the individual
pollsters that call into the libvirt daemon to gather the disk stats etc.
It appears that the libvirt virDomainBlockStats() call fails with "domain
is not running" for suspended instances.
This would only occur intermittently as it requires the instance to
remain in the suspended state across a polling interval boundary.
So we need tighten up our logic there to avoid spewing needless errors
when a very normal event occurs (i.e. instance suspension).
I've filed a bug[1] which some ideas for addressing the issue - this
will require a bit of discussion before agreeing a way forward, but I'll
prioritize getting this knocked on the head asap.
Great! Thanks. The change I pushed yesterday should help prevent this
sort of thing from creeping in across all projects. But as Julian
observed, the process of removing entries from the whitelist that are no
longer needed due to bug fixes is not so easy and automatic. I'm trying
to put together a script that will check the whitelist entries against
the last two weeks of builds using logstash but it is not so simple to
do that since general regexps cannot be used with logstash.
-David
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
