On Mon, Mar 24, 2014 at 09:56:09PM +0100, Salvatore Orlando wrote: > Thanks a lot! > > We now need to get on these bugs, and define with QA an acceptable failure > rate criterion for switching the full job to voting. > It would be good to have a chance to only run the tests against code which > is already in master. > To this aim we might push a dummy patch, and keep it spinning in the check > queue.
Honestly, there isn't really a number. I had a thread trying to get consensus on that back when I first made tempest run in parallel. What I ended up doing back then and what we've done since for this kind of change is to just pick a slower week for the gate and just green light it, of course after checking to make sure if it blows up we're not blocking anything critical. If it looks like it's passing at roughly the same rate as everything else and you guys think it's ready. 25% is definitely too high, for comparison when I looked at a couple of min. ago at the numbers for the past 4 days on the equivalent job with nova-network it only failed 4% of the time. (12 out of 300) But that number does fluctuate quite a bit for example looking at the past week the number grows to 11.6%. (171 out of 1480) Doing it this way doesn't seem like the best, but until it's gating things really don't get the attention they deserve and more bugs will just slip in while you wait. There will most likely be initial pain after it merges, but it's the only real way to lock it down and make forward progress. -Matt Treinish > > > On 24 March 2014 21:45, Rossella Sblendido <[email protected]> wrote: > > > Hello all, > > > > here is an update regarding the Neutron full parallel job. > > I used the following Logstash query [1] that checks the failures of the > > last > > 4 days (the last bug fix related with the full job was merged 4 days ago). > > These are the results: > > > > 123 failure (25% of the total) > > > > I took a sample of 50 failures and I obtained the following: > > > > 22% legitimate failures (they are due to the code change introduced by the > > patch) > > 22% infra issues > > 12% https://bugs.launchpad.net/openstack-ci/+bug/1291611 > > 12% https://bugs.launchpad.net/tempest/+bug/1281969 > > 8% https://bugs.launchpad.net/tempest/+bug/1294603 > > 3% https://bugs.launchpad.net/neutron/+bug/1283522 > > 3% https://bugs.launchpad.net/neutron/+bug/1291920 > > 3% https://bugs.launchpad.net/nova/+bug/1290642 > > 3% https://bugs.launchpad.net/tempest/+bug/1252971 > > 3% https://bugs.launchpad.net/horizon/+bug/1257885 > > 3% https://bugs.launchpad.net/tempest/+bug/1292242 > > 3% https://bugs.launchpad.net/neutron/+bug/1277439 > > 3% https://bugs.launchpad.net/neutron/+bug/1283599 > > > > cheers, > > > > Rossella > > > > [1] http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOi > > BcImNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWZ1bGxcIiBBTkQgbWVzc2 > > FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIgQU5EIHRhZ3M6Y29uc29sZSIsIm > > ZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiY3VzdG9tIiwiZ3 > > JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7ImZyb20iOiIyMDE0LTAzLTIwVD > > EzOjU0OjI1KzAwOjAwIiwidG8iOiIyMDE0LTAzLTI0VDEzOjU0OjI1KzAwOj > > AwIiwidXNlcl9pbnRlcnZhbCI6IjAifSwibW9kZSI6IiIsImFuYWx5emVfZm > > llbGQiOiIiLCJzdGFtcCI6MTM5NTY3MDY2ODc0OX0= > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
