Re: [openstack-dev] [QA][Nova]Making gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial non voting

[Matt Riedemann]

On 1/10/2017 10:02 AM, Jordan Pittier wrote:

Hi,
I don't know if you've noticed but
the gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial job has a
high rate of false negative. I've queried Gerrit and analysed all the
"Verified -2" messages left by Jenkins (i.e Gate failure) for the last
30 days. (script is here [1]).

On project openstack/nova: For the last 58 times where
gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial ran AND
jenkins left a 'Verified -2' message, the job failed 48 times and
succeeded 10 times.

On project openstack/tempest: For the last 25 times where
gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial ran AND
jenkins left a 'Verified -2' message, the job failed 14 times and
succeeded 11 times.

In order words, when there's a gate failure
gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial is the main
culprit, by a significant margin.

I am Tempest core reviewer and this bugs me because it slows the
development of the project I care for reasons that I don't really
understand. I am going to propose a change to make this job non voting
on openstack/tempest.

Jordan

[1]
: 
https://github.com/JordanP/openstack-snippets/blob/master/analyse-gate-failures/analyse_gate_failures.py





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



The ceph job has had a high failure rate the last month or more. It's 
been very whack-a-mole from what I've seen when I'm digging into issues. 
There are still some open unresolved bugs being tracked against that job 
in the e-r status page:


http://status.openstack.org//elastic-recheck/index.html

We've fixed a few issues already (device not found on volume detach race 
in nova was one, and some cinder capacity filter issues in another), but 
what's out there now is still an issue and I'm not aware of a ton of 
focus on fixing those. jbernard probably knows the latest but unless 
there are good fixes just waiting for review, then I'm probably OK with 
making it non-voting too.


The most recent bug I reported against that job was due to the c-vol 
service completely dropping out in that job for some reason and then the 
cinder scheduler couldn't build any volumes. Very weird.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [QA][Nova]Making gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial non voting

[Jordan Pittier]

Hi,
I don't know if you've noticed but
the gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial job has a
high rate of false negative. I've queried Gerrit and analysed all the
"Verified -2" messages left by Jenkins (i.e Gate failure) for the last 30
days. (script is here [1]).

On project openstack/nova: For the last 58 times where
gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial ran AND jenkins
left a 'Verified -2' message, the job failed 48 times and succeeded 10
times.

On project openstack/tempest: For the last 25 times where
gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial ran AND jenkins
left a 'Verified -2' message, the job failed 14 times and succeeded 11
times.

In order words, when there's a gate failure
gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial is the main
culprit, by a significant margin.

I am Tempest core reviewer and this bugs me because it slows the
development of the project I care for reasons that I don't really
understand. I am going to propose a change to make this job non voting on
openstack/tempest.

Jordan

[1] :
https://github.com/JordanP/openstack-snippets/blob/master/analyse-gate-failures/analyse_gate_failures.py

-- 
 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev