Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-04-02 Thread Mike Perez
On 00:21 Tue 31 Mar , Rochelle Grober wrote: Top posting… I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed. I don't know what you're talking about here. Are you saying there was an issue with the Tempest tests

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-31 Thread Duncan Thomas
On 31 March 2015 at 01:35, John Griffith john.griffi...@gmail.com wrote: On Mon, Mar 30, 2015 at 4:06 PM, Doug Wiegley doug...@parksidesoftware.com wrote: ​ - Test relies on some “optional” feature, like overlapping IP subnets that the backend doesn’t support. I’d argue it’s another

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
On Mon, Mar 30, 2015 at 4:06 PM, Doug Wiegley doug...@parksidesoftware.com wrote: A few reasons, I’m sure there are others: - Broken tests that hardcode something about the ref implementation. The test needs to be fixed, of course, but in the meantime, a constantly failing CI is worthless

[openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
This may have already been raised/discussed, but I'm kinda confused so thought I'd ask on the ML here. The whole point of third party CI as I recall was to run the same tests that we run in the official Gate against third party drivers. To me that would imply that a CI system/device that marks

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread Doug Wiegley
A few reasons, I’m sure there are others: - Broken tests that hardcode something about the ref implementation. The test needs to be fixed, of course, but in the meantime, a constantly failing CI is worthless (hello, lbaas scenario test.) - Test relies on some “optional” feature, like

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread Rochelle Grober
Top posting… I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed. Unfortunately, we haven’t yet established a good process to notify third parties when skipped tests are fixed and should be “unskipped”. Maybe tagging

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests On Mon, Mar 30, 2015 at 6:21 PM, Rochelle Grober rochelle.gro...@huawei.com wrote: Top posting… I believe the main issue was a problem

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
On Mon, Mar 30, 2015 at 6:21 PM, Rochelle Grober rochelle.gro...@huawei.com wrote: Top posting… I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed. Unfortunately, we haven’t yet established a good process to notify

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread Arkady_Kanevsky
. That is easier to track, can be cleanly used by defcore and for tagging. Thanks, Arkady From: John Griffith [mailto:john.griffi...@gmail.com] Sent: Monday, March 30, 2015 8:12 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [OpenStack-Dev] [third-party-ci