Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-04-02 Thread Mike Perez
On 00:21 Tue 31 Mar , Rochelle Grober wrote: > Top posting… I believe the main issue was a problem with snapshots that > caused false negatives for most cinder drivers. But, that got fixed. I don't know what you're talking about here. Are you saying there was an issue with the Tempest tests t

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread Duncan Thomas
On 31 March 2015 at 01:35, John Griffith wrote: > On Mon, Mar 30, 2015 at 4:06 PM, Doug Wiegley < > doug...@parksidesoftware.com> wrote: > >> ​ >> >> > - Test relies on some “optional” feature, like overlapping IP subnets that >> the backend doesn’t support. I’d argue it’s another case of broken

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
Griffith [mailto:john.griffi...@gmail.com] > *Sent:* Monday, March 30, 2015 8:12 PM > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] > Clarifications on the goal and skipping tests > > > > >

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread Arkady_Kanevsky
] Clarifications on the goal and skipping tests On Mon, Mar 30, 2015 at 6:21 PM, Rochelle Grober mailto:rochelle.gro...@huawei.com>> wrote: Top posting… I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed. Unfortu

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
On Mon, Mar 30, 2015 at 6:21 PM, Rochelle Grober wrote: > Top posting… I believe the main issue was a problem with snapshots that > caused false negatives for most cinder drivers. But, that got fixed. > Unfortunately, we haven’t yet established a good process to notify third > parties when skip

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread Rochelle Grober
Top posting… I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed. Unfortunately, we haven’t yet established a good process to notify third parties when skipped tests are fixed and should be “unskipped”. Maybe tagging t

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
On Mon, Mar 30, 2015 at 4:06 PM, Doug Wiegley wrote: > A few reasons, I’m sure there are others: > > - Broken tests that hardcode something about the ref implementation. The > test needs to be fixed, of course, but in the meantime, a constantly > failing CI is worthless (hello, lbaas scenario tes

Re: [openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread Doug Wiegley
A few reasons, I’m sure there are others: - Broken tests that hardcode something about the ref implementation. The test needs to be fixed, of course, but in the meantime, a constantly failing CI is worthless (hello, lbaas scenario test.) - Test relies on some “optional” feature, like overlapping

[openstack-dev] [OpenStack-Dev] [third-party-ci] Clarifications on the goal and skipping tests

2015-03-30 Thread John Griffith
This may have already been raised/discussed, but I'm kinda confused so thought I'd ask on the ML here. The whole point of third party CI as I recall was to run the same tests that we run in the official Gate against third party drivers. To me that would imply that a CI system/device that marks it