Yea, the situation described below is imagable.
And all things seem to be a measure and tradeoff, i.e., if the feature is
supported by a big part of the backends and can be deemed as something like
"main trend", then we should test it in Tempest, though inevitably we will
suffer the procedure of fail-＞troubleshoot-＞skip in some third party CI, but
not everything can be perfect in the world, again, it's a tradeoff.
And as to how to decide what feature is "main trend" feature, maybe we can have
a mechanism of voting, just as how we decide the core-functionality set.
Original Mail
Sender: ＜sean.mcgin...@gmx.com＞
To: ＜openstack-dev@lists.openstack.org＞
Date: 2017/05/03 21:35
Subject: Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests thebackend
specific feature?
On Wed, May 03, 2017 at 01:14:18PM +0000, Andrea Frittoli wrote:
＞
＞ ＞ ＞
＞
＞ ＞ Now that this is all in place, I think it's working well and I would like
＞ ＞ to see it continue that way. IMO, tempest proper should not have anything
＞ ＞ that isn't universally applicable to real world deployments. Not just for
＞ ＞ things like Ceph, but things like the manage/unmanage backend specific
＞ ＞ tests that were added and broke a large majority of third party CI.
＞ ＞
＞
＞ Is there a policy in Cinder that a backend must implement a certain set of
＞ APIs? If so we could think of testing only that set of APIs in Tempest, so
＞ that
＞ any app developer knows that he/she can rely on that minimum set of APIs.
＞
Yes, there is a minimum feature set that all backend drivers must support.
That base functionality can be found here [0] and here [1].
[0]
https://docs.openstack.org/developer/cinder/devref/drivers.html#core-functionality
[1]
https://github.com/openstack/cinder/blob/master/cinder/interface/volume_driver.py
The issue we've had with some of the tempest tests isn't actually around
whether the given backend supports the API. It's the way the API is used
that becomes a challenge.
I'll use the manage/unmanage snapshot one as an example. This is not one
of the required APIs, but many backends do support it. But the way this
API works is you are basically saying "manage this snapshot that you
identify as X", where "X" can be different things depending on the volume
driver being used. It's the storage device's native way of identifying
a snapshot, which may or may not be our Cinder snapshot ID.
So that test was added based on the way the LVM driver works for this.
That part is great, we get some more code coverage in the gate. But then
each volume driver that uses a different ID had to first a) start failing
CI, b) troubleshoot what happened to cause this new failure, c) reconfig
their CI to skip this test. Repeat cycle for each test added that does
something specific to how one or a small subset of backends work.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
