On Wed, May 03, 2017 at 11:09:32PM -0400, Monty Taylor wrote: > On 05/02/2017 11:49 AM, Sean McGinnis wrote: > > On Tue, May 02, 2017 at 03:36:20PM +0200, Jordan Pittier wrote: > > > On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann <[email protected]> > > > wrote: > > > > > > > In Cinder, there are many features/APIs which are backend specific and > > > > will return 405 or 501 if same is not implemented on any backend [1]. > > > > If such tests are implemented in Tempest, then it will break some gate > > > > where that backend job is voting. like ceph job in glance_store gate. > > > > > > > > There been many such cases recently where ceph jobs were broken due to > > > > such tests and recently it is for force-delete backup feature[2]. > > > > Reverting force-delete tests in [3]. To resolve such cases at some > > > > extend, Jon is going to add a white/black list of tests which can run > > > > on ceph job [4] depends on what all feature ceph implemented. But this > > > > does not resolve it completely due to many reason like > > > > 1. External use of Tempest become difficult where user needs to know > > > > what all tests to skip for which backend > > > > 2. Tempest tests become too specific to backend. > > > > > > > > Now there are few options to resolve this: > > > > 1. Tempest should not tests such API/feature which are backend > > > > specific like mentioned by api-ref like[1]. > > > > > > > So basically, if one of the 50 Cinder driver doesn't support a feature, we > > > should never test that feature ? What about the 49 other drivers ? If a > > > feature exists and can be tested in the Gate (with whatever default > > > config/driver is shipped) then I think we should test it. > > > > > 50? Over 100 as of Ocata. > > > > Well, is tempest's purpose in life to provide complete gate test coverage, > > or is tempest's purpose in life to give operators a tool to validate that > > their deployment is working as expected? > > I'd actually like to suggest that such a scenario actually points out a > thing that is ultimately potential pain passed to the end user in the real > world, so this question about what/how to test this in tempest is a good one > to have. > > If there is a feature which is only provisionally available depending on the > backend driver such that it's hard to test in tempest without an out of band > config - then it's a feature that a user will have no clue whether it works > on a given cloud. > > As we find these, I'd love it if we could expose discovery in the API for > viability of the feature. Like: > > GET /capabilities > > { > "capabilities": { > "has_force_delete": true > } > } > > (I know we've talked about that concept generally, but this is a specific > example) > > If such a thing existed, then the user can know whether they can use a thing > .. and so can tempest. A tempest test to validate force_delete working could > check the capability reported by the API and validate that if the API says > "true" that the feature work as expected, and if it says "false" validate > that attempting to call it returns a 405 (or whatever is appropriate) > > Ultimately, every config we need to feed to tempest is potentially a place > where an end user is unable to know whether or not to expect a call to work > - and an opportunity for us to provide our API consumers with a richer > experience.
Heh, well I've been saying all of these things for years. In fact I got so tired of repeating it all the time I just put it in the tempest configuration guide: (although I remember it being a lot snarkier) https://docs.openstack.org/developer/tempest/configuration.html#service-feature-configuration So, I'll be very happy when the capabilities work actually becomes a thing you can use. But, it feels like we've been talking about around the problem for a very long time... -Matt Treinish > > > In attempting to do things in the past, I've received push back based on > > the argument that it was the latter. For this reason, in-tree tempest tests > > were added to Cinder to give us a way to get better test coverage for our > > own sake. > > > > Now that this is all in place, I think it's working well and I would like > > to see it continue that way. IMO, tempest proper should not have anything > > that isn't universally applicable to real world deployments. Not just for > > things like Ceph, but things like the manage/unmanage backend specific > > tests that were added and broke a large majority of third party CI. > > > > Backend specific things should not be part of tempest in my opinion. We > > should cover those things through in-tree tempest plugins and our own > > testing. > > > > > > > 2. Tempest test can be disabled/skip based on backend. - This is not > > > > good idea as it increase config options and overhead of setting those. > > > > > > > Using regex and blacklist, any 3rd party CI can skip any test based on the > > > test ID. Without introducing a config flag. See: > > > https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871 > > > > > > > > > > 3. Tempest test can verify behavior with if else condition as per > > > > backend. This is bad idea and lose the test strength. > > > > > > > Yeah, that's bad. > > > > > > > > > > > IMO options 1 is better options. More feedback are welcome. > > > > > > > > > > ..1 https://developer.openstack.org/api-ref/block-storage/v3/? > > > > expanded=force-delete-a-backup-detail#force-delete-a-backup > > > > ..2 https://bugs.launchpad.net/glance/+bug/1687538 > > > > ..3 https://review.openstack.org/#/c/461625/ > > > > ..4 http://lists.openstack.org/pipermail/openstack-dev/2017- > > > > April/115229.html > > > > > > > > -gmann
signature.asc
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
