Hi David, 2014/1/13 David Kranz <dkr...@redhat.com>: > On 01/12/2014 10:14 PM, Matthew Treinish wrote: >> >> <snip> >>>> >>>> Last, is a question, is it possible to currently run the full API an >>>> build a >>>> json schema for it all? Or to query these validating schemas? We >>>> *really* >>>> want that over in tempest, so we can completely drop the manual creation >>>> of >>>> negative testing, and just fuzz known bad against the schema >>>> definitions. >>> >>> Sorry, I'm not sure I understand this question correctly. >>> We need to define schemas for each API with the separated schema patches. >>> It is impossible to define one schema for all APIs. >>> >>> >>> Hi David, Marc, >>> >>> I guess the negative test generator of Tempest would need each API >>> definition. >>> Glance can provide API definitions through API with jsonschema format, >>> but >>> Nova does not have such feature. >>> We need to port these API schema from Nova to Tempest, I guess. right? >>> >> As I understand things after reviewing the first versions of the negative >> test >> generator patch is that we have to hard code the schema into a file (right >> now >> it's in the test file, but eventually it'll be an external input file). >> One of >> my issues with doing that is it's highly manual process, essentially a >> copy and >> paste from the nova tree. I think what we're looking for from this >> jsonschema >> validation work is an API which we can query the API and get the >> jsonschema >> definitions; similar to what the glance API offers. > > I looked at the glance schema api and it seemed to just return the schema > for the json returned by the call, not for the arguments to the request. Am > I wrong about that?
I think you are right. Glance feature provides the schemas of request body only. That does not contain the http method type and the endpoint. > Additionally, the schema for the request json in these patches is not enough > for the negative test generator. The schema for negative generation also > needs the http method type and a description of the resources that are part > of the url of the request. Thanks for your explanation, I see. Current Nova's validation feature does not seem enough for the negative test generator. It would be necessary to hard-code API schemas in Tempest. Thanks Ken'ichi Ohmichi --- > The negative test patch in the shared fork > https://github.com/mkoderer/tempest/tree/feature/negative_tests defines such > a schema. The tempest patch will be updated from this fork on Monday. > >> >> I wouldn't advocate making the negative test generator be fully dynamic >> for the >> same reason we don't autodetect which API versions and extensions/features >> are >> enabled in tempest, but rather rely on the config file. But, instead have >> an >> additional tool which could query the schema for all the endpoints in nova >> and >> generate an input file for the negative test generator. That way we'll >> still >> catch breaking API changes in the gate, but it's not a manual process to >> update >> the input file with the schema definitions in tempest when there is a >> breaking >> API change. (Which hopefully should almost never happen) > > I agree, it would be ideal if there were a way for tempest to grab the > appropriate schemas from somewhere else rather than hard-coding them in > tempest itself. But as far as I can see, most of the services don't even > have json schemas defined. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
