Re: [openstack-dev] [nova] [qa] api schema validation pattern changes
Hi David, 2014/1/13 David Kranz dkr...@redhat.com: On 01/12/2014 10:14 PM, Matthew Treinish wrote: snip Last, is a question, is it possible to currently run the full API an build a json schema for it all? Or to query these validating schemas? We *really* want that over in tempest, so we can completely drop the manual creation of negative testing, and just fuzz known bad against the schema definitions. Sorry, I'm not sure I understand this question correctly. We need to define schemas for each API with the separated schema patches. It is impossible to define one schema for all APIs. Hi David, Marc, I guess the negative test generator of Tempest would need each API definition. Glance can provide API definitions through API with jsonschema format, but Nova does not have such feature. We need to port these API schema from Nova to Tempest, I guess. right? As I understand things after reviewing the first versions of the negative test generator patch is that we have to hard code the schema into a file (right now it's in the test file, but eventually it'll be an external input file). One of my issues with doing that is it's highly manual process, essentially a copy and paste from the nova tree. I think what we're looking for from this jsonschema validation work is an API which we can query the API and get the jsonschema definitions; similar to what the glance API offers. I looked at the glance schema api and it seemed to just return the schema for the json returned by the call, not for the arguments to the request. Am I wrong about that? I think you are right. Glance feature provides the schemas of request body only. That does not contain the http method type and the endpoint. Additionally, the schema for the request json in these patches is not enough for the negative test generator. The schema for negative generation also needs the http method type and a description of the resources that are part of the url of the request. Thanks for your explanation, I see. Current Nova's validation feature does not seem enough for the negative test generator. It would be necessary to hard-code API schemas in Tempest. Thanks Ken'ichi Ohmichi --- The negative test patch in the shared fork https://github.com/mkoderer/tempest/tree/feature/negative_tests defines such a schema. The tempest patch will be updated from this fork on Monday. I wouldn't advocate making the negative test generator be fully dynamic for the same reason we don't autodetect which API versions and extensions/features are enabled in tempest, but rather rely on the config file. But, instead have an additional tool which could query the schema for all the endpoints in nova and generate an input file for the negative test generator. That way we'll still catch breaking API changes in the gate, but it's not a manual process to update the input file with the schema definitions in tempest when there is a breaking API change. (Which hopefully should almost never happen) I agree, it would be ideal if there were a way for tempest to grab the appropriate schemas from somewhere else rather than hard-coding them in tempest itself. But as far as I can see, most of the services don't even have json schemas defined. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [qa] api schema validation pattern changes
Von: David Kranz [mailto:dkr...@redhat.com] Gesendet: Montag, 13. Januar 2014 05:04 An: openstack-dev@lists.openstack.org Betreff: Re: [openstack-dev] [nova] [qa] api schema validation pattern changes On 01/12/2014 10:14 PM, Matthew Treinish wrote: snip Last, is a question, is it possible to currently run the full API an build a json schema for it all? Or to query these validating schemas? We *really* want that over in tempest, so we can completely drop the manual creation of negative testing, and just fuzz known bad against the schema definitions. Sorry, I'm not sure I understand this question correctly. We need to define schemas for each API with the separated schema patches. It is impossible to define one schema for all APIs. Hi David, Marc, I guess the negative test generator of Tempest would need each API definition. Glance can provide API definitions through API with jsonschema format, but Nova does not have such feature. We need to port these API schema from Nova to Tempest, I guess. right? As I understand things after reviewing the first versions of the negative test generator patch is that we have to hard code the schema into a file (right now it's in the test file, but eventually it'll be an external input file). One of my issues with doing that is it's highly manual process, essentially a copy and paste from the nova tree. I think what we're looking for from this jsonschema validation work is an API which we can query the API and get the jsonschema definitions; similar to what the glance API offers. I looked at the glance schema api and it seemed to just return the schema for the json returned by the call, not for the arguments to the request. Am I wrong about that? Additionally, the schema for the request json in these patches is not enough for the negative test generator. The schema for negative generation also needs the http method type and a description of the resources that are part of the url of the request. The negative test patch in the shared fork https://github.com/mkoderer/tempest/tree/feature/negative_tests defines such a schema. The tempest patch will be updated from this fork on Monday. I wouldn't advocate making the negative test generator be fully dynamic for the same reason we don't autodetect which API versions and extensions/features are enabled in tempest, but rather rely on the config file. But, instead have an additional tool which could query the schema for all the endpoints in nova and generate an input file for the negative test generator. That way we'll still catch breaking API changes in the gate, but it's not a manual process to update the input file with the schema definitions in tempest when there is a breaking API change. (Which hopefully should almost never happen) I agree, it would be ideal if there were a way for tempest to grab the appropriate schemas from somewhere else rather than hard-coding them in tempest itself. But as far as I can see, most of the services don't even have json schemas defined. I think at the end the hard coded json schema should be optional. If the endpoint supports an export of the schema it simply takes it and put the result automatically into the description of the negative test. For me this is a second step since currently we already duplicating somehow this in our manual negative tests. What we would need is at least one interface that exposes the schema. Marc ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [qa] api schema validation pattern changes
On 01/12/2014 10:14 PM, Matthew Treinish wrote: snip Last, is a question, is it possible to currently run the full API an build a json schema for it all? Or to query these validating schemas? We *really* want that over in tempest, so we can completely drop the manual creation of negative testing, and just fuzz known bad against the schema definitions. Sorry, I'm not sure I understand this question correctly. We need to define schemas for each API with the separated schema patches. It is impossible to define one schema for all APIs. Hi David, Marc, I guess the negative test generator of Tempest would need each API definition. Glance can provide API definitions through API with jsonschema format, but Nova does not have such feature. We need to port these API schema from Nova to Tempest, I guess. right? As I understand things after reviewing the first versions of the negative test generator patch is that we have to hard code the schema into a file (right now it's in the test file, but eventually it'll be an external input file). One of my issues with doing that is it's highly manual process, essentially a copy and paste from the nova tree. I think what we're looking for from this jsonschema validation work is an API which we can query the API and get the jsonschema definitions; similar to what the glance API offers. I looked at the glance schema api and it seemed to just return the schema for the json returned by the call, not for the arguments to the request. Am I wrong about that? Additionally, the schema for the request json in these patches is not enough for the negative test generator. The schema for negative generation also needs the http method type and a description of the resources that are part of the url of the request. The negative test patch in the shared fork https://github.com/mkoderer/tempest/tree/feature/negative_tests defines such a schema. The tempest patch will be updated from this fork on Monday. I wouldn't advocate making the negative test generator be fully dynamic for the same reason we don't autodetect which API versions and extensions/features are enabled in tempest, but rather rely on the config file. But, instead have an additional tool which could query the schema for all the endpoints in nova and generate an input file for the negative test generator. That way we'll still catch breaking API changes in the gate, but it's not a manual process to update the input file with the schema definitions in tempest when there is a breaking API change. (Which hopefully should almost never happen) I agree, it would be ideal if there were a way for tempest to grab the appropriate schemas from somewhere else rather than hard-coding them in tempest itself. But as far as I can see, most of the services don't even have json schemas defined. -Daivd ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
