Re: [openstack-dev] [qa][nova] Ownership and path to schema definitions

2014-03-06 Thread David Kranz 

On 03/05/2014 10:25 PM, Christopher Yeoh wrote:

On Tue, 04 Mar 2014 13:31:07 -0500
David Kranz dkr...@redhat.com wrote:

I think it would be a good time to have at least an initial
discussion about the requirements for theses schemas and where they
will live. The next step in tempest around this is to replace the
existing negative test files with auto-gen versions, and most of the
work in doing that is to define the schemas.

The tempest framework needs to know the http method, url part,
expected error codes, and payload description. I believe only the
last is covered by the current nova schema definitions, with the
others being some kind of attribute or data associated with the
method that is doing the validation. Ideally the information being
used to do the validation could be auto-converted to a more general
schema that could be used by tempest. I'm interested in what folks
have to say about this and especially from the folks who are core
members of both nova and tempest. See below for one example (note
that the tempest generator does not yet handle pattern).


So as you've seen a lot of what is wanted for the tempest framework is
implicitly known already within the method context which is why its not
again explicitly stated in the schema. Not actually having thought
about it a lot, but I suspect the expected errors decorator is
something that would fit just as well in the validation framework
however.

Some of the other stuff such as url part, descriptions etc, not so much
as it would be purely duplicate information that would get out of date.
However for documentation auto generation it is something that we do
also want to have available in an automated fashion.  I did a bit of
exploration early in Icehouse in generating this within the context of
api samples tests where we have access to this sort of stuff and I think
together we'd have all the info we need, I'm just not sure mashing them
together is the right way to do it.
To be clear, I was advocating for some solution where all the 
information needed to create negative tests, api docs, (part of client 
libraries?), etc. could be derived from the source code. The pieces of 
information could come from explicit schema definitions, scanning the 
code, python introspection. I have done this sort of thing in other 
languages but am not enough of a Python wiz to say exactly how this 
could work. Certainly the schema should not have duplicate information.

And from the documentation point of view we need to have a bit of a
think about whether doc strings on methods should be the canonical way
we produce descriptional information about API methods. One hand its
appealing, on the other hand they tend to be not very useful or very
internals Nova focussed. But we could get much better at it.
Agreed. IMO, doc strings on methods have far more benefits (automation, 
not getting out of sync with code) than real costs (non-coders have to 
edit the code to review or improve the docs). This works best when 
developers write at least drafts of doc strings. The REST API case is a 
little more tricky because it does not map as directly to methods.


 -David


Short version - yea I think we want to get to the point where tempest
doesn't generate these manually. But I'm not sure about how we
should do it.

Chris


   -David

  From nova:

get_console_output = {
  'type': 'object',
  'properties': {
  'get_console_output': {
  'type': 'object',
  'properties': {
  'length': {
  'type': ['integer', 'string'],
  'minimum': 0,
  'pattern': '^[0-9]+$',
  },
  },
  'additionalProperties': False,
  },
  },
  'required': ['get_console_output'],
  'additionalProperties': False,
}

  From tempest:

{
  name: get-console-output,
  http-method: POST,
  url: servers/%s/action,
  resources: [
  {name:server, expected_result: 404}
  ],
  json-schema: {
  type: object,
  properties: {
  os-getConsoleOutput: {
  type: object,
  properties: {
  length: {
  type: [integer, string],
  minimum: 0
  }
  }
  }
  },
  additionalProperties: false
  }
}


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][nova] Ownership and path to schema definitions

2014-03-06 Thread Jay Pipes 
On Thu, 2014-03-06 at 13:55 +1030, Christopher Yeoh wrote:
 On Tue, 04 Mar 2014 13:31:07 -0500
 David Kranz dkr...@redhat.com wrote:
  I think it would be a good time to have at least an initial
  discussion about the requirements for theses schemas and where they
  will live. The next step in tempest around this is to replace the
  existing negative test files with auto-gen versions, and most of the
  work in doing that is to define the schemas.
  
  The tempest framework needs to know the http method, url part,
  expected error codes, and payload description. I believe only the
  last is covered by the current nova schema definitions, with the
  others being some kind of attribute or data associated with the
  method that is doing the validation. Ideally the information being
  used to do the validation could be auto-converted to a more general
  schema that could be used by tempest. I'm interested in what folks
  have to say about this and especially from the folks who are core
  members of both nova and tempest. See below for one example (note
  that the tempest generator does not yet handle pattern).
 
 So as you've seen a lot of what is wanted for the tempest framework is
 implicitly known already within the method context which is why its not
 again explicitly stated in the schema. Not actually having thought
 about it a lot, but I suspect the expected errors decorator is
 something that would fit just as well in the validation framework
 however. 
 
 Some of the other stuff such as url part, descriptions etc, not so much
 as it would be purely duplicate information that would get out of date.
 However for documentation auto generation it is something that we do
 also want to have available in an automated fashion.  I did a bit of
 exploration early in Icehouse in generating this within the context of
 api samples tests where we have access to this sort of stuff and I think
 together we'd have all the info we need, I'm just not sure mashing them
 together is the right way to do it.

JSON-Home is a perfect complement to JSONSchema in this regard. You
would expose the object model that underpins the request payload
validation using JSONSchema. And you would expose the REST API contract
(things like what HTTP methods are allowed, what parameters are allowed
for a method, what content-types are accepted, etc) using JSON-Home.

See Marconi for an example of using JSON-Home for API discovery:

https://github.com/openstack/marconi/blob/master/marconi/queues/transport/wsgi/v1_1/homedoc.py

See Glance for an example of JSONSchema for object model discovery:

https://github.com/openstack/glance/blob/master/glance/api/v2/image_members.py#L264

Best,
-jay

 And from the documentation point of view we need to have a bit of a
 think about whether doc strings on methods should be the canonical way
 we produce descriptional information about API methods. One hand its
 appealing, on the other hand they tend to be not very useful or very
 internals Nova focussed. But we could get much better at it.
 
 Short version - yea I think we want to get to the point where tempest
 doesn't generate these manually. But I'm not sure about how we
 should do it.
 
 Chris
 
  
-David
  
   From nova:
  
  get_console_output = {
   'type': 'object',
   'properties': {
   'get_console_output': {
   'type': 'object',
   'properties': {
   'length': {
   'type': ['integer', 'string'],
   'minimum': 0,
   'pattern': '^[0-9]+$',
   },
   },
   'additionalProperties': False,
   },
   },
   'required': ['get_console_output'],
   'additionalProperties': False,
  }
  
   From tempest:
  
  {
   name: get-console-output,
   http-method: POST,
   url: servers/%s/action,
   resources: [
   {name:server, expected_result: 404}
   ],
   json-schema: {
   type: object,
   properties: {
   os-getConsoleOutput: {
   type: object,
   properties: {
   length: {
   type: [integer, string],
   minimum: 0
   }
   }
   }
   },
   additionalProperties: false
   }
  }
  
  
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][nova] Ownership and path to schema definitions

2014-03-05 Thread Kenichi Oomichi 

Hi David,

That is an interesting idea, I'd like to join in :-)

 -Original Message-
 From: David Kranz [mailto:dkr...@redhat.com]
 Sent: Wednesday, March 05, 2014 3:31 AM
 To: OpenStack Development Mailing List
 Subject: [openstack-dev] [qa][nova] Ownership and path to schema definitions
 
 Given that
 
 1. there is an ongoing api discussion in which using json schemas is an
 important part
 2. tempest now has a schema-based auto-generate feature for negative tests
 
 I think it would be a good time to have at least an initial discussion
 about the requirements for theses schemas and where they will live.
 The next step in tempest around this is to replace the existing negative
 test files with auto-gen versions, and most of the work in doing that
 is to define the schemas.
 
 The tempest framework needs to know the http method, url part, expected
 error codes, and payload description. I believe only the last is covered
 by the current nova schema definitions, with the others being some kind
 of attribute or data associated with the method that is doing the
 validation.

Now the schema definitions of Nova cover *request* payload description only.
If Tempest will verify *response* payload in many cases(I hope that), it
would be good to add the schema definitions of response into Tempest.


 Ideally the information being used to do the validation
 could be auto-converted to a more general schema that could be used by
 tempest. I'm interested in what folks have to say about this and
 especially from the folks who are core members of both nova and tempest.
 See below for one example (note that the tempest generator does not yet
 handle pattern).
 
   -David
 
  From nova:
 
 get_console_output = {
  'type': 'object',
  'properties': {
  'get_console_output': {
  'type': 'object',
  'properties': {
  'length': {
  'type': ['integer', 'string'],
  'minimum': 0,
  'pattern': '^[0-9]+$',
  },
  },
  'additionalProperties': False,
  },
  },
  'required': ['get_console_output'],
  'additionalProperties': False,
 }
 
  From tempest:
 
 {
  name: get-console-output,
  http-method: POST,
  url: servers/%s/action,
  resources: [
  {name:server, expected_result: 404}
  ],

The above name, http-method and url would be useful for API
documentation also. So I feel it would be great for Tempest and
API doc if Nova's schemas contain their info. The above resources
also would be useful if we can define status code of successful access.


Thanks
Ken'ichi Ohmichi

---

  json-schema: {
  type: object,
  properties: {
  os-getConsoleOutput: {
  type: object,
  properties: {
  length: {
  type: [integer, string],
  minimum: 0
  }
  }
  }
  },
  additionalProperties: false
  }
 }
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][nova] Ownership and path to schema definitions

2014-03-05 Thread Christopher Yeoh 
On Tue, 04 Mar 2014 13:31:07 -0500
David Kranz dkr...@redhat.com wrote:
 I think it would be a good time to have at least an initial
 discussion about the requirements for theses schemas and where they
 will live. The next step in tempest around this is to replace the
 existing negative test files with auto-gen versions, and most of the
 work in doing that is to define the schemas.
 
 The tempest framework needs to know the http method, url part,
 expected error codes, and payload description. I believe only the
 last is covered by the current nova schema definitions, with the
 others being some kind of attribute or data associated with the
 method that is doing the validation. Ideally the information being
 used to do the validation could be auto-converted to a more general
 schema that could be used by tempest. I'm interested in what folks
 have to say about this and especially from the folks who are core
 members of both nova and tempest. See below for one example (note
 that the tempest generator does not yet handle pattern).


So as you've seen a lot of what is wanted for the tempest framework is
implicitly known already within the method context which is why its not
again explicitly stated in the schema. Not actually having thought
about it a lot, but I suspect the expected errors decorator is
something that would fit just as well in the validation framework
however. 

Some of the other stuff such as url part, descriptions etc, not so much
as it would be purely duplicate information that would get out of date.
However for documentation auto generation it is something that we do
also want to have available in an automated fashion.  I did a bit of
exploration early in Icehouse in generating this within the context of
api samples tests where we have access to this sort of stuff and I think
together we'd have all the info we need, I'm just not sure mashing them
together is the right way to do it.

And from the documentation point of view we need to have a bit of a
think about whether doc strings on methods should be the canonical way
we produce descriptional information about API methods. One hand its
appealing, on the other hand they tend to be not very useful or very
internals Nova focussed. But we could get much better at it.

Short version - yea I think we want to get to the point where tempest
doesn't generate these manually. But I'm not sure about how we
should do it.

Chris

 
   -David
 
  From nova:
 
 get_console_output = {
  'type': 'object',
  'properties': {
  'get_console_output': {
  'type': 'object',
  'properties': {
  'length': {
  'type': ['integer', 'string'],
  'minimum': 0,
  'pattern': '^[0-9]+$',
  },
  },
  'additionalProperties': False,
  },
  },
  'required': ['get_console_output'],
  'additionalProperties': False,
 }
 
  From tempest:
 
 {
  name: get-console-output,
  http-method: POST,
  url: servers/%s/action,
  resources: [
  {name:server, expected_result: 404}
  ],
  json-schema: {
  type: object,
  properties: {
  os-getConsoleOutput: {
  type: object,
  properties: {
  length: {
  type: [integer, string],
  minimum: 0
  }
  }
  }
  },
  additionalProperties: false
  }
 }
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev