Re: [openstack-dev] [qa][tempest] Where to do response body validation
On Thu, 13 Mar 2014 08:17:50 +0200 Valeriy Ponomaryov wrote: > I disagree to moving this logic to "tempest/services/*". The idea of > these modules - assemble requests and return responses. Testing and > verification should be wrapped over it. Either base class or tests, > it depends on situation... > So I can't think of situation where the Nova API would be returning a success code and the format of the response (which is all we'd really be checking) is not valid. We can test at a higher level, but it means calling the validation from quite a few spots (everywhere the client is called) rather than just in one spot. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest] Where to do response body validation
On Thu, 13 Mar 2014 06:40:58 + Kenichi Oomichi wrote: > > Hi Chris, > > Thank you for picking it up, > > > -Original Message- > > From: Christopher Yeoh [mailto:cbky...@gmail.com] > > Sent: Thursday, March 13, 2014 1:56 PM > > To: OpenStack Development Mailing List > > Subject: [openstack-dev] [qa][tempest] Where to do response body > > validation > > > > The new tempest body response validation is being added to > > individual testcases. See this as an example: > > > > https://review.openstack.org/#/c/78149 > > > > After having a look at https://review.openstack.org/#/c/80174/ > > I'm now thinking that perhaps we should be doing the response > > validation in the tempest/services/compute classes. And only check > > the response body if the status code is a success code (and then > > check that it is an appropriate success code). > > > > I think this will lead to fewer changes needed in the end as the > > response body checking will not needed to be added to individual > > tests. > > > > There may be some complications with handling extensions, but I > > think they are all implement backwards compatible behaviour so > > should be ok. > > > > Anyone have any thoughts about this alternative approach? > > I like the above idea that the response body validation will be > operated in REST client. Tempest will be able to check response body > anytime and reduce the test code. > > One concern is that Nova API returns different response body when > admin user or not. I'd like to check response body containing > attributes what admin user can get. For example, "get server info" > API returns a response including "OS-EXT-STS" and "OS-EXT-SRV-ATT" > attributes when an admin user. Yes that's true, I think that's a case of extensions and policy changing what is returned. > > So how about operating the basic validation(without special > attributes) only in each REST client? > The the special validation(such as the above admin info) would be > operated in each test. The schema size for the special cases could be > reduced. That sounds reasonable to me. We'd just check for the additional parameters that we expect to be present. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest] Where to do response body validation
Hi Chris, Thank you for picking it up, > -Original Message- > From: Christopher Yeoh [mailto:cbky...@gmail.com] > Sent: Thursday, March 13, 2014 1:56 PM > To: OpenStack Development Mailing List > Subject: [openstack-dev] [qa][tempest] Where to do response body validation > > The new tempest body response validation is being added to individual > testcases. See this as an example: > > https://review.openstack.org/#/c/78149 > > After having a look at https://review.openstack.org/#/c/80174/ > I'm now thinking that perhaps we should be doing the response validation > in the tempest/services/compute classes. And only check the > response body if the status code is a success code (and then check that > it is an appropriate success code). > > I think this will lead to fewer changes needed in the end as the > response body checking will not needed to be added to individual tests. > > There may be some complications with handling extensions, but I think > they are all implement backwards compatible behaviour so should be ok. > > Anyone have any thoughts about this alternative approach? I like the above idea that the response body validation will be operated in REST client. Tempest will be able to check response body anytime and reduce the test code. One concern is that Nova API returns different response body when admin user or not. I'd like to check response body containing attributes what admin user can get. For example, "get server info" API returns a response including "OS-EXT-STS" and "OS-EXT-SRV-ATT" attributes when an admin user. So how about operating the basic validation(without special attributes) only in each REST client? The the special validation(such as the above admin info) would be operated in each test. The schema size for the special cases could be reduced. Thanks Ken'ichi Ohmichi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][tempest] Where to do response body validation
I disagree to moving this logic to "tempest/services/*". The idea of these modules - assemble requests and return responses. Testing and verification should be wrapped over it. Either base class or tests, it depends on situation... -- Kind Regards Valeriy Ponomaryov On Thu, Mar 13, 2014 at 6:55 AM, Christopher Yeoh wrote: > Hi, > > The new tempest body response validation is being added to individual > testcases. See this as an example: > > https://review.openstack.org/#/c/78149 > > After having a look at https://review.openstack.org/#/c/80174/ > I'm now thinking that perhaps we should be doing the response validation > in the tempest/services/compute classes. And only check the > response body if the status code is a success code (and then check that > it is an appropriate success code). > > I think this will lead to fewer changes needed in the end as the > response body checking will not needed to be added to individual tests. > > There may be some complications with handling extensions, but I think > they are all implement backwards compatible behaviour so should be ok. > > Anyone have any thoughts about this alternative approach? > > Regards, > > Chris > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
