Re: [openstack-dev] [Ceilometer] [qa] [oslo] Declarative HTTP Tests
On Thu, 23 Oct 2014, Doug Hellmann wrote: Thanks for the feedback Doug, it's useful. WebTest isn’t quite what you’re talking about, but does provide a way to talk to a WSGI app from within a test suite rather simply. Can you expand a little on why “declarative” tests are better suited for this than the more usual sorts of tests we write? I'll add a bit more on why I think declarative tests are useful, but it basically comes down to explicit transparency of the on-the-wire HTTP requests and responses. At least within Ceilometer and at least for me, unittest form tests are very difficult to read as a significant portion of the action is somewhere in test setup or a super class. This may not have much impact on the effectiveness of the tests for computers, but it's a total killer of the effectiveness of the tests as tools for discovery by a developer who needs to make changes or, heaven forbid, is merely curious. I think we can agree that a more informed developer, via a more learnable codebase, is a good thing? I did look at WebTest and while it looks pretty good I don't particular care for its grammar. This might be seen as a frivolous point (because under the hood the same thing is happening, and its not this aspect that would be exposed the declarations) but calling a method on the `app` provided by WebTest feels different than using an http library to make a request of what appears to be a web server hosting the app. If people prefer WebTest over wsgi-intercept, that's fine, it's not something worth fighting about. The positions I'm taking in the spec, as you'll have seen from my responses to Eoghan, are trying to emphasize the focus on HTTP rather than the app itself. I think this could lead, eventually, to better HTTP APIs. I definitely don’t think the ceilometer team should build something completely new for this without a lot more detail in the spec about which projects on PyPI were evaluated and rejected as not meeting the requirements. If we do need/want something like this I would expect it to be built within the QA program. I don’t know if it’s appropriate to put it in tempestlib or if we need a completely new tool. I don't think anyone wants to build a new thing unless that turns out to be necessary, thus this thread. I'm hoping to get input from people who have thought about or explored this before. I'm hoping we can build on the shoulders of giants and all that. I'm also hoping to short circuit extensive personal research by collaborating with others who may have already done it before. I have an implementation of the concept that I've used in a previous project (linked from the spec) which could work as a starting point from which we could iterate but if there is something better out there I'd prefer to start there. So the questions from the original post still stand, all input welcome, please and thank you. * Is this a good idea? * Do other projects have similar ideas in progress? * Is this concept something for which a generic tool should be created _prior_ to implementation in an individual project? * Is there prior art? What's a good format? -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] [qa] [oslo] Declarative HTTP Tests
On 10/23/2014 06:27 AM, Chris Dent wrote: I've proposed a spec to Ceilometer https://review.openstack.org/#/c/129669/ for a suite of declarative HTTP tests that would be runnable both in gate check jobs and in local dev environments. There's been some discussion that this may be generally applicable and could be best served by a generic tool. My original assertion was let's make something work and then see if people like it but I thought I also better check with the larger world: * Is this a good idea? I think so * Do other projects have similar ideas in progress? Tempest faced a similar problem around negative tests in particular. We have code in tempest that automatically generates a series of negative test cases based on illegal variations of a schema. If you want to look at it the NegativeAutoTest class is probably a good place to start. We have discussed using a similar methodology for positive test cases but never did anything with that. Currently only a few of the previous negative tests have been replaced with auto-gen tests. In addition to the issue of how to represent the schema, the other major issue we encountered was the need to create resources used by the auto-generated tests and a way to integrate a resource description into the schema. We use json for the schema and hoped one day to be able to receive base schemas from the projects themselves. * Is this concept something for which a generic tool should be created _prior_ to implementation in an individual project? * Is there prior art? What's a good format? Marc Koderer and I did a lot of searching and asking folks if there was some python code that we could use as a starting point but in the end did not find anything. I do not have a list of what we considered and rejected. -David Thanks. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] [qa] [oslo] Declarative HTTP Tests
On Oct 24, 2014, at 6:58 AM, Chris Dent chd...@redhat.com wrote: On Thu, 23 Oct 2014, Doug Hellmann wrote: Thanks for the feedback Doug, it's useful. WebTest isn’t quite what you’re talking about, but does provide a way to talk to a WSGI app from within a test suite rather simply. Can you expand a little on why “declarative” tests are better suited for this than the more usual sorts of tests we write? I'll add a bit more on why I think declarative tests are useful, but it basically comes down to explicit transparency of the on-the-wire HTTP requests and responses. At least within Ceilometer and at least for me, unittest form tests are very difficult to read as a significant portion of the action is somewhere in test setup or a super class. This may not have much impact on the effectiveness of the tests for computers, but it's a total killer of the effectiveness of the tests as tools for discovery by a developer who needs to make changes or, heaven forbid, is merely curious. I think we can agree that a more informed developer, via a more learnable codebase, is a good thing? OK, at first I thought you were talking about writing out literal HTTP request/response sets, but looking at the example YAML file in [1] I see you’re doing something more abstract. I was worried about minor changes in a serialization library somewhere breaking a bunch of tests by changing their formatting in insignificant ways, but that shouldn’t be a problem if you’re testing the semantic contents rather than the literal contents of the response. [1] https://github.com/tiddlyweb/tiddlyweb/blob/master/test/httptest.yaml I did look at WebTest and while it looks pretty good I don't particular care for its grammar. This might be seen as a frivolous point (because under the hood the same thing is happening, and its not this aspect that would be exposed the declarations) but calling a method on the `app` provided by WebTest feels different than using an http library to make a request of what appears to be a web server hosting the app. If people prefer WebTest over wsgi-intercept, that's fine, it's not something worth fighting about. The positions I'm taking in the spec, as you'll have seen from my responses to Eoghan, are trying to emphasize the focus on HTTP rather than the app itself. I think this could lead, eventually, to better HTTP APIs. I’m not familiar with wsgi-intercept, so I can’t really comment on the difference there. I find WebTest tests to be reasonably easy to read, but since (as I understand it) I would write YAML files instead of Python tests, I’m not sure I care which library is used to build the tool as long as we don’t have to actually spin up a web server listening on a network port in order to run tests. I definitely don’t think the ceilometer team should build something completely new for this without a lot more detail in the spec about which projects on PyPI were evaluated and rejected as not meeting the requirements. If we do need/want something like this I would expect it to be built within the QA program. I don’t know if it’s appropriate to put it in tempestlib or if we need a completely new tool. I don't think anyone wants to build a new thing unless that turns out to be necessary, thus this thread. I'm hoping to get input from people who have thought about or explored this before. I'm hoping we can build on the shoulders of giants and all that. I'm also hoping to short circuit extensive personal research by collaborating with others who may have already done it before. I have an implementation of the concept that I've used in a previous project (linked from the spec) which could work as a starting point from which we could iterate but if there is something better out there I'd prefer to start there. So the questions from the original post still stand, all input welcome, please and thank you. * Is this a good idea? * Do other projects have similar ideas in progress? * Is this concept something for which a generic tool should be created _prior_ to implementation in an individual project? * Is there prior art? What's a good format? -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer] [qa] [oslo] Declarative HTTP Tests
I've proposed a spec to Ceilometer https://review.openstack.org/#/c/129669/ for a suite of declarative HTTP tests that would be runnable both in gate check jobs and in local dev environments. There's been some discussion that this may be generally applicable and could be best served by a generic tool. My original assertion was let's make something work and then see if people like it but I thought I also better check with the larger world: * Is this a good idea? * Do other projects have similar ideas in progress? * Is this concept something for which a generic tool should be created _prior_ to implementation in an individual project? * Is there prior art? What's a good format? Thanks. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] [qa] [oslo] Declarative HTTP Tests
On Oct 23, 2014, at 6:27 AM, Chris Dent chd...@redhat.com wrote: I've proposed a spec to Ceilometer https://review.openstack.org/#/c/129669/ for a suite of declarative HTTP tests that would be runnable both in gate check jobs and in local dev environments. There's been some discussion that this may be generally applicable and could be best served by a generic tool. My original assertion was let's make something work and then see if people like it but I thought I also better check with the larger world: * Is this a good idea? * Do other projects have similar ideas in progress? * Is this concept something for which a generic tool should be created _prior_ to implementation in an individual project? * Is there prior art? What's a good format? WebTest isn’t quite what you’re talking about, but does provide a way to talk to a WSGI app from within a test suite rather simply. Can you expand a little on why “declarative” tests are better suited for this than the more usual sorts of tests we write? I definitely don’t think the ceilometer team should build something completely new for this without a lot more detail in the spec about which projects on PyPI were evaluated and rejected as not meeting the requirements. If we do need/want something like this I would expect it to be built within the QA program. I don’t know if it’s appropriate to put it in tempestlib or if we need a completely new tool. Doug Thanks. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
