Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-04-21 Thread Everett Toews
On Apr 20, 2015, at 2:45 PM, Chris Dent chd...@redhat.com wrote: I wanted to make a quick update on the latest happenings with gabbi[0], the tool I've created to do declarative testing of OpenStack APIs (starting with Ceilometer and Gnocchi). * Jay Pipes and I are doing a presentation API

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-04-20 Thread Chris Dent
I wanted to make a quick update on the latest happenings with gabbi[0], the tool I've created to do declarative testing of OpenStack APIs (starting with Ceilometer and Gnocchi). * Jay Pipes and I are doing a presentation API Matters at summit. The latter of half of that will be me noodling

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-03-03 Thread Chris Dent
On Tue, 3 Mar 2015, Steve Baker wrote: This looks very useful, I'd like to use this in the heat functional tests job. Awesome, let me know if you need any assistance getting started. Is it possible to write tests which do a POST/PUT then a loop of GETs until some condition is met (a

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-03-02 Thread Steve Baker
On 03/03/15 00:56, Chris Dent wrote: I (and a few others) have been using gabbi[1] for a couple of months now and it has proven very useful and evolved a bit so I thought it would be worthwhile to followup my original message and give an update. Some recent reviews[1] give a sample of how it

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-03-02 Thread Chris Dent
I (and a few others) have been using gabbi[1] for a couple of months now and it has proven very useful and evolved a bit so I thought it would be worthwhile to followup my original message and give an update. Some recent reviews[1] give a sample of how it can be used to validate an existing API

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-03-02 Thread Jay Pipes
On 03/02/2015 03:56 AM, Chris Dent wrote: I (and a few others) have been using gabbi[1] for a couple of months now and it has proven very useful and evolved a bit so I thought it would be worthwhile to followup my original message and give an update. Some recent reviews[1] give a sample of how

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-18 Thread Jay Pipes
On 01/12/2015 02:20 PM, Chris Dent wrote: After some discussion with Sean Dague and a few others it became clear that it would be a good idea to introduce a new tool I've been working on to the list to get a sense of its usefulness generally, work towards getting it into global requirements,

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-13 Thread Chris Dent
On Mon, 12 Jan 2015, Sean Dague wrote: I think it's important to look at this in the narrower context, we're not testing full environments here, this is custom crafting HTTP req / resp in a limited context to make sure components are completing a contract. Yes, exactly. In fact one of the

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Eoghan Glynn
After some discussion with Sean Dague and a few others it became clear that it would be a good idea to introduce a new tool I've been working on to the list to get a sense of its usefulness generally, work towards getting it into global requirements, and get the documentation fleshed out so

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Dean Troyer
Thanks for this Chris, I'm hoping to get my fingers dirty with it Real Soon Now. On Mon, Jan 12, 2015 at 1:54 PM, Eoghan Glynn egl...@redhat.com wrote: I'd be interested in hearing the api-wg viewpoint, specifically whether that working group intends to recommend any best practices around the

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Sean Dague
On 01/12/2015 03:11 PM, Dean Troyer wrote: Thanks for this Chris, I'm hoping to get my fingers dirty with it Real Soon Now. On Mon, Jan 12, 2015 at 1:54 PM, Eoghan Glynn egl...@redhat.com mailto:egl...@redhat.com wrote: I'd be interested in hearing the api-wg viewpoint, specifically

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Sean Dague
On 01/12/2015 03:18 PM, Boris Pavlovic wrote: Chris, The Idea is brilliant. I may steal it! =) But there are some issues that will be faced: 1) Using as a base unittest: python -m subunit.run discover -f gabbi | subunit2pyunit So rally team won't be able to reuse it for

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Boris Pavlovic
Sean, I definitely like the direction that gabbi seems to be headed. It feels like a much cleaner version of what nova tried to do with API samples. As long as multiple projects think this is an interesting direction, I think it's probably fine to add it to global-requirements and let them

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Gregory Haynes
Excerpts from Chris Dent's message of 2015-01-12 19:20:18 +: After some discussion with Sean Dague and a few others it became clear that it would be a good idea to introduce a new tool I've been working on to the list to get a sense of its usefulness generally, work towards getting it

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Chris Dent
On Mon, 12 Jan 2015, Gregory Haynes wrote: Awesome! I was discussing trying to add extensions to RAML[1] so we could do something like this the other day. Is there any reason you didnt use an existing modeling language like this? Glad you like it. I chose to go with my own model in the YAML

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Boris Pavlovic
Chris, The Idea is brilliant. I may steal it! =) But there are some issues that will be faced: 1) Using as a base unittest: python -m subunit.run discover -f gabbi | subunit2pyunit So rally team won't be able to reuse it for load testing (if we directly integrate it) because we will have

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Chris Dent
On Tue, 13 Jan 2015, Boris Pavlovic wrote: The Idea is brilliant. I may steal it! =) Feel free. But there are some issues that will be faced: 1) Using as a base unittest: python -m subunit.run discover -f gabbi | subunit2pyunit So rally team won't be able to reuse it for load testing

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Boris Pavlovic
Hi Chris, If there's sufficient motivation and time it might make sense to separate the part of gabbi that builds TestCases from the part that runs (and evaluates) HTTP requests and responses. If that happens then integration with tools like Rally and runners is probably possible. Having

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Sean Dague
On 01/12/2015 05:00 PM, Boris Pavlovic wrote: Hi Chris, If there's sufficient motivation and time it might make sense to separate the part of gabbi that builds TestCases from the part that runs (and evaluates) HTTP requests and responses. If that happens then integration

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Anne Gentle
On Mon, Jan 12, 2015 at 1:20 PM, Chris Dent chd...@redhat.com wrote: After some discussion with Sean Dague and a few others it became clear that it would be a good idea to introduce a new tool I've been working on to the list to get a sense of its usefulness generally, work towards getting

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Chris Dent
On Mon, 12 Jan 2015, Anne Gentle wrote: I'm interested, sure. What did you use to write the HTTP tests, as in, what was the source of truth for what the requests and responses should be? That is an _extremely_ good question and one I really struggled with as I started integrating gabbi with

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Boris Pavlovic
Sean, So I'd say let's focus on that problem right now, and get some traction on this as part of functional test suites in OpenStack. Genericizing it too much just turns this back into a version of every other full stack testing tool, which we know isn't sufficient for having quality

Re: [openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

2015-01-12 Thread Chris Dent
On Tue, 13 Jan 2015, Boris Pavlovic wrote: Having separated engine seems like a good idea. It will really simplify stuff I'm not certain that's the case, but it may be worth exploration. This seems like a huge duplication of efforts. I mean operators will write own tools developers own...