Perhaps it's just me, but given if I was developing in a different language, I would not want to use a command line tool to interact with my application. What is the point then of developing RESTful APIs if the primary client is not it, but these command line tools instead?
While it may appear that the approach I'm advocating is extra work, let me try explain the purpose of this approach. If these aren't clear, I'd be more than glad to give some concrete examples where these techniques have been useful. A few benefits that come to mind are: * Testing of XML implementations of the API. While this could be built into the clients, I don't see many folks who would rush to that cause * Direct access to the response. The clients hide any header/response code information from the recipient, which can be very important. For example, the response headers for Nova contain a token that can be captured and used when troubleshooting issues. * Avoiding the user friendliness of the clients. While retries and user friendly exception handling are great for clients, they are not what I want as a tester. As a tester, while I do want a minimal layer of abstraction between my AUT and my code so that I'm not making HTTP requests directly from my tests, from what I've seen the clients can make more efforts than I'd prefer to be helpful. * The ability to inject other metrics gathering into my clients for the purpose of troubleshooting/logging/information handling While perhaps the idea is that only the smoke tests would use this approach, I'm hesitant to the idea of developing tests using multiple approaches simply for the sake of using the clients for certain tests. I'm assuming these were things that were talked about during the CLI portions of OpenStack summit, which I wasn't able to attend. I wasn't aware of this or even some of the new parallel testing efforts which somehow did not come up during the QA track. The purpose of Tempest in the first place was to unify the functional and integration testing efforts for OpenStack projects, and I'm dedicated to doing everything I can to make that happen. If everyone is in agreement on the other side, I certainly don't want to be the one in the way against the majority. However, I just wanted to state my concerns before we take any further actions. Daryl On May 3, 2012, at 9:54 PM, Maru Newby wrote: The rest api is the default interface, and the client tools target that interface. Since the clients are cli more than python api, they can be used by any language that can use a shell. What exactly does reimplementing the clients for the sake of testing accomplish? Double the maintenance effort for the same result, imho. Cheers, Maru On 2012-05-03, at 12:54 PM, Daryl Walleck wrote: So my first question is around this. So is the claim is that the client tools are the default interface for the applications? While that works for coders in python, what about people using other languages? Even then, there's no guarantee that the clients in different languages are implemented in the same way. Tempest was designed originally because while it does use an abstraction between the API and the tests, there is nothing to "assist" the user by retrying and the like. While I think there's a place for writing tests using the command line clients, to me that would be a smoke test of a client and not as much a smoke test of the API. Daryl On May 3, 2012, at 12:01 PM, Jay Pipes wrote: However, before this can happen, a number of improvements need to be made to Tempest. The issue with the "smoke tests" in Tempest is that they aren't really smoke tests. They do not use the default client tools (like novaclient, keystoneclient, etc) and are not annotated consistently. _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net<mailto:openstack@lists.launchpad.net> Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
