On 02/04/2013 12:15 AM, Koper, Dies wrote: > Hi Ronelle, Marios, all > > Last week we modified the api tests slightly so they would pass on fgcp. > I would like to do the same with the cimi tests. > Before I complete my patch, I'd like to consult with you to ensure the > direction I'd like to take is acceptable with you. > > Unlike the api tests, I appreciate that the cimi tests (at least the > part* ones) are based on documented scenarios that we need to follow. So > I'm trying to come up with a solution that complies with the scenarios. > > To get the tests to pass on the fgcp I have to work around the same > restriction with the fgcp endpoint API that affected the api tests: when > you create a resource (machine/volume) or delete one, it does not accept > any other resource creation/deletion requests in that system until the > creation/deletion has completed. > > Currently, I'm considering to make the following changes: > > 1) For tests that create resources and deletion is not part of the test > scenario, perform the deletion in a teardown operation (as is already > done in most cases). The teardown method would loop through the > resources to stop and destroy them. When a destroy operation returns a > 405 (Method Not Allowed) or 409 (Conflict), the operation is retried > again and again after a number of seconds. > > As the teardown is not part of the scenario, I hope this non-ideal > method is acceptable. > > 2) For tests that create resources, I'd like to add a checkpoint: that > the resource has actually been created (by sleeping and performing a GET > on it until its state becomes AVAILABLE/ STARTED/ STOPPED. The test then > continues as per the scenario. > > I would say this is actually a better implementation of the scenario: > Where e.g. the scenario says the success criteria is "A new Machine > resource is created.", our current test is just checking that the > response of the creation request is 201. There is no check whether the > resource has been created. If it failed during the creation process, our > test would not catch that. With my proposal it would, because we'd > actually be checking that the machine left the CREATING state and > transitioned into a stable success state. > > I expect again that added sleeps will not affect performance with the > mock driver. I can imagine the extra check introduced above does incur a > performance impact, depending on the performance of the backend cloud > provider under testing. > > What do you think? > > Cheers, > Dies Koper >
Dies, I think this proposal sounds great. The only concern I have is how to address retires that could potentially never succeed. I would like to suggest having a retry MAX with a default value that can be customized by someone running the tests. This way the test could be run in a shorter time when the user realizes they may have to do some manual clean-up. I would also like to suggest that the test log the state for each iteration of the retry, how many retries were required and the time it took to detect the expected result. This may help us gather some insight into where we may need to try to work to improve the user experience. Let me know if I am wrong but I would expect addressing my 2 suggestions would not be overly complex. Hope this helps and thanks, Joe VLcek
