Re: [Openstack] Integration tests
Matt Dietz wrote: Ditto On 9/12/11 2:10 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Jay Pipes [jaypi...@gmail.com] Can we discuss pulling novaclient into Nova's source tree at the design summit? +1 Someone should file it at summit.openstack.org, then :) -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
done From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Thierry Carrez [thie...@openstack.org] Sent: Tuesday, September 13, 2011 6:57 AM To: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Matt Dietz wrote: Ditto On 9/12/11 2:10 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Jay Pipes [jaypi...@gmail.com] Can we discuss pulling novaclient into Nova's source tree at the design summit? +1 Someone should file it at summit.openstack.org, then :) -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
Wow, another integration test framework emerges out of the woodwork :) Looking forward to getting all of these into a single place... and clearing up the lack of communication between teams on this subject! There's Kong, Stacktester, and Backfire. Any others out there we should know about? Cheers, jay On Sun, Sep 11, 2011 at 9:09 PM, Matt Dietz matt.di...@rackspace.com wrote: Suite looks great, Soren! Wanted to mention that we actually developed our own suite of tests, which you can find at https://github.com/ohthree/backfire I'm planning to reconcile the difference so we can stop the independent efforts, but it's going to take time. Something else I'd like to see in the suite, that we currently have in backfire via a custom test module, is the ability to parallelize the tests for execution. From: Soren Hansen so...@linux2go.dk Date: Sat, 10 Sep 2011 00:21:10 +0200 To: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests That link shouldn't have included the +bug bit. Copy/paste fail. :( Sent from my phone. Pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
I'm with the Reddwarf (Database as a Service) team at Rackspace. We employ a large set of integration / functional tests which run in a Vagrant controlled VM. Our tests use python-novaclient as well as a similar client for Reddwarf we've written ourselves. The motivation is to eat our own dog food- if the novaclient is the official rich client for Python, why shouldn't the test make use of it? We also use a library called Proboscis so we can order our tests without prefixing numbers to them, automatically skip related tests when something in a dependency fails, and avoid global variables even as we pass state between related tests. I think the openstack-integration-tests would definitely benefit from it. I'd love to have a conversation about getting the traits outlined above adopted into this standardized testing solution. :) Output of our tests running: http://pastie.org/2521835 Proboscis documentation: http://packages.python.org/proboscis/ From: openstack-bounces+tim.simpson=rackspace@lists.launchpad.net [openstack-bounces+tim.simpson=rackspace@lists.launchpad.net] on behalf of Jay Pipes [jaypi...@gmail.com] Sent: Monday, September 12, 2011 8:20 AM To: Matt Dietz Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Wow, another integration test framework emerges out of the woodwork :) Looking forward to getting all of these into a single place... and clearing up the lack of communication between teams on this subject! There's Kong, Stacktester, and Backfire. Any others out there we should know about? Cheers, jay On Sun, Sep 11, 2011 at 9:09 PM, Matt Dietz matt.di...@rackspace.com wrote: Suite looks great, Soren! Wanted to mention that we actually developed our own suite of tests, which you can find at https://github.com/ohthree/backfire I'm planning to reconcile the difference so we can stop the independent efforts, but it's going to take time. Something else I'd like to see in the suite, that we currently have in backfire via a custom test module, is the ability to parallelize the tests for execution. From: Soren Hansen so...@linux2go.dk Date: Sat, 10 Sep 2011 00:21:10 +0200 To: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests That link shouldn't have included the +bug bit. Copy/paste fail. :( Sent from my phone. Pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
Regarding novaclient, I was debating this myself last night. Our backfire suite currently uses it. However, I can see scenarios where nova has advanced but novaclient has yet to, and because all three projects are independently developed, we're stuck temporarily. I can see utility in tests specifically for Novaclient, and ones that use a built in client (as the suite that Soren provided currently does.) Also, Backfire uses a module Kevin Mitchell wrote named Dtest that actually provides a lot of that same functionality you mention, Tim. It also gives you the ability to spawn the tests individually in greenthreads so they execute in parallel. We got push back when we initially tried to merge our tests into Nova, partially because it uses a 3rd party library for executing the tests. We can try merging these things together, if that's what everyone wants. On 9/12/11 11:39 AM, Tim Simpson tim.simp...@rackspace.com wrote: I'm with the Reddwarf (Database as a Service) team at Rackspace. We employ a large set of integration / functional tests which run in a Vagrant controlled VM. Our tests use python-novaclient as well as a similar client for Reddwarf we've written ourselves. The motivation is to eat our own dog food- if the novaclient is the official rich client for Python, why shouldn't the test make use of it? We also use a library called Proboscis so we can order our tests without prefixing numbers to them, automatically skip related tests when something in a dependency fails, and avoid global variables even as we pass state between related tests. I think the openstack-integration-tests would definitely benefit from it. I'd love to have a conversation about getting the traits outlined above adopted into this standardized testing solution. :) Output of our tests running: http://pastie.org/2521835 Proboscis documentation: http://packages.python.org/proboscis/ From: openstack-bounces+tim.simpson=rackspace@lists.launchpad.net [openstack-bounces+tim.simpson=rackspace@lists.launchpad.net] on behalf of Jay Pipes [jaypi...@gmail.com] Sent: Monday, September 12, 2011 8:20 AM To: Matt Dietz Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Wow, another integration test framework emerges out of the woodwork :) Looking forward to getting all of these into a single place... and clearing up the lack of communication between teams on this subject! There's Kong, Stacktester, and Backfire. Any others out there we should know about? Cheers, jay On Sun, Sep 11, 2011 at 9:09 PM, Matt Dietz matt.di...@rackspace.com wrote: Suite looks great, Soren! Wanted to mention that we actually developed our own suite of tests, which you can find at https://github.com/ohthree/backfire I'm planning to reconcile the difference so we can stop the independent efforts, but it's going to take time. Something else I'd like to see in the suite, that we currently have in backfire via a custom test module, is the ability to parallelize the tests for execution. From: Soren Hansen so...@linux2go.dk Date: Sat, 10 Sep 2011 00:21:10 +0200 To: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests That link shouldn't have included the +bug bit. Copy/paste fail. :( Sent from my phone. Pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
On Mon, Sep 12, 2011 at 1:22 PM, Matt Dietz matt.di...@rackspace.com wrote: Regarding novaclient, I was debating this myself last night. Our backfire suite currently uses it. However, I can see scenarios where nova has advanced but novaclient has yet to, and because all three projects are independently developed, we're stuck temporarily. This, I believe, is a problem. novaclient is currently the only client that isn't in the core project itself. Can we discuss pulling novaclient into Nova's source tree at the design summit? I can see utility in tests specifically for Novaclient, and ones that use a built in client (as the suite that Soren provided currently does.) Sure. This is actually something that Glance's functional test suite does. It uses httplib2 as a direct HTTP client and uses bin/glance to test the supplied Glance client. Also, Backfire uses a module Kevin Mitchell wrote named Dtest that actually provides a lot of that same functionality you mention, Tim. It also gives you the ability to spawn the tests individually in greenthreads so they execute in parallel. We got push back when we initially tried to merge our tests into Nova, partially because it uses a 3rd party library for executing the tests. We can try merging these things together, if that's what everyone wants. I'd be totally cool with bringing DTest goodness into https://github.com/sorenh/openstack-integration-tests. :) Less duplication of effort, the better IMHO. openstack-integration-tests should be in the business of writing integration tests, not multi-processing testing libraries. -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
From: Jay Pipes [jaypi...@gmail.com] Can we discuss pulling novaclient into Nova's source tree at the design summit? +1 This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
It would be advantageous for the tests to only use one client, whatever it turns out to be. I think it would be most practical to use a bleeding edge version of Nova client and add any features necessary to it in its own repo directly or by somehow mixing it in with the test code. As for Dtest- I guess great minds think alike? It looks like both projects started at the same time (Proboscis began back in February in an internal Rackspace repo) so its unfortunate we're only just now discovering each other's work. Dtest looks very interesting and the ability to run tests in parallel is something I've been thought about doing in Proboscis, but unsure of how to proceed since Proboscis orders things before sending the list to Nose (for the purposes of backwards compatibility). It may be possible however for Proboscis to feed Dtest's execution routines. At first glance the ordering functionality in dtest is similar to Proboscis but after reviewing some of the code in backfire I believe there are features of Proboscis not present there. Proboscis in general is TestNG in Python, and anytime I've broken away from how TestNG handles these things (usually on accident) I've found it to be a mistake-TestNG really did think out how to do this pretty well. We should talk sometime, I think each project could gain a lot from it. From: Matt Dietz Sent: Monday, September 12, 2011 12:22 PM To: Tim Simpson; Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Regarding novaclient, I was debating this myself last night. Our backfire suite currently uses it. However, I can see scenarios where nova has advanced but novaclient has yet to, and because all three projects are independently developed, we're stuck temporarily. I can see utility in tests specifically for Novaclient, and ones that use a built in client (as the suite that Soren provided currently does.) Also, Backfire uses a module Kevin Mitchell wrote named Dtest that actually provides a lot of that same functionality you mention, Tim. It also gives you the ability to spawn the tests individually in greenthreads so they execute in parallel. We got push back when we initially tried to merge our tests into Nova, partially because it uses a 3rd party library for executing the tests. We can try merging these things together, if that's what everyone wants. On 9/12/11 11:39 AM, Tim Simpson tim.simp...@rackspace.com wrote: I'm with the Reddwarf (Database as a Service) team at Rackspace. We employ a large set of integration / functional tests which run in a Vagrant controlled VM. Our tests use python-novaclient as well as a similar client for Reddwarf we've written ourselves. The motivation is to eat our own dog food- if the novaclient is the official rich client for Python, why shouldn't the test make use of it? We also use a library called Proboscis so we can order our tests without prefixing numbers to them, automatically skip related tests when something in a dependency fails, and avoid global variables even as we pass state between related tests. I think the openstack-integration-tests would definitely benefit from it. I'd love to have a conversation about getting the traits outlined above adopted into this standardized testing solution. :) Output of our tests running: http://pastie.org/2521835 Proboscis documentation: http://packages.python.org/proboscis/ From: openstack-bounces+tim.simpson=rackspace@lists.launchpad.net [openstack-bounces+tim.simpson=rackspace@lists.launchpad.net] on behalf of Jay Pipes [jaypi...@gmail.com] Sent: Monday, September 12, 2011 8:20 AM To: Matt Dietz Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Wow, another integration test framework emerges out of the woodwork :) Looking forward to getting all of these into a single place... and clearing up the lack of communication between teams on this subject! There's Kong, Stacktester, and Backfire. Any others out there we should know about? Cheers, jay On Sun, Sep 11, 2011 at 9:09 PM, Matt Dietz matt.di...@rackspace.com wrote: Suite looks great, Soren! Wanted to mention that we actually developed our own suite of tests, which you can find at https://github.com/ohthree/backfire I'm planning to reconcile the difference so we can stop the independent efforts, but it's going to take time. Something else I'd like to see in the suite, that we currently have in backfire via a custom test module, is the ability to parallelize the tests for execution. From: Soren Hansen so...@linux2go.dk Date: Sat, 10 Sep 2011 00:21:10 +0200 To: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests That link shouldn't have included the +bug bit. Copy/paste fail. :( Sent from my phone. Pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post
Re: [Openstack] Integration tests
Ditto On 9/12/11 2:10 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Jay Pipes [jaypi...@gmail.com] Can we discuss pulling novaclient into Nova's source tree at the design summit? +1 This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
On 9/12/11 2:54 PM, Tim Simpson tim.simp...@rackspace.com wrote: It would be advantageous for the tests to only use one client, whatever it turns out to be. I think it would be most practical to use a bleeding edge version of Nova client and add any features necessary to it in its own repo directly or by somehow mixing it in with the test code. My biggest concern in using novaclient exclusively is masking errors in both the API and novaclient. If this is resolved another way, then I'm ok. There are a few different ways to do this, including running through a separate python client that hits the API directly (as kong and stacktester do, I believe), or using language bindings from other languages as a part of the tests to make sure everyone is in sync. Of course it is important that we maintain novaclient as a part of the end to end tests as well. Gabe As for Dtest- I guess great minds think alike? It looks like both projects started at the same time (Proboscis began back in February in an internal Rackspace repo) so its unfortunate we're only just now discovering each other's work. Dtest looks very interesting and the ability to run tests in parallel is something I've been thought about doing in Proboscis, but unsure of how to proceed since Proboscis orders things before sending the list to Nose (for the purposes of backwards compatibility). It may be possible however for Proboscis to feed Dtest's execution routines. At first glance the ordering functionality in dtest is similar to Proboscis but after reviewing some of the code in backfire I believe there are features of Proboscis not present there. Proboscis in general is TestNG in Python, and anytime I've broken away from how TestNG handles these things (usually on accident) I've found it to be a mistake-TestNG really did think out how to do this pretty well. We should talk sometime, I think each project could gain a lot from it. From: Matt Dietz Sent: Monday, September 12, 2011 12:22 PM To: Tim Simpson; Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Regarding novaclient, I was debating this myself last night. Our backfire suite currently uses it. However, I can see scenarios where nova has advanced but novaclient has yet to, and because all three projects are independently developed, we're stuck temporarily. I can see utility in tests specifically for Novaclient, and ones that use a built in client (as the suite that Soren provided currently does.) Also, Backfire uses a module Kevin Mitchell wrote named Dtest that actually provides a lot of that same functionality you mention, Tim. It also gives you the ability to spawn the tests individually in greenthreads so they execute in parallel. We got push back when we initially tried to merge our tests into Nova, partially because it uses a 3rd party library for executing the tests. We can try merging these things together, if that's what everyone wants. On 9/12/11 11:39 AM, Tim Simpson tim.simp...@rackspace.com wrote: I'm with the Reddwarf (Database as a Service) team at Rackspace. We employ a large set of integration / functional tests which run in a Vagrant controlled VM. Our tests use python-novaclient as well as a similar client for Reddwarf we've written ourselves. The motivation is to eat our own dog food- if the novaclient is the official rich client for Python, why shouldn't the test make use of it? We also use a library called Proboscis so we can order our tests without prefixing numbers to them, automatically skip related tests when something in a dependency fails, and avoid global variables even as we pass state between related tests. I think the openstack-integration-tests would definitely benefit from it. I'd love to have a conversation about getting the traits outlined above adopted into this standardized testing solution. :) Output of our tests running: http://pastie.org/2521835 Proboscis documentation: http://packages.python.org/proboscis/ From: openstack-bounces+tim.simpson=rackspace@lists.launchpad.net [openstack-bounces+tim.simpson=rackspace@lists.launchpad.net] on behalf of Jay Pipes [jaypi...@gmail.com] Sent: Monday, September 12, 2011 8:20 AM To: Matt Dietz Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Wow, another integration test framework emerges out of the woodwork :) Looking forward to getting all of these into a single place... and clearing up the lack of communication between teams on this subject! There's Kong, Stacktester, and Backfire. Any others out there we should know about? Cheers, jay On Sun, Sep 11, 2011 at 9:09 PM, Matt Dietz matt.di...@rackspace.com wrote: Suite looks great, Soren! Wanted to mention that we actually developed our own suite of tests, which you can find at https://github.com/ohthree/backfire I'm planning to reconcile the difference so we can stop the independent efforts
Re: [Openstack] Integration tests
Actually to extend this, Novaclient is used for zone communication inside of nova. It not being included is silly, at this point On 9/12/11 1:37 PM, Jay Pipes jaypi...@gmail.com wrote: On Mon, Sep 12, 2011 at 1:22 PM, Matt Dietz matt.di...@rackspace.com wrote: Regarding novaclient, I was debating this myself last night. Our backfire suite currently uses it. However, I can see scenarios where nova has advanced but novaclient has yet to, and because all three projects are independently developed, we're stuck temporarily. This, I believe, is a problem. novaclient is currently the only client that isn't in the core project itself. Can we discuss pulling novaclient into Nova's source tree at the design summit? I can see utility in tests specifically for Novaclient, and ones that use a built in client (as the suite that Soren provided currently does.) Sure. This is actually something that Glance's functional test suite does. It uses httplib2 as a direct HTTP client and uses bin/glance to test the supplied Glance client. Also, Backfire uses a module Kevin Mitchell wrote named Dtest that actually provides a lot of that same functionality you mention, Tim. It also gives you the ability to spawn the tests individually in greenthreads so they execute in parallel. We got push back when we initially tried to merge our tests into Nova, partially because it uses a 3rd party library for executing the tests. We can try merging these things together, if that's what everyone wants. I'd be totally cool with bringing DTest goodness into https://github.com/sorenh/openstack-integration-tests. :) Less duplication of effort, the better IMHO. openstack-integration-tests should be in the business of writing integration tests, not multi-processing testing libraries. -jay This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
One solution is to make a second version of the Nova client with the same interface, so that the test config could use one or the other on different test runs. We could make this client communicate via XML to ensure we hit the XML serialization code. To me the biggest benefit of using the client directly is it would make everyone have to buy into how it behaves and feels. As for testing it itself, I agree thats a secondary goal which is arguably negated by the risk of hiding bugs immune to nova-client which are not caught. From: Gabe Westmaas Sent: Monday, September 12, 2011 3:16 PM To: Tim Simpson; Matt Dietz; Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests On 9/12/11 2:54 PM, Tim Simpson tim.simp...@rackspace.com wrote: It would be advantageous for the tests to only use one client, whatever it turns out to be. I think it would be most practical to use a bleeding edge version of Nova client and add any features necessary to it in its own repo directly or by somehow mixing it in with the test code. My biggest concern in using novaclient exclusively is masking errors in both the API and novaclient. If this is resolved another way, then I'm ok. There are a few different ways to do this, including running through a separate python client that hits the API directly (as kong and stacktester do, I believe), or using language bindings from other languages as a part of the tests to make sure everyone is in sync. Of course it is important that we maintain novaclient as a part of the end to end tests as well. Gabe As for Dtest- I guess great minds think alike? It looks like both projects started at the same time (Proboscis began back in February in an internal Rackspace repo) so its unfortunate we're only just now discovering each other's work. Dtest looks very interesting and the ability to run tests in parallel is something I've been thought about doing in Proboscis, but unsure of how to proceed since Proboscis orders things before sending the list to Nose (for the purposes of backwards compatibility). It may be possible however for Proboscis to feed Dtest's execution routines. At first glance the ordering functionality in dtest is similar to Proboscis but after reviewing some of the code in backfire I believe there are features of Proboscis not present there. Proboscis in general is TestNG in Python, and anytime I've broken away from how TestNG handles these things (usually on accident) I've found it to be a mistake-TestNG really did think out how to do this pretty well. We should talk sometime, I think each project could gain a lot from it. From: Matt Dietz Sent: Monday, September 12, 2011 12:22 PM To: Tim Simpson; Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests Regarding novaclient, I was debating this myself last night. Our backfire suite currently uses it. However, I can see scenarios where nova has advanced but novaclient has yet to, and because all three projects are independently developed, we're stuck temporarily. I can see utility in tests specifically for Novaclient, and ones that use a built in client (as the suite that Soren provided currently does.) Also, Backfire uses a module Kevin Mitchell wrote named Dtest that actually provides a lot of that same functionality you mention, Tim. It also gives you the ability to spawn the tests individually in greenthreads so they execute in parallel. We got push back when we initially tried to merge our tests into Nova, partially because it uses a 3rd party library for executing the tests. We can try merging these things together, if that's what everyone wants. On 9/12/11 11:39 AM, Tim Simpson tim.simp...@rackspace.com wrote: I'm with the Reddwarf (Database as a Service) team at Rackspace. We employ a large set of integration / functional tests which run in a Vagrant controlled VM. Our tests use python-novaclient as well as a similar client for Reddwarf we've written ourselves. The motivation is to eat our own dog food- if the novaclient is the official rich client for Python, why shouldn't the test make use of it? We also use a library called Proboscis so we can order our tests without prefixing numbers to them, automatically skip related tests when something in a dependency fails, and avoid global variables even as we pass state between related tests. I think the openstack-integration-tests would definitely benefit from it. I'd love to have a conversation about getting the traits outlined above adopted into this standardized testing solution. :) Output of our tests running: http://pastie.org/2521835 Proboscis documentation: http://packages.python.org/proboscis/ From: openstack-bounces+tim.simpson=rackspace@lists.launchpad.net [openstack-bounces+tim.simpson=rackspace@lists.launchpad.net] on behalf of Jay Pipes [jaypi...@gmail.com
Re: [Openstack] Integration tests
On Sep 12, 2011, at 3:16 PM, Gabe Westmaas wrote: It would be advantageous for the tests to only use one client, whatever it turns out to be. I think it would be most practical to use a bleeding edge version of Nova client and add any features necessary to it in its own repo directly or by somehow mixing it in with the test code. My biggest concern in using novaclient exclusively is masking errors in both the API and novaclient. If this is resolved another way, then I'm ok. There were many who wanted to separate the two functions: inter-zone communication would use a highly specialized, internally-sourced client, while 'novaclient' would be an independent project managed outside of OpenStack, and would serve as to keep the API honest. While this would cause some duplication, the trade-off in cleanliness might be worth it. -- Ed Leafe This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
Suite looks great, Soren! Wanted to mention that we actually developed our own suite of tests, which you can find at https://github.com/ohthree/backfire I'm planning to reconcile the difference so we can stop the independent efforts, but it's going to take time. Something else I'd like to see in the suite, that we currently have in backfire via a custom test module, is the ability to parallelize the tests for execution. From: Soren Hansen so...@linux2go.dkmailto:so...@linux2go.dk Date: Sat, 10 Sep 2011 00:21:10 +0200 To: openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Subject: Re: [Openstack] Integration tests That link shouldn't have included the +bug bit. Copy/paste fail. :( Sent from my phone. Pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration tests
That link shouldn't have included the +bug bit. Copy/paste fail. :( Sent from my phone. Pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
