Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
Responses inline. On Mon, Dec 9, 2013 at 10:07 AM, Sean Dague s...@dague.net wrote: On 12/09/2013 10:12 AM, Brant Knudson wrote: Monty - Thanks for doing the work already to get the infrastructure set up. Looks like I've got the easy part here. I posted an initial patch that has one test from keystone in https://review.openstack.org/#/c/60724/ . I hope to be able to move all the tests over unchanged. The tricky part is getting all the fixtures set up the same way that keystone does. I think a direct port of the keystone fixtures is the wrong approach. These really need to act more like the scenario tests that exist over there. And if the intent is just a dump of the keystone tests we need to step back... because that's not going to get accepted. The reason I'd like to keep keystone's client tests as they are is that they provide us with coverage of keystone and keystoneclient functionality. This doesn't mean they have to stay that way forever, since once they're moved out of Keystone we can start refactoring them. An alternative approach is to clean up Keystone's client tests as much as possible first to make them essentially the scenario tests that tempest would accept. This would leave the tests in keystone longer than we'd like since we'd like them out of there ASAP. I actually think that we should solve #4 first - how you test the thing you actually want to test in the gate. Which is about getting devstack-gate to setup the world that you want to test. I really think the location of the tests all flow from there. Because right now it seems like the cart is before the horse. OK. Let's solve #4. If the tests as they are aren't going to be accepted in Tempest then we can put them elsewhere (leave them in keystone (just run differently), move to keystoneclient or a new project). To me this testing has some similarities to Grenade since that involves testing with multiple versions too, so I'll look into how Grenade works. Is testing multiple versions of keystoneclient actually worth it? If the other projects don't feel the need for this then why does Keystone? It's actually caught problems so it's proved useful to Keystone, and we're making changes to the client so this type of testing seems important, but maybe it's not useful enough to continue to do the multiple version testing. If we're going to support backwards compatibility we should test it. If we put something together to test multiple versions of keystoneclient would other projects want to use it for their clients? - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On 2013-12-09 11:07:50 -0600 (-0600), Brant Knudson wrote: [...] Is testing multiple versions of keystoneclient actually worth it? If the other projects don't feel the need for this then why does Keystone? It's actually caught problems so it's proved useful to Keystone, and we're making changes to the client so this type of testing seems important, but maybe it's not useful enough to continue to do the multiple version testing. If we're going to support backwards compatibility we should test it. [...] Well, at a minimum we should be testing both that the tip of master for the client works with servers, and also that the tip of server branches (master, stable/x, stable/y) work with the latest released version of the client. There are already plans in motion to solve this in integration testing for all clients, because not doing so allows us to break ourselves in unfortunate ways. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On Mon, Dec 9, 2013 at 11:18 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2013-12-09 11:07:50 -0600 (-0600), Brant Knudson wrote: [...] Is testing multiple versions of keystoneclient actually worth it? If the other projects don't feel the need for this then why does Keystone? It's actually caught problems so it's proved useful to Keystone, and we're making changes to the client so this type of testing seems important, but maybe it's not useful enough to continue to do the multiple version testing. If we're going to support backwards compatibility we should test it. [...] Well, at a minimum we should be testing both that the tip of master for the client works with servers, and also that the tip of server branches (master, stable/x, stable/y) work with the latest released version of the client. There are already plans in motion to solve this in integration testing for all clients, because not doing so allows us to break ourselves in unfortunate ways. This isn't the testing that Keystone's client tests do. Keystone's client tests verify that the client doesn't change in an incompatible way. For example, we've got a test that verifies that you can do client.tenants.list() ; so if someone decides to change tenants to projects then this test will fail. I don't know of any specific testing that verifies that the client works against older (stable/*) servers. Maybe it happens when you submit a change to stable/ later. - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On 2013-12-09 11:50:31 -0600 (-0600), Brant Knudson wrote: [...] I don't know of any specific testing that verifies that the client works against older (stable/*) servers. Maybe it happens when you submit a change to stable/ later. When a change is submitted or approved for a stable/foo server branch, Tempest tests run with the other stable servers from that release along with the master branch tip of all client libraries. This helps prevent a stable branch update in a server from doing something which won't work with the latest client versions (as long as it's being exercised in Tempest). Work is underway to do the same for changes to the clients so they don't break stable branches of servers (as happened recently with the iso8601/grizzly debacle). The other missing piece of the puzzle is testing the servers against the latest release of the client, so we don't land changes to servers which require unreleased client features. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On 12/09/2013 11:07 AM, Sean Dague wrote: On 12/09/2013 10:12 AM, Brant Knudson wrote: Monty - Thanks for doing the work already to get the infrastructure set up. Looks like I've got the easy part here. I posted an initial patch that has one test from keystone in https://review.openstack.org/#/c/60724/ . I hope to be able to move all the tests over unchanged. The tricky part is getting all the fixtures set up the same way that keystone does. I think a direct port of the keystone fixtures is the wrong approach. These really need to act more like the scenario tests that exist over there. And if the intent is just a dump of the keystone tests we need to step back... because that's not going to get accepted. I actually think that we should solve #4 first - how you test the thing you actually want to test in the gate. Which is about getting devstack-gate to setup the world that you want to test. I really think the location of the tests all flow from there. Because right now it seems like the cart is before the horse. I think we can rework the Keystone tests to meet the Tempest standard without making it inot a Major rewrite. The biggest Sin of the current tests is that it creates self.user_foo with an id of 'foo'. But the test should never be looking for the string 'foo'. Instead, They should be doing the equivalent to: assertEquals(self.user_foo.id, some_user.id) If we make the fixture Setup create user_foo, but with an ID generated for each test using uuid4, we should be able to have the proper test semantics. -Sean ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On Sun, Dec 8, 2013 at 5:20 PM, Monty Taylor mord...@inaugust.com wrote: Hi! Thanks - I've been wanting to kill this for a long time. Thanks for starting the discussion... On 12/08/2013 07:26 PM, Brant Knudson wrote: We'd like to get the keystoneclient tests out of keystone. They're serving a useful purpose of catching problems with non-backwards compatible changes in keystoneclient so we still want them run. Problem is they're running at the wrong time -- only on changes to keystone and not changes to keystoneclient. The tests need to be run: When keystoneclient changes - run the tests against the change When the tests change - run the change against the current keystoneclient and also old clients When keystone changes - run the tests against the change with current client You're talking about this: https://review.openstack.org/#/c/41931/ Which is almost ready to go. So here's what I think we need to do to get keystone client tests out of keystone: 1) Figure out where to put the tests - is it tempest or something else? Tempest. They should be moved to tempest. 2) Write up a test and put it there 3) Have a job that when there's a change in the tests it runs against current client lib Don't need most of that - the generalized test clients against stable versions matrix is about to land - as long as tempest has your tests in the scenarios, you'll be fine. 4) Expand the job to also run against old clients - or is there 1 job per version? - what versions? (keystone does master, essex-3, and 0.1.1) - e.g. tox -e master,essex-3,0.1.1 essex and 0.1.1 are both older than dirt. We just removed folsom from the gate. I'd got ahead and remove these. - suggest start with these versions and then consider what to use in future OpenStack has a clear support policy - the gate infrastructure will be covering that. Done! 5) Now we can start adding tests Tempest scenarios. 6) Have a job that when there's a change in keystoneclient it runs these tests against the change 7) When there's a change in keystone, run these tests against the change Yup. Already accounted for. 8) Copy the keystoneclient tests from keystone to the new location -- will require some changes 9) Remove the tests from keystone \o/ 10) Move tests back to keystone where makes sense -- use webtest like v3 tests I created an etherpad with this same info so it's easier to discuss: https://etherpad.openstack.org/p/KeystoneTestsToTempest - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev THANK YOU TO EVERYONE HELPING TO MAKE THIS HAPPEN! To cross-link, there was also a summit discussion on the topic: https://etherpad.openstack.org/p/icehouse-summit-qa-keystone -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
Monty - Thanks for doing the work already to get the infrastructure set up. Looks like I've got the easy part here. I posted an initial patch that has one test from keystone in https://review.openstack.org/#/c/60724/ . I hope to be able to move all the tests over unchanged. The tricky part is getting all the fixtures set up the same way that keystone does. I'm not sure what directory the tests go in Tempest. Maybe it's a new one like shardy's attempts: https://review.openstack.org/#/c/54810/ . I'll leave that to the tempest team. - Brant On Sun, Dec 8, 2013 at 5:20 PM, Monty Taylor mord...@inaugust.com wrote: Hi! Thanks - I've been wanting to kill this for a long time. Thanks for starting the discussion... On 12/08/2013 07:26 PM, Brant Knudson wrote: We'd like to get the keystoneclient tests out of keystone. They're serving a useful purpose of catching problems with non-backwards compatible changes in keystoneclient so we still want them run. Problem is they're running at the wrong time -- only on changes to keystone and not changes to keystoneclient. The tests need to be run: When keystoneclient changes - run the tests against the change When the tests change - run the change against the current keystoneclient and also old clients When keystone changes - run the tests against the change with current client You're talking about this: https://review.openstack.org/#/c/41931/ Which is almost ready to go. So here's what I think we need to do to get keystone client tests out of keystone: 1) Figure out where to put the tests - is it tempest or something else? Tempest. They should be moved to tempest. 2) Write up a test and put it there 3) Have a job that when there's a change in the tests it runs against current client lib Don't need most of that - the generalized test clients against stable versions matrix is about to land - as long as tempest has your tests in the scenarios, you'll be fine. 4) Expand the job to also run against old clients - or is there 1 job per version? - what versions? (keystone does master, essex-3, and 0.1.1) - e.g. tox -e master,essex-3,0.1.1 essex and 0.1.1 are both older than dirt. We just removed folsom from the gate. I'd got ahead and remove these. - suggest start with these versions and then consider what to use in future OpenStack has a clear support policy - the gate infrastructure will be covering that. Done! What is the support policy for the client libs? Nobody's tried to remove anything since I started reviewing. 5) Now we can start adding tests Tempest scenarios. 6) Have a job that when there's a change in keystoneclient it runs these tests against the change 7) When there's a change in keystone, run these tests against the change Yup. Already accounted for. 8) Copy the keystoneclient tests from keystone to the new location -- will require some changes 9) Remove the tests from keystone \o/ 10) Move tests back to keystone where makes sense -- use webtest like v3 tests I created an etherpad with this same info so it's easier to discuss: https://etherpad.openstack.org/p/KeystoneTestsToTempest - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On 12/09/2013 10:12 AM, Brant Knudson wrote: Monty - Thanks for doing the work already to get the infrastructure set up. Looks like I've got the easy part here. I posted an initial patch that has one test from keystone in https://review.openstack.org/#/c/60724/ . I hope to be able to move all the tests over unchanged. The tricky part is getting all the fixtures set up the same way that keystone does. I think a direct port of the keystone fixtures is the wrong approach. These really need to act more like the scenario tests that exist over there. And if the intent is just a dump of the keystone tests we need to step back... because that's not going to get accepted. I actually think that we should solve #4 first - how you test the thing you actually want to test in the gate. Which is about getting devstack-gate to setup the world that you want to test. I really think the location of the tests all flow from there. Because right now it seems like the cart is before the horse. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][keystone] Keystoneclient tests to tempest
We'd like to get the keystoneclient tests out of keystone. They're serving a useful purpose of catching problems with non-backwards compatible changes in keystoneclient so we still want them run. Problem is they're running at the wrong time -- only on changes to keystone and not changes to keystoneclient. The tests need to be run: When keystoneclient changes - run the tests against the change When the tests change - run the change against the current keystoneclient and also old clients When keystone changes - run the tests against the change with current client So here's what I think we need to do to get keystone client tests out of keystone: 1) Figure out where to put the tests - is it tempest or something else? 2) Write up a test and put it there 3) Have a job that when there's a change in the tests it runs against current client lib 4) Expand the job to also run against old clients - or is there 1 job per version? - what versions? (keystone does master, essex-3, and 0.1.1) - e.g. tox -e master,essex-3,0.1.1 - suggest start with these versions and then consider what to use in future 5) Now we can start adding tests 6) Have a job that when there's a change in keystoneclient it runs these tests against the change 7) When there's a change in keystone, run these tests against the change 8) Copy the keystoneclient tests from keystone to the new location -- will require some changes 9) Remove the tests from keystone \o/ 10) Move tests back to keystone where makes sense -- use webtest like v3 tests I created an etherpad with this same info so it's easier to discuss: https://etherpad.openstack.org/p/KeystoneTestsToTempest - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On Sunday, December 08, 2013 11:26:07 AM, Brant Knudson wrote: We'd like to get the keystoneclient tests out of keystone. They're serving a useful purpose of catching problems with non-backwards compatible changes in keystoneclient so we still want them run. Problem is they're running at the wrong time -- only on changes to keystone and not changes to keystoneclient. The tests need to be run: When keystoneclient changes - run the tests against the change When the tests change - run the change against the current keystoneclient and also old clients When keystone changes - run the tests against the change with current client So here's what I think we need to do to get keystone client tests out of keystone: 1) Figure out where to put the tests - is it tempest or something else? 2) Write up a test and put it there 3) Have a job that when there's a change in the tests it runs against current client lib 4) Expand the job to also run against old clients - or is there 1 job per version? - what versions? (keystone does master, essex-3, and 0.1.1) - e.g. tox -e master,essex-3,0.1.1 - suggest start with these versions and then consider what to use in future 5) Now we can start adding tests 6) Have a job that when there's a change in keystoneclient it runs these tests against the change 7) When there's a change in keystone, run these tests against the change 8) Copy the keystoneclient tests from keystone to the new location -- will require some changes 9) Remove the tests from keystone \o/ 10) Move tests back to keystone where makes sense -- use webtest like v3 tests I created an etherpad with this same info so it's easier to discuss: https://etherpad.openstack.org/p/KeystoneTestsToTempest - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I'll ask the super obvious question, why not move the keystoneclient tests to keystoneclient? -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
On Sun, Dec 8, 2013 at 3:37 PM, Matt Riedemann mrie...@linux.vnet.ibm.comwrote: On Sunday, December 08, 2013 11:26:07 AM, Brant Knudson wrote: We'd like to get the keystoneclient tests out of keystone. They're serving a useful purpose of catching problems with non-backwards compatible changes in keystoneclient so we still want them run. Problem is they're running at the wrong time -- only on changes to keystone and not changes to keystoneclient. The tests need to be run: When keystoneclient changes - run the tests against the change When the tests change - run the change against the current keystoneclient and also old clients When keystone changes - run the tests against the change with current client So here's what I think we need to do to get keystone client tests out of keystone: 1) Figure out where to put the tests - is it tempest or something else? 2) Write up a test and put it there 3) Have a job that when there's a change in the tests it runs against current client lib 4) Expand the job to also run against old clients - or is there 1 job per version? - what versions? (keystone does master, essex-3, and 0.1.1) - e.g. tox -e master,essex-3,0.1.1 - suggest start with these versions and then consider what to use in future 5) Now we can start adding tests 6) Have a job that when there's a change in keystoneclient it runs these tests against the change 7) When there's a change in keystone, run these tests against the change 8) Copy the keystoneclient tests from keystone to the new location -- will require some changes 9) Remove the tests from keystone \o/ 10) Move tests back to keystone where makes sense -- use webtest like v3 tests I created an etherpad with this same info so it's easier to discuss: https://etherpad.openstack.org/p/KeystoneTestsToTempest - Brant I'll ask the super obvious question, why not move the keystoneclient tests to keystoneclient? I believe Brant is talking about the tests that use different versions of the keystone client against the keystone server. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek www: http://dstanek.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][keystone] Keystoneclient tests to tempest
Hi! Thanks - I've been wanting to kill this for a long time. Thanks for starting the discussion... On 12/08/2013 07:26 PM, Brant Knudson wrote: We'd like to get the keystoneclient tests out of keystone. They're serving a useful purpose of catching problems with non-backwards compatible changes in keystoneclient so we still want them run. Problem is they're running at the wrong time -- only on changes to keystone and not changes to keystoneclient. The tests need to be run: When keystoneclient changes - run the tests against the change When the tests change - run the change against the current keystoneclient and also old clients When keystone changes - run the tests against the change with current client You're talking about this: https://review.openstack.org/#/c/41931/ Which is almost ready to go. So here's what I think we need to do to get keystone client tests out of keystone: 1) Figure out where to put the tests - is it tempest or something else? Tempest. They should be moved to tempest. 2) Write up a test and put it there 3) Have a job that when there's a change in the tests it runs against current client lib Don't need most of that - the generalized test clients against stable versions matrix is about to land - as long as tempest has your tests in the scenarios, you'll be fine. 4) Expand the job to also run against old clients - or is there 1 job per version? - what versions? (keystone does master, essex-3, and 0.1.1) - e.g. tox -e master,essex-3,0.1.1 essex and 0.1.1 are both older than dirt. We just removed folsom from the gate. I'd got ahead and remove these. - suggest start with these versions and then consider what to use in future OpenStack has a clear support policy - the gate infrastructure will be covering that. Done! 5) Now we can start adding tests Tempest scenarios. 6) Have a job that when there's a change in keystoneclient it runs these tests against the change 7) When there's a change in keystone, run these tests against the change Yup. Already accounted for. 8) Copy the keystoneclient tests from keystone to the new location -- will require some changes 9) Remove the tests from keystone \o/ 10) Move tests back to keystone where makes sense -- use webtest like v3 tests I created an etherpad with this same info so it's easier to discuss: https://etherpad.openstack.org/p/KeystoneTestsToTempest - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
