Re: [openstack-dev] [qa] Status of account creds in the [identity] section of tempest.conf
On Fri, Jun 19, 2015 at 10:30 AM Matthew Treinish mtrein...@kortar.org wrote: On Thu, Jun 18, 2015 at 02:13:56PM -0400, David Kranz wrote: We had a discussion about this at the qa meeting today around the following proposal: tl;dr The test accounts feature provides the same functionality as the embedded credentials. We should deprecate the account information embedded directly in tempest.conf in favor of test-accounts, and remove those options at the beginning of the M cycle. We would also rework the non-isolated jobs to use parallel test accounts, with and without admin creds. +1 Starting now, new features such as cleanup and tempest config will not be required to work well (or at all) if the embedded creds are used instead of test accounts. So this was always the long term plan when we started work on the test-accounts mechanism about a year ago. We were holding off on deprecating the original config option based approach until finished the role and network support for test accounts and we had jobs setup using the mechanism. Now that the we finished both role and network support all that's left is setting up the jobs. I don't think there would be any opposition to marking the user and alt user options as deprecated after that. Also leaving in line comments (and maybe emit a warning) marking the non-locking provider mechanism as going away, probably in M. That way we start clearly marking to users that these options will be going away. +1 because the options are completely moving out of conf, we cannot use the deprecation mechanism from oslo.config - emitting a warning is a good idea We have (at least) three use cases that are important, and we want tempest to work well with all of them, but that means something different in each case: 1. throw-away clouds (ci, gate) 2. test clouds 3. production clouds Well, tempest is designed to and tries to support running against any OpenStack cloud. I'm not sure if there are more deployment types than these 3 categories but we should also be supporting those too. For (1), the most important thing is that failing tests not cause false negatives in other tests due to re-using a tenant. This makes tenant isolation continue to be a good choice here, and requiring admin is not an issue. In a perfect world where tempest never left behind any resources regardless of an error at any line of code, test accounts could be used. But we are probably a long way from that. So the cleanup issue here is actually a wider openstack issue. Tempest will *always* call delete on created projects and users. This was a requirement for making test accounts work. (the mechanism for calling delete or freeing a credential set from the list is shared) With tenant isolation this means we'll be deleting a project and users and resources scoped to either may not be deleted first. (if there is a tempest or openstack bug) This is a wider issue for all OpenStack projects that there was a thread a few months ago discussing. For (3), we cannot use admin creds for tempest runs, and test accounts with cleanup allow parallel execution, accepting the risk of a leak causing a false negative. The only way to avoid that risk is to stamp out all leak bugs in tempest. Well depending on the resource in leak in question test accounts would likely catch the issues if there is a list on that resource in a later test. But, I agree, resource leaks have always been treated as bugs and we'll continue to do so. For (2), either isolation or test accounts with cleanup can be used The tempest.conf values are not used in any of these scenarios. Is there a reason they are needed for anything? So the only thing that uses config options for credentials is actually tenant isolation, which uses them to provide admin credentials to do the dynamic creation of accounts. The real advantage of tenant isolation, besides not reusing anything, is actually its configuration simplicity. Using an accounts file can be tricky to use, there are a lot of little gotchas and assumptions in how you write the file. (which we try to document in both the config guide and the sample accounts.yaml file) It also requires a large number of users to be provided depending on the concurrency you're running with. While tenant isolation requires just setting 3-5 config options and you're fine after that. I don't think requiring the use of an accounts file for tenant isolation makes much sense, it's really heavyweight for 1 set of admin creds. Which probably means we should keep the admin user config option around. Although it might make more sense to move those options to the auth section. (and maybe rename them to make it clear that it's for tenant isolation only) I would also keep the admin account in tempest.conf. And +1 to renaming the config options. If tenant isolation is disabled, we will
Re: [openstack-dev] [qa] Status of account creds in the [identity] section of tempest.conf
On Thu, Jun 18, 2015 at 02:13:56PM -0400, David Kranz wrote: We had a discussion about this at the qa meeting today around the following proposal: tl;dr The test accounts feature provides the same functionality as the embedded credentials. We should deprecate the account information embedded directly in tempest.conf in favor of test-accounts, and remove those options at the beginning of the M cycle. We would also rework the non-isolated jobs to use parallel test accounts, with and without admin creds. Starting now, new features such as cleanup and tempest config will not be required to work well (or at all) if the embedded creds are used instead of test accounts. So this was always the long term plan when we started work on the test-accounts mechanism about a year ago. We were holding off on deprecating the original config option based approach until finished the role and network support for test accounts and we had jobs setup using the mechanism. Now that the we finished both role and network support all that's left is setting up the jobs. I don't think there would be any opposition to marking the user and alt user options as deprecated after that. Also leaving in line comments (and maybe emit a warning) marking the non-locking provider mechanism as going away, probably in M. That way we start clearly marking to users that these options will be going away. We have (at least) three use cases that are important, and we want tempest to work well with all of them, but that means something different in each case: 1. throw-away clouds (ci, gate) 2. test clouds 3. production clouds Well, tempest is designed to and tries to support running against any OpenStack cloud. I'm not sure if there are more deployment types than these 3 categories but we should also be supporting those too. For (1), the most important thing is that failing tests not cause false negatives in other tests due to re-using a tenant. This makes tenant isolation continue to be a good choice here, and requiring admin is not an issue. In a perfect world where tempest never left behind any resources regardless of an error at any line of code, test accounts could be used. But we are probably a long way from that. So the cleanup issue here is actually a wider openstack issue. Tempest will *always* call delete on created projects and users. This was a requirement for making test accounts work. (the mechanism for calling delete or freeing a credential set from the list is shared) With tenant isolation this means we'll be deleting a project and users and resources scoped to either may not be deleted first. (if there is a tempest or openstack bug) This is a wider issue for all OpenStack projects that there was a thread a few months ago discussing. For (3), we cannot use admin creds for tempest runs, and test accounts with cleanup allow parallel execution, accepting the risk of a leak causing a false negative. The only way to avoid that risk is to stamp out all leak bugs in tempest. Well depending on the resource in leak in question test accounts would likely catch the issues if there is a list on that resource in a later test. But, I agree, resource leaks have always been treated as bugs and we'll continue to do so. For (2), either isolation or test accounts with cleanup can be used The tempest.conf values are not used in any of these scenarios. Is there a reason they are needed for anything? So the only thing that uses config options for credentials is actually tenant isolation, which uses them to provide admin credentials to do the dynamic creation of accounts. The real advantage of tenant isolation, besides not reusing anything, is actually its configuration simplicity. Using an accounts file can be tricky to use, there are a lot of little gotchas and assumptions in how you write the file. (which we try to document in both the config guide and the sample accounts.yaml file) It also requires a large number of users to be provided depending on the concurrency you're running with. While tenant isolation requires just setting 3-5 config options and you're fine after that. I don't think requiring the use of an accounts file for tenant isolation makes much sense, it's really heavyweight for 1 set of admin creds. Which probably means we should keep the admin user config option around. Although it might make more sense to move those options to the auth section. (and maybe rename them to make it clear that it's for tenant isolation only) -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] Status of account creds in the [identity] section of tempest.conf
We had a discussion about this at the qa meeting today around the following proposal: tl;dr The test accounts feature provides the same functionality as the embedded credentials. We should deprecate the account information embedded directly in tempest.conf in favor of test-accounts, and remove those options at the beginning of the M cycle. We would also rework the non-isolated jobs to use parallel test accounts, with and without admin creds. Starting now, new features such as cleanup and tempest config will not be required to work well (or at all) if the embedded creds are used instead of test accounts. We have (at least) three use cases that are important, and we want tempest to work well with all of them, but that means something different in each case: 1. throw-away clouds (ci, gate) 2. test clouds 3. production clouds For (1), the most important thing is that failing tests not cause false negatives in other tests due to re-using a tenant. This makes tenant isolation continue to be a good choice here, and requiring admin is not an issue. In a perfect world where tempest never left behind any resources regardless of an error at any line of code, test accounts could be used. But we are probably a long way from that. For (3), we cannot use admin creds for tempest runs, and test accounts with cleanup allow parallel execution, accepting the risk of a leak causing a false negative. The only way to avoid that risk is to stamp out all leak bugs in tempest. For (2), either isolation or test accounts with cleanup can be used The tempest.conf values are not used in any of these scenarios. Is there a reason they are needed for anything? -David __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
