Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
On Sun, Apr 06, 2014 at 10:22:15PM -0400, Michael Elder wrote: If Keystone is configured with an external identity provider (LDAP, OpenID, etc), how does the creation of a new user per resource affect that external identity source? My understanding is that it should be possible to configure keystone to use multiple (domain specific) identity backends. So a possible configuration could be to have real users backed by the LDAP directory, and have all projects/users associated with heat (which are created in a special heat domain, completely separated from real users) backed by some other identity backend, e.g Sql. http://docs.openstack.org/developer/keystone/configuration.html#domain-specific-drivers This is something we should definitely test, and I'd welcome feedback from the keystone folks, or anyone who has experience with this functionality, as to how well it works in Icehouse. My suggestion is broader, but in the same spirit: Could we consider defining an _authorization_ stack token (thanks Adam), which acts like an OAuth token (by delegating a set of actionable behaviors that a token holder may perform). The stack token would be managed within the stack in some protected form and used for any activities later performed on resources which are managed by the stack. Instead of imposing user administration tasks like creating users, deleting users, etc against the Keystone database, Heat would instead provide these stack tokens to any service which it connects to when managing a resource. In fact, there's no real reason that the stack token couldn't piggyback on the existing Keystone token mechanism, except that it would be potentially longer lived and restricted to the specific set of resources for which it was granted. So oauth was considered before we implemented the domain-isolated users, but it was not possible to persue due to lack of client support: https://wiki.openstack.org/wiki/Heat/Blueprints/InstanceUsers https://blueprints.launchpad.net/heat/+spec/oauth-credentials-resource The main issue with tokens as provided by keystone today, is that they will expire. That is the main reason for chosing to create a user rather than e.g a token limited in scope via a trust - if you put it in the instance, you have to refresh it before expiry, which may not always be possible. Additionally, you don't really want the credentials deployed inside a (implicitly untrusted) instance derived from the operator who created the stack - you want something associated with the stack but completely isolated from real users Your stack token approach above appears to indicate that Heat would somehow generate, and maintain the lifecycle of, some special token which is not owned by keystone. This idea has been discussed, and rejected, because we would prefer to make use of keystone functionality instead of having the burden of maintaining our own bespoke authorization system. If implementing something based on oauth, or some sort of scope-limited non-expiring token, becomes possible, it's quite likely we can provide the option to do something other than the domain isolated users which has been impelmented for Icehouse. Ultimately, we had to use what was available in keystone *now* to enable delivery of something which worked for Icehouse, hence the decision to use what is available in the keystone v3 API. Not sure if email is the best medium for this discussion, so if there's a better option, I'm happy to follow that path as well. I think it's fine, and I'm happy to get constructive feedback on the the current approach, along with ideas for roadmap items which can potentially improve it. I have proposed this summit session which may provide more opportunity for discussion, if accepted: http://summit.openstack.org/cfp/details/190 Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
Hi Steve, Thank you -- this clarifies things quite a bit. I'd like to join that discussion at the summit if possible. -M Kind Regards, Michael D. Elder STSM | Master Inventor mdel...@us.ibm.com | linkedin.com/in/mdelder Success is not delivering a feature; success is learning how to solve the customer’s problem.” -Mark Cook From: Steven Hardy sha...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 04/07/2014 12:00 PM Subject:Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2 On Sun, Apr 06, 2014 at 10:22:15PM -0400, Michael Elder wrote: If Keystone is configured with an external identity provider (LDAP, OpenID, etc), how does the creation of a new user per resource affect that external identity source? My understanding is that it should be possible to configure keystone to use multiple (domain specific) identity backends. So a possible configuration could be to have real users backed by the LDAP directory, and have all projects/users associated with heat (which are created in a special heat domain, completely separated from real users) backed by some other identity backend, e.g Sql. http://docs.openstack.org/developer/keystone/configuration.html#domain-specific-drivers This is something we should definitely test, and I'd welcome feedback from the keystone folks, or anyone who has experience with this functionality, as to how well it works in Icehouse. My suggestion is broader, but in the same spirit: Could we consider defining an _authorization_ stack token (thanks Adam), which acts like an OAuth token (by delegating a set of actionable behaviors that a token holder may perform). The stack token would be managed within the stack in some protected form and used for any activities later performed on resources which are managed by the stack. Instead of imposing user administration tasks like creating users, deleting users, etc against the Keystone database, Heat would instead provide these stack tokens to any service which it connects to when managing a resource. In fact, there's no real reason that the stack token couldn't piggyback on the existing Keystone token mechanism, except that it would be potentially longer lived and restricted to the specific set of resources for which it was granted. So oauth was considered before we implemented the domain-isolated users, but it was not possible to persue due to lack of client support: https://wiki.openstack.org/wiki/Heat/Blueprints/InstanceUsers https://blueprints.launchpad.net/heat/+spec/oauth-credentials-resource The main issue with tokens as provided by keystone today, is that they will expire. That is the main reason for chosing to create a user rather than e.g a token limited in scope via a trust - if you put it in the instance, you have to refresh it before expiry, which may not always be possible. Additionally, you don't really want the credentials deployed inside a (implicitly untrusted) instance derived from the operator who created the stack - you want something associated with the stack but completely isolated from real users Your stack token approach above appears to indicate that Heat would somehow generate, and maintain the lifecycle of, some special token which is not owned by keystone. This idea has been discussed, and rejected, because we would prefer to make use of keystone functionality instead of having the burden of maintaining our own bespoke authorization system. If implementing something based on oauth, or some sort of scope-limited non-expiring token, becomes possible, it's quite likely we can provide the option to do something other than the domain isolated users which has been impelmented for Icehouse. Ultimately, we had to use what was available in keystone *now* to enable delivery of something which worked for Icehouse, hence the decision to use what is available in the keystone v3 API. Not sure if email is the best medium for this discussion, so if there's a better option, I'm happy to follow that path as well. I think it's fine, and I'm happy to get constructive feedback on the the current approach, along with ideas for roadmap items which can potentially improve it. I have proposed this summit session which may provide more opportunity for discussion, if accepted: http://summit.openstack.org/cfp/details/190 Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
Adam, I was imprecise, thank you for correcting that error. I think the net of the statement still holds though: the Keystone token mechanism defines a mechanism for authorization, why doesn't the heat stack manage a token for any behavior that requires authorization? -M Kind Regards, Michael D. Elder STSM | Master Inventor mdel...@us.ibm.com | linkedin.com/in/mdelder Success is not delivering a feature; success is learning how to solve the customer’s problem.” -Mark Cook From: Adam Young ayo...@redhat.com To: openstack-dev@lists.openstack.org Date: 04/04/2014 09:54 PM Subject:Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2 On 04/04/2014 02:46 PM, Clint Byrum wrote: Excerpts from Michael Elder's message of 2014-04-04 07:16:55 -0700: Opened in Launchpad: https://bugs.launchpad.net/heat/+bug/1302624 I still have concerns though about the design approach of creating a new project for every stack and new users for every resource. If I provision 1000 patterns a day with an average of 10 resources per pattern, you're looking at 10,000 users per day. How can that scale? If that can't scale, then keystone is not viable at all. I like to think we can scale keystone to the many millions of users level. How can we ensure that all stale projects and users are cleaned up as instances are destroy? When users choose to go through horizon or nova to tear down instances, what cleans up the project users associated with that heat stack? So, they created these things via Heat, but have now left the dangling references in Heat, and expect things to work properly? If they create it via Heat, they need to delete it via Heat. Keystone defines the notion of tokens to support authentication, why doesn't the design provision and store a token for the stack and its equivalent management? Tokens are _authentication_, not _authorization_. Tokens are authorization, not authentication. For Authentication you should be using a real cryptographically secure authentication mechanism: either Kerberos or X509. For the latter, we need to have a way to lock down access to an individual resource in Heat. This allows putting secrets in deployments and knowing that only the instance which has been deployed to will have access to the secrets. I do see an optimization possible, which is to just create a user for the box that is given access to any deployments on the box. That would make sense if users are going to create many many deployments per server. But even at 10 per server, having 10 users is simpler than trying to manage shared users and edit their authorization rules. Now, I actually think that OAUTH tokens _are_ intended to be authorization as well as authentication, so that is probably where the focus should be long term. But really, you're talking about the same thing: a single key lookup in keystone. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
On 07/04/14 12:52, Michael Elder wrote: I think the net of the statement still holds though: the Keystone token mechanism defines a mechanism for authorization, why doesn't the heat stack manage a token for any behavior that requires authorization? Heat does use a token, but that token is associated with a user which can only perform limited operations on one heat resource. This reduces the risk that an unauthorized action can be performed due to using some form of shared user. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
If Keystone is configured with an external identity provider (LDAP, OpenID, etc), how does the creation of a new user per resource affect that external identity source? My suggestion is broader, but in the same spirit: Could we consider defining an _authorization_ stack token (thanks Adam), which acts like an OAuth token (by delegating a set of actionable behaviors that a token holder may perform). The stack token would be managed within the stack in some protected form and used for any activities later performed on resources which are managed by the stack. Instead of imposing user administration tasks like creating users, deleting users, etc against the Keystone database, Heat would instead provide these stack tokens to any service which it connects to when managing a resource. In fact, there's no real reason that the stack token couldn't piggyback on the existing Keystone token mechanism, except that it would be potentially longer lived and restricted to the specific set of resources for which it was granted. Not sure if email is the best medium for this discussion, so if there's a better option, I'm happy to follow that path as well. -M Kind Regards, Michael D. Elder STSM | Master Inventor mdel...@us.ibm.com | linkedin.com/in/mdelder Success is not delivering a feature; success is learning how to solve the customer’s problem.” -Mark Cook From: Steve Baker sba...@redhat.com To: openstack-dev@lists.openstack.org Date: 04/06/2014 09:16 PM Subject:Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2 On 07/04/14 12:52, Michael Elder wrote: I think the net of the statement still holds though: the Keystone token mechanism defines a mechanism for authorization, why doesn't the heat stack manage a token for any behavior that requires authorization? Heat does use a token, but that token is associated with a user which can only perform limited operations on one heat resource. This reduces the risk that an unauthorized action can be performed due to using some form of shared user.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
Opened in Launchpad: https://bugs.launchpad.net/heat/+bug/1302624 I still have concerns though about the design approach of creating a new project for every stack and new users for every resource. If I provision 1000 patterns a day with an average of 10 resources per pattern, you're looking at 10,000 users per day. How can that scale? How can we ensure that all stale projects and users are cleaned up as instances are destroy? When users choose to go through horizon or nova to tear down instances, what cleans up the project users associated with that heat stack? Keystone defines the notion of tokens to support authentication, why doesn't the design provision and store a token for the stack and its equivalent management? -M Kind Regards, Michael D. Elder STSM | Master Inventor mdel...@us.ibm.com | linkedin.com/in/mdelder Success is not delivering a feature; success is learning how to solve the customer’s problem.” -Mark Cook From: Steve Baker sba...@redhat.com To: openstack-dev@lists.openstack.org Date: 04/03/2014 10:13 PM Subject:Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2 On 04/04/14 14:05, Michael Elder wrote: Hello, I'm looking for insights about the interaction between keystone and the software configuration work that's gone into Icehouse in the last month or so. I've found that when using software configuration, the KeystoneV2 is broken because the server.py#_create_transport_credentials() explicitly depends on KeystoneV3 methods. Here's what I've come across: In the following commit, the introduction of _create_transport_credentials() on server.py begins to create a user for each OS::Nova::Server resource in the template: commit b776949ae94649b4a1eebd72fabeaac61b404e0f Author: Steve Baker sba...@redhat.com Date: Mon Mar 3 16:39:57 2014 +1300 Change: https://review.openstack.org/#/c/77798/ server.py lines 470-471: if self.user_data_software_config(): self._create_transport_credentials() With the introduction of this change, each server resource which is provisioned results in the creation of a new user ID. The call delegates through to stack_user.py lines 40-54: def _create_user(self): # Check for stack user project, create if not yet set if not self.stack.stack_user_project_id: project_id = self.keystone().create_stack_domain_project( self.stack.id) self.stack.set_stack_user_project_id(project_id) # Create a keystone user in the stack domain project user_id = self.keystone().create_stack_domain_user( username=self.physical_resource_name(),## HERE THE USERNAME IS SET TO THE RESOURCE NAME password=self.password, project_id=self.stack.stack_user_project_id) # Store the ID in resource data, for compatibility with SignalResponder db_api.resource_data_set(self, 'user_id', user_id) My concerns with this approach: - Each resource is going to result in the creation of a unique user in Keystone. That design point seems hardly teneble if you're provisioning a large number of templates by an organization every day. Compared to the resources consumed by creating a new nova server (or a keystone token!), I don't think creating new users will present a significant overhead. As for creating users bound to resources, this is something heat has done previously but we're doing it with more resources now. With havana heat (or KeystoneV2) those users will be created in the same project as the stack launching user, and the stack launching user needs admin permissions to create these users. - If you attempt to set your resource names to some human-readable string (like web_server), you get one shot to provision the template, wherein future attempts to provision it will result in exceptions due to duplicate user ids. This needs a bug raised. This isn't an issue on KeystoneV3 since the users are created in a project which is specific to the stack. Also for v3 operations the username is ignored as the user_id is used exclusively. - The change prevents compatibility between Heat on Icehouse and KeystoneV2. Please continue to test this with KeystoneV2. However any typical icehouse OpenStack should really have the keystone v3 API enabled. Can you explain the reasons why yours isn't? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
Excerpts from Michael Elder's message of 2014-04-04 07:16:55 -0700: Opened in Launchpad: https://bugs.launchpad.net/heat/+bug/1302624 I still have concerns though about the design approach of creating a new project for every stack and new users for every resource. If I provision 1000 patterns a day with an average of 10 resources per pattern, you're looking at 10,000 users per day. How can that scale? If that can't scale, then keystone is not viable at all. I like to think we can scale keystone to the many millions of users level. How can we ensure that all stale projects and users are cleaned up as instances are destroy? When users choose to go through horizon or nova to tear down instances, what cleans up the project users associated with that heat stack? So, they created these things via Heat, but have now left the dangling references in Heat, and expect things to work properly? If they create it via Heat, they need to delete it via Heat. Keystone defines the notion of tokens to support authentication, why doesn't the design provision and store a token for the stack and its equivalent management? Tokens are _authentication_, not _authorization_. For the latter, we need to have a way to lock down access to an individual resource in Heat. This allows putting secrets in deployments and knowing that only the instance which has been deployed to will have access to the secrets. I do see an optimization possible, which is to just create a user for the box that is given access to any deployments on the box. That would make sense if users are going to create many many deployments per server. But even at 10 per server, having 10 users is simpler than trying to manage shared users and edit their authorization rules. Now, I actually think that OAUTH tokens _are_ intended to be authorization as well as authentication, so that is probably where the focus should be long term. But really, you're talking about the same thing: a single key lookup in keystone. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
Excerpts from Adam Young's message of 2014-04-04 18:48:40 -0700: On 04/04/2014 02:46 PM, Clint Byrum wrote: Excerpts from Michael Elder's message of 2014-04-04 07:16:55 -0700: Opened in Launchpad: https://bugs.launchpad.net/heat/+bug/1302624 I still have concerns though about the design approach of creating a new project for every stack and new users for every resource. If I provision 1000 patterns a day with an average of 10 resources per pattern, you're looking at 10,000 users per day. How can that scale? If that can't scale, then keystone is not viable at all. I like to think we can scale keystone to the many millions of users level. How can we ensure that all stale projects and users are cleaned up as instances are destroy? When users choose to go through horizon or nova to tear down instances, what cleans up the project users associated with that heat stack? So, they created these things via Heat, but have now left the dangling references in Heat, and expect things to work properly? If they create it via Heat, they need to delete it via Heat. Keystone defines the notion of tokens to support authentication, why doesn't the design provision and store a token for the stack and its equivalent management? Tokens are _authentication_, not _authorization_. Tokens are authorization, not authentication. For Authentication you should be using a real cryptographically secure authentication mechanism: either Kerberos or X509. Indeed, I may have used the terms incorrectly. Unless I'm mistaken, a token is valid wherever it is presented. It is simply proving that you authenticated yourself to keystone and that you have xyz roles. Perhaps the roles are authorization. But those roles aren't scoped to a token, they're scoped to a user, so it still remains that it serves as authentication for what you have and what you're authorized to do as a whole user. That is why I suggest OAUTH, because that is a scheme which offers tokens with limited scope. We kind of have the same thing with trusts, but that also doesn't really offer the kind of isolation what we want, nor does it really offer advantages over user-per-deployment. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
Hello, I'm looking for insights about the interaction between keystone and the software configuration work that's gone into Icehouse in the last month or so. I've found that when using software configuration, the KeystoneV2 is broken because the server.py#_create_transport_credentials() explicitly depends on KeystoneV3 methods. Here's what I've come across: In the following commit, the introduction of _create_transport_credentials() on server.py begins to create a user for each OS::Nova::Server resource in the template: commit b776949ae94649b4a1eebd72fabeaac61b404e0f Author: Steve Baker sba...@redhat.com Date: Mon Mar 3 16:39:57 2014 +1300 Change: https://review.openstack.org/#/c/77798/ server.py lines 470-471: if self.user_data_software_config(): self._create_transport_credentials() With the introduction of this change, each server resource which is provisioned results in the creation of a new user ID. The call delegates through to stack_user.py lines 40-54: def _create_user(self): # Check for stack user project, create if not yet set if not self.stack.stack_user_project_id: project_id = self.keystone().create_stack_domain_project( self.stack.id) self.stack.set_stack_user_project_id(project_id) # Create a keystone user in the stack domain project user_id = self.keystone().create_stack_domain_user( username=self.physical_resource_name(), ## HERE THE USERNAME IS SET TO THE RESOURCE NAME password=self.password, project_id=self.stack.stack_user_project_id) # Store the ID in resource data, for compatibility with SignalResponder db_api.resource_data_set(self, 'user_id', user_id) My concerns with this approach: - Each resource is going to result in the creation of a unique user in Keystone. That design point seems hardly teneble if you're provisioning a large number of templates by an organization every day. - If you attempt to set your resource names to some human-readable string (like web_server), you get one shot to provision the template, wherein future attempts to provision it will result in exceptions due to duplicate user ids. - The change prevents compatibility between Heat on Icehouse and KeystoneV2. The change comments were a bit sparse on the design reasoning behind this approach, and my search of the mail archives was unsuccessful. http://openstack.markmail.org/search/?q=heat+on+keystone+v2 -M Kind Regards, Michael D. Elder STSM | Master Inventor mdel...@us.ibm.com | linkedin.com/in/mdelder Success is not delivering a feature; success is learning how to solve the customer’s problem.” -Mark Cook ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Problems with Heat software configurations and KeystoneV2
On 04/04/14 14:05, Michael Elder wrote: Hello, I'm looking for insights about the interaction between keystone and the software configuration work that's gone into Icehouse in the last month or so. I've found that when using software configuration, the KeystoneV2 is broken because the server.py#_create_transport_credentials() explicitly depends on KeystoneV3 methods. Here's what I've come across: In the following commit, the introduction of _create_transport_credentials() on server.py begins to create a user for each OS::Nova::Server resource in the template: commit b776949ae94649b4a1eebd72fabeaac61b404e0f Author: Steve Baker sba...@redhat.com Date: Mon Mar 3 16:39:57 2014 +1300 Change: https://review.openstack.org/#/c/77798/ server.py lines 470-471: if/self/.user_data_software_config(): /self/._create_transport_credentials() With the introduction of this change, each server resource which is provisioned results in the creation of a new user ID. The call delegates through to stack_user.py lines 40-54: def*_create_user*(/self/): # Check for stack user project, create if not yet set ifnot/self/.stack.stack_user_project_id: project_id = /self/.keystone().create_stack_domain_project( /self/.stack.id) /self/.stack.set_stack_user_project_id(project_id) # Create a _keystone_ user in the stack domain project user_id = /self/.keystone().create_stack_domain_user( username=/self/.physical_resource_name(),## HERE THE USERNAME IS SET TO THE RESOURCE NAME password=/self/.password, project_id=/self/.stack.stack_user_project_id) # Store the ID in resource data, for compatibility with SignalResponder db_api.resource_data_set(/self/, /'user_id'/, user_id) My concerns with this approach: - Each resource is going to result in the creation of a unique user in Keystone. That design point seems hardly teneble if you're provisioning a large number of templates by an organization every day. Compared to the resources consumed by creating a new nova server (or a keystone token!), I don't think creating new users will present a significant overhead. As for creating users bound to resources, this is something heat has done previously but we're doing it with more resources now. With havana heat (or KeystoneV2) those users will be created in the same project as the stack launching user, and the stack launching user needs admin permissions to create these users. - If you attempt to set your resource names to some human-readable string (like web_server), you get one shot to provision the template, wherein future attempts to provision it will result in exceptions due to duplicate user ids. This needs a bug raised. This isn't an issue on KeystoneV3 since the users are created in a project which is specific to the stack. Also for v3 operations the username is ignored as the user_id is used exclusively. - The change prevents compatibility between Heat on Icehouse and KeystoneV2. Please continue to test this with KeystoneV2. However any typical icehouse OpenStack should really have the keystone v3 API enabled. Can you explain the reasons why yours isn't? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
