Re: [openstack-dev] [keystone] Feature Status and Exceptions
Hello, On Fri, Jul 13, 2018 at 03:50:33PM -0500, Lance Bragstad wrote: > On 07/13/2018 03:37 PM, Johannes Grassler wrote: > > On Fri, Jul 13, 2018 at 02:19:35PM -0500, Lance Bragstad wrote: > >> *Capability Lists** > >> * > >> The capability lists involves a lot of work, not just within keystone, > >> but also keystonemiddleware, which will freeze next week. I think it's > >> reasonable to say that this will be something that has to be pushed to > >> Stein [3]. > > I was was planning to email you about that, too...I didn't have much > > time for it lately (rushing to get a few changes in Monasca in plus a > > whole bunch of packaging stuff) and with the deadline this close I > > didn't see much of a chance to get anything meaningful in. > > > > So +1 for Stein from my side. This time I can plan for and accomodate it > > by having less Monasca stuff on my plate... > > +1 > > Thanks for confirming. There still seems to be quite a bit of discussion > around the data model and layout. We can use the PTG to focus on that as > a group if needed (and if you'll be there). For now I'll try to remain cautiously optimistic that this discussion can mostly be resolved by starting from the controller end and making my way to the data model from that side, as people suggested :-) As for the PTG: until now I was planning on skipping it. Lots of travel already this year and I need some quiet time without jetlag to work on the code, too... Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] Feature Status and Exceptions
Hello, On Fri, Jul 13, 2018 at 02:19:35PM -0500, Lance Bragstad wrote: > *Capability Lists** > * > The capability lists involves a lot of work, not just within keystone, > but also keystonemiddleware, which will freeze next week. I think it's > reasonable to say that this will be something that has to be pushed to > Stein [3]. I was was planning to email you about that, too...I didn't have much time for it lately (rushing to get a few changes in Monasca in plus a whole bunch of packaging stuff) and with the deadline this close I didn't see much of a chance to get anything meaningful in. So +1 for Stein from my side. This time I can plan for and accomodate it by having less Monasca stuff on my plate... Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone][Design Session] Where to propose extensions to trusts?
Hello, On 10/21/2016 03:07 PM, Steve Martinelli wrote: On Fri, Oct 21, 2016 at 5:01 AM, Johannes Grassler <jgrass...@suse.de> wrote: I've got a last minute proposal, or rather two of them for the Keystone side of the design sessions in Barcelona. [...] The Authorization work session is more than appropriate, please go ahead and add the items to the etherpad. Ok, will do. Thank you! Do try to be there in person, it'll make things easier. If you can't make the session then you can always pop into another keystone session or lurk around until the end of one where we can talk. I'll try to make the Authorization session. I think I can find an arrangement for the Barbican session. Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Keystone][Design Session] Where to propose extensions to trusts?
Hello, I've got a last minute proposal, or rather two of them for the Keystone side of the design sessions in Barcelona. I guess it's something that would fit the Authorization work session on Friday (09:00-09:40) but I'm not sure I can simply add it on the Etherpad. Also, I may not able to attend the session in person since I'll need to join the Barbican session in the same time slot as well[0], so I'd appreciate an alternative venue if that's possible. Hence I'll put them forth here for now: 1. Scope extensions for trusts == Various services (Magnum, Heat, maybe Neutron LBaaS soon) use Keystone trusts to grant their service users or dedicated trustee users limited rights for various purposes, such as deferred operations on behalf of the user or providing access to certificate containers. These trusts can only be limited to a set of roles in a given project right now. The Admin guide mentions an endpoint limitation on top of that, but I haven't found any code in Keystone for handling that. I played with it a bit and found that every keyword argument Keystone doesn't know what to do with ends up in the trust table's `extra` column, but there's no code for doing anything with it as far as I can tell. It would be a useful thing, though and implementing it goes hand in hand with my proposal: I'd like to see an ability to scope trusts to: 1) A subset of endpoints (i.e. "This trust may only access Barbican's internalURL and nothing else") 2) A fixed path (i.e. "This trust may only access /v1/secrets/238ca94d-6115-4fe6-b41f-c445eeb47df8) 3) A fixed URL (i.e. "This trust may only access http://192.168.123.20:9311/v1/secrets/238ca94d-6115-4fe6-b41f-c445eeb47df8) The main thing I'm currently interested in is whether this is feasible. If so, I'd be happy to work on a blueprint and implementation. I believe this is really important to allow services and users to limit trusts to exactly the access level needed and not a bit more. 2. Lightweight trusts = When Keystone trusts are created the current practice is to either 1) Delegate the trust to a service user using the trust (example: Heat) 2) Create a dedicated trustee user for delegating the trust to (example: Magnum) (1) is fine as far as I'm concerned, but I think (2) could do with some improvement. The dedicated trustee user will have a user name and password that needs to be used to authenticate as that user (along with a trust ID to consume the trust). I'd rather see a long-lived trust token scoped to the trust that can be extended upon expiry for that purpose. Just like a regular token, in other words. For the following reasons: 1) Everybody who creates such a user will have different idea of a secure password length. A token is generated by keystone in a centralized manner and always of a sufficient length to make for a secure secret. 2) It requires third party software that may need to authenticate as the trustee to be aware of trusts. Example: Kubernetes' Cinder volume driver (used by Magnum). Most such software should be able to handle an auth token, on the other hand. 3) There is less cleanup required after the trust has served its purpose: only the trust needs to be deleted at that point, but no trustee user. Comments on these two proposals (and advice on a suitable forum for discussing them at the summit) would be greatly appreciated. Thank you! Cheers, Johannes Footnotes: [0] In a pinch I could probably ask a colleague to stand in for me, but I'd prefer a solution where I can be present for both discussions. -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] keystoneclient.client.v3.Client: extract identity endpoint
Hello, On 10/14/2016 02:27 AM, Jamie Lennox wrote: On 13 October 2016 at 23:19, Johannes Grassler <jgrass...@suse.de> wrote: [Is there a canonical way to extract the identity URL being used by python-keystoneclient?] [...] keystone_service=client.services.list(type='identity')[0] client.endpoints.list(service=keystone_service, interface='admin', region=client.region_name) ...but that feels rather dirty since it independently looks up the admin endpoint rather than plucking the identity endpoint from the keystone client instance. [...] From the session you can do: session.get_endpoint(service_type='identity', interface='admin', region='region') to get the URL from the catalog. Ok, that is at least a little more elegant. Thank you! Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [keystone] keystoneclient.client.v3.Client: extract identity endpoint
Hello, I've got an existing keystoneclient.client.v3.Client object with an authenticated session. Now I'd like to get the identity URL this object uses for requesting things from Keystone. I want to use that URL in a trust's endpoint list in order to allow the user the client is authenticated as to talk to Keystone on the trustor's behalf. The client is authenticated as a service user and issues a GET to GET http://192.168.123.20/identity_admin/v3/OS-TRUST/trusts when the following code snippet is executed: client.trusts.list() (`client` is my keystoneclient.client.v3.Client instance). Initially I thought I could use the auth_url from the client's session object, i.e. client.session.auth.auth_url but that turned out to be a dead end because it's the internal endpoint: http://192.168.123.20/identity/v3 This will be useless for a trust's endpoint URL list if the trustee (my service user) ends up using http://192.168.123.20/identity_admin/v3 to talk to Keystone. I could look up the admin URL from the catalog like this... keystone_service=client.services.list(type='identity')[0] client.endpoints.list(service=keystone_service, interface='admin', region=client.region_name) ...but that feels rather dirty since it independently looks up the admin endpoint rather than plucking the identity endpoint from the keystone client instance. Is there a cleaner way to get that information directly from the keystoneclient.client.v3.Client instance? Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Listing Domain roles (or retrieving them by name)
Hello, On 09/20/2016 10:15 AM, Johannes Grassler wrote: is there a canonical way to either * list roles in a given domain * or retrieve a role from a given domain by name (preferred) Looks like there is a way: osc_lib.utils.find_resource(admin_client.roles, role_name, domain_id=domain_id) returns a role object for the role I'm looking for. (admin_client is the Keystone client's RoleManager, role_name contains the role's name). Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Keystone] Listing Domain roles (or retrieving them by name)
Hello, is there a canonical way to either * list roles in a given domain * or retrieve a role from a given domain by name (preferred) keystoneclient.v3.roles.RoleManager.list() does not appear to do the trick. While it takes a `domain` argument, it only returns roles with a domain_id=None attribute but none of the roles in the domain I specified. Also, it appears to be deprecated if this comment[0] in python-openstackclient is anything to go by. As for why I want to do this: I attempt to create the role in question and catch the Conflict exception that happens if a role with that name exists already. To use that existing role I need its UUID though (or a role object as keystoneclient.v3.roles.RoleManager.create() would have returned if it were successful). The name does not help since I cannot use that for keystoneclient.v3.roles.RoleManager.grant(). Come to think of it, a way to grant roles on a domain by name would also solve the problem... Cheers, Johannes [0] https://github.com/openstack/python-openstackclient/blob/master/openstackclient/identity/v3/role.py#L241 -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [magnum] Use Keystone trusts in Magnum?
Hello, On 07/06/2016 09:52 PM, Hongbin Lu wrote: Magnum generates Keystone trust for each bay: https://blueprints.launchpad.net/magnum/+spec/create-trustee-user-for-each-bay . Possibly, you can reuse the trust stored in the bay for this purpose. Oh...good to know that, thanks! That would make it a lot easier. I'll give it a try... Cheers, Johannes Best regards, Hongbin -Original Message- From: Johannes Grassler [mailto:jgrass...@suse.de] Sent: July-06-16 9:40 AM To: OpenStack Development Mailing List Subject: [openstack-dev] [magnum] Use Keystone trusts in Magnum? Hello, I submitted https://review.openstack.org/#/c/326428 a while ago to get around having to configure Heat's policy.json in a very permissive manner[0]. I naively only tested it as one user, but gating caught that omission and dutifully failed (a user cannot stack-get another user's Heat stack, even if it's the Magnum service user). Ordinarily, that is. Beyond the ordinary, Heat uses[1] Keystone trusts[2] to handle what is basically the same problem (acting on a user's behalf way past the time of the stack-create when the token used for the stack-create may have expired already). I propose doing the same thing in Magnum to get the Magnum service user the ability to perform a stack-get on all of its bays' stacks. That way the hairy problems with the wide-open permissions neccessary for a global stack-list can be avoided entirely. I'd be willing to implement this, either as part of the existing change referenced above or with a blueprint and all the bells and whistles. So I have two questions: 1) Is this an acceptable way to handle the issue? 2) If so, is it blueprint material or can I get away with adding the code required for Keystone trusts to the existing change? Cheers, Johannes Footnotes: [0] See Steven Hardy's excellent dissection of the problem at the root of it: http://lists.openstack.org/pipermail/openstack-dev/2016- July/098742.html [1] http://hardysteven.blogspot.de/2014/04/heat-auth-model-updates- part-1-trusts.html [2] https://wiki.openstack.org/wiki/Keystone/Trusts -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany ___ ___ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev- requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [magnum] Use Keystone trusts in Magnum?
Hello, I submitted https://review.openstack.org/#/c/326428 a while ago to get around having to configure Heat's policy.json in a very permissive manner[0]. I naively only tested it as one user, but gating caught that omission and dutifully failed (a user cannot stack-get another user's Heat stack, even if it's the Magnum service user). Ordinarily, that is. Beyond the ordinary, Heat uses[1] Keystone trusts[2] to handle what is basically the same problem (acting on a user's behalf way past the time of the stack-create when the token used for the stack-create may have expired already). I propose doing the same thing in Magnum to get the Magnum service user the ability to perform a stack-get on all of its bays' stacks. That way the hairy problems with the wide-open permissions neccessary for a global stack-list can be avoided entirely. I'd be willing to implement this, either as part of the existing change referenced above or with a blueprint and all the bells and whistles. So I have two questions: 1) Is this an acceptable way to handle the issue? 2) If so, is it blueprint material or can I get away with adding the code required for Keystone trusts to the existing change? Cheers, Johannes Footnotes: [0] See Steven Hardy's excellent dissection of the problem at the root of it: http://lists.openstack.org/pipermail/openstack-dev/2016-July/098742.html [1] http://hardysteven.blogspot.de/2014/04/heat-auth-model-updates-part-1-trusts.html [2] https://wiki.openstack.org/wiki/Keystone/Trusts -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat][Senlin] Deprecation roadmap for Heat's Autoscaling resources?
Hello, On 07/04/2016 02:52 PM, Thomas Herve wrote: On Mon, Jul 4, 2016 at 2:06 PM, Johannes Grassler <jgrass...@suse.de> wrote: [...] Right now OS::Heat::AutoScalingGroup and friends still exist, even in the master branch. Are they slated for removal at some stage or will they remain available even as Senlin becomes the go-to mechanism for autoscaling? They will remain available for the foreseeable future. We just don't have any great plans to improve them currently. Ok, good to know that. Thank you! Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat][Senlin] Deprecation roadmap for Heat's Autoscaling resources?
Hello, I just noticed the note at the top of <https://wiki.openstack.org/wiki/Heat/AutoScaling>: | The content on this page is obsolete. The autoscaling solution is offloaded from Heat to Senlin since Mitaka. Right now OS::Heat::AutoScalingGroup and friends still exist, even in the master branch. Are they slated for removal at some stage or will they remain available even as Senlin becomes the go-to mechanism for autoscaling? Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [magnum][heat] Global stack-list for Magnum service user
Hello, Thanks for the exhaustive comment on the issue. Won't help much in the short term, but it's good to see there will eventually be a way to sort this out properly! On 07/04/2016 12:50 PM, Steven Hardy wrote: On Mon, Jul 04, 2016 at 11:43:47AM +0200, Johannes Grassler wrote: [Magnum's global stack-list versus Heat's policy.json] Yes, this sort of problem is the reason we disabled global_index by default[1] - because of the somewhat notorious keystone bug #968696[2], we could not create a default rule which correctly communicated that this should be a cloud-admin only action. So, instead of creating an insecure-by-default rule, we disabled it for everybody, so that deployers could make an explicit choice about how to enable access to this potentially sensitive API. Ok, that's fair enough. | stacks:global_index: "role:service", [...] I don't think this is feasible, because it implies a level of admin-ness for service users that I think isn't desirable (even it if may be the status-quo, I don't personally think trusting all services to have global access to heat by default is a good model from a security isolation perspective). Yes...also, it just shifts the problem as Pavlo pointed out: an admin user can just assign themselves the 'service' role in their own tenant. So that's no advantage over using role:admin :-) [...] So, in summary, I think we should fix our integration with the new keystone is_admin_project stuff, then potentially switch the global_index to use the is_admin rule as defined by our policy.json. Indeed, that sounds a lot better. Then, you'd just need to add the magnum service user to whatever project is defined in keystone as being the admin project, but it would not require exposing this API to every other service by default. Hope that helps! Definitely does, thanks! Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [magnum][heat] Global stack-list for Magnum service user
Hello, Magnum has a periodic task that checks the state of the Heat stacks it creates for its bays. It does this across all users/tenants that have Magnum bays. Currently it uses a global stack-list operation to query these Heat stacks: https://github.com/openstack/magnum/blob/master/magnum/service/periodic.py#L83 Now the Magnum service user does not normally have permission to perform this operation, hence the Magnum documentation currently suggests the following change to Heat's policy.json: | stacks:global_index: "role:admin", This is less than optimal since it allows any tenant's admin user to perform a global stack-list. Would it be an option to have something like this in Heat's default policy.json? | stacks:global_index: "role:service", That way the global stack-list would be restricted to service users and seting Magnum (or other services that use Heat internally) wouldn't need a change to Heat's policy.json. If that kind of approach is feasible I'd be happy to submit a change. Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Dealing with nonexistent resources during resource-list / stack-delete
On 03/08/2016 06:26 PM, Zane Bitter wrote: On 08/03/16 08:03, Johannes Grassler wrote: I think you're being a little too pessimistic. I was going to explain one approach, but it turned out to be easier to just submit a patch: https://review.openstack.org/290027 Ah, that one looks good, thanks :-) I believe this satisfies requirements 1, 2, and 4. I agree that there's no way we can really tackle 3 as far as third-party plugins go, but at least all explicit dependencies are guaranteed to be added. For the implicit ones, it's up to plugin authors not to screw it up. That's fair enough and makes for a good warning to include in the docs. BTW I created https://bugs.launchpad.net/heat/+bug/1554625 to cover this larger issue (as opposed to the specific problem you are fixing in bug 1442121); that would probably be a good one to reference in your docs changes. Ok, thanks. That link will prevent a lot of unwieldy problem explanation people are unlikely to read anyway :-) Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Dealing with nonexistent resources during resource-list / stack-delete
Hello, On 03/08/2016 04:57 PM, Zane Bitter wrote: On 08/03/16 10:40, Johannes Grassler wrote: On 03/07/2016 04:48 PM, Zane Bitter wrote: On 04/03/16 04:35, Johannes Grassler wrote: [Uncaught client exceptions in resource plugins' add_dependencies() methods] In the meantime, we need to find and squash every instance of this problem wherever we can like you said. It might also be a good idea to caution against unchecked API client invocations in http://docs.openstack.org/developer/heat/developing_guides/pluginguide.html [...] It's best if they *do* omit it entirely. The only reason we override it in the Neutron resources is that the Neutron API is terrible for orchestration purposes[1]. It adds a bunch of invisible, fragile magic that breaks in subtle ways when e.g. resources are moved into nested stacks. I never saw the Neutron API that way before (I guess I just got used to the unintuitive bits), but seeing it spelled out in your rant makes it very obvious, yes. I didn't know that was the root cause for overriding add_dependencies() and that very ignorance of mine... The default implementation provides everything that we *ought* to need, so if we document anything I think it should be that plugin developers should not touch add_dependencies() at all. ...suggests mentioning that much is probably a good idea (lest somebody pick one of the Neutron plugins as a template to base their own resource plugin on). Definitely not big enough to require a spec IMO. Yes, I can see that, given how it's not something plugin writers should do anyway. Then I'll just write a little paragraph cautioning against overriding add_dependcies() and add a Related-Bug: line. Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Dealing with nonexistent resources during resource-list / stack-delete
Hello, On 03/07/2016 04:48 PM, Zane Bitter wrote: On 04/03/16 04:35, Johannes Grassler wrote: [Uncaught client exceptions in resource plugins' add_dependencies() methods] In the meantime, we need to find and squash every instance of this problem wherever we can like you said. It might also be a good idea to caution against unchecked API client invocations in http://docs.openstack.org/developer/heat/developing_guides/pluginguide.html until a real fix for the problem is in place. Also, that document does not even mention the add_dependencies() method, leaving plugin developers that much more room to come up with dodgy code that does weird stuff in add_dependencies() or even omits it entirely. I haven't written any plugins so far, but I suppose I could update that part of the documentation - I've become rather familiar this part of the code while fixing <https://bugs.launchpad.net/bugs/1442121>. If I wanted to add a section on add_dependencies() and its pitfalls to pluginguide.rst, how would I go about it? Just submit a change with a "Partial-Bug: #1442121" or is that big enough to require a spec? Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Dealing with nonexistent resources during resource-list / stack-delete
Hello, On 03/07/2016 04:48 PM, Zane Bitter wrote: On 04/03/16 04:35, Johannes Grassler wrote: [Uncaught client exceptions in resource plugins' add_dependencies() methods] Yes, you're right and this sucks. That's not the only problem we've had in this area recently - for example there was also: https://bugs.launchpad.net/heat/+bug/1536515. The fact that we have to have these hacked in implicit dependencies at all is terrible, but we really need to make sure they can't break basic operations like loading a stack from the DB so we can show or delete it. The ideal would be: * We can guarantee to catch all (non-exit) exceptions, no matter what kind of crazy stuff people write in add_dependencies() * The plugin developer doesn't have to do anything special to get this behaviour Just these two are a tall order already. One could have the client plugins default to ignoring 404s (maybe by adding a `ignore_defaults` keyword argument to the client_plugin() method that defaults to True). That will break third-party plugins that need to handle this exception themselves for one reason or another. Are there such plugins, or could there conceivably be such plugins? I can't think of a likely scenario off the top of my head, but since I'm not familiar with any third party plugins that may not mean much. Also, and probably more serious, it may require a potentially largish number of adaptations to core heat code in places where this behaviour is undesirable (and that might in turn cause new bugs). * The code remains backwards compatible with any third-party resource plugins circulating out there ...and that rules out handling the exception at a higher level (i.e. whereever add_dependencies() is called). Doing that creates a lot of possibilities to end up with a messy dependency graph. * We always add as many dependencies as possible (i.e. all non-exception-raising dependencies are added) That pretty much means code in the add_dependcies() method, and the only feasible way I see of influencing this code is finding a way to selectively get the client plugins to default to ignoring dependencies. * Genuine dependency problems (e.g. non-existent target of get_resource/get_attr) are still surfaced, preferably on CREATE only Ignoring the 404s at a higher level may be feasible in the DELETE case, but I'm not sure how bad a problem a possibly incomplete dependency graph creates for stack-delete: is it a complete show stopper or just a matter of issuing multiple stack-delete requests until all resources are gone? Cheers, Johannes -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat] Dealing with nonexistent resources during resource-list / stack-delete
Hello, I am currently looking into https://bugs.launchpad.net/heat/+bug/1442121 and not quite sure how to tackle it, since the problematic code is used for lots of things[0]: the root cause of the problem are API clients in resource plugins that do not anticipate a resource with an entry in Heat's database having been deleted in the implementing service's database[1]. Here's an example: https://github.com/openstack/heat/blob/e4dc942ce1a8c79b450345c7afae326c80d8a5d6/heat/engine/resources/openstack/neutron/floatingip.py#L179 If that happens[1] an uncaught exception will be thrown that among other things breaks the very operations one would need for cleaning up the mess. As far as I can see it, the cleanest way would be to go through all resources with a fine comb and add exception handling to the API calls in the add_dependencies() method where it is missing (just return False for any resource that no longer exists). Or is there a better way? Cheers, Johannes Footnotes: [0] Whenever a stack's resources are being listed using heat.engine.service.list_stack_resources(). resource-list and stack-delete, all invoke list_stack_resources()). stack-abandon does so indirectly (it appears to trigger stack-delete judging by the log, but it yields the desired output, at least in Liberty). These are just the ones I tested, there are probably more. [1] It can happen for a number of reasons, either due to resource dependency problems upon stack-delete as it happened in the original bug report or due to an operator accidently deleting resources that are managed by Heat. -- Johannes Grassler, Cloud Developer SUSE Linux GmbH, HRB 21284 (AG Nürnberg) GF: Felix Imendörffer, Jane Smithard, Graham Norton Maxfeldstr. 5, 90409 Nürnberg, Germany __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
