Re: [openstack-dev] [ironic] using keystone right - catalog, endpoints, tokens and noauth

[Pavlo Shchelokovskyy]

Hi all,

thanks Monty for the feedback. I've started this set of patches in ironic
[0] (very WiP currently), and would really like some eyes on them (not
right now, as I plan to rewrite those basing on this conversation :) but
soon I hope). Also I have some questions/comments inline:

On Thu, May 25, 2017 at 12:36 AM, Monty Taylor  wrote:

> On 05/24/2017 12:51 PM, Eric Fried wrote:
>
>> Pavlo-
>>
>> There's a blueprint [1] whereby we're trying to address a bunch of
>> these same concerns in nova.  You can see the first part in action here
>> [2].  However, it has become clear that nova is just one of the many
>> services that would benefit from get_service_url().  With the full
>> support of mordred (let's call it The Full Monty), we've got our sights
>> on moving that method into ksa itself for that purpose.
>>
>
> Yes - this has started with documenting how to consume Keystone Catalog
> and discovery properly.
>
> https://review.openstack.org/#/q/topic:version-discovery
>
> (it's a big stack)
>
> Once we're good with that - the next step is getting ksa updated to be
> able to handle the end-to-end. It does most of it today, but there are
> enough edgecases it doesn't that you wind up having to do something else,
> like efried just did in nova. The goal is to make that not necessary - and
> so that it's both possible and EASY for everyone to CORRECTLY consume
> catalog and version discovery.
>
> (more comments inline below)
>
>
> Please have a look at this blueprint and change set.  Let us know
>> if
>> your concerns would be addressed if this were available to you from ksa.
>>
>> [1]
>> https://specs.openstack.org/openstack/nova-specs/specs/pike/
>> approved/use-service-catalog-for-endpoints.html
>> [2] https://review.openstack.org/#/c/458257/
>>
>> Thanks,
>> efried
>>
>> On 05/24/2017 04:46 AM, Pavlo Shchelokovskyy wrote:
>>
>>> Hi all,
>>>
>>> There are several problems or inefficiencies in how we are dealing with
>>> auth to other services. Although it became much better in Newton, some
>>> things are still to be improved and I like to discuss how to tackle
>>> those and my ideas for that.
>>>
>>> Keystone endpoints
>>> ===
>>>
>>> Apparently since February-ish DevStack no longer sets up 'internal'
>>> endpoints for most of the services in core devstack [0].
>>> Luckily we were not broken by that right away - although when
>>> discovering a service endpoint from keystone catalog we default to
>>> 'internal' endpoint [1], for most services our devstack plugin still
>>> configures explicit service URL in the corresponding config section, and
>>> thus the service discovery from keystone never takes place (or that code
>>> path is not tested by functional/integration testing).
>>>
>>> AFAIK different endpoint types (internal vs public) are still quite used
>>> by deployments (and IMO rightfully so), so we have to continue
>>> supporting that. I propose to take the following actions:
>>>
>>
> I agree you should continue supporting it.
>
> I'm not sure it's important for you to change your defaults ... as long at
> it's possible to consistently set "interface=public" or
> "interface=internal" and have the results be correct, I think that's the
> big win.
>
> - in our devstack plugin, stop setting up the direct service URLs in
>>> config, always use keystone catalog for discovery
>>>
>>
> YES
>
> - in every conf section related to external service add
>>> 'endpoint_type=[internal|public]' option, defaulting to 'internal', with
>>> a warning in option description (and validated on conductor start) that
>>> it will be changed to 'public' in the next release
>>>
>>
> efried just added a call to keystoneauth which will register all of the
> appropriate CONF options that are needed to request a service endpoint from
> the catalog - register_adapter_conf_options:
>
> http://git.openstack.org/cgit/openstack/keystoneauth/tree/ke
> ystoneauth1/loading/__init__.py#n39
>
> The word "adapter" in this case isn't directly important - but there are
> three general concepts in keystoneauth that relate to how you connect:
>
> auth
>  - how you authenticate - auth_type, username, password, etc.
> session
>  - how the transport layer connects - certs, timeouts, etc.
> adapter
>  - what base endpoint to mount from the catalog - service_type, interface,
> endpoint_override, api_version


Currently I'm trying to understand how to use adapters for creating
clients. It seems that not all of clients of interest for me support this,
as some ignore 'interface', 'endpoint_override'  etc of the adapter
instance if I'm passing it instead of a session, and always rely on the
same/similar options passed to client. Are there any examples of how to use
them? Also, some clients (e.g. a neutron [1]) already base their
SessionClient on keystoneauth1.adapter.Adapter, so how could one create
such client instance with adapter options loaded from 

Re: [openstack-dev] [ironic] using keystone right - catalog, endpoints, tokens and noauth

[Dmitry Tantsur]

On 05/24/2017 11:46 AM, Pavlo Shchelokovskyy wrote:

Hi all,

There are several problems or inefficiencies in how we are dealing with auth to 
other services. Although it became much better in Newton, some things are still 
to be improved and I like to discuss how to tackle those and my ideas for that.


Keystone endpoints
===

Apparently since February-ish DevStack no longer sets up 'internal' endpoints 
for most of the services in core devstack [0].
Luckily we were not broken by that right away - although when discovering a 
service endpoint from keystone catalog we default to 'internal' endpoint [1], 
for most services our devstack plugin still configures explicit service URL in 
the corresponding config section, and thus the service discovery from keystone 
never takes place (or that code path is not tested by functional/integration 
testing).


AFAIK different endpoint types (internal vs public) are still quite used by 
deployments (and IMO rightfully so), so we have to continue supporting that. I 
propose to take the following actions:


- in our devstack plugin, stop setting up the direct service URLs in config, 
always use keystone catalog for discovery
- in every conf section related to external service add 
'endpoint_type=[internal|public]' option, defaulting to 'internal', with a 
warning in option description (and validated on conductor start) that it will be 
changed to 'public' in the next release
- use those values from CONF wherever we ask for service URL from catalog or 
instantiate client with session.

- populate these options in our devstack plugin to be 'public'


+1 to all this

- in Queens, switch the default to 'public' and use defaults in devstack plugin, 
remove warnings.


-1 here. The semantic of "internal" is much closer to what we want. What we 
actually need is a way to provide a prioritized list of interfaces. So this 
would be ["internal", "public"], which means "take internal if it exists, 
otherwise take public". I'm pretty sure no clients support anything like that.




Unify clients creation


again, in those config sections related to service clients, we have many options 
to instantiate clients from (especially glance section, see my other recent ML 
about our image service code). Many of those seem to be from the time when 
keystone catalog was missing some functionality or not existing at all, and 
keystoneauth lib abstracting identity and client sessions was not there either.


To simplify setup and unify as much code as possible I'd like to propose the 
following:


- in each config section for service client add (if missing) a '_url' 
option that should point to the API of given service and will be used *only in 
noauth mode* when there's no Keystone catalog to discover the service endpoint from


As Monty noticed, we seem to have endpoint_override already.

- in the code creating service clients, always create a keystoneauth session 
from config sections, using appropriate keystoneauth identity plugin - 
'token_endpoint' with fake token _url for noauth mode, 'password' for 
service user client, 'token' when using a token from incoming request. The 
latter will have a benefit to make it possible for the session to reauth itself 
when user token is about to expire, but might require changes in some public 
methods to pass in the full task.context instead of just token
- always create clients from sessions. Although AFAIK all clients ironic uses 
already support this, some in ironic code (e.g. glance) still always create a 
client from token and endpoint directly.


+1000

- deprecate some options explicitly registered by ironic in those sections that 
are becoming redundant - including those that relate to HTTP session settings 
(like timeout, retries, SSL certs and settings) as those will be used from 
options registered by keystoneauth Session, and those multiple options that 
piece together a single service URL.


+1



This will decrease the complexity of service client-related code and will make 
configuring those cleaner.


Of course all of this has to be done minding proper deprecation process, 
although that might complicate things (as usual :/).


Legacy auth
=

Probably not worth specific mention, but we implemented a proper 
keystoneauth-based loading of client auth options back in Newton almost a year 
ago, so the code attempting to load auth for clients in a deprecated way from 
"[keystone_authtoken]" section can be safely removed already.


Yes please. We've already removed it from ironic-inspector.



As always, I'm eager to hear your comments.

[0] https://review.openstack.org/#/c/433272/ 

[1] 
http://git.openstack.org/cgit/openstack/ironic/tree/ironic/common/keystone.py#n118 



Best regards,
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com 

Re: [openstack-dev] [ironic] using keystone right - catalog, endpoints, tokens and noauth

[Monty Taylor]

On 05/24/2017 12:51 PM, Eric Fried wrote:

Pavlo-

There's a blueprint [1] whereby we're trying to address a bunch of
these same concerns in nova.  You can see the first part in action here
[2].  However, it has become clear that nova is just one of the many
services that would benefit from get_service_url().  With the full
support of mordred (let's call it The Full Monty), we've got our sights
on moving that method into ksa itself for that purpose.


Yes - this has started with documenting how to consume Keystone Catalog 
and discovery properly.


https://review.openstack.org/#/q/topic:version-discovery

(it's a big stack)

Once we're good with that - the next step is getting ksa updated to be 
able to handle the end-to-end. It does most of it today, but there are 
enough edgecases it doesn't that you wind up having to do something 
else, like efried just did in nova. The goal is to make that not 
necessary - and so that it's both possible and EASY for everyone to 
CORRECTLY consume catalog and version discovery.


(more comments inline below)


Please have a look at this blueprint and change set.  Let us know if
your concerns would be addressed if this were available to you from ksa.

[1]
https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/use-service-catalog-for-endpoints.html
[2] https://review.openstack.org/#/c/458257/

Thanks,
efried

On 05/24/2017 04:46 AM, Pavlo Shchelokovskyy wrote:

Hi all,

There are several problems or inefficiencies in how we are dealing with
auth to other services. Although it became much better in Newton, some
things are still to be improved and I like to discuss how to tackle
those and my ideas for that.

Keystone endpoints
===

Apparently since February-ish DevStack no longer sets up 'internal'
endpoints for most of the services in core devstack [0].
Luckily we were not broken by that right away - although when
discovering a service endpoint from keystone catalog we default to
'internal' endpoint [1], for most services our devstack plugin still
configures explicit service URL in the corresponding config section, and
thus the service discovery from keystone never takes place (or that code
path is not tested by functional/integration testing).

AFAIK different endpoint types (internal vs public) are still quite used
by deployments (and IMO rightfully so), so we have to continue
supporting that. I propose to take the following actions:


I agree you should continue supporting it.

I'm not sure it's important for you to change your defaults ... as long 
at it's possible to consistently set "interface=public" or 
"interface=internal" and have the results be correct, I think that's the 
big win.



- in our devstack plugin, stop setting up the direct service URLs in
config, always use keystone catalog for discovery


YES


- in every conf section related to external service add
'endpoint_type=[internal|public]' option, defaulting to 'internal', with
a warning in option description (and validated on conductor start) that
it will be changed to 'public' in the next release


efried just added a call to keystoneauth which will register all of the 
appropriate CONF options that are needed to request a service endpoint 
from the catalog - register_adapter_conf_options:


http://git.openstack.org/cgit/openstack/keystoneauth/tree/keystoneauth1/loading/__init__.py#n39

The word "adapter" in this case isn't directly important - but there are 
three general concepts in keystoneauth that relate to how you connect:


auth
 - how you authenticate - auth_type, username, password, etc.
session
 - how the transport layer connects - certs, timeouts, etc.
adapter
 - what base endpoint to mount from the catalog - service_type, 
interface, endpoint_override, api_version



- use those values from CONF wherever we ask for service URL from
catalog or instantiate client with session.


YES


- populate these options in our devstack plugin to be 'public'
- in Queens, switch the default to 'public' and use defaults in devstack
plugin, remove warnings.

Unify clients creation


again, in those config sections related to service clients, we have many
options to instantiate clients from (especially glance section, see my
other recent ML about our image service code). Many of those seem to be
from the time when keystone catalog was missing some functionality or
not existing at all, and keystoneauth lib abstracting identity and
client sessions was not there either.

To simplify setup and unify as much code as possible I'd like to propose
the following:

- in each config section for service client add (if missing) a
'_url' option that should point to the API of given service and
will be used *only in noauth mode* when there's no Keystone catalog to
discover the service endpoint from


I disagre with this one.

The option exists and is called "endpoint_override" and it skips the 
catalog completely. It 

Re: [openstack-dev] [ironic] using keystone right - catalog, endpoints, tokens and noauth

[Eric Fried]

Pavlo-

There's a blueprint [1] whereby we're trying to address a bunch of
these same concerns in nova.  You can see the first part in action here
[2].  However, it has become clear that nova is just one of the many
services that would benefit from get_service_url().  With the full
support of mordred (let's call it The Full Monty), we've got our sights
on moving that method into ksa itself for that purpose.

Please have a look at this blueprint and change set.  Let us know if
your concerns would be addressed if this were available to you from ksa.

[1]
https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/use-service-catalog-for-endpoints.html
[2] https://review.openstack.org/#/c/458257/

Thanks,
efried

On 05/24/2017 04:46 AM, Pavlo Shchelokovskyy wrote:
> Hi all,
> 
> There are several problems or inefficiencies in how we are dealing with
> auth to other services. Although it became much better in Newton, some
> things are still to be improved and I like to discuss how to tackle
> those and my ideas for that.
> 
> Keystone endpoints
> ===
> 
> Apparently since February-ish DevStack no longer sets up 'internal'
> endpoints for most of the services in core devstack [0].
> Luckily we were not broken by that right away - although when
> discovering a service endpoint from keystone catalog we default to
> 'internal' endpoint [1], for most services our devstack plugin still
> configures explicit service URL in the corresponding config section, and
> thus the service discovery from keystone never takes place (or that code
> path is not tested by functional/integration testing).
> 
> AFAIK different endpoint types (internal vs public) are still quite used
> by deployments (and IMO rightfully so), so we have to continue
> supporting that. I propose to take the following actions:
> 
> - in our devstack plugin, stop setting up the direct service URLs in
> config, always use keystone catalog for discovery
> - in every conf section related to external service add
> 'endpoint_type=[internal|public]' option, defaulting to 'internal', with
> a warning in option description (and validated on conductor start) that
> it will be changed to 'public' in the next release
> - use those values from CONF wherever we ask for service URL from
> catalog or instantiate client with session.
> - populate these options in our devstack plugin to be 'public'
> - in Queens, switch the default to 'public' and use defaults in devstack
> plugin, remove warnings.
> 
> Unify clients creation
> 
> 
> again, in those config sections related to service clients, we have many
> options to instantiate clients from (especially glance section, see my
> other recent ML about our image service code). Many of those seem to be
> from the time when keystone catalog was missing some functionality or
> not existing at all, and keystoneauth lib abstracting identity and
> client sessions was not there either.
> 
> To simplify setup and unify as much code as possible I'd like to propose
> the following:
> 
> - in each config section for service client add (if missing) a
> '_url' option that should point to the API of given service and
> will be used *only in noauth mode* when there's no Keystone catalog to
> discover the service endpoint from
> - in the code creating service clients, always create a keystoneauth
> session from config sections, using appropriate keystoneauth identity
> plugin - 'token_endpoint' with fake token _url for noauth mode,
> 'password' for service user client, 'token' when using a token from
> incoming request. The latter will have a benefit to make it possible for
> the session to reauth itself when user token is about to expire, but
> might require changes in some public methods to pass in the full
> task.context instead of just token
> - always create clients from sessions. Although AFAIK all clients ironic
> uses already support this, some in ironic code (e.g. glance) still
> always create a client from token and endpoint directly.
> - deprecate some options explicitly registered by ironic in those
> sections that are becoming redundant - including those that relate to
> HTTP session settings (like timeout, retries, SSL certs and settings) as
> those will be used from options registered by keystoneauth Session, and
> those multiple options that piece together a single service URL.
> 
> This will decrease the complexity of service client-related code and
> will make configuring those cleaner.
> 
> Of course all of this has to be done minding proper deprecation process,
> although that might complicate things (as usual :/).
> 
> Legacy auth
> =
> 
> Probably not worth specific mention, but we implemented a proper
> keystoneauth-based loading of client auth options back in Newton almost
> a year ago, so the code attempting to load auth for clients in a
> deprecated way from "[keystone_authtoken]" section can be safely