Re: [openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-03-02 Thread Chris Dent 

On Mon, 27 Feb 2017, Sean Dague wrote:


However, when there is magic applied it means that stops being true. And
now folks think the APIs work like the magic works, not realizing it's
all client side magic, and when they try to do this in node next month,
it will all fall apart.


+many

It's good we have a plan (elsewhere in the thread) to get things
smooth again, but we should also see if we can articulate something
along the lines of "design goals" so that this kind of thing is
decreasingly common.

We've become relatively good at identifying when the problem exists:
If you find yourself justifying some cruft on side A for behavior on
side B we know that's a problem for other users of B. What we're less
good at is evolving B quickly enough such that A doesn't have to
compensate. There's likely no easy solution that also accounts for
compatibility.

--
Chris Dent ¯\_(ツ)_/¯   https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-02-27 Thread Jamie Lennox 
On 27 February 2017 at 08:56, Sean Dague  wrote:

> We recently implemented a Nova feature around validating that project_id
> for quotas we real in keystone. After that merged, trippleo builds
> started to fail because their undercloud did not specify the 'identity'
> service as the unversioned endpoint.
>
> https://github.com/openstack/nova/blob/8b498ce199ac4acd94eb33a7f47c05
> ee0c743c34/nova/api/openstack/identity.py#L34-L36
> - (code merged in Nova).
>
> After some debug, it was clear that '/v2.0/v3/projects/...' was what was
> being called. And after lots of conferring in the Keystone room, we
> definitely made sure that the code in question was correct. The thing I
> wanted to do was make the failure more clear.
>
> The suggestion was made to use the following code approach:
>
> https://review.openstack.org/#/c/438049/6/nova/api/openstack/identity.py
>
> resp = sess.get('/projects/%s' % project_id,
> endpoint_filter={
> 'service_type': 'identity',
> 'version': (3, 0)
> },
> raise_exc=False)
>
>
> However, I tested that manually with an identity =>
> http:///v2.0 endpoint, and it passes. Which confused me.
>
> Until I found this -
> https://github.com/openstack/keystoneauth/blob/
> 3364703d3b0e529f7c1b7d1d8ea81726c4f5f121/keystoneauth1/discover.py#L313
>
> keystonauth is specifically coding around the keystone transition from a
> versioned /v2.0 endpoint to an unversioned one.
>
>
> While that is good for the python ecosystem using it, it's actually
> *quite* bad for the rest of our ecosystem (direct REST, java, ruby, go,
> js, php), because it means that all other facilities need the same work
> around. I actually wonder if this is one of the in the field reasons for
> why the transition from v2 -> v3 is going slow. That's actually going to
> potentially break a lot of software.
>
> It feels like this whole discovery version hack bit should be removed -
> https://review.openstack.org/#/c/438483/. It also feels like a migration
> path for non python software in changing the catalog entries needs to be
> figured out as well.
>
> I think on the Nova side we need to go back to looking for bogus
> endpoint because we don't want issues like this hidden from us.
>
> -Sean


So I would completely agree, I would like to see this behaviour
removed. However
it was done very intentionally - and at the time it was written it was
needed.

This is one of a number of situations where keystoneauth tried its best to
paper over inconsistencies in OpenStack APIs because to various levels of
effectiveness almost all the python clients were doing this. Any whilst we
have slowly pushed the documentation and standard deployment procedures to
unversioned URLs whilst this hack was maintained in keystoneauth we didn't
have to fix it individually for every client.

Where python and keystoneauth are different from every other language is
that the services themselves are written in python and using these
libraries and inter-service communication had to continue to work
throughout the transition. You may remember the fun we had trying to change
to v3 auth and unversioned URLs in devstack? This hack is what made it
possible at all. As you say this is extremely difficult for other
languages, but something there isn't a solution for whilst this transition
is in place.

Anyway a few cycles later we are in a different position and a new service
such as the placement API can decide that it shouldn't work at all if the
catalog isn't configured as OpenStack advises. This is great! We can
effectively force deployments to transition to unversioned URLs. We can't
change the default behaviour in keystoneauth but it should be relatively
easy to give you an adapter that doesn't do this. Probably something like
[1]. I also filed it as a bug, which links to this thread [2], but could
otherwise do with some more detail.

Long story short, sorry but it'll have to be a new flag. Yes, keystoneauth
is supposed to be a low-level request maker, but it is also trying to paper
over a number of historical bad decisions so at the very least the user
experience is correct and we don't have clients re-inventing it themselves.

[1] https://review.openstack.org/#/c/438788/
[2] https://bugs.launchpad.net/keystoneauth/+bug/1668484
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-02-27 Thread Sean Dague 
On 02/27/2017 10:49 AM, Monty Taylor wrote:
> On 02/27/2017 09:36 AM, Morgan Fainberg wrote:
>>
>>
>> On Mon, Feb 27, 2017 at 7:26 AM, Sean Dague > > wrote:
>>
>> On 02/27/2017 10:22 AM, Morgan Fainberg wrote:
>> 
>> > I agree we should kill the discovery hack, however that is a break in
>> > the keystoneauth contract. Simply put, we cannot. Keystoneauth is one 
>> of
>> > the few things (similar to how shade works) where behavior, exposed
>> > elements, etc are considered a strict contract that will not change. If
>> > we could have avoided stevedore and PBR we would have.
>> >
>> > The best we can provide is a way to build the instances from
>> > keystoneauth that does not include that hack.
>> >
>> > The short is, we can't remove it. Similar to how we cannot change the
>> > raise of exceptions for non-200 responses (the behavior is already 
>> encoded).
>>
>> Ok, I'm going to go back to not using the version= parameter then.
>> Because it's not actually doing the right thing.
>>
>> I also am a bit concerned that basically through some client changes
>> that people didn't understand, we've missed a break in the upstream
>> transition that will impact real clouds. :(
>>
>>
>> We can make an adapter that does what you want (requests adapters are
>> cool). I was just chatting with Monty about this, and we can help you
>> out on this front.
>>
>> The adapter should make things a lot easier once the lifting is done. 
> 
> Yah. Consider me to be on this. Looking at the code you've got to make
> intra-openstack rest calls makes me want to poke out my own eyeballs. It
> does _not_ have to be this hard or this brittle.
> 
> It'll likely take a few days and a thing or two to unwind.

I'm definitely happy if there are better ways to do it.

But, I'm also concerned about the bigger picture. I thought keystoneauth
was giving a pretty low level REST interface, which is good, because it
means we can use and think about the services as they are documented in
the api-ref.

However, when there is magic applied it means that stops being true. And
now folks think the APIs work like the magic works, not realizing it's
all client side magic, and when they try to do this in node next month,
it will all fall apart.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-02-27 Thread Monty Taylor 
On 02/27/2017 09:36 AM, Morgan Fainberg wrote:
> 
> 
> On Mon, Feb 27, 2017 at 7:26 AM, Sean Dague  > wrote:
> 
> On 02/27/2017 10:22 AM, Morgan Fainberg wrote:
> 
> > I agree we should kill the discovery hack, however that is a break in
> > the keystoneauth contract. Simply put, we cannot. Keystoneauth is one of
> > the few things (similar to how shade works) where behavior, exposed
> > elements, etc are considered a strict contract that will not change. If
> > we could have avoided stevedore and PBR we would have.
> >
> > The best we can provide is a way to build the instances from
> > keystoneauth that does not include that hack.
> >
> > The short is, we can't remove it. Similar to how we cannot change the
> > raise of exceptions for non-200 responses (the behavior is already 
> encoded).
> 
> Ok, I'm going to go back to not using the version= parameter then.
> Because it's not actually doing the right thing.
> 
> I also am a bit concerned that basically through some client changes
> that people didn't understand, we've missed a break in the upstream
> transition that will impact real clouds. :(
> 
> 
> We can make an adapter that does what you want (requests adapters are
> cool). I was just chatting with Monty about this, and we can help you
> out on this front.
> 
> The adapter should make things a lot easier once the lifting is done. 

Yah. Consider me to be on this. Looking at the code you've got to make
intra-openstack rest calls makes me want to poke out my own eyeballs. It
does _not_ have to be this hard or this brittle.

It'll likely take a few days and a thing or two to unwind.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-02-27 Thread Morgan Fainberg 
On Mon, Feb 27, 2017 at 7:26 AM, Sean Dague  wrote:

> On 02/27/2017 10:22 AM, Morgan Fainberg wrote:
> 
> > I agree we should kill the discovery hack, however that is a break in
> > the keystoneauth contract. Simply put, we cannot. Keystoneauth is one of
> > the few things (similar to how shade works) where behavior, exposed
> > elements, etc are considered a strict contract that will not change. If
> > we could have avoided stevedore and PBR we would have.
> >
> > The best we can provide is a way to build the instances from
> > keystoneauth that does not include that hack.
> >
> > The short is, we can't remove it. Similar to how we cannot change the
> > raise of exceptions for non-200 responses (the behavior is already
> encoded).
>
> Ok, I'm going to go back to not using the version= parameter then.
> Because it's not actually doing the right thing.
>
> I also am a bit concerned that basically through some client changes
> that people didn't understand, we've missed a break in the upstream
> transition that will impact real clouds. :(
>
>
We can make an adapter that does what you want (requests adapters are
cool). I was just chatting with Monty about this, and we can help you out
on this front.

The adapter should make things a lot easier once the lifting is done.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-02-27 Thread Morgan Fainberg 
On Mon, Feb 27, 2017 at 5:56 AM, Sean Dague  wrote:

> We recently implemented a Nova feature around validating that project_id
> for quotas we real in keystone. After that merged, trippleo builds
> started to fail because their undercloud did not specify the 'identity'
> service as the unversioned endpoint.
>
> https://github.com/openstack/nova/blob/8b498ce199ac4acd94eb33a7f47c05
> ee0c743c34/nova/api/openstack/identity.py#L34-L36
> - (code merged in Nova).
>
> After some debug, it was clear that '/v2.0/v3/projects/...' was what was
> being called. And after lots of conferring in the Keystone room, we
> definitely made sure that the code in question was correct. The thing I
> wanted to do was make the failure more clear.
>
> The suggestion was made to use the following code approach:
>
> https://review.openstack.org/#/c/438049/6/nova/api/openstack/identity.py
>
> resp = sess.get('/projects/%s' % project_id,
> endpoint_filter={
> 'service_type': 'identity',
> 'version': (3, 0)
> },
> raise_exc=False)
>
>
> However, I tested that manually with an identity =>
> http:///v2.0 endpoint, and it passes. Which confused me.
>
> Until I found this -
> https://github.com/openstack/keystoneauth/blob/
> 3364703d3b0e529f7c1b7d1d8ea81726c4f5f121/keystoneauth1/discover.py#L313
>
> keystonauth is specifically coding around the keystone transition from a
> versioned /v2.0 endpoint to an unversioned one.
>
>
> While that is good for the python ecosystem using it, it's actually
> *quite* bad for the rest of our ecosystem (direct REST, java, ruby, go,
> js, php), because it means that all other facilities need the same work
> around. I actually wonder if this is one of the in the field reasons for
> why the transition from v2 -> v3 is going slow. That's actually going to
> potentially break a lot of software.
>
> It feels like this whole discovery version hack bit should be removed -
> https://review.openstack.org/#/c/438483/. It also feels like a migration
> path for non python software in changing the catalog entries needs to be
> figured out as well.
>
> I think on the Nova side we need to go back to looking for bogus
> endpoint because we don't want issues like this hidden from us.
>
>
I agree we should kill the discovery hack, however that is a break in the
keystoneauth contract. Simply put, we cannot. Keystoneauth is one of the
few things (similar to how shade works) where behavior, exposed elements,
etc are considered a strict contract that will not change. If we could have
avoided stevedore and PBR we would have.

The best we can provide is a way to build the instances from keystoneauth
that does not include that hack.

The short is, we can't remove it. Similar to how we cannot change the raise
of exceptions for non-200 responses (the behavior is already encoded).

--Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-02-27 Thread Sean Dague 
On 02/27/2017 10:22 AM, Morgan Fainberg wrote:

> I agree we should kill the discovery hack, however that is a break in
> the keystoneauth contract. Simply put, we cannot. Keystoneauth is one of
> the few things (similar to how shade works) where behavior, exposed
> elements, etc are considered a strict contract that will not change. If
> we could have avoided stevedore and PBR we would have.
> 
> The best we can provide is a way to build the instances from
> keystoneauth that does not include that hack.
> 
> The short is, we can't remove it. Similar to how we cannot change the
> raise of exceptions for non-200 responses (the behavior is already encoded).

Ok, I'm going to go back to not using the version= parameter then.
Because it's not actually doing the right thing.

I also am a bit concerned that basically through some client changes
that people didn't understand, we've missed a break in the upstream
transition that will impact real clouds. :(

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone] [nova] keystonauth catalog work arounds hiding transition issues

2017-02-27 Thread Sean Dague 
We recently implemented a Nova feature around validating that project_id
for quotas we real in keystone. After that merged, trippleo builds
started to fail because their undercloud did not specify the 'identity'
service as the unversioned endpoint.

https://github.com/openstack/nova/blob/8b498ce199ac4acd94eb33a7f47c05ee0c743c34/nova/api/openstack/identity.py#L34-L36
- (code merged in Nova).

After some debug, it was clear that '/v2.0/v3/projects/...' was what was
being called. And after lots of conferring in the Keystone room, we
definitely made sure that the code in question was correct. The thing I
wanted to do was make the failure more clear.

The suggestion was made to use the following code approach:

https://review.openstack.org/#/c/438049/6/nova/api/openstack/identity.py

resp = sess.get('/projects/%s' % project_id,
endpoint_filter={
'service_type': 'identity',
'version': (3, 0)
},
raise_exc=False)


However, I tested that manually with an identity =>
http:///v2.0 endpoint, and it passes. Which confused me.

Until I found this -
https://github.com/openstack/keystoneauth/blob/3364703d3b0e529f7c1b7d1d8ea81726c4f5f121/keystoneauth1/discover.py#L313

keystonauth is specifically coding around the keystone transition from a
versioned /v2.0 endpoint to an unversioned one.


While that is good for the python ecosystem using it, it's actually
*quite* bad for the rest of our ecosystem (direct REST, java, ruby, go,
js, php), because it means that all other facilities need the same work
around. I actually wonder if this is one of the in the field reasons for
why the transition from v2 -> v3 is going slow. That's actually going to
potentially break a lot of software.

It feels like this whole discovery version hack bit should be removed -
https://review.openstack.org/#/c/438483/. It also feels like a migration
path for non python software in changing the catalog entries needs to be
figured out as well.

I think on the Nova side we need to go back to looking for bogus
endpoint because we don't want issues like this hidden from us.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev