Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Morgan Fainberg]

On Mon, Apr 18, 2016 at 5:43 PM, Matt Fischer  wrote:

> On Mon, Apr 18, 2016 at 12:52 PM, Morgan Fainberg <
> morgan.fainb...@gmail.com> wrote:
>
>>
>>
>> On Mon, Apr 18, 2016 at 7:29 AM, Brant Knudson  wrote:
>>
>>>
>>>
>>> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>>>
 We all want Fernet to be a reality.  We ain't there yet (Except for
 mfish who has no patience) but we are getting closer.  The goal is to get
 Fernet as the default token provider as soon as possible. The review to do
 this has uncovered a few details that need to be fixed before we can do
 this.

 Trusts for V2 tokens were not working correctly.  Relatively easy fix.
 https://review.openstack.org/#/c/278693/ Patch is still failing on
 Python 3.  The tests are kindof racy due to the revocation event 1 second
 granularity.  Some of the tests here have A sleep (1) in them still, but
 all should be using the time control aspect of the unit test fixtures.

 Some of the tests also use the same user to validate a token as that
 have, for example, a role unassigned.  These expose a problem that the
 revocation events are catching too many tokens, some of which should not be
 treated as revoked.

 Also, some of the logic for revocation checking has to change. Before,
 if a user had two roles, and had one removed, the token would be revoked.
 Now, however, the token will validate successful, but the response will
 only have the single assigned role in it.


 Python 3 tests are failing because the Fernet formatter is insisting
 that all project-ids be valid UUIDs, but some of the old tests have "FOO"
 and "BAR" as ids.  These either need to be converted to UUIDS, or the
 formatter needs to be more forgiving.

 Caching of token validations was messing with revocation checking.
 Tokens that were valid once were being reported as always valid. Thus, the
 current review  removes all caching on token validations, a change we
 cannot maintain.  Once all the test are successfully passing, we will
 re-introduce the cache, and be far more aggressive about cache 
 invalidation.

 Tempest tests are currently failing due to Devstack not properly
 identifying Fernet as the default token provider, and creating the Fernet
 key repository.  I'm tempted to just force devstack to always create the
 directory, as a user would need it if they ever switched the token provider
 post launch anyway.


>>> There's a review to change devstack to default to fernet:
>>> https://review.openstack.org/#/c/195780/ . This was mostly to show that
>>> tempest still passes with fernet configured. It uncovered a couple of test
>>> issues (similar in nature to the revocation checking issues mentioned in
>>> the original note) that have since been fixed.
>>>
>>> We'd prefer to not have devstack overriding config options and instead
>>> use keystone's defaults. The problem is if fernet is the default in
>>> keystone then it won't work out of the box since the key database won't
>>> exist. One option that I think we should investigate is to have keystone
>>> create the key database on startup if it doesn't exist.
>>>
>>>
>> I am unsure if this is the right path, unless we consider possibly moving
>> the key-DB for fernet into the SQL backend (possible?) notably so we can
>> control a cluster of keystones.
>>
>
> -1 this idea. Now we've got to figure out how to lock the DB and where
> will those keys/passwords go? The current model is apparently painful for
> devstack,  but it works fine for anyone who has an automated deployment
> system, or something else simple like rsync/scp can be used. Fernet keys
> are simpler to update/replicate than PKI certs and we didn't store those in
> the DB to my knowledge. Let's please not optimize this solution for
> devstack.
>
>
The DB solution is actually presented as explicitly not AIO install, it
would be to solve the "cluster case" as the default value for keystone's
config, since keystone can reliably share the data. DB tables can actually
be locked (this is separate from select for update) and would be less
painful.

The Devstack solution today (and looking to the future) would absolutely be
file-based. No change there. This conversation is revolving around how to
reduce the operational overhead of making keystone itself (not in
devstack's context) default to fernet and not produce wildly (or
potentially wildly) broken states.

This is a convo that will need to occur at the summit and we can hash out
the details of what makes sense more easily. I expect the DB solution would
be a bad one in any case, but before we make keystone's config default to
Fernet, we need to make the non-devstack case(s) (and existing deployments)
not suck.

--Morgan
__
OpenStack Development Mailing List (not for usage question

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Matt Fischer]

On Mon, Apr 18, 2016 at 12:52 PM, Morgan Fainberg  wrote:

>
>
> On Mon, Apr 18, 2016 at 7:29 AM, Brant Knudson  wrote:
>
>>
>>
>> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>>
>>> We all want Fernet to be a reality.  We ain't there yet (Except for
>>> mfish who has no patience) but we are getting closer.  The goal is to get
>>> Fernet as the default token provider as soon as possible. The review to do
>>> this has uncovered a few details that need to be fixed before we can do
>>> this.
>>>
>>> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
>>> https://review.openstack.org/#/c/278693/ Patch is still failing on
>>> Python 3.  The tests are kindof racy due to the revocation event 1 second
>>> granularity.  Some of the tests here have A sleep (1) in them still, but
>>> all should be using the time control aspect of the unit test fixtures.
>>>
>>> Some of the tests also use the same user to validate a token as that
>>> have, for example, a role unassigned.  These expose a problem that the
>>> revocation events are catching too many tokens, some of which should not be
>>> treated as revoked.
>>>
>>> Also, some of the logic for revocation checking has to change. Before,
>>> if a user had two roles, and had one removed, the token would be revoked.
>>> Now, however, the token will validate successful, but the response will
>>> only have the single assigned role in it.
>>>
>>>
>>> Python 3 tests are failing because the Fernet formatter is insisting
>>> that all project-ids be valid UUIDs, but some of the old tests have "FOO"
>>> and "BAR" as ids.  These either need to be converted to UUIDS, or the
>>> formatter needs to be more forgiving.
>>>
>>> Caching of token validations was messing with revocation checking.
>>> Tokens that were valid once were being reported as always valid. Thus, the
>>> current review  removes all caching on token validations, a change we
>>> cannot maintain.  Once all the test are successfully passing, we will
>>> re-introduce the cache, and be far more aggressive about cache invalidation.
>>>
>>> Tempest tests are currently failing due to Devstack not properly
>>> identifying Fernet as the default token provider, and creating the Fernet
>>> key repository.  I'm tempted to just force devstack to always create the
>>> directory, as a user would need it if they ever switched the token provider
>>> post launch anyway.
>>>
>>>
>> There's a review to change devstack to default to fernet:
>> https://review.openstack.org/#/c/195780/ . This was mostly to show that
>> tempest still passes with fernet configured. It uncovered a couple of test
>> issues (similar in nature to the revocation checking issues mentioned in
>> the original note) that have since been fixed.
>>
>> We'd prefer to not have devstack overriding config options and instead
>> use keystone's defaults. The problem is if fernet is the default in
>> keystone then it won't work out of the box since the key database won't
>> exist. One option that I think we should investigate is to have keystone
>> create the key database on startup if it doesn't exist.
>>
>>
> I am unsure if this is the right path, unless we consider possibly moving
> the key-DB for fernet into the SQL backend (possible?) notably so we can
> control a cluster of keystones.
>

-1 this idea. Now we've got to figure out how to lock the DB and where will
those keys/passwords go? The current model is apparently painful for
devstack,  but it works fine for anyone who has an automated deployment
system, or something else simple like rsync/scp can be used. Fernet keys
are simpler to update/replicate than PKI certs and we didn't store those in
the DB to my knowledge. Let's please not optimize this solution for
devstack.




>
> If we aren't making the data shared by default, I would rather have
> devstack override the keystone default as UUID still seems like the sanest
> default due to other config overhead (with filesystem-based fernet keys).
>
> --Morgan
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Dolph Mathews]

On Mon, Apr 18, 2016 at 5:14 PM, Adam Young  wrote:

> On 04/18/2016 10:29 AM, Brant Knudson wrote:
>
>
>
> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>
>> We all want Fernet to be a reality.  We ain't there yet (Except for mfish
>> who has no patience) but we are getting closer.  The goal is to get Fernet
>> as the default token provider as soon as possible. The review to do this
>> has uncovered a few details that need to be fixed before we can do this.
>>
>> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
>> https://review.openstack.org/#/c/278693/ Patch is still failing on
>> Python 3.  The tests are kindof racy due to the revocation event 1 second
>> granularity.  Some of the tests here have A sleep (1) in them still, but
>> all should be using the time control aspect of the unit test fixtures.
>>
>> Some of the tests also use the same user to validate a token as that
>> have, for example, a role unassigned.  These expose a problem that the
>> revocation events are catching too many tokens, some of which should not be
>> treated as revoked.
>>
>> Also, some of the logic for revocation checking has to change. Before, if
>> a user had two roles, and had one removed, the token would be revoked.
>> Now, however, the token will validate successful, but the response will
>> only have the single assigned role in it.
>>
>>
>> Python 3 tests are failing because the Fernet formatter is insisting that
>> all project-ids be valid UUIDs, but some of the old tests have "FOO" and
>> "BAR" as ids.  These either need to be converted to UUIDS, or the formatter
>> needs to be more forgiving.
>>
>> Caching of token validations was messing with revocation checking. Tokens
>> that were valid once were being reported as always valid. Thus, the current
>> review  removes all caching on token validations, a change we cannot
>> maintain.  Once all the test are successfully passing, we will re-introduce
>> the cache, and be far more aggressive about cache invalidation.
>>
>> Tempest tests are currently failing due to Devstack not properly
>> identifying Fernet as the default token provider, and creating the Fernet
>> key repository.  I'm tempted to just force devstack to always create the
>> directory, as a user would need it if they ever switched the token provider
>> post launch anyway.
>>
>>
> There's a review to change devstack to default to fernet:
> https://review.openstack.org/#/c/195780/ . This was mostly to show that
> tempest still passes with fernet configured. It uncovered a couple of test
> issues (similar in nature to the revocation checking issues mentioned in
> the original note) that have since been fixed.
>
> We'd prefer to not have devstack overriding config options and instead use
> keystone's defaults. The problem is if fernet is the default in keystone
> then it won't work out of the box since the key database won't exist. One
> option that I think we should investigate is to have keystone create the
> key database on startup if it doesn't exist.
>
>
> In some deployment, they should be owned by different users.  In general,
> a system/daemon user should not be writing to /etc.  Key rotation/etc is
> likely to be handled by an external Content management system, so it might
> not be the right default.
>

+1 Besides race conditions, this is why keystone doesn't try to
"automatically" populate /etc/keystone/fernet-keys/ on startup. Fernet keys
only need to be readable by the user running keystone.


>
>
>
> - Brant
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Fox, Kevin M]

Its also could be more difficult for ops to debug if it autocreates it on a 
cluster when config management is broken and you don't realize it, and you see 
keys that were created, but totally wrong for the cluster.

Though maybe there should be a generic ha=True option in keystone to override 
that behavior and never try creating keys. Then default ha=False.

Thanks,
Kevin

From: Adam Young [ayo...@redhat.com]
Sent: Monday, April 18, 2016 3:14 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Keystone] State of Fernet Token deployment

On 04/18/2016 10:29 AM, Brant Knudson wrote:


On Fri, Apr 15, 2016 at 9:04 PM, Adam Young 
mailto:ayo...@redhat.com>> wrote:
We all want Fernet to be a reality.  We ain't there yet (Except for mfish who 
has no patience) but we are getting closer.  The goal is to get Fernet as the 
default token provider as soon as possible. The review to do this has uncovered 
a few details that need to be fixed before we can do this.

Trusts for V2 tokens were not working correctly.  Relatively easy fix. 
https://review.openstack.org/#/c/278693/ Patch is still failing on Python 3.  
The tests are kindof racy due to the revocation event 1 second granularity.  
Some of the tests here have A sleep (1) in them still, but all should be using 
the time control aspect of the unit test fixtures.

Some of the tests also use the same user to validate a token as that have, for 
example, a role unassigned.  These expose a problem that the revocation events 
are catching too many tokens, some of which should not be treated as revoked.

Also, some of the logic for revocation checking has to change. Before, if a 
user had two roles, and had one removed, the token would be revoked.  Now, 
however, the token will validate successful, but the response will only have 
the single assigned role in it.


Python 3 tests are failing because the Fernet formatter is insisting that all 
project-ids be valid UUIDs, but some of the old tests have "FOO" and "BAR" as 
ids.  These either need to be converted to UUIDS, or the formatter needs to be 
more forgiving.

Caching of token validations was messing with revocation checking. Tokens that 
were valid once were being reported as always valid. Thus, the current review  
removes all caching on token validations, a change we cannot maintain.  Once 
all the test are successfully passing, we will re-introduce the cache, and be 
far more aggressive about cache invalidation.

Tempest tests are currently failing due to Devstack not properly identifying 
Fernet as the default token provider, and creating the Fernet key repository.  
I'm tempted to just force devstack to always create the directory, as a user 
would need it if they ever switched the token provider post launch anyway.


There's a review to change devstack to default to fernet: 
https://review.openstack.org/#/c/195780/ . This was mostly to show that tempest 
still passes with fernet configured. It uncovered a couple of test issues 
(similar in nature to the revocation checking issues mentioned in the original 
note) that have since been fixed.

We'd prefer to not have devstack overriding config options and instead use 
keystone's defaults. The problem is if fernet is the default in keystone then 
it won't work out of the box since the key database won't exist. One option 
that I think we should investigate is to have keystone create the key database 
on startup if it doesn't exist.

In some deployment, they should be owned by different users.  In general, a 
system/daemon user should not be writing to /etc.  Key rotation/etc is likely 
to be handled by an external Content management system, so it might not be the 
right default.



- Brant




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Adam Young]

On 04/18/2016 10:29 AM, Brant Knudson wrote:



On Fri, Apr 15, 2016 at 9:04 PM, Adam Young > wrote:


We all want Fernet to be a reality.  We ain't there yet (Except
for mfish who has no patience) but we are getting closer.  The
goal is to get Fernet as the default token provider as soon as
possible. The review to do this has uncovered a few details that
need to be fixed before we can do this.

Trusts for V2 tokens were not working correctly. Relatively easy
fix. https://review.openstack.org/#/c/278693/ Patch is still
failing on Python 3.  The tests are kindof racy due to the
revocation event 1 second granularity. Some of the tests here have
A sleep (1) in them still, but all should be using the time
control aspect of the unit test fixtures.

Some of the tests also use the same user to validate a token as
that have, for example, a role unassigned.  These expose a problem
that the revocation events are catching too many tokens, some of
which should not be treated as revoked.

Also, some of the logic for revocation checking has to change.
Before, if a user had two roles, and had one removed, the token
would be revoked.  Now, however, the token will validate
successful, but the response will only have the single assigned
role in it.


Python 3 tests are failing because the Fernet formatter is
insisting that all project-ids be valid UUIDs, but some of the old
tests have "FOO" and "BAR" as ids.  These either need to be
converted to UUIDS, or the formatter needs to be more forgiving.

Caching of token validations was messing with revocation checking.
Tokens that were valid once were being reported as always valid.
Thus, the current review  removes all caching on token
validations, a change we cannot maintain.  Once all the test are
successfully passing, we will re-introduce the cache, and be far
more aggressive about cache invalidation.

Tempest tests are currently failing due to Devstack not properly
identifying Fernet as the default token provider, and creating the
Fernet key repository.  I'm tempted to just force devstack to
always create the directory, as a user would need it if they ever
switched the token provider post launch anyway.


There's a review to change devstack to default to fernet: 
https://review.openstack.org/#/c/195780/ . This was mostly to show 
that tempest still passes with fernet configured. It uncovered a 
couple of test issues (similar in nature to the revocation checking 
issues mentioned in the original note) that have since been fixed.


We'd prefer to not have devstack overriding config options and instead 
use keystone's defaults. The problem is if fernet is the default in 
keystone then it won't work out of the box since the key database 
won't exist. One option that I think we should investigate is to have 
keystone create the key database on startup if it doesn't exist.


In some deployment, they should be owned by different users.  In 
general, a system/daemon user should not be writing to /etc.  Key 
rotation/etc is likely to be handled by an external Content management 
system, so it might not be the right default.





- Brant



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Adam Young]

On 04/18/2016 02:10 PM, Matt Fischer wrote:

Thanks Brant,

I will missing that distinction.

On Mon, Apr 18, 2016 at 9:43 AM, Brant Knudson > wrote:




On Mon, Apr 18, 2016 at 10:20 AM, Matt Fischer
mailto:m...@mattfischer.com>> wrote:

On Mon, Apr 18, 2016 at 8:29 AM, Brant Knudson mailto:b...@acm.org>> wrote:



On Fri, Apr 15, 2016 at 9:04 PM, Adam Young
mailto:ayo...@redhat.com>> wrote:

We all want Fernet to be a reality.  We ain't there
yet (Except for mfish who has no patience) but we are
getting closer.  The goal is to get Fernet as the
default token provider as soon as possible. The review
to do this has uncovered a few details that need to be
fixed before we can do this.

Trusts for V2 tokens were not working correctly.
Relatively easy fix.
https://review.openstack.org/#/c/278693/ Patch is
still failing on Python 3.  The tests are kindof racy
due to the revocation event 1 second granularity. 
Some of the tests here have A sleep (1) in them still,

but all should be using the time control aspect of the
unit test fixtures.

Some of the tests also use the same user to validate a
token as that have, for example, a role unassigned. 
These expose a problem that the revocation events are

catching too many tokens, some of which should not be
treated as revoked.

Also, some of the logic for revocation checking has to
change. Before, if a user had two roles, and had one
removed, the token would be revoked.  Now, however,
the token will validate successful, but the response
will only have the single assigned role in it.


Python 3 tests are failing because the Fernet
formatter is insisting that all project-ids be valid
UUIDs, but some of the old tests have "FOO" and "BAR"
as ids.  These either need to be converted to UUIDS,
or the formatter needs to be more forgiving.

Caching of token validations was messing with
revocation checking. Tokens that were valid once were
being reported as always valid. Thus, the current
review  removes all caching on token validations, a
change we cannot maintain.  Once all the test are
successfully passing, we will re-introduce the cache,
and be far more aggressive about cache invalidation.

Tempest tests are currently failing due to Devstack
not properly identifying Fernet as the default token
provider, and creating the Fernet key repository.  I'm
tempted to just force devstack to always create the
directory, as a user would need it if they ever
switched the token provider post launch anyway.


There's a review to change devstack to default to fernet:
https://review.openstack.org/#/c/195780/ . This was mostly
to show that tempest still passes with fernet configured.
It uncovered a couple of test issues (similar in nature to
the revocation checking issues mentioned in the original
note) that have since been fixed.

We'd prefer to not have devstack overriding config options
and instead use keystone's defaults. The problem is if
fernet is the default in keystone then it won't work out
of the box since the key database won't exist. One option
that I think we should investigate is to have keystone
create the key database on startup if it doesn't exist.

- Brant



I'm not a devstack user, but as I mentioned before, I assume
devstack called keystone-manage db_sync? Why couldn't it also
call keystone-manage fernet_setup?


When you tell devstack that it's using fernet then it does
keystone-manage fernet_setup. When you tell devstack to use the
default, it doesn't fernet_setup because for now it thinks the
default is UUID and doesn't require keys. One way to have devstack
work when fernet is the default is to have devstack always do
keystone-manage fernet_setup.

My thought was to have this as a temporary fix as the default changes.  
Once we settle in to Fernet, we can swap to "only Fernet if Fernet"


There is no reason Devstack can't read the config option from Keystone, 
but that is a larger change than I want to make for this.





Really what we want to do is have devstack work like other
deployment methods. We can reasonably expect featureful dep

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Morgan Fainberg]

On Mon, Apr 18, 2016 at 7:29 AM, Brant Knudson  wrote:

>
>
> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>
>> We all want Fernet to be a reality.  We ain't there yet (Except for mfish
>> who has no patience) but we are getting closer.  The goal is to get Fernet
>> as the default token provider as soon as possible. The review to do this
>> has uncovered a few details that need to be fixed before we can do this.
>>
>> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
>> https://review.openstack.org/#/c/278693/ Patch is still failing on
>> Python 3.  The tests are kindof racy due to the revocation event 1 second
>> granularity.  Some of the tests here have A sleep (1) in them still, but
>> all should be using the time control aspect of the unit test fixtures.
>>
>> Some of the tests also use the same user to validate a token as that
>> have, for example, a role unassigned.  These expose a problem that the
>> revocation events are catching too many tokens, some of which should not be
>> treated as revoked.
>>
>> Also, some of the logic for revocation checking has to change. Before, if
>> a user had two roles, and had one removed, the token would be revoked.
>> Now, however, the token will validate successful, but the response will
>> only have the single assigned role in it.
>>
>>
>> Python 3 tests are failing because the Fernet formatter is insisting that
>> all project-ids be valid UUIDs, but some of the old tests have "FOO" and
>> "BAR" as ids.  These either need to be converted to UUIDS, or the formatter
>> needs to be more forgiving.
>>
>> Caching of token validations was messing with revocation checking. Tokens
>> that were valid once were being reported as always valid. Thus, the current
>> review  removes all caching on token validations, a change we cannot
>> maintain.  Once all the test are successfully passing, we will re-introduce
>> the cache, and be far more aggressive about cache invalidation.
>>
>> Tempest tests are currently failing due to Devstack not properly
>> identifying Fernet as the default token provider, and creating the Fernet
>> key repository.  I'm tempted to just force devstack to always create the
>> directory, as a user would need it if they ever switched the token provider
>> post launch anyway.
>>
>>
> There's a review to change devstack to default to fernet:
> https://review.openstack.org/#/c/195780/ . This was mostly to show that
> tempest still passes with fernet configured. It uncovered a couple of test
> issues (similar in nature to the revocation checking issues mentioned in
> the original note) that have since been fixed.
>
> We'd prefer to not have devstack overriding config options and instead use
> keystone's defaults. The problem is if fernet is the default in keystone
> then it won't work out of the box since the key database won't exist. One
> option that I think we should investigate is to have keystone create the
> key database on startup if it doesn't exist.
>
>
I am unsure if this is the right path, unless we consider possibly moving
the key-DB for fernet into the SQL backend (possible?) notably so we can
control a cluster of keystones.

If we aren't making the data shared by default, I would rather have
devstack override the keystone default as UUID still seems like the sanest
default due to other config overhead (with filesystem-based fernet keys).

--Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Matt Fischer]

Thanks Brant,

I will missing that distinction.

On Mon, Apr 18, 2016 at 9:43 AM, Brant Knudson  wrote:

>
>
> On Mon, Apr 18, 2016 at 10:20 AM, Matt Fischer 
> wrote:
>
>> On Mon, Apr 18, 2016 at 8:29 AM, Brant Knudson  wrote:
>>
>>>
>>>
>>> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>>>
 We all want Fernet to be a reality.  We ain't there yet (Except for
 mfish who has no patience) but we are getting closer.  The goal is to get
 Fernet as the default token provider as soon as possible. The review to do
 this has uncovered a few details that need to be fixed before we can do
 this.

 Trusts for V2 tokens were not working correctly.  Relatively easy fix.
 https://review.openstack.org/#/c/278693/ Patch is still failing on
 Python 3.  The tests are kindof racy due to the revocation event 1 second
 granularity.  Some of the tests here have A sleep (1) in them still, but
 all should be using the time control aspect of the unit test fixtures.

 Some of the tests also use the same user to validate a token as that
 have, for example, a role unassigned.  These expose a problem that the
 revocation events are catching too many tokens, some of which should not be
 treated as revoked.

 Also, some of the logic for revocation checking has to change. Before,
 if a user had two roles, and had one removed, the token would be revoked.
 Now, however, the token will validate successful, but the response will
 only have the single assigned role in it.


 Python 3 tests are failing because the Fernet formatter is insisting
 that all project-ids be valid UUIDs, but some of the old tests have "FOO"
 and "BAR" as ids.  These either need to be converted to UUIDS, or the
 formatter needs to be more forgiving.

 Caching of token validations was messing with revocation checking.
 Tokens that were valid once were being reported as always valid. Thus, the
 current review  removes all caching on token validations, a change we
 cannot maintain.  Once all the test are successfully passing, we will
 re-introduce the cache, and be far more aggressive about cache 
 invalidation.

 Tempest tests are currently failing due to Devstack not properly
 identifying Fernet as the default token provider, and creating the Fernet
 key repository.  I'm tempted to just force devstack to always create the
 directory, as a user would need it if they ever switched the token provider
 post launch anyway.


>>> There's a review to change devstack to default to fernet:
>>> https://review.openstack.org/#/c/195780/ . This was mostly to show that
>>> tempest still passes with fernet configured. It uncovered a couple of test
>>> issues (similar in nature to the revocation checking issues mentioned in
>>> the original note) that have since been fixed.
>>>
>>> We'd prefer to not have devstack overriding config options and instead
>>> use keystone's defaults. The problem is if fernet is the default in
>>> keystone then it won't work out of the box since the key database won't
>>> exist. One option that I think we should investigate is to have keystone
>>> create the key database on startup if it doesn't exist.
>>>
>>> - Brant
>>>
>>>
>>
>> I'm not a devstack user, but as I mentioned before, I assume devstack
>> called keystone-manage db_sync? Why couldn't it also call keystone-manage
>> fernet_setup?
>>
>>
> When you tell devstack that it's using fernet then it does keystone-manage
> fernet_setup. When you tell devstack to use the default, it doesn't
> fernet_setup because for now it thinks the default is UUID and doesn't
> require keys. One way to have devstack work when fernet is the default is
> to have devstack always do keystone-manage fernet_setup.
>
> Really what we want to do is have devstack work like other deployment
> methods. We can reasonably expect featureful deployers like puppet to
> keystone-manage fernet_setup in the course of setting up keystone. There's
> more basic deployers like RPMs or debs that in the past have said they like
> the defaults to "just work" (like UUID tokens) and not require extra
> commands.
>
> - Brant
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Brant Knudson]

On Mon, Apr 18, 2016 at 10:20 AM, Matt Fischer  wrote:

> On Mon, Apr 18, 2016 at 8:29 AM, Brant Knudson  wrote:
>
>>
>>
>> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>>
>>> We all want Fernet to be a reality.  We ain't there yet (Except for
>>> mfish who has no patience) but we are getting closer.  The goal is to get
>>> Fernet as the default token provider as soon as possible. The review to do
>>> this has uncovered a few details that need to be fixed before we can do
>>> this.
>>>
>>> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
>>> https://review.openstack.org/#/c/278693/ Patch is still failing on
>>> Python 3.  The tests are kindof racy due to the revocation event 1 second
>>> granularity.  Some of the tests here have A sleep (1) in them still, but
>>> all should be using the time control aspect of the unit test fixtures.
>>>
>>> Some of the tests also use the same user to validate a token as that
>>> have, for example, a role unassigned.  These expose a problem that the
>>> revocation events are catching too many tokens, some of which should not be
>>> treated as revoked.
>>>
>>> Also, some of the logic for revocation checking has to change. Before,
>>> if a user had two roles, and had one removed, the token would be revoked.
>>> Now, however, the token will validate successful, but the response will
>>> only have the single assigned role in it.
>>>
>>>
>>> Python 3 tests are failing because the Fernet formatter is insisting
>>> that all project-ids be valid UUIDs, but some of the old tests have "FOO"
>>> and "BAR" as ids.  These either need to be converted to UUIDS, or the
>>> formatter needs to be more forgiving.
>>>
>>> Caching of token validations was messing with revocation checking.
>>> Tokens that were valid once were being reported as always valid. Thus, the
>>> current review  removes all caching on token validations, a change we
>>> cannot maintain.  Once all the test are successfully passing, we will
>>> re-introduce the cache, and be far more aggressive about cache invalidation.
>>>
>>> Tempest tests are currently failing due to Devstack not properly
>>> identifying Fernet as the default token provider, and creating the Fernet
>>> key repository.  I'm tempted to just force devstack to always create the
>>> directory, as a user would need it if they ever switched the token provider
>>> post launch anyway.
>>>
>>>
>> There's a review to change devstack to default to fernet:
>> https://review.openstack.org/#/c/195780/ . This was mostly to show that
>> tempest still passes with fernet configured. It uncovered a couple of test
>> issues (similar in nature to the revocation checking issues mentioned in
>> the original note) that have since been fixed.
>>
>> We'd prefer to not have devstack overriding config options and instead
>> use keystone's defaults. The problem is if fernet is the default in
>> keystone then it won't work out of the box since the key database won't
>> exist. One option that I think we should investigate is to have keystone
>> create the key database on startup if it doesn't exist.
>>
>> - Brant
>>
>>
>
> I'm not a devstack user, but as I mentioned before, I assume devstack
> called keystone-manage db_sync? Why couldn't it also call keystone-manage
> fernet_setup?
>
>
When you tell devstack that it's using fernet then it does keystone-manage
fernet_setup. When you tell devstack to use the default, it doesn't
fernet_setup because for now it thinks the default is UUID and doesn't
require keys. One way to have devstack work when fernet is the default is
to have devstack always do keystone-manage fernet_setup.

Really what we want to do is have devstack work like other deployment
methods. We can reasonably expect featureful deployers like puppet to
keystone-manage fernet_setup in the course of setting up keystone. There's
more basic deployers like RPMs or debs that in the past have said they like
the defaults to "just work" (like UUID tokens) and not require extra
commands.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Lance Bragstad]

It looks like it does [0].


[0]
https://github.com/openstack-dev/devstack/blob/4e7804431ada7e2cc0db63bd4c52b17782d33b5b/lib/keystone#L494-L497

On Mon, Apr 18, 2016 at 10:20 AM, Matt Fischer  wrote:

> On Mon, Apr 18, 2016 at 8:29 AM, Brant Knudson  wrote:
>
>>
>>
>> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>>
>>> We all want Fernet to be a reality.  We ain't there yet (Except for
>>> mfish who has no patience) but we are getting closer.  The goal is to get
>>> Fernet as the default token provider as soon as possible. The review to do
>>> this has uncovered a few details that need to be fixed before we can do
>>> this.
>>>
>>> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
>>> https://review.openstack.org/#/c/278693/ Patch is still failing on
>>> Python 3.  The tests are kindof racy due to the revocation event 1 second
>>> granularity.  Some of the tests here have A sleep (1) in them still, but
>>> all should be using the time control aspect of the unit test fixtures.
>>>
>>> Some of the tests also use the same user to validate a token as that
>>> have, for example, a role unassigned.  These expose a problem that the
>>> revocation events are catching too many tokens, some of which should not be
>>> treated as revoked.
>>>
>>> Also, some of the logic for revocation checking has to change. Before,
>>> if a user had two roles, and had one removed, the token would be revoked.
>>> Now, however, the token will validate successful, but the response will
>>> only have the single assigned role in it.
>>>
>>>
>>> Python 3 tests are failing because the Fernet formatter is insisting
>>> that all project-ids be valid UUIDs, but some of the old tests have "FOO"
>>> and "BAR" as ids.  These either need to be converted to UUIDS, or the
>>> formatter needs to be more forgiving.
>>>
>>> Caching of token validations was messing with revocation checking.
>>> Tokens that were valid once were being reported as always valid. Thus, the
>>> current review  removes all caching on token validations, a change we
>>> cannot maintain.  Once all the test are successfully passing, we will
>>> re-introduce the cache, and be far more aggressive about cache invalidation.
>>>
>>> Tempest tests are currently failing due to Devstack not properly
>>> identifying Fernet as the default token provider, and creating the Fernet
>>> key repository.  I'm tempted to just force devstack to always create the
>>> directory, as a user would need it if they ever switched the token provider
>>> post launch anyway.
>>>
>>>
>> There's a review to change devstack to default to fernet:
>> https://review.openstack.org/#/c/195780/ . This was mostly to show that
>> tempest still passes with fernet configured. It uncovered a couple of test
>> issues (similar in nature to the revocation checking issues mentioned in
>> the original note) that have since been fixed.
>>
>> We'd prefer to not have devstack overriding config options and instead
>> use keystone's defaults. The problem is if fernet is the default in
>> keystone then it won't work out of the box since the key database won't
>> exist. One option that I think we should investigate is to have keystone
>> create the key database on startup if it doesn't exist.
>>
>> - Brant
>>
>>
>
> I'm not a devstack user, but as I mentioned before, I assume devstack
> called keystone-manage db_sync? Why couldn't it also call keystone-manage
> fernet_setup?
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Matt Fischer]

On Mon, Apr 18, 2016 at 8:29 AM, Brant Knudson  wrote:

>
>
> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>
>> We all want Fernet to be a reality.  We ain't there yet (Except for mfish
>> who has no patience) but we are getting closer.  The goal is to get Fernet
>> as the default token provider as soon as possible. The review to do this
>> has uncovered a few details that need to be fixed before we can do this.
>>
>> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
>> https://review.openstack.org/#/c/278693/ Patch is still failing on
>> Python 3.  The tests are kindof racy due to the revocation event 1 second
>> granularity.  Some of the tests here have A sleep (1) in them still, but
>> all should be using the time control aspect of the unit test fixtures.
>>
>> Some of the tests also use the same user to validate a token as that
>> have, for example, a role unassigned.  These expose a problem that the
>> revocation events are catching too many tokens, some of which should not be
>> treated as revoked.
>>
>> Also, some of the logic for revocation checking has to change. Before, if
>> a user had two roles, and had one removed, the token would be revoked.
>> Now, however, the token will validate successful, but the response will
>> only have the single assigned role in it.
>>
>>
>> Python 3 tests are failing because the Fernet formatter is insisting that
>> all project-ids be valid UUIDs, but some of the old tests have "FOO" and
>> "BAR" as ids.  These either need to be converted to UUIDS, or the formatter
>> needs to be more forgiving.
>>
>> Caching of token validations was messing with revocation checking. Tokens
>> that were valid once were being reported as always valid. Thus, the current
>> review  removes all caching on token validations, a change we cannot
>> maintain.  Once all the test are successfully passing, we will re-introduce
>> the cache, and be far more aggressive about cache invalidation.
>>
>> Tempest tests are currently failing due to Devstack not properly
>> identifying Fernet as the default token provider, and creating the Fernet
>> key repository.  I'm tempted to just force devstack to always create the
>> directory, as a user would need it if they ever switched the token provider
>> post launch anyway.
>>
>>
> There's a review to change devstack to default to fernet:
> https://review.openstack.org/#/c/195780/ . This was mostly to show that
> tempest still passes with fernet configured. It uncovered a couple of test
> issues (similar in nature to the revocation checking issues mentioned in
> the original note) that have since been fixed.
>
> We'd prefer to not have devstack overriding config options and instead use
> keystone's defaults. The problem is if fernet is the default in keystone
> then it won't work out of the box since the key database won't exist. One
> option that I think we should investigate is to have keystone create the
> key database on startup if it doesn't exist.
>
> - Brant
>
>

I'm not a devstack user, but as I mentioned before, I assume devstack
called keystone-manage db_sync? Why couldn't it also call keystone-manage
fernet_setup?
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Brant Knudson]

On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:

> We all want Fernet to be a reality.  We ain't there yet (Except for mfish
> who has no patience) but we are getting closer.  The goal is to get Fernet
> as the default token provider as soon as possible. The review to do this
> has uncovered a few details that need to be fixed before we can do this.
>
> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
> https://review.openstack.org/#/c/278693/ Patch is still failing on Python
> 3.  The tests are kindof racy due to the revocation event 1 second
> granularity.  Some of the tests here have A sleep (1) in them still, but
> all should be using the time control aspect of the unit test fixtures.
>
> Some of the tests also use the same user to validate a token as that have,
> for example, a role unassigned.  These expose a problem that the revocation
> events are catching too many tokens, some of which should not be treated as
> revoked.
>
> Also, some of the logic for revocation checking has to change. Before, if
> a user had two roles, and had one removed, the token would be revoked.
> Now, however, the token will validate successful, but the response will
> only have the single assigned role in it.
>
>
> Python 3 tests are failing because the Fernet formatter is insisting that
> all project-ids be valid UUIDs, but some of the old tests have "FOO" and
> "BAR" as ids.  These either need to be converted to UUIDS, or the formatter
> needs to be more forgiving.
>
> Caching of token validations was messing with revocation checking. Tokens
> that were valid once were being reported as always valid. Thus, the current
> review  removes all caching on token validations, a change we cannot
> maintain.  Once all the test are successfully passing, we will re-introduce
> the cache, and be far more aggressive about cache invalidation.
>
> Tempest tests are currently failing due to Devstack not properly
> identifying Fernet as the default token provider, and creating the Fernet
> key repository.  I'm tempted to just force devstack to always create the
> directory, as a user would need it if they ever switched the token provider
> post launch anyway.
>
>
There's a review to change devstack to default to fernet:
https://review.openstack.org/#/c/195780/ . This was mostly to show that
tempest still passes with fernet configured. It uncovered a couple of test
issues (similar in nature to the revocation checking issues mentioned in
the original note) that have since been fixed.

We'd prefer to not have devstack overriding config options and instead use
keystone's defaults. The problem is if fernet is the default in keystone
then it won't work out of the box since the key database won't exist. One
option that I think we should investigate is to have keystone create the
key database on startup if it doesn't exist.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Matt Fischer]

On Fri, Apr 15, 2016 at 8:04 PM, Adam Young  wrote:

> We all want Fernet to be a reality.  We ain't there yet (Except for mfish
> who has no patience) but we are getting closer.  The goal is to get Fernet
> as the default token provider as soon as possible. The review to do this
> has uncovered a few details that need to be fixed before we can do this.
>

I exist to beta test for you!


>
> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
> https://review.openstack.org/#/c/278693/ Patch is still failing on Python
> 3.  The tests are kindof racy due to the revocation event 1 second
> granularity.  Some of the tests here have A sleep (1) in them still, but
> all should be using the time control aspect of the unit test fixtures.
>
> Some of the tests also use the same user to validate a token as that have,
> for example, a role unassigned.  These expose a problem that the revocation
> events are catching too many tokens, some of which should not be treated as
> revoked.
>
> Also, some of the logic for revocation checking has to change. Before, if
> a user had two roles, and had one removed, the token would be revoked.
> Now, however, the token will validate successful, but the response will
> only have the single assigned role in it.
>
>
> Python 3 tests are failing because the Fernet formatter is insisting that
> all project-ids be valid UUIDs, but some of the old tests have "FOO" and
> "BAR" as ids.  These either need to be converted to UUIDS, or the formatter
> needs to be more forgiving.
>
> Caching of token validations was messing with revocation checking. Tokens
> that were valid once were being reported as always valid. Thus, the current
> review  removes all caching on token validations, a change we cannot
> maintain.  Once all the test are successfully passing, we will re-introduce
> the cache, and be far more aggressive about cache invalidation.
>

Which review removes this, the one above?


>
> Tempest tests are currently failing due to Devstack not properly
> identifying Fernet as the default token provider, and creating the Fernet
> key repository.  I'm tempted to just force devstack to always create the
> directory, as a user would need it if they ever switched the token provider
> post launch anyway.
>

Wouldn't devstack handle the same way it might call keystone manage
bootstrap or db_sync? Devstack could just call fernet_setup if that's the
configured provider?

Also I know that there is a session on this at the summit too, but for me,
I want Fernet validation to be faster. I know work is underway for this in
Newton, but I'd consider adding it to the to-do pile.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Keystone] State of Fernet Token deployment

[Adam Young]

We all want Fernet to be a reality.  We ain't there yet (Except for 
mfish who has no patience) but we are getting closer.  The goal is to 
get Fernet as the default token provider as soon as possible. The review 
to do this has uncovered a few details that need to be fixed before we 
can do this.


Trusts for V2 tokens were not working correctly.  Relatively easy fix. 
https://review.openstack.org/#/c/278693/ Patch is still failing on 
Python 3.  The tests are kindof racy due to the revocation event 1 
second granularity.  Some of the tests here have A sleep (1) in them 
still, but all should be using the time control aspect of the unit test 
fixtures.


Some of the tests also use the same user to validate a token as that 
have, for example, a role unassigned.  These expose a problem that the 
revocation events are catching too many tokens, some of which should not 
be treated as revoked.


Also, some of the logic for revocation checking has to change. Before, 
if a user had two roles, and had one removed, the token would be 
revoked.  Now, however, the token will validate successful, but the 
response will only have the single assigned role in it.



Python 3 tests are failing because the Fernet formatter is insisting 
that all project-ids be valid UUIDs, but some of the old tests have 
"FOO" and "BAR" as ids.  These either need to be converted to UUIDS, or 
the formatter needs to be more forgiving.


Caching of token validations was messing with revocation checking. 
Tokens that were valid once were being reported as always valid. Thus, 
the current review  removes all caching on token validations, a change 
we cannot maintain.  Once all the test are successfully passing, we will 
re-introduce the cache, and be far more aggressive about cache 
invalidation.


Tempest tests are currently failing due to Devstack not properly 
identifying Fernet as the default token provider, and creating the 
Fernet key repository.  I'm tempted to just force devstack to always 
create the directory, as a user would need it if they ever switched the 
token provider post launch anyway.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev