Re: [openstack-dev] [keystone] Case for renewability of tokens, increasing expiration time

[Fox, Kevin M]

Trusts are a part of the solution, yes, but without finer grained roles, the 
trust model doesn't help too much. You still must delegate far too much access 
at present. :/

Thanks,
Kevin

From: Jamie Lennox [jamielen...@gmail.com]
Sent: Saturday, November 21, 2015 9:49 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [keystone] Case for renewability of tokens, 
increasing expiration time


I realize this has been mostly closed up, but just a few additions.

On 19 November 2015 at 08:06, Dolph Mathews 
<dolph.math...@gmail.com<mailto:dolph.math...@gmail.com>> wrote:

On Tue, Nov 17, 2015 at 2:56 PM, Lindsay Pallickal 
<pallic...@gmail.com<mailto:pallic...@gmail.com>> wrote:


On Tue, Nov 17, 2015 at 5:31 AM, Dolph Mathews 
<dolph.math...@gmail.com<mailto:dolph.math...@gmail.com>> wrote:


On Tuesday, November 17, 2015, Lindsay Pallickal 
<pallic...@gmail.com<mailto:pallic...@gmail.com>> wrote:

It looks like expiration is done this way to limit the damage from stolen 
tokens, but if a token were stolen, it could be stolen from the same location a 
username and password will now have to sit, to get around having to re-supply 
them manually once an hour. Yes, forcing re-auth hourly with a username and 
password, at the Keystone level, can limit damage if any of the core Openstack 
services are compromised for the mass theft of tokens. But limited damage will 
depend just as much on how quickly the compromise is detected. The more time an 
attacker has in the system, the less his actions will be hampered by a lack of 
token renewals. VMs can be compromised, and data exported or destroyed given 
just a small window.

This first part of this is only a "maybe", not a completely true assertion. 
There are *many* more opportunities for individual tokens to be exposed than 
for the credentials used to create them. In the case if a mass token 
compromise, which I consider to be a completely different scenario, token 
expiration probably isn't going it be any help because there's probably always 
a fresher token available to the attacker, anyway, until the exploit is closed. 
Instead, keystone has several means of quickly revoking tokens in bulk 
(revocation events, truncating the UUID token table, restarting the memcache 
token backend, or rotating all your Fernet keys away... all depending on your 
deployment).

The token does get sent a lot more often than the username/password 
credentials, but as long as both are sent via SSL/TLS, shouldn't the 
opportunity for exposure be similar? Although, I could see it being easier to 
accidentally send a token in the clear, or in a way vulnerable to a mitm 
(ignoring ssl cert issues), as there are frequently more bits of code in the 
client that deal using a token, versus a just few bits to secure when deal with 
logging in. Tokens get passed around to far more services with potential 
vulnerabilities as well, but I see that as a separate issue. I agree with your 
comment that token expiration will not really help in a mass compromise 
scenario.

Similar, yes, but with a couple significant differences. Off the top of my head:

In keystone's v2 API, bearer tokens are a part of the URL in token validation 
calls and token revocation calls. Of course, access logs are everywhere and 
many deployments do these operations over HTTP! The v3 API relegates them to 
headers, at least. I wonder if it would be possible to have keystoneauth / 
keystoneclient log a client-side warning when it's asked to send a bearer token 
over HTTP?

And certainly, the number of possible attack vectors on a bearer token being 
passed around to a bunch of services is increased by an order of magnitude.

OpenStack is also fairly free with its tokens. You send it to every service and 
every service that needs to do work elsewhere forwards it to someone else. At 
least keeping the token system the only service that your username/password is 
ever exposed to is keystone. An attacker able to compromise another service 
still has a lot of power due to the bearer-ness of tokens but they didn't get 
your user/pass.




On the other hand, if a user facing application, or a long running service 
supporting one, needs the API and is to remain convenient, they have to store 
usernames and passwords. Sometimes that long running service is connected to 
the outside world and faces security threats, such as when relaying Openstack 
API requests to their respective internal network endpoints - similar to what 
Horizon does. I wonder if I use Horizon continuously, whether it will force me 
to manually re-authenticate hourly, and if not, how it is getting around this 
problem, with or without storing the username and password.

The solution that was developed for this situation in heat was trusts. With 
this you delegate certain roles to the service user from the original user to 

Re: [openstack-dev] [keystone] Case for renewability of tokens, increasing expiration time

[Jamie Lennox]

I realize this has been mostly closed up, but just a few additions.

On 19 November 2015 at 08:06, Dolph Mathews  wrote:


> On Tue, Nov 17, 2015 at 2:56 PM, Lindsay Pallickal 
> wrote:
>
>>
>>
>> On Tue, Nov 17, 2015 at 5:31 AM, Dolph Mathews 
>> wrote:
>>
>>>
>>>
>>> On Tuesday, November 17, 2015, Lindsay Pallickal 
>>> wrote:
>>>

 It looks like expiration is done this way to limit the damage from
 stolen tokens, but if a token were stolen, it could be stolen from the same
 location a username and password will now have to sit, to get around having
 to re-supply them manually once an hour. Yes, forcing re-auth hourly with a
 username and password, at the Keystone level, can limit damage if any of
 the core Openstack services are compromised for the mass theft of tokens.
 But limited damage will depend just as much on how quickly the compromise
 is detected. The more time an attacker has in the system, the less his
 actions will be hampered by a lack of token renewals. VMs can be
 compromised, and data exported or destroyed given just a small window.

>>>
>>> This first part of this is only a "maybe", not a completely true
>>> assertion. There are *many* more opportunities for individual tokens to be
>>> exposed than for the credentials used to create them. In the case if a mass
>>> token compromise, which I consider to be a completely different scenario,
>>> token expiration probably isn't going it be any help because there's
>>> probably always a fresher token available to the attacker, anyway, until
>>> the exploit is closed. Instead, keystone has several means of quickly
>>> revoking tokens in bulk (revocation events, truncating the UUID token
>>> table, restarting the memcache token backend, or rotating all your Fernet
>>> keys away... all depending on your deployment).
>>>
>>
>> The token does get sent a lot more often than the username/password
>> credentials, but as long as both are sent via SSL/TLS, shouldn't the
>> opportunity for exposure be similar? Although, I could see it being easier
>> to accidentally send a token in the clear, or in a way vulnerable to a mitm
>> (ignoring ssl cert issues), as there are frequently more bits of code in
>> the client that deal using a token, versus a just few bits to secure when
>> deal with logging in. Tokens get passed around to far more services with
>> potential vulnerabilities as well, but I see that as a separate issue. I
>> agree with your comment that token expiration will not really help in a
>> mass compromise scenario.
>>
>
> Similar, yes, but with a couple significant differences. Off the top of my
> head:
>
> In keystone's v2 API, bearer tokens are a part of the URL in token
> validation calls and token revocation calls. Of course, access logs are
> everywhere and many deployments do these operations over HTTP! The v3 API
> relegates them to headers, at least. I wonder if it would be possible to
> have keystoneauth / keystoneclient log a client-side warning when it's
> asked to send a bearer token over HTTP?
>
> And certainly, the number of possible attack vectors on a bearer token
> being passed around to a bunch of services is increased by an order of
> magnitude.
>

OpenStack is also fairly free with its tokens. You send it to every service
and every service that needs to do work elsewhere forwards it to someone
else. At least keeping the token system the only service that your
username/password is ever exposed to is keystone. An attacker able to
compromise another service still has a lot of power due to the bearer-ness
of tokens but they didn't get your user/pass.


>>
>>>

 On the other hand, if a user facing application, or a long running
 service supporting one, needs the API and is to remain convenient, they
 have to store usernames and passwords. Sometimes that long running service
 is connected to the outside world and faces security threats, such as when
 relaying Openstack API requests to their respective internal network
 endpoints - similar to what Horizon does. I wonder if I use Horizon
 continuously, whether it will force me to manually re-authenticate hourly,
 and if not, how it is getting around this problem, with or without storing
 the username and password.

>>>
The solution that was developed for this situation in heat was trusts. With
this you delegate certain roles to the service user from the original user
to act on your behalf. This lets those service users perform actions on
behalf of users after the initial request has expired. Now there are
similarly all sorts of security implications to trusts, the ability to
steal trust ids, who sets them up, whats delegated, however if done right
they are a much more secure alternative than simply soring user/pass as at
least they are already project and role restricted.



>
>>> Bearer tokens are 

Re: [openstack-dev] [keystone] Case for renewability of tokens, increasing expiration time

[Dolph Mathews]

On Tue, Nov 17, 2015 at 2:56 PM, Lindsay Pallickal 
wrote:

>
>
> On Tue, Nov 17, 2015 at 5:31 AM, Dolph Mathews 
> wrote:
>
>>
>>
>> On Tuesday, November 17, 2015, Lindsay Pallickal 
>> wrote:
>>
>>>
>>> It looks like expiration is done this way to limit the damage from
>>> stolen tokens, but if a token were stolen, it could be stolen from the same
>>> location a username and password will now have to sit, to get around having
>>> to re-supply them manually once an hour. Yes, forcing re-auth hourly with a
>>> username and password, at the Keystone level, can limit damage if any of
>>> the core Openstack services are compromised for the mass theft of tokens.
>>> But limited damage will depend just as much on how quickly the compromise
>>> is detected. The more time an attacker has in the system, the less his
>>> actions will be hampered by a lack of token renewals. VMs can be
>>> compromised, and data exported or destroyed given just a small window.
>>>
>>
>> This first part of this is only a "maybe", not a completely true
>> assertion. There are *many* more opportunities for individual tokens to be
>> exposed than for the credentials used to create them. In the case if a mass
>> token compromise, which I consider to be a completely different scenario,
>> token expiration probably isn't going it be any help because there's
>> probably always a fresher token available to the attacker, anyway, until
>> the exploit is closed. Instead, keystone has several means of quickly
>> revoking tokens in bulk (revocation events, truncating the UUID token
>> table, restarting the memcache token backend, or rotating all your Fernet
>> keys away... all depending on your deployment).
>>
>
> The token does get sent a lot more often than the username/password
> credentials, but as long as both are sent via SSL/TLS, shouldn't the
> opportunity for exposure be similar? Although, I could see it being easier
> to accidentally send a token in the clear, or in a way vulnerable to a mitm
> (ignoring ssl cert issues), as there are frequently more bits of code in
> the client that deal using a token, versus a just few bits to secure when
> deal with logging in. Tokens get passed around to far more services with
> potential vulnerabilities as well, but I see that as a separate issue. I
> agree with your comment that token expiration will not really help in a
> mass compromise scenario.
>

Similar, yes, but with a couple significant differences. Off the top of my
head:

In keystone's v2 API, bearer tokens are a part of the URL in token
validation calls and token revocation calls. Of course, access logs are
everywhere and many deployments do these operations over HTTP! The v3 API
relegates them to headers, at least. I wonder if it would be possible to
have keystoneauth / keystoneclient log a client-side warning when it's
asked to send a bearer token over HTTP?

And certainly, the number of possible attack vectors on a bearer token
being passed around to a bunch of services is increased by an order of
magnitude.


>
>
>>
>>>
>>> On the other hand, if a user facing application, or a long running
>>> service supporting one, needs the API and is to remain convenient, they
>>> have to store usernames and passwords. Sometimes that long running service
>>> is connected to the outside world and faces security threats, such as when
>>> relaying Openstack API requests to their respective internal network
>>> endpoints - similar to what Horizon does. I wonder if I use Horizon
>>> continuously, whether it will force me to manually re-authenticate hourly,
>>> and if not, how it is getting around this problem, with or without storing
>>> the username and password.
>>>
>>
>> Bearer tokens are simply not the right solution to this use case. Unless
>> horizon were to store your credentials client-side, you will be logged out
>> after your keystone.conf [token] expiration passes.
>>
>
> I just checked and Horizon re-requests login after an hour, despite
> activity. So yeah, Horizon is not storing credentials, which is good.
>

Good to hear the confirmation that's still true - it's been a long while
since I've looked at that code, and I almost guarantee we don't test that
behavior anywhere.


>
>
>>
>>
>>>
>>> I suspect it is more probable to button up security closer to the core
>>> of Openstack, rather than relying on many third party API consuming
>>> apps/services to secure even more compromising credentials. Perhaps effort
>>> could be put into auditing for atypical mass token renewal events, like a
>>> sudden increase, or accumulating rate of renewal requests in proportion
>>> with the number of tokens streamed into compromised state.
>>>
>>> Allowing a maximum of 3 hours, in 1 hour renewal increments via
>>> Keystone, and making that configurable, would be a good compromise for
>>> user/outside facing apps/apis. Reducing temptation to adopt the security
>>> burden of storing 

Re: [openstack-dev] [keystone] Case for renewability of tokens, increasing expiration time

[Dolph Mathews]

On Tuesday, November 17, 2015, Lindsay Pallickal 
wrote:

> I was having an issue extending the expiration on unscoped and
> tenant/project scoped tokens retrieved with an existing token. I now
> realize this is a feature, not a bug, but I've got some points to argue
> that extending token expiration when getting a new token with an existing
> one (renewal), is worth considering. Maybe as on option with a cap on
> maximum renewal time.
>
> Problem (as state before I realized this was a feature):
>
> When I "renew" an unscoped token using an existing unscoped token, a new
> token (access.token.id is different) is issued, but the expiration is
> always the same as the first unscoped token that was returned on
> authentication with a username and password. I expected the expiration to
> extend with each new token, but this only seems to happen when an unscoped
> token is retrieved by authenticating with a username and password, not when
> an existing unscoped token is used to authenticate the request for a new
> token.
>
> Tenant/project scoped tokens, similarly, always return the expiration from
> the unscoped token used to retrieve them. Since any unscoped tokens can't
> get an updated expiration, the problem extends here.
>
> I have an application where the user enters their username and password
> once, and whenever a token is used, I mark it "dirty" so it is "renewed"
> within 30 seconds. This way they don't have to re-login until they haven't
> performed any actions for the ~1 hour expiration time Keystone grants. Now
> clearly the expiration time is not moving ahead with these "renewals", so
> it doesn't work as intended, and that leads me to seek some remedy. Storing
> a username and password in browser local storage/cookies, or multiple on a
> server that relays browser requests to api endpoints, would open a can of
> worms.
>
> Reasons to reconsider the expiration policy:
>
> It looks like expiration is done this way to limit the damage from stolen
> tokens, but if a token were stolen, it could be stolen from the same
> location a username and password will now have to sit, to get around having
> to re-supply them manually once an hour. Yes, forcing re-auth hourly with a
> username and password, at the Keystone level, can limit damage if any of
> the core Openstack services are compromised for the mass theft of tokens.
> But limited damage will depend just as much on how quickly the compromise
> is detected. The more time an attacker has in the system, the less his
> actions will be hampered by a lack of token renewals. VMs can be
> compromised, and data exported or destroyed given just a small window.
>

This first part of this is only a "maybe", not a completely true assertion.
There are *many* more opportunities for individual tokens to be exposed
than for the credentials used to create them. In the case if a mass token
compromise, which I consider to be a completely different scenario, token
expiration probably isn't going it be any help because there's probably
always a fresher token available to the attacker, anyway, until the exploit
is closed. Instead, keystone has several means of quickly revoking tokens
in bulk (revocation events, truncating the UUID token table, restarting the
memcache token backend, or rotating all your Fernet keys away... all
depending on your deployment).


>
> On the other hand, if a user facing application, or a long running service
> supporting one, needs the API and is to remain convenient, they have to
> store usernames and passwords. Sometimes that long running service is
> connected to the outside world and faces security threats, such as when
> relaying Openstack API requests to their respective internal network
> endpoints - similar to what Horizon does. I wonder if I use Horizon
> continuously, whether it will force me to manually re-authenticate hourly,
> and if not, how it is getting around this problem, with or without storing
> the username and password.
>

Bearer tokens are simply not the right solution to this use case. Unless
horizon were to store your credentials client-side, you will be logged out
after your keystone.conf [token] expiration passes.


>
> I suspect it is more probable to button up security closer to the core of
> Openstack, rather than relying on many third party API consuming
> apps/services to secure even more compromising credentials. Perhaps effort
> could be put into auditing for atypical mass token renewal events, like a
> sudden increase, or accumulating rate of renewal requests in proportion
> with the number of tokens streamed into compromised state.
>
> Allowing a maximum of 3 hours, in 1 hour renewal increments via Keystone,
> and making that configurable, would be a good compromise for user/outside
> facing apps/apis. Reducing temptation to adopt the security burden of
> storing usernames and passwords, especially on a server exposed to the
> outside or in storage that is outside (cookies & local 

Re: [openstack-dev] [keystone] Case for renewability of tokens, increasing expiration time

[Lindsay Pallickal]

On Tue, Nov 17, 2015 at 5:31 AM, Dolph Mathews 
wrote:

>
>
> On Tuesday, November 17, 2015, Lindsay Pallickal 
> wrote:
>
>>
>> It looks like expiration is done this way to limit the damage from stolen
>> tokens, but if a token were stolen, it could be stolen from the same
>> location a username and password will now have to sit, to get around having
>> to re-supply them manually once an hour. Yes, forcing re-auth hourly with a
>> username and password, at the Keystone level, can limit damage if any of
>> the core Openstack services are compromised for the mass theft of tokens.
>> But limited damage will depend just as much on how quickly the compromise
>> is detected. The more time an attacker has in the system, the less his
>> actions will be hampered by a lack of token renewals. VMs can be
>> compromised, and data exported or destroyed given just a small window.
>>
>
> This first part of this is only a "maybe", not a completely true
> assertion. There are *many* more opportunities for individual tokens to be
> exposed than for the credentials used to create them. In the case if a mass
> token compromise, which I consider to be a completely different scenario,
> token expiration probably isn't going it be any help because there's
> probably always a fresher token available to the attacker, anyway, until
> the exploit is closed. Instead, keystone has several means of quickly
> revoking tokens in bulk (revocation events, truncating the UUID token
> table, restarting the memcache token backend, or rotating all your Fernet
> keys away... all depending on your deployment).
>

The token does get sent a lot more often than the username/password
credentials, but as long as both are sent via SSL/TLS, shouldn't the
opportunity for exposure be similar? Although, I could see it being easier
to accidentally send a token in the clear, or in a way vulnerable to a mitm
(ignoring ssl cert issues), as there are frequently more bits of code in
the client that deal using a token, versus a just few bits to secure when
deal with logging in. Tokens get passed around to far more services with
potential vulnerabilities as well, but I see that as a separate issue. I
agree with your comment that token expiration will not really help in a
mass compromise scenario.


>
>>
>> On the other hand, if a user facing application, or a long running
>> service supporting one, needs the API and is to remain convenient, they
>> have to store usernames and passwords. Sometimes that long running service
>> is connected to the outside world and faces security threats, such as when
>> relaying Openstack API requests to their respective internal network
>> endpoints - similar to what Horizon does. I wonder if I use Horizon
>> continuously, whether it will force me to manually re-authenticate hourly,
>> and if not, how it is getting around this problem, with or without storing
>> the username and password.
>>
>
> Bearer tokens are simply not the right solution to this use case. Unless
> horizon were to store your credentials client-side, you will be logged out
> after your keystone.conf [token] expiration passes.
>

I just checked and Horizon re-requests login after an hour, despite
activity. So yeah, Horizon is not storing credentials, which is good.


>
>
>>
>> I suspect it is more probable to button up security closer to the core of
>> Openstack, rather than relying on many third party API consuming
>> apps/services to secure even more compromising credentials. Perhaps effort
>> could be put into auditing for atypical mass token renewal events, like a
>> sudden increase, or accumulating rate of renewal requests in proportion
>> with the number of tokens streamed into compromised state.
>>
>> Allowing a maximum of 3 hours, in 1 hour renewal increments via Keystone,
>> and making that configurable, would be a good compromise for user/outside
>> facing apps/apis. Reducing temptation to adopt the security burden of
>> storing usernames and passwords, especially on a server exposed to the
>> outside or in storage that is outside (cookies & local storage), could
>> itself be a security boon.
>>
>
> Have you tried setting keystone.conf [token] expiration to 10800 (3 hours,
> in seconds)? If so, why doesn't this satisfy your use case? And why expect
> ordinary users to do busy work to hit the same token lifespan that an
> attacker will trivially achieve anyway?
>
>

I wish I had seen that expiration setting earlier. Just what I needed,
thank you.

In retrospect, it does not make sense to force users to do the busy work of
renewal when an attacker could just do the same. Even if the token has
expired a earlier due to non-renewal, causing an intruder misses the
window, it will only be a matter of time before a fresh token is available
through the same vector. I think a 3 hour expiration window will not add a
lot of risk, and the option is already built in, so all is well. Thanks
again.

Lindsay