Re: [openstack-dev] [keystone][nova] Persistent application credentials

[James Penick]

Big +1 to re-evaluating this. In my environment we have many users
deploying and managing a number of different apps in different tenants.
Some of our users, such as Yahoo Mail service engineers could be in up to
40 different tenants. Those service engineers may change products as their
careers develop. Having to re-deploy part of an application stack because
Sally SE changed products would be unnecessarily disruptive.

 I regret that I missed the bus on this back in June. But at Oath we've
built a system (Called Copper Argos) on top of Athenz (it's open source:
www.athenz.io) to provide instance identity in a way that is both unique
but doesn't have all of the problems of a static persistent identity.

 The really really really* high level overview is:
1. Users pass application identity data to Nova as metadata during the boot
process.
2. Our vendor-data driver works with a service called HostSignd to validate
that data and create a one time use attestation document which is injected
into the instance's config drive.
3. On boot an agent within the instance will use that time-limited host
attestation document to identify itself to the Athenz identity service,
which will then exchange the document for a unique certificate containing
the application data passed in the boot call.
4. From then on the instance identity (TLS certificate) is periodically
exchanged by the agent for a new certificate.
5. The host attestation document and the instance TLS certificate can each
only be used a single time to exchange for another certificate. The
attestation document has a very short ttl, and the instance identity is set
to live slightly longer than the planned rotation frequency. So if you
rotate your certificates once an hour, the ttl on the cert should be 2
hours. This gives some wiggle room in the event the identity service is
down for any reason.

The agent is also capable of supporting SSH CA by passing the SSH host key
up to be re-signed whenever it exchanges the TLS certificate. All instances
leveraging Athens identity can communicate to one another using TLS mutual
auth.

If there's any interest i'd be happy to go into more detail here on the ML
and/or at the summit in Sydney

-James
* With several more zoolander-style Really's thrown in for good measure.


On Tue, Oct 10, 2017 at 12:34 PM, Fox, Kevin M  wrote:

> Big +1 for reevaluating the bigger picture. We have a pile of api's that
> together don't always form the most useful of api's due to lack of big
> picture analysis.
>
> +1 to thinking through the dev's/devops use case.
>
> Another one to really think over is single user that != application
> developer. IE, Pure user type person deploying cloud app in their tenant
> written by dev not employees by user's company. User shouldn't have to go
> to Operator to provision service accounts and other things. App dev should
> be able to give everything needed to let OpenStack launch say a heat
> template that provisions the service accounts for the User, not making the
> user twiddle the api themselves. It should be a "here, launch this" kind of
> thing, and they fill out the heat form, and out pops a working app. If they
> have to go prevision a bunch of stuff themselves before passing stuff to
> the form, game over. Likewise, if they have to look at yaml, game over. How
> do app credentials fit into this?
>
> Thanks,
> Kevin
>
> ____
> From: Zane Bitter [zbit...@redhat.com]
> Sent: Monday, October 09, 2017 9:39 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [keystone][nova] Persistent application
> credentials
>
> On 12/09/17 18:58, Colleen Murphy wrote:
> > While it's fresh in our minds, I wanted to write up a short recap of
> > where we landed in the Application Credentials discussion in the BM/VM
> > room today. For convenience the (as of yet unrevised) spec is here:
>
> Thanks so much for staying on this Colleen, it's tremendously helpful to
> have someone from the core team keeping an eye on it :)
>
> > http://specs.openstack.org/openstack/keystone-specs/
> specs/keystone/backlog/application-credentials.html
> >
> > Attached are images of the whiteboarded notes.
> >
> > On the contentious question of the lifecycle of an application
> > credential, we re-landed in the same place we found ourselves in when
> > the spec originally landed, which is that the credential becomes invalid
> > when its creating user is disabled or deleted. The risk involved in
> > allowing a credential to continue to be valid after its creating user
> > has been disabled is not really surmountable, and we are basically
> > giving up on this feature. The benefits we still get from not having to
>

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Fox, Kevin M]

Big +1 for reevaluating the bigger picture. We have a pile of api's that 
together don't always form the most useful of api's due to lack of big picture 
analysis.

+1 to thinking through the dev's/devops use case.

Another one to really think over is single user that != application developer. 
IE, Pure user type person deploying cloud app in their tenant written by dev 
not employees by user's company. User shouldn't have to go to Operator to 
provision service accounts and other things. App dev should be able to give 
everything needed to let OpenStack launch say a heat template that provisions 
the service accounts for the User, not making the user twiddle the api 
themselves. It should be a "here, launch this" kind of thing, and they fill out 
the heat form, and out pops a working app. If they have to go prevision a bunch 
of stuff themselves before passing stuff to the form, game over. Likewise, if 
they have to look at yaml, game over. How do app credentials fit into this?

Thanks,
Kevin


From: Zane Bitter [zbit...@redhat.com]
Sent: Monday, October 09, 2017 9:39 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [keystone][nova] Persistent application credentials

On 12/09/17 18:58, Colleen Murphy wrote:
> While it's fresh in our minds, I wanted to write up a short recap of
> where we landed in the Application Credentials discussion in the BM/VM
> room today. For convenience the (as of yet unrevised) spec is here:

Thanks so much for staying on this Colleen, it's tremendously helpful to
have someone from the core team keeping an eye on it :)

> http://specs.openstack.org/openstack/keystone-specs/specs/keystone/backlog/application-credentials.html
>
> Attached are images of the whiteboarded notes.
>
> On the contentious question of the lifecycle of an application
> credential, we re-landed in the same place we found ourselves in when
> the spec originally landed, which is that the credential becomes invalid
> when its creating user is disabled or deleted. The risk involved in
> allowing a credential to continue to be valid after its creating user
> has been disabled is not really surmountable, and we are basically
> giving up on this feature. The benefits we still get from not having to
> embed user passwords in config files, especially for LDAP or federated
> users, is still a vast improvement over the situation today, as is the
> ability to rotate credentials.

OK, there were lots of smart people in the room so I trust that y'all
made the right decision.

I'd just like to step back for a moment though and ask: how exactly do
we expect users to make use of Keystone?

When I think about a typical OpenStack user of the near future, they
looks something like this: there's a team with a handful of developers,
with maybe one or two devops engineers. This team is responsible for a
bunch of applications, at various stages in their lifecycles. They work
in a department with several such teams, in an organisation with several
such departments. People regularly join or leave the team - whether
because they join or leave the organisation or just transfer between
different teams. The applications are deployed with Heat and are at
least partly self-managing (e.g. they use auto-scaling, or auto-healing,
or have automated backups, or all of the above), but also require
occasional manual intervention (beyond just a Heat stack-update). The
applications may be deployed to a private OpenStack cloud, a public
OpenStack cloud, or both, with minimal differences in how they work when
moving back and forth.

(Obviously the beauty of Open Source is that we don't think about only
one set of users. But I think if we can serve this set of users as a
baseline then we have built something pretty generically useful.)

So my question is: how do we recommend these users use Keystone? We
definitely _can_ support them. But the most workable way I can think of
would be to create a long-lived application user account for each
project in LDAP/ActiveDirectory/whatever and have that account manage
the application. Then things will work basically the same way in the
public cloud, where you also get a user per project. Hopefully some
auditability is maintained by having Jenkins/Zuul/Solum/whatever do the
pushing of changes to Heat, although realistically many users will not
be that sophisticated. Once we have application credentials, the folks
doing manual intervention would be able to do so in the same way on
public clouds as on private clouds, without being given the account
credentials.

Some observations about this scenario:
* The whole user/role infrastructure is completely unused - 'Users' are
1:1 with projects. We might as well not have built it.
* Having Keystone backed by LDAP/ActiveDirectory is arguably worse than
useless - it just means there

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Colleen Murphy]

On Mon, Oct 9, 2017 at 6:39 PM, Zane Bitter  wrote:

> On 12/09/17 18:58, Colleen Murphy wrote:
>
>> While it's fresh in our minds, I wanted to write up a short recap of
>> where we landed in the Application Credentials discussion in the BM/VM room
>> today. For convenience the (as of yet unrevised) spec is here:
>>
>
> Thanks so much for staying on this Colleen, it's tremendously helpful to
> have someone from the core team keeping an eye on it :)

No problem :)

>
>
> http://specs.openstack.org/openstack/keystone-specs/specs/ke
>> ystone/backlog/application-credentials.html
>>
>> Attached are images of the whiteboarded notes.
>>
>> On the contentious question of the lifecycle of an application
>> credential, we re-landed in the same place we found ourselves in when the
>> spec originally landed, which is that the credential becomes invalid when
>> its creating user is disabled or deleted. The risk involved in allowing a
>> credential to continue to be valid after its creating user has been
>> disabled is not really surmountable, and we are basically giving up on this
>> feature. The benefits we still get from not having to embed user passwords
>> in config files, especially for LDAP or federated users, is still a vast
>> improvement over the situation today, as is the ability to rotate
>> credentials.
>>
>
> OK, there were lots of smart people in the room so I trust that y'all made
> the right decision.
>
> I'd just like to step back for a moment though and ask: how exactly do we
> expect users to make use of Keystone?
>
> When I think about a typical OpenStack user of the near future, they looks
> something like this: there's a team with a handful of developers, with
> maybe one or two devops engineers. This team is responsible for a bunch of
> applications, at various stages in their lifecycles. They work in a
> department with several such teams, in an organisation with several such
> departments. People regularly join or leave the team - whether because they
> join or leave the organisation or just transfer between different teams.
> The applications are deployed with Heat and are at least partly
> self-managing (e.g. they use auto-scaling, or auto-healing, or have
> automated backups, or all of the above), but also require occasional manual
> intervention (beyond just a Heat stack-update). The applications may be
> deployed to a private OpenStack cloud, a public OpenStack cloud, or both,
> with minimal differences in how they work when moving back and forth.
>
> (Obviously the beauty of Open Source is that we don't think about only one
> set of users. But I think if we can serve this set of users as a baseline
> then we have built something pretty generically useful.)
>
> So my question is: how do we recommend these users use Keystone? We
> definitely _can_ support them. But the most workable way I can think of
> would be to create a long-lived application user account for each project
> in LDAP/ActiveDirectory/whatever and have that account manage the
> application. Then things will work basically the same way in the public
> cloud, where you also get a user per project. Hopefully some auditability
> is maintained by having Jenkins/Zuul/Solum/whatever do the pushing of
> changes to Heat, although realistically many users will not be that
> sophisticated. Once we have application credentials, the folks doing manual
> intervention would be able to do so in the same way on public clouds as on
> private clouds, without being given the account credentials.
>
> Some observations about this scenario:
> * The whole user/role infrastructure is completely unused - 'Users' are
> 1:1 with projects. We might as well not have built it.
> * Having Keystone backed by LDAP/ActiveDirectory is arguably worse than
> useless - it just means there are two different places to set things up
> when creating a project and an extra layer of indirection. (I won't say we
> might as well not have built it, because many organisations have compliance
> rules that, ahem, well let's just say they were developed in a different
> context :)
> * We're missing an essential feature of cloud (or even of VPSs): you
> shouldn't need to raise a ticket with IT to be able to deploy a new
> application. Any involvement from them should be asynchronous (e.g. setting
> quotas - although even that is an OpenStack-specific thing: in public
> clouds excessive use is discouraged by _billing_ and in non-OpenStack
> clouds users set their _own_ quotas); we don't want humans in the loop.
> * AFAIK it's not documented anywhere that this is the way we expect you to
> use OpenStack. Anybody would think it's all about the Users and Roles.
>
Another observation - if all the members of the team know the application
user's username and password (which they must, in order to use it to create
application credentials), then the team members who leave the team would
continue to have access to everything the application user has access to,
and also still be able t

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Zane Bitter]

On 12/09/17 18:58, Colleen Murphy wrote:
While it's fresh in our minds, I wanted to write up a short recap of 
where we landed in the Application Credentials discussion in the BM/VM 
room today. For convenience the (as of yet unrevised) spec is here:


Thanks so much for staying on this Colleen, it's tremendously helpful to 
have someone from the core team keeping an eye on it :)



http://specs.openstack.org/openstack/keystone-specs/specs/keystone/backlog/application-credentials.html

Attached are images of the whiteboarded notes.

On the contentious question of the lifecycle of an application 
credential, we re-landed in the same place we found ourselves in when 
the spec originally landed, which is that the credential becomes invalid 
when its creating user is disabled or deleted. The risk involved in 
allowing a credential to continue to be valid after its creating user 
has been disabled is not really surmountable, and we are basically 
giving up on this feature. The benefits we still get from not having to 
embed user passwords in config files, especially for LDAP or federated 
users, is still a vast improvement over the situation today, as is the 
ability to rotate credentials.


OK, there were lots of smart people in the room so I trust that y'all 
made the right decision.


I'd just like to step back for a moment though and ask: how exactly do 
we expect users to make use of Keystone?


When I think about a typical OpenStack user of the near future, they 
looks something like this: there's a team with a handful of developers, 
with maybe one or two devops engineers. This team is responsible for a 
bunch of applications, at various stages in their lifecycles. They work 
in a department with several such teams, in an organisation with several 
such departments. People regularly join or leave the team - whether 
because they join or leave the organisation or just transfer between 
different teams. The applications are deployed with Heat and are at 
least partly self-managing (e.g. they use auto-scaling, or auto-healing, 
or have automated backups, or all of the above), but also require 
occasional manual intervention (beyond just a Heat stack-update). The 
applications may be deployed to a private OpenStack cloud, a public 
OpenStack cloud, or both, with minimal differences in how they work when 
moving back and forth.


(Obviously the beauty of Open Source is that we don't think about only 
one set of users. But I think if we can serve this set of users as a 
baseline then we have built something pretty generically useful.)


So my question is: how do we recommend these users use Keystone? We 
definitely _can_ support them. But the most workable way I can think of 
would be to create a long-lived application user account for each 
project in LDAP/ActiveDirectory/whatever and have that account manage 
the application. Then things will work basically the same way in the 
public cloud, where you also get a user per project. Hopefully some 
auditability is maintained by having Jenkins/Zuul/Solum/whatever do the 
pushing of changes to Heat, although realistically many users will not 
be that sophisticated. Once we have application credentials, the folks 
doing manual intervention would be able to do so in the same way on 
public clouds as on private clouds, without being given the account 
credentials.


Some observations about this scenario:
* The whole user/role infrastructure is completely unused - 'Users' are 
1:1 with projects. We might as well not have built it.
* Having Keystone backed by LDAP/ActiveDirectory is arguably worse than 
useless - it just means there are two different places to set things up 
when creating a project and an extra layer of indirection. (I won't say 
we might as well not have built it, because many organisations have 
compliance rules that, ahem, well let's just say they were developed in 
a different context :)
* We're missing an essential feature of cloud (or even of VPSs): you 
shouldn't need to raise a ticket with IT to be able to deploy a new 
application. Any involvement from them should be asynchronous (e.g. 
setting quotas - although even that is an OpenStack-specific thing: in 
public clouds excessive use is discouraged by _billing_ and in 
non-OpenStack clouds users set their _own_ quotas); we don't want humans 
in the loop.
* AFAIK it's not documented anywhere that this is the way we expect you 
to use OpenStack. Anybody would think it's all about the Users and Roles.


Perhaps someone can suggest a better scenario for this group of users? I 
can't think of one that doesn't involve radical differences between 
public & private clouds (something we're explicitly trying to prevent, 
according to our mission statement), and/or risk total application 
breakage when personnel change.


My worry is that in this and other areas, there's a disconnect between 
the needs of the people whom we say we're building OpenStack for and 
what we're actually building. (Speculati

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Brant Knudson]

On Thu, Jul 20, 2017 at 8:02 PM, Zane Bitter  wrote:

>
> * If Keystone supported either a public-key or a Kerberos-style
> authentication mechanism to get a token


Keystone (via support for accepting authentication from the web server
hosting it) can be configured to accept X.509 and kerberos, see
http://www.jamielennox.net/blog/2015/02/12/step-by-step-kerberized-keystone/

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Zane Bitter]

On 19/07/17 23:19, Monty Taylor wrote:


Instance users do not solve this. Instance users can be built with this- 
but instance users are themselves not sufficient. Instance users are 
only sufficient in single-cloud ecosystems where it is possible to grant 
permissions on all the resources in the single-cloud ecosystem to an 
instance. We are not a single-cloud ecosystem.


Good point. Actually, nobody lives in a single-cloud ecosystem any more. 
So the 'public' side of any hybrid-cloud arrangement (including the big 
3 public clouds, not just OpenStack) will always need a way to deal with 
this.


Nodepool runs in Rackspace's DFW region. It has accounts across nine 
different clouds. If this were only solved with Instance users we'd have 
to boot a VM in each cloud so that we could call the publicly-accessible 
REST APIs of the clouds to boot VMs in each cloud.


I'm glad you're here, because I don't spend a lot of time thinking about 
such use cases (if we can get cloud applications to work on even one 
cloud then I can retire to my goat farm happy) and this one would have 
escaped me :)


So let's boil this down to 4 types of 'users' who need to authenticate 
to a given cloud:


1) Actual, corporeal humans
2) Services that are part of the cloud itself (e.g. autoscaling)
3) Hybrid-cloud applications running elsewhere (e.g. nodepool)
4) Applications running in the cloud

Looking at how AWS handles these cases AIUI:

1) For each tenant there is a 'root' account with access to billing &c. 
Best practice is not to create API credentials for this account at all. 
Instead, you create IAM Users for all of the humans who need to access 
the tenant and give permissions to them (bootstrapped by the root 
account) using IAM Policies. To make management easier, you can 
aggregate Users into Groups. If a user leaves the organisation, you 
delete their IAM User. If the owner leaves the organisation, somebody 
else becomes the owner and you rotate the root password.


2) Cloud services can be named as principals in IAM policies, so 
permissions can be given to them in the same way that they are to human 
users.


3) You create an IAM User for the application and give it the 
appropriate permissions. The credential they get is actually a private 
key, not a password, so in theory you could store it in an HSM that just 
signs stuff with it and not provide it directly to the application. 
Otherwise, the credentials are necessarily disclosed to the team 
maintaining the application. If somebody who has/had access to private 
key leaves, you need to rotate the credentials. It's possible to 
automate the mechanics of this, but ultimately it has to be triggered by 
a human using their own credentials otherwise it's turtles all the way 
down. The AWS cloud has no way of ensuring that you rotate the 
credentials at appropriate times, or even knowing when those times are.


4) Instance users. You can give permissions to a VM that you have 
created in the cloud. It automatically receives credentials in its 
metadata. The credentials expire quite rapidly and are automatically 
replaced with new ones, also accessible through the metadata server. The 
application just reads the latest credentials from metadata and uses 
them. If someone leaves the organisation, you don't care. If an attacker 
breaches your server, the damage is limited to a relatively short window 
once you've evicted them again. There's no way to do the Wrong Thing 
even if you're trying.


And in OpenStack:

1) Works great provided you only have one user per project. Your 
password may, and probably will, be shared with your billing account 
(public cloud), or will be shared with pretty much your whole life 
(private cloud). If multiple humans need to work on the project, you'll 
generally need to share passwords or do something out-of-band to set it 
up (e.g. open a ticket with IT). If somebody leaves the organisation, 
same deal.


Application credentials could greatly improve this in the public cloud 
scenario.


2) Cloud services can create trusts that allow them to act on behalf of 
a particular user. If that user leaves the organisation, your 
application is hosed until someone else redeploys it to get a new trust.


Persistent application credentials could potentially replace trusts and 
solve this problem, although they'd need to be stored somewhere more 
secure (i.e. Barbican) than trust IDs are currently stored. A better 
solution might be to allow the service user to be granted permissions by 
the forthcoming fine-grained authorisation mechanism (independently of 
an application credential) - but this would require changes to the 
Keystone policies, because it would currently be blocked by the 
Scoped-RBAC system.


3) The credentials are necessarily disclosed to the team maintaining the 
application. Your password may, and probably will, be shared with your 
billing account. If somebody leaves the organisation, you have to rotate 
the password. This 

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Zane Bitter]

On 19/07/17 22:27, Monty Taylor wrote:
I propose we set aside time at the PTG to dig in to this. Between Zane 
and I and the Keystone core team I have confidence we can find a way out.


This may be a bad time to mention that regrettably I won't be attending 
the PTG, due to (happy!) family reasons.


It sounds like you and I are on the same page already in terms of the 
requirements though. I'm fairly relaxed about what the solution looks 
like, as long as we actually address those requirements.


- ZB

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Lance Bragstad]

On 07/19/2017 09:27 PM, Monty Taylor wrote:
> On 07/19/2017 12:18 AM, Zane Bitter wrote:
>> On 18/07/17 10:55, Lance Bragstad wrote:

 Would Keystone folks be happy to allow persistent credentials once
 we have a way to hand out only the minimum required privileges?


 If I'm understanding correctly, this would make application
 credentials dependent on several cycles of policy work. Right?
>>>
>>> I think having the ability to communicate deprecations though
>>> oslo.policy would help here. We could use it to move towards better
>>> default roles, which requires being able to set minimum privileges.
>>>
>>> Using the current workflow requires operators to define the minimum
>>> privileges for whatever is using the application credential, and
>>> work that into their policy. Is that the intended workflow that we
>>> want to put on the users and operators of application credentials?
>>
>> The plan is to add an authorisation mechanism that is user-controlled
>> and independent of the (operator-controlled) policy. The beginnings
>> of this were included in earlier drafts of the spec, but were removed
>> in patch set 19 in favour of leaving them for a future spec:
>>
>> https://review.openstack.org/#/c/450415/18..19/specs/keystone/pike/application-credentials.rst
>
>
> Yes - that's right - and I expect to start work on that again as soon
> as this next keystoneauth release with version discovery is out the door.
>
> It turns out there are different POVs on this topic, and it's VERY
> important to be clear which one we're talking about at any given point
> in time. A bunch of the confusion just in getting as far as we've
> gotten so far came from folks saying words like "policy" or "trusts"
> or "ACLs" or "RBAC" - but not clarifying which group of cloud users
> they were discussing and from what context.
>
> The problem that Zane and I are are discussing and advocating for are
> for UNPRIVILEDGED users who neither deploy nor operate the cloud but
> who use the cloud to run applications.
>
> Unfortunately, neither the current policy system nor trusts are useful
> in any way shape or form for those humans. Policy and trusts are tools
> for cloud operators to take a certain set of actions.
>
> Similarly, the concern from the folks who are not in favor of
> project-lifecycled application credentials is the one that Zane
> outlined - that there will be $someone with access to those
> credentials after a User change event, and thus $security will be
> compromised.
>
> There is a balance that can and must be found. The use case Zane and I
> are talking about is ESSENTIAL, and literally ever single human who is
> a actually using OpenStack to run applications needs it. Needed it
> last year in fact, and they are, in fact doing things like writing
> ssh-agent like daemons in which they can store their corporate LDAP
> credentials so that their automation will work because we're not
> giving them a workable option.
>
> That said, the concerns about not letting a thing out the door that is
> insecure by design like PHP4's globally scoped URL variables is also
> super important.
>
> So we need to find a design that meets both goals.
>
> I have thoughts on the topic, but have been holding off until
> version-discovery is out the door. My hunch is that, like application
> credentials, we're not going to make significant headway without
> getting humans in the room - because the topic is WAY too fraught with
> peril.
>
> I propose we set aside time at the PTG to dig in to this. Between Zane
> and I and the Keystone core team I have confidence we can find a way out.

Done. I've added this thread to keystone's planning etherpad under
cross-project things we need to talk about [0]. Feel free to elaborate
and fill in context as you see fit. I'll make sure the content makes
it's way into a dedicated etherpad before we have that discussion
(usually as I go through each topic and plan the schedule).


[0] https://etherpad.openstack.org/p/keystone-queens-ptg

>
> Monty
>
> PS. It will not help to solve limited-scope before we solve this.
> Limited scope is an end-user opt-in action and having it does not
> remove the concerns that have been expressed.
>
> __
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Sean Dague]

On 07/19/2017 10:00 PM, Adrian Turjak wrote:
> The problem is then entirely procedural within a team. Do they rotate
> all keys when one person leaves? Anything less is the same problem. All
> we can do is make rotation less of a pain, but it will still be painful
> no matter what, and depending on the situation the team makes the choice
> of how to handle rotation if at all.
> 
> The sole reason for project level ownership of these application
> credentials is so that a user leaving/being deleted isn't a scramble to
> replace keys, and a team has the option/time to do it if they care about
> the possibility of that person having known the keys (again, not our
> problem, not a security flaw in code). Anything else, pretty much makes
> this feature useless for teams. :(
> 
> Having both options (owned by project vs user) is useful, but the
> 'security issues' are kind of implied by using project owned app creds.
> It's a very useful feature with some 'use at your own risk' attached.

I think this is a pretty good summary.

In many situations the situation of removing people from projects
(termination) will also physically remove their path to said clouds (as
they are beyond the firewall). It's not true with public clouds, but
it's not making the situation any worse, because right now it's shared
passwords to accounts.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Monty Taylor]

On 07/19/2017 12:11 AM, Zane Bitter wrote:

On 17/07/17 23:12, Lance Bragstad wrote:

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?


If I'm understanding correctly, this would make application 
credentials dependent on several cycles of policy work. Right?


My thought here was that if this were the case (i.e. persistent 
credentials are OK provided the user can lock down the privileges) then 
you could make a case that the current spec is on the right track. For 
now we implement the application credentials as non-persistent, people 
who know about it use at their own risk, and for people who don't 
there's no exposure. Later on we add the authorisation stuff and relax 
the non-persistence requirement.


On further reflection, I'm not convinced by this - if we care about 
protecting people who don't intentionally use/know about the feature 
now, then we should probably still care once the tools are in place for 
the people who are using it intentionally to lock it down tightly.


So I'm increasingly convinced that we need to do one of two things. Either:

* Agree with Colleen (elsewhere in the thread) that persistent 
application credentials are still better than the status quo and 
reinstate the project-scoped lifecycle in accordance with original 
intent of the spec; or


* Agree that the concerns raised by Morgan & Adam will always apply, and 
look for a solution that gives us automatic key rotation - which might 
be quite different in shape (I can elaborate if necessary).


(That said, I chatted about this briefly with Monty yesterday and he 
said that his recollection was that there is a long-term solution that 
will keep everyone happy. He'll try to remember what it is once he's 
finished on the version discovery stuff he's currently working on.)


Part of this comes down to the fact that there are actually multiple 
scenarios and persistent credentials actually only applies to a scenario 
that typically requires a human with elevated credentials.


SO - I think we can get a long way forward by divvying up some 
responsibilities clearly.


What I mean is:

* The simple consume case ("typical public cloud") is User-per-Project 
with User lifecycle tied to Project lifecycle. In this case the idea of 
a 'persistent' credential is meaningless, because there is no 'other' 
User with access to the Project If the User in this scenario creates a 
Credential it doesn't actually matter what the Credential lifecycle is 
because the act of Account is ultimately about disabling or deleting 
access to the Project in question. We can and should help the folks who 
are running clouds in this model with $something (we need to talk 
details) so that if they are running in this model they don't 
accidentally or by default leave a door open when they think they've 
disabled someone's User as part of shutting off their Account. But in 
this scenario OpenStack adding project-persistent credentials is not a 
big deal - it doesn't provide value. (While a User in that scenario, who 
typically does not have the Role to create a new User being able to 
manage Application Credentials is a HUGE win)


* The other scenario is where there is more than one Human who has a 
User that have been granted Roles on a Project. This is the one where 
project-lifecycle credentials are meaningful and valuable, but it's also 
one that involves some Human with elevated admin-style privileges having 
been involved at some point because that is required to assign Users 
Roles in the Project in the first place.


I believe if we divide application credentials into two kinds:

1) Application Credentials with lifecycle tied to User
2) Application Credentials with lifecycle tied to Project

Then I think it's ok for the ability to do (2) to require a specific 
Role in policy. If we do that, then whatever Human it is that is mapping 
multiple Users into a single Project can decide whether any of those 
Users should be granted the ability to make Project-lifecycle 
Application Credentials. Such a Human is already a Human who has a User 
with elevated permissions, as you have to be to assign Roles to Users 
Projects.


In any case, as I mentioned in the other mail, I think there are a bunch 
of details here that are going to require us being in the room - and 
everyone realizing that everyones use cases and everyones concerns are 
important. If we dig in, I'm sure we can come out on the other side with 
happiness and joy.




I'm trying to avoid taking a side here because everyone is right. 

++

Currently anybody who want to do anything remotely 'cloudy' (i.e. have 
the application talk to OpenStack APIs) has to either share their 
personal password with the application (and by extension their whole 


Or - create an account in the Team's name and by storing the password 
for that account realizing that everyone on the team has access to the 
password so tha

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Monty Taylor]

On 07/19/2017 12:18 AM, Zane Bitter wrote:

On 18/07/17 10:55, Lance Bragstad wrote:


Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?


If I'm understanding correctly, this would make application 
credentials dependent on several cycles of policy work. Right?


I think having the ability to communicate deprecations though 
oslo.policy would help here. We could use it to move towards better 
default roles, which requires being able to set minimum privileges.


Using the current workflow requires operators to define the minimum 
privileges for whatever is using the application credential, and work 
that into their policy. Is that the intended workflow that we want to 
put on the users and operators of application credentials?


The plan is to add an authorisation mechanism that is user-controlled 
and independent of the (operator-controlled) policy. The beginnings of 
this were included in earlier drafts of the spec, but were removed in 
patch set 19 in favour of leaving them for a future spec:


https://review.openstack.org/#/c/450415/18..19/specs/keystone/pike/application-credentials.rst 


Yes - that's right - and I expect to start work on that again as soon as 
this next keystoneauth release with version discovery is out the door.


It turns out there are different POVs on this topic, and it's VERY 
important to be clear which one we're talking about at any given point 
in time. A bunch of the confusion just in getting as far as we've gotten 
so far came from folks saying words like "policy" or "trusts" or "ACLs" 
or "RBAC" - but not clarifying which group of cloud users they were 
discussing and from what context.


The problem that Zane and I are are discussing and advocating for are 
for UNPRIVILEDGED users who neither deploy nor operate the cloud but who 
use the cloud to run applications.


Unfortunately, neither the current policy system nor trusts are useful 
in any way shape or form for those humans. Policy and trusts are tools 
for cloud operators to take a certain set of actions.


Similarly, the concern from the folks who are not in favor of 
project-lifecycled application credentials is the one that Zane outlined 
- that there will be $someone with access to those credentials after a 
User change event, and thus $security will be compromised.


There is a balance that can and must be found. The use case Zane and I 
are talking about is ESSENTIAL, and literally ever single human who is a 
actually using OpenStack to run applications needs it. Needed it last 
year in fact, and they are, in fact doing things like writing ssh-agent 
like daemons in which they can store their corporate LDAP credentials so 
that their automation will work because we're not giving them a workable 
option.


That said, the concerns about not letting a thing out the door that is 
insecure by design like PHP4's globally scoped URL variables is also 
super important.


So we need to find a design that meets both goals.

I have thoughts on the topic, but have been holding off until 
version-discovery is out the door. My hunch is that, like application 
credentials, we're not going to make significant headway without getting 
humans in the room - because the topic is WAY too fraught with peril.


I propose we set aside time at the PTG to dig in to this. Between Zane 
and I and the Keystone core team I have confidence we can find a way out.


Monty

PS. It will not help to solve limited-scope before we solve this. 
Limited scope is an end-user opt-in action and having it does not remove 
the concerns that have been expressed.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Adrian Turjak]

On 19/07/17 01:23, Colleen Murphy wrote:
> On Tue, Jul 18, 2017 at 1:39 AM, Zane Bitter  > wrote:
>
> So the application credentials spec has merged - huge thanks to
> Monty and the Keystone team for getting this done:
>
> https://review.openstack.org/#/c/450415/
> 
> 
> http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html
> 
> 
>
> However, it appears that there was a disconnect in how two groups
> of folks were reading the spec that only became apparent towards
> the end of the process. Specifically, at this exact moment:
>
> 
> http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59
> 
> 
>
> To summarise, Keystone folks are uncomfortable with the idea of
> application credentials that share the lifecycle of the project
> (rather than the user that created them), because a consumer could
> surreptitiously create an application credential and continue to
> use that to access the OpenStack APIs even after their User
> account is deleted. The agreed solution was to delete the
> application credentials when the User that created them is
> deleted, thus tying the lifecycle to that of the User.
>
> This means that teams using this feature will need to audit all of
> their applications for credential usage and rotate any credentials
> created by a soon-to-be-former team member *before* removing said
> team member's User account, or risk breakage. Basically we're
> relying on users to do the Right Thing (bad), but when they don't
> we're defaulting to breaking [some of] their apps over leaving
> them insecure (all things being equal, good).
>
> Unfortunately, if we do regard this as a serious problem, I don't
> think this solution is sufficient. Assuming that application
> credentials are stored on VMs in the project for use by the
> applications running on them, then anyone with access to those
> servers can obtain the credentials and continue to use them even
> if their own account is deleted. The solution to this is to rotate
> *all* application keys when a user is deleted. So really we're
> relying on users to do the Right Thing (bad), but when they don't
> we're defaulting to breaking [some of] their apps *and*
> [potentially] leaving them insecure (worst possible combination).
>
> (We're also being inconsistent, because according to the spec if
> you revoke a role from a User then any application credentials
> they've created that rely on that role continue to work. It's only
> if you delete the User that they're revoked.)
>
>
> As far as I can see, there are only two solutions to the
> fundamental problem:
>
> 1) Fine-grained user-defined access control. We can minimise the
> set of things that the application credentials are authorised to
> do. That's out of scope for this spec, but something we're already
> planning as a future enhancement.
> 2) Automated regular rotation of credentials. We can make sure
> that whatever a departing team member does manage to hang onto
> quickly becomes useless.
>
> By way of comparison, AWS does both. There's fine-grained defined
> access control in the form of IAM Roles, and these Roles can be
> associated with EC2 servers. The servers have an account with
> rotating keys provided through the metadata server. I can't find
> the exact period of rotation documented, but it's on the order of
> magnitude of 1 hour.
>
> There's plenty not to like about this design. Specifically, it's
> 2017 not 2007 and the idea that there's no point offering to
> segment permissions at a finer grained level than that of a VM no
> longer holds water IMHO, thanks to SELinux and containers. It'd be
> nice to be able to provide multiple sets of credentials to
> different services running on a VM, and it's probably essential to
> our survival that we find a way to provide individual credentials
> to containers. Nevertheless, what they have does solve the problem.
>
> Note that there's pretty much no sane way for the user to automate
> credential rotation themselves, because it's turtles all the way
> down. e.g. it's easy in principle to set up a Heat template with a
> Mistral workflow that will rotate the credentials for you, but
> they'll do so using trusts that are, in turn, tied back to the
> consumer who created the stack. (It suddenly occurs to me that
> this is a problem that all services using trusts are g

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Zane Bitter]

On 18/07/17 10:55, Lance Bragstad wrote:


Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?


If I'm understanding correctly, this would make application 
credentials dependent on several cycles of policy work. Right?


I think having the ability to communicate deprecations though 
oslo.policy would help here. We could use it to move towards better 
default roles, which requires being able to set minimum privileges.


Using the current workflow requires operators to define the minimum 
privileges for whatever is using the application credential, and work 
that into their policy. Is that the intended workflow that we want to 
put on the users and operators of application credentials?


The plan is to add an authorisation mechanism that is user-controlled 
and independent of the (operator-controlled) policy. The beginnings of 
this were included in earlier drafts of the spec, but were removed in 
patch set 19 in favour of leaving them for a future spec:


https://review.openstack.org/#/c/450415/18..19/specs/keystone/pike/application-credentials.rst

- ZB

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Zane Bitter]

On 17/07/17 23:12, Lance Bragstad wrote:

Would Keystone folks be happy to allow persistent credentials once
we have a way to hand out only the minimum required privileges?


If I'm understanding correctly, this would make application credentials 
dependent on several cycles of policy work. Right?


My thought here was that if this were the case (i.e. persistent 
credentials are OK provided the user can lock down the privileges) then 
you could make a case that the current spec is on the right track. For 
now we implement the application credentials as non-persistent, people 
who know about it use at their own risk, and for people who don't 
there's no exposure. Later on we add the authorisation stuff and relax 
the non-persistence requirement.


On further reflection, I'm not convinced by this - if we care about 
protecting people who don't intentionally use/know about the feature 
now, then we should probably still care once the tools are in place for 
the people who are using it intentionally to lock it down tightly.


So I'm increasingly convinced that we need to do one of two things. Either:

* Agree with Colleen (elsewhere in the thread) that persistent 
application credentials are still better than the status quo and 
reinstate the project-scoped lifecycle in accordance with original 
intent of the spec; or


* Agree that the concerns raised by Morgan & Adam will always apply, and 
look for a solution that gives us automatic key rotation - which might 
be quite different in shape (I can elaborate if necessary).


(That said, I chatted about this briefly with Monty yesterday and he 
said that his recollection was that there is a long-term solution that 
will keep everyone happy. He'll try to remember what it is once he's 
finished on the version discovery stuff he's currently working on.)



I'm trying to avoid taking a side here because everyone is right. 
Currently anybody who want to do anything remotely 'cloudy' (i.e. have 
the application talk to OpenStack APIs) has to either share their 
personal password with the application (and by extension their whole 
team) or to do the thing that is the polar opposite of cloud: file a 
ticket with IT to get a service user account added desk> and share that password instead. And this really is a disaster for 
OpenStack. On the other hand, allowing the creation of persistent 
application credentials in the absence of regular automatic rotation 
does create risk for those folks who are not aggressively auditing them 
(perhaps because they have no legitimate use for them) and the result is 
likely to be lots of clouds disabling them by policy, keeping their 
users in the dark age of IT-ticket-filing  and frustrating 
our interoperability goals.


It is possible in theory to satisfy both via the 'instance users' 
concept, but the Nova team's response to this has consistently been 
"prove to us that this has to be in Nova". Well, here's your answer.


cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Lance Bragstad]

On 07/17/2017 10:12 PM, Lance Bragstad wrote:
>
>
> On Mon, Jul 17, 2017 at 6:39 PM, Zane Bitter  > wrote:
>
> So the application credentials spec has merged - huge thanks to
> Monty and the Keystone team for getting this done:
>
> https://review.openstack.org/#/c/450415/
> 
> 
> http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html
> 
> 
>
> However, it appears that there was a disconnect in how two groups
> of folks were reading the spec that only became apparent towards
> the end of the process. Specifically, at this exact moment:
>
> 
> http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59
> 
> 
>
> To summarise, Keystone folks are uncomfortable with the idea of
> application credentials that share the lifecycle of the project
> (rather than the user that created them), because a consumer could
> surreptitiously create an application credential and continue to
> use that to access the OpenStack APIs even after their User
> account is deleted. The agreed solution was to delete the
> application credentials when the User that created them is
> deleted, thus tying the lifecycle to that of the User.
>
> This means that teams using this feature will need to audit all of
> their applications for credential usage and rotate any credentials
> created by a soon-to-be-former team member *before* removing said
> team member's User account, or risk breakage. Basically we're
> relying on users to do the Right Thing (bad), but when they don't
> we're defaulting to breaking [some of] their apps over leaving
> them insecure (all things being equal, good).
>
> Unfortunately, if we do regard this as a serious problem, I don't
> think this solution is sufficient. Assuming that application
> credentials are stored on VMs in the project for use by the
> applications running on them, then anyone with access to those
> servers can obtain the credentials and continue to use them even
> if their own account is deleted. The solution to this is to rotate
> *all* application keys when a user is deleted. So really we're
> relying on users to do the Right Thing (bad), but when they don't
> we're defaulting to breaking [some of] their apps *and*
> [potentially] leaving them insecure (worst possible combination).
>
> (We're also being inconsistent, because according to the spec if
> you revoke a role from a User then any application credentials
> they've created that rely on that role continue to work. It's only
> if you delete the User that they're revoked.)
>
>
> As far as I can see, there are only two solutions to the
> fundamental problem:
>
> 1) Fine-grained user-defined access control. We can minimise the
> set of things that the application credentials are authorised to
> do. That's out of scope for this spec, but something we're already
> planning as a future enhancement.
> 2) Automated regular rotation of credentials. We can make sure
> that whatever a departing team member does manage to hang onto
> quickly becomes useless.
>
> By way of comparison, AWS does both. There's fine-grained defined
> access control in the form of IAM Roles, and these Roles can be
> associated with EC2 servers. The servers have an account with
> rotating keys provided through the metadata server. I can't find
> the exact period of rotation documented, but it's on the order of
> magnitude of 1 hour.
>
> There's plenty not to like about this design. Specifically, it's
> 2017 not 2007 and the idea that there's no point offering to
> segment permissions at a finer grained level than that of a VM no
> longer holds water IMHO, thanks to SELinux and containers. It'd be
> nice to be able to provide multiple sets of credentials to
> different services running on a VM, and it's probably essential to
> our survival that we find a way to provide individual credentials
> to containers. Nevertheless, what they have does solve the problem.
>
> Note that there's pretty much no sane way for the user to automate
> credential rotation themselves, because it's turtles all the way
> down. e.g. it's easy in principle to set up a Heat template with a
> Mistral workflow that will rotate the credentials for you, but
> they'll do so using trusts that are, in turn, tied back to the
> consumer who created the stack. (It suddenly occurs to me that
> this is a problem that all services using tru

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Colleen Murphy]

On Tue, Jul 18, 2017 at 1:39 AM, Zane Bitter  wrote:

> So the application credentials spec has merged - huge thanks to Monty and
> the Keystone team for getting this done:
>
> https://review.openstack.org/#/c/450415/
> http://specs.openstack.org/openstack/keystone-specs/specs/
> keystone/pike/application-credentials.html
>
> However, it appears that there was a disconnect in how two groups of folks
> were reading the spec that only became apparent towards the end of the
> process. Specifically, at this exact moment:
>
> http://eavesdrop.openstack.org/irclogs/%23openstack-keystone
> /%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59
>
> To summarise, Keystone folks are uncomfortable with the idea of
> application credentials that share the lifecycle of the project (rather
> than the user that created them), because a consumer could surreptitiously
> create an application credential and continue to use that to access the
> OpenStack APIs even after their User account is deleted. The agreed
> solution was to delete the application credentials when the User that
> created them is deleted, thus tying the lifecycle to that of the User.
>
> This means that teams using this feature will need to audit all of their
> applications for credential usage and rotate any credentials created by a
> soon-to-be-former team member *before* removing said team member's User
> account, or risk breakage. Basically we're relying on users to do the Right
> Thing (bad), but when they don't we're defaulting to breaking [some of]
> their apps over leaving them insecure (all things being equal, good).
>
> Unfortunately, if we do regard this as a serious problem, I don't think
> this solution is sufficient. Assuming that application credentials are
> stored on VMs in the project for use by the applications running on them,
> then anyone with access to those servers can obtain the credentials and
> continue to use them even if their own account is deleted. The solution to
> this is to rotate *all* application keys when a user is deleted. So really
> we're relying on users to do the Right Thing (bad), but when they don't
> we're defaulting to breaking [some of] their apps *and* [potentially]
> leaving them insecure (worst possible combination).
>
> (We're also being inconsistent, because according to the spec if you
> revoke a role from a User then any application credentials they've created
> that rely on that role continue to work. It's only if you delete the User
> that they're revoked.)
>
>
> As far as I can see, there are only two solutions to the fundamental
> problem:
>
> 1) Fine-grained user-defined access control. We can minimise the set of
> things that the application credentials are authorised to do. That's out of
> scope for this spec, but something we're already planning as a future
> enhancement.
> 2) Automated regular rotation of credentials. We can make sure that
> whatever a departing team member does manage to hang onto quickly becomes
> useless.
>
> By way of comparison, AWS does both. There's fine-grained defined access
> control in the form of IAM Roles, and these Roles can be associated with
> EC2 servers. The servers have an account with rotating keys provided
> through the metadata server. I can't find the exact period of rotation
> documented, but it's on the order of magnitude of 1 hour.
>
> There's plenty not to like about this design. Specifically, it's 2017 not
> 2007 and the idea that there's no point offering to segment permissions at
> a finer grained level than that of a VM no longer holds water IMHO, thanks
> to SELinux and containers. It'd be nice to be able to provide multiple sets
> of credentials to different services running on a VM, and it's probably
> essential to our survival that we find a way to provide individual
> credentials to containers. Nevertheless, what they have does solve the
> problem.
>
> Note that there's pretty much no sane way for the user to automate
> credential rotation themselves, because it's turtles all the way down. e.g.
> it's easy in principle to set up a Heat template with a Mistral workflow
> that will rotate the credentials for you, but they'll do so using trusts
> that are, in turn, tied back to the consumer who created the stack. (It
> suddenly occurs to me that this is a problem that all services using trusts
> are going to need to solve.) Somewhere it all has to be tied back to
> something that survives the entire lifecycle of the project.
>
> Would Keystone folks be happy to allow persistent credentials once we have
> a way to hand out only the minimum required privileges?
>

I agree that in the haste of getting this approved before the spec freeze
deadline we took this in the wrong direction. I think that this spec was
fine before the addition of "Will be deleted when the associated User is
deleted" constraint.

As I understand it, the worry coming from the team is that a user who
sneakily copies the secret keys to an offsite location before their 

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Lance Bragstad]

On Mon, Jul 17, 2017 at 6:39 PM, Zane Bitter  wrote:

> So the application credentials spec has merged - huge thanks to Monty and
> the Keystone team for getting this done:
>
> https://review.openstack.org/#/c/450415/
> http://specs.openstack.org/openstack/keystone-specs/specs/
> keystone/pike/application-credentials.html
>
> However, it appears that there was a disconnect in how two groups of folks
> were reading the spec that only became apparent towards the end of the
> process. Specifically, at this exact moment:
>
> http://eavesdrop.openstack.org/irclogs/%23openstack-keystone
> /%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59
>
> To summarise, Keystone folks are uncomfortable with the idea of
> application credentials that share the lifecycle of the project (rather
> than the user that created them), because a consumer could surreptitiously
> create an application credential and continue to use that to access the
> OpenStack APIs even after their User account is deleted. The agreed
> solution was to delete the application credentials when the User that
> created them is deleted, thus tying the lifecycle to that of the User.
>
> This means that teams using this feature will need to audit all of their
> applications for credential usage and rotate any credentials created by a
> soon-to-be-former team member *before* removing said team member's User
> account, or risk breakage. Basically we're relying on users to do the Right
> Thing (bad), but when they don't we're defaulting to breaking [some of]
> their apps over leaving them insecure (all things being equal, good).
>
> Unfortunately, if we do regard this as a serious problem, I don't think
> this solution is sufficient. Assuming that application credentials are
> stored on VMs in the project for use by the applications running on them,
> then anyone with access to those servers can obtain the credentials and
> continue to use them even if their own account is deleted. The solution to
> this is to rotate *all* application keys when a user is deleted. So really
> we're relying on users to do the Right Thing (bad), but when they don't
> we're defaulting to breaking [some of] their apps *and* [potentially]
> leaving them insecure (worst possible combination).
>
> (We're also being inconsistent, because according to the spec if you
> revoke a role from a User then any application credentials they've created
> that rely on that role continue to work. It's only if you delete the User
> that they're revoked.)
>
>
> As far as I can see, there are only two solutions to the fundamental
> problem:
>
> 1) Fine-grained user-defined access control. We can minimise the set of
> things that the application credentials are authorised to do. That's out of
> scope for this spec, but something we're already planning as a future
> enhancement.
> 2) Automated regular rotation of credentials. We can make sure that
> whatever a departing team member does manage to hang onto quickly becomes
> useless.
>
> By way of comparison, AWS does both. There's fine-grained defined access
> control in the form of IAM Roles, and these Roles can be associated with
> EC2 servers. The servers have an account with rotating keys provided
> through the metadata server. I can't find the exact period of rotation
> documented, but it's on the order of magnitude of 1 hour.
>
> There's plenty not to like about this design. Specifically, it's 2017 not
> 2007 and the idea that there's no point offering to segment permissions at
> a finer grained level than that of a VM no longer holds water IMHO, thanks
> to SELinux and containers. It'd be nice to be able to provide multiple sets
> of credentials to different services running on a VM, and it's probably
> essential to our survival that we find a way to provide individual
> credentials to containers. Nevertheless, what they have does solve the
> problem.
>
> Note that there's pretty much no sane way for the user to automate
> credential rotation themselves, because it's turtles all the way down. e.g.
> it's easy in principle to set up a Heat template with a Mistral workflow
> that will rotate the credentials for you, but they'll do so using trusts
> that are, in turn, tied back to the consumer who created the stack. (It
> suddenly occurs to me that this is a problem that all services using trusts
> are going to need to solve.) Somewhere it all has to be tied back to
> something that survives the entire lifecycle of the project.
>
> Would Keystone folks be happy to allow persistent credentials once we have
> a way to hand out only the minimum required privileges?
>

If I'm understanding correctly, this would make application credentials
dependent on several cycles of policy work. Right?


>
> If not I think we're back to https://review.openstack.org/#/c/93/
>
> cheers,
> Zane.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: open

[openstack-dev] [keystone][nova] Persistent application credentials

[Zane Bitter]

So the application credentials spec has merged - huge thanks to Monty 
and the Keystone team for getting this done:


https://review.openstack.org/#/c/450415/
http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html

However, it appears that there was a disconnect in how two groups of 
folks were reading the spec that only became apparent towards the end of 
the process. Specifically, at this exact moment:


http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59

To summarise, Keystone folks are uncomfortable with the idea of 
application credentials that share the lifecycle of the project (rather 
than the user that created them), because a consumer could 
surreptitiously create an application credential and continue to use 
that to access the OpenStack APIs even after their User account is 
deleted. The agreed solution was to delete the application credentials 
when the User that created them is deleted, thus tying the lifecycle to 
that of the User.


This means that teams using this feature will need to audit all of their 
applications for credential usage and rotate any credentials created by 
a soon-to-be-former team member *before* removing said team member's 
User account, or risk breakage. Basically we're relying on users to do 
the Right Thing (bad), but when they don't we're defaulting to breaking 
[some of] their apps over leaving them insecure (all things being equal, 
good).


Unfortunately, if we do regard this as a serious problem, I don't think 
this solution is sufficient. Assuming that application credentials are 
stored on VMs in the project for use by the applications running on 
them, then anyone with access to those servers can obtain the 
credentials and continue to use them even if their own account is 
deleted. The solution to this is to rotate *all* application keys when a 
user is deleted. So really we're relying on users to do the Right Thing 
(bad), but when they don't we're defaulting to breaking [some of] their 
apps *and* [potentially] leaving them insecure (worst possible combination).


(We're also being inconsistent, because according to the spec if you 
revoke a role from a User then any application credentials they've 
created that rely on that role continue to work. It's only if you delete 
the User that they're revoked.)



As far as I can see, there are only two solutions to the fundamental 
problem:


1) Fine-grained user-defined access control. We can minimise the set of 
things that the application credentials are authorised to do. That's out 
of scope for this spec, but something we're already planning as a future 
enhancement.
2) Automated regular rotation of credentials. We can make sure that 
whatever a departing team member does manage to hang onto quickly 
becomes useless.


By way of comparison, AWS does both. There's fine-grained defined access 
control in the form of IAM Roles, and these Roles can be associated with 
EC2 servers. The servers have an account with rotating keys provided 
through the metadata server. I can't find the exact period of rotation 
documented, but it's on the order of magnitude of 1 hour.


There's plenty not to like about this design. Specifically, it's 2017 
not 2007 and the idea that there's no point offering to segment 
permissions at a finer grained level than that of a VM no longer holds 
water IMHO, thanks to SELinux and containers. It'd be nice to be able to 
provide multiple sets of credentials to different services running on a 
VM, and it's probably essential to our survival that we find a way to 
provide individual credentials to containers. Nevertheless, what they 
have does solve the problem.


Note that there's pretty much no sane way for the user to automate 
credential rotation themselves, because it's turtles all the way down. 
e.g. it's easy in principle to set up a Heat template with a Mistral 
workflow that will rotate the credentials for you, but they'll do so 
using trusts that are, in turn, tied back to the consumer who created 
the stack. (It suddenly occurs to me that this is a problem that all 
services using trusts are going to need to solve.) Somewhere it all has 
to be tied back to something that survives the entire lifecycle of the 
project.


Would Keystone folks be happy to allow persistent credentials once we 
have a way to hand out only the minimum required privileges?


If not I think we're back to https://review.openstack.org/#/c/93/

cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev