Re: [openstack-dev] [Heat]Heat template parameters encryption
I tend to agree with you Keith, securing Heat is Heat's problem. Securing Nova is nova's problem. And I too would expect that those with admin access to Heat, would not have admin access to Nova. That is why we split these things up with API's. I still prefer that users encrypt secrets on the client side, and store said secrets in Barbican, passing only a temporary handle into templates for consumption. But until we have that, just encrypting hidden parameters would be simple to do and I wouldn't even mind it being on by default in devstack because only a small percentage of parameters are "hidden". My initial reluctance to the plan was in encrypting everything, as that makes verifying things a lot harder. But just encrypting the passwords.. I think that's a decent plan. A couple of ideas: * Provide a utility to change the key (must update the entire database). * Allow multiple decryption keys (to enable tool above to work slowly). Excerpts from Keith Bray's message of 2014-06-11 22:29:13 -0700: > > On 6/11/14 2:43 AM, "Steven Hardy" wrote: > > >>IMO, when a template author marks a parameter as hidden/secret, it seems > >>incorrect to store that information in plain text. > > > >Well I'd still question why we're doing this, as my previous questions > >have > >not been answered: > >- AFAIK nova user-data is not encrypted, so surely you're just shifting > >the > > attack vector from one DB to another in nearly all cases? > > Having one system (e.g. Nova) not as secure as it could be isn't a reason > to not secure another system as best we can. For every attack vector you > close, you have another one to chase. I'm concerned that the merit of the > feature is being debated, so let me see if I can address that: > > We want to use Heat to launch customer facing stacks. In a UI, we would > prompt customers for Template inputs, including for example: Desired > Wordpress Admin Password, Desired MySQL password, etc. The UI then makes > an API call to Heat to orchestrate instantiation of the stack. With Heat > as it is today, these customer specified credentials (as template > parameters) would be stored in Heat's database in plain text. As a Heat > Service Administrator, I do not need nor do I want the customer's > Wordpress application password to be accessible to me. The application > belongs to the customer, not to the infrastructure provider. Sure, I > could blow the customer's entire instance away as the service provider. > But, if I get fired or leave the company, I could no longer can blow away > their instance... If I leave the company, however, I could have taken a > copy of the Heat DB with me, or had looked that info up in the Heat DB > before my exit, and I could then externally attack the customer's > Wordpress instance. It makes no sense for us to store user specified > creds unencrypted unless we are administering the customer's Wordpress > instance for them, which we are not. We are administering the > infrastructure only. I realize the encryption key could also be stolen, > but in a "production" system the encryption key access gets locked down to > a VERY small set of folks and not all the people that administer Heat > (that's part of good security practices and makes auditing of a leaked > encryption key much easier). > > >- Is there any known way for heat to "leak sensitive user data", other > >than > > a cloud operator with admin access to the DB stealing it? Surely cloud > > operators can trivially access all your resources anyway, including > > instances and the nova DB/API so they have this data anyway. > > Encrypting the data in the DB also helps in case if a leak of arbitrary DB > data does surface in Heat. We are not aware of any issues with Heat today > that could leak that data... But, we never know what vulnerabilities will > be introduced or discovered in the future. > > > At Rackspace, individual cloud operators can not trivially access all > customer cloud resources. When operating a large cloud at scale, service > administrator's operations and capabilities are limited to the systems > they work on. While I could impersonate a user via Heat and do lot's of > bad things across many of their resources, each of the other systems > (Nova, Databases, Auth, etc.) audit the who is doing what on behave of > what customer, so I can't do something malicious to a customer's Nova > instance without the Auth System Administrators ensuring that HR knows I > would be the person to blame. Similarly, a Nova system administrator > can't delete a customer's Heat stack without our Heat administrators > knowing who is to blame. We have checks and balances across our systems > and purposefully segment our possible attack vectors. > > Leaving sensitive customer data unencrypted at rest provides many more > options for that data to get in the wrong hands or be taken outside the > company. It is quick and easy to do a MySQL dump if the DB linux system > is compromised,
Re: [openstack-dev] [Heat]Heat template parameters encryption
On 6/11/14 2:43 AM, "Steven Hardy" wrote: >>IMO, when a template author marks a parameter as hidden/secret, it seems >>incorrect to store that information in plain text. > >Well I'd still question why we're doing this, as my previous questions >have >not been answered: >- AFAIK nova user-data is not encrypted, so surely you're just shifting >the > attack vector from one DB to another in nearly all cases? Having one system (e.g. Nova) not as secure as it could be isn't a reason to not secure another system as best we can. For every attack vector you close, you have another one to chase. I'm concerned that the merit of the feature is being debated, so let me see if I can address that: We want to use Heat to launch customer facing stacks. In a UI, we would prompt customers for Template inputs, including for example: Desired Wordpress Admin Password, Desired MySQL password, etc. The UI then makes an API call to Heat to orchestrate instantiation of the stack. With Heat as it is today, these customer specified credentials (as template parameters) would be stored in Heat's database in plain text. As a Heat Service Administrator, I do not need nor do I want the customer's Wordpress application password to be accessible to me. The application belongs to the customer, not to the infrastructure provider. Sure, I could blow the customer's entire instance away as the service provider. But, if I get fired or leave the company, I could no longer can blow away their instance... If I leave the company, however, I could have taken a copy of the Heat DB with me, or had looked that info up in the Heat DB before my exit, and I could then externally attack the customer's Wordpress instance. It makes no sense for us to store user specified creds unencrypted unless we are administering the customer's Wordpress instance for them, which we are not. We are administering the infrastructure only. I realize the encryption key could also be stolen, but in a "production" system the encryption key access gets locked down to a VERY small set of folks and not all the people that administer Heat (that's part of good security practices and makes auditing of a leaked encryption key much easier). >- Is there any known way for heat to "leak sensitive user data", other >than > a cloud operator with admin access to the DB stealing it? Surely cloud > operators can trivially access all your resources anyway, including > instances and the nova DB/API so they have this data anyway. Encrypting the data in the DB also helps in case if a leak of arbitrary DB data does surface in Heat. We are not aware of any issues with Heat today that could leak that data... But, we never know what vulnerabilities will be introduced or discovered in the future. At Rackspace, individual cloud operators can not trivially access all customer cloud resources. When operating a large cloud at scale, service administrator's operations and capabilities are limited to the systems they work on. While I could impersonate a user via Heat and do lot's of bad things across many of their resources, each of the other systems (Nova, Databases, Auth, etc.) audit the who is doing what on behave of what customer, so I can't do something malicious to a customer's Nova instance without the Auth System Administrators ensuring that HR knows I would be the person to blame. Similarly, a Nova system administrator can't delete a customer's Heat stack without our Heat administrators knowing who is to blame. We have checks and balances across our systems and purposefully segment our possible attack vectors. Leaving sensitive customer data unencrypted at rest provides many more options for that data to get in the wrong hands or be taken outside the company. It is quick and easy to do a MySQL dump if the DB linux system is compromised, which has nothing to do with Heat having a vulnerability. Our ask is to provide an optional way for the service operator to allow template authors to choose what data is sensitive, and if the data is marked sensitive, it gets encrypted by the Heat system as opposed to storing in plain txt. I hope this helps. Vijendar, feel free to take some of this write-up and include it in a gerit review of the feature blueprint. Thanks, -Keith ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat]Heat template parameters encryption
On Tue, Jun 10, 2014 at 05:24:36PM +, Vijendar Komalla wrote: > Hi Devs/All, > Does any one have comments/objections for following interim solution? > 1. Add a config option to enable/disable parameter encryption and set > default value to disable > 2. Encrypt parameters that were marked as hidden or encrypt all parameters Provided it's configurable, and probably defaulted to disabled (or at least disabled for devstack), that sounds OK to me. Obviously you'll have to figure out a way to do this in a backwards compatible way, so we handle existing non-encrypted DB content on upgrade. > IMO, when a template author marks a parameter as hidden/secret, it seems > incorrect to store that information in plain text. Well I'd still question why we're doing this, as my previous questions have not been answered: - AFAIK nova user-data is not encrypted, so surely you're just shifting the attack vector from one DB to another in nearly all cases? - Is there any known way for heat to "leak sensitive user data", other than a cloud operator with admin access to the DB stealing it? Surely cloud operators can trivially access all your resources anyway, including instances and the nova DB/API so they have this data anyway. IMO a bit more context explaining why would be helpful here. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat]Heat template parameters encryption
Hi Devs/All,
Does any one have comments/objections for following interim solution?
1. Add a config option to enable/disable parameter encryption and set
default value to disable
2. Encrypt parameters that were marked as hidden or encrypt all parameters
IMO, when a template author marks a parameter as hidden/secret, it seems
incorrect to store that information in plain text.
Thanks,
Vijendar
On 6/5/14 11:54 AM, "Vijendar Komalla"
wrote:
>I am not sure when Barbican would be stable/ready. As an interim solution,
>what do you guys think about having a config option to enable/disable
>parameter encryption (along with my current implementation)?
>
>
>
>On 6/5/14 4:23 AM, "Steven Hardy" wrote:
>
>>On Thu, Jun 05, 2014 at 12:17:07AM +, Randall Burt wrote:
>>> On Jun 4, 2014, at 7:05 PM, Clint Byrum
>>> wrote:
>>>
>>> > Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
>>> >> On 04/06/14 15:58, Vijendar Komalla wrote:
>>> >>> Hi Devs,
>>> >>> I have submitted an WIP review
>>>(https://review.openstack.org/#/c/97900/)
>>> >>> for Heat parameters encryption blueprint
>>> >>>
>>>https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
>>> >>> This quick and dirty implementation encrypts all the parameters on
>>>on
>>> >>> Stack 'store' and decrypts on on Stack 'load'.
>>> >>> Following are couple of improvements I am thinking about;
>>> >>> 1. Instead of encrypting individual parameters, on Stack 'store'
>>>encrypt
>>> >>> all the parameters together as a dictionary [something like
>>> >>> crypt.encrypt(json.dumps(param_dictionary))]
>>> >>
>>> >> Yeah, definitely don't encrypt them individually.
>>> >>
>>> >>> 2. Just encrypt parameters that were marked as 'hidden', instead of
>>> >>> encrypting all parameters
>>> >>>
>>> >>> I would like to hear your feedback/suggestions.
>>> >>
>>> >> Just as a heads-up, we will soon need to store the properties of
>>> >> resources too, at which point parameters become the least of our
>>> >> problems. (In fact, in theory we wouldn't even need to store
>>> >> parameters... and probably by the time convergence is completely
>>> >> implemented, we won't.) Which is to say that there's almost
>>>certainly no
>>> >> point in discriminating between hidden and non-hidden parameters.
>>> >>
>>> >> I'll refrain from commenting on whether the extra security this
>>>affords
>>> >> is worth the giant pain it causes in debugging, except to say that
>>>IMO
>>> >> there should be a config option to disable the feature (and if it's
>>> >> enabled by default, it should probably be disabled by default in
>>>e.g.
>>> >> devstack).
>>> >
>>> > Storing secrets seems like a job for Barbican. That handles the giant
>>> > pain problem because in devstack you can just tell Barbican to have
>>>an
>>> > open read policy.
>>> >
>>> > I'd rather see good hooks for Barbican than blanket encryption. I've
>>> > worked with a few things like this and they are despised and worked
>>> > around universally because of the reason Zane has expressed concern
>>>about:
>>> > debugging gets ridiculous.
>>> >
>>> > How about this:
>>> >
>>> > parameters:
>>> > secrets:
>>> >type: sensitive
>>> > resources:
>>> > sensitive_deployment:
>>> >type: OS::Heat::StructuredDeployment
>>> >properties:
>>> > config: weverConfig
>>> > server: myserver
>>> > input_values:
>>> >secret_handle: { get_param: secrets }
>>> >
>>> > The sensitive type would, on the client side, store the value in
>>>Barbican,
>>> > never in Heat. Instead it would just pass in a handle which the user
>>> > can then build policy around. Obviously this implies the user would
>>>set
>>> > up Barbican's in-instance tools to access the secrets value. But the
>>> > idea is, let Heat worry about being high performing and
>>>introspectable,
>>> > and then let Barbican worry about sensitive things.
>>>
>>> While certainly ideal, it doesn't solve the current problem since we
>>>can't yet guarantee Barbican will even be available in a given release
>>>of OpenStack. In the meantime, Heat continues to store sensitive user
>>>information unencrypted in its database. Once Barbican is integrated,
>>>I'd be all for changing this implementation, but until then, we do need
>>>an interim solution. Sure, debugging is a pain and as developers we can
>>>certainly grumble, but leaking sensitive user information because we
>>>were too fussed to protect data at rest seems worse IMO. Additionally,
>>>the solution as described sounds like we're imposing a pretty awkward
>>>process on a user to save ourselves from having to decrypt some data in
>>>the cases where we can't access the stack information directly from the
>>>API or via debugging running Heat code (where the data isn't encrypted
>>>anymore).
>>
>>Under what circumstances are we "leaking sensitive user information"?
>>
>>Are you just trying to mitigate a potential attack vector, in the event
>>of
>>a bug which leaks data from the D
Re: [openstack-dev] [Heat]Heat template parameters encryption
I am not sure when Barbican would be stable/ready. As an interim solution,
what do you guys think about having a config option to enable/disable
parameter encryption (along with my current implementation)?
On 6/5/14 4:23 AM, "Steven Hardy" wrote:
>On Thu, Jun 05, 2014 at 12:17:07AM +, Randall Burt wrote:
>> On Jun 4, 2014, at 7:05 PM, Clint Byrum
>> wrote:
>>
>> > Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
>> >> On 04/06/14 15:58, Vijendar Komalla wrote:
>> >>> Hi Devs,
>> >>> I have submitted an WIP review
>>(https://review.openstack.org/#/c/97900/)
>> >>> for Heat parameters encryption blueprint
>> >>>
>>https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
>> >>> This quick and dirty implementation encrypts all the parameters on
>>on
>> >>> Stack 'store' and decrypts on on Stack 'load'.
>> >>> Following are couple of improvements I am thinking about;
>> >>> 1. Instead of encrypting individual parameters, on Stack 'store'
>>encrypt
>> >>> all the parameters together as a dictionary [something like
>> >>> crypt.encrypt(json.dumps(param_dictionary))]
>> >>
>> >> Yeah, definitely don't encrypt them individually.
>> >>
>> >>> 2. Just encrypt parameters that were marked as 'hidden', instead of
>> >>> encrypting all parameters
>> >>>
>> >>> I would like to hear your feedback/suggestions.
>> >>
>> >> Just as a heads-up, we will soon need to store the properties of
>> >> resources too, at which point parameters become the least of our
>> >> problems. (In fact, in theory we wouldn't even need to store
>> >> parameters... and probably by the time convergence is completely
>> >> implemented, we won't.) Which is to say that there's almost
>>certainly no
>> >> point in discriminating between hidden and non-hidden parameters.
>> >>
>> >> I'll refrain from commenting on whether the extra security this
>>affords
>> >> is worth the giant pain it causes in debugging, except to say that
>>IMO
>> >> there should be a config option to disable the feature (and if it's
>> >> enabled by default, it should probably be disabled by default in
>>e.g.
>> >> devstack).
>> >
>> > Storing secrets seems like a job for Barbican. That handles the giant
>> > pain problem because in devstack you can just tell Barbican to have an
>> > open read policy.
>> >
>> > I'd rather see good hooks for Barbican than blanket encryption. I've
>> > worked with a few things like this and they are despised and worked
>> > around universally because of the reason Zane has expressed concern
>>about:
>> > debugging gets ridiculous.
>> >
>> > How about this:
>> >
>> > parameters:
>> > secrets:
>> >type: sensitive
>> > resources:
>> > sensitive_deployment:
>> >type: OS::Heat::StructuredDeployment
>> >properties:
>> > config: weverConfig
>> > server: myserver
>> > input_values:
>> >secret_handle: { get_param: secrets }
>> >
>> > The sensitive type would, on the client side, store the value in
>>Barbican,
>> > never in Heat. Instead it would just pass in a handle which the user
>> > can then build policy around. Obviously this implies the user would
>>set
>> > up Barbican's in-instance tools to access the secrets value. But the
>> > idea is, let Heat worry about being high performing and
>>introspectable,
>> > and then let Barbican worry about sensitive things.
>>
>> While certainly ideal, it doesn't solve the current problem since we
>>can't yet guarantee Barbican will even be available in a given release
>>of OpenStack. In the meantime, Heat continues to store sensitive user
>>information unencrypted in its database. Once Barbican is integrated,
>>I'd be all for changing this implementation, but until then, we do need
>>an interim solution. Sure, debugging is a pain and as developers we can
>>certainly grumble, but leaking sensitive user information because we
>>were too fussed to protect data at rest seems worse IMO. Additionally,
>>the solution as described sounds like we're imposing a pretty awkward
>>process on a user to save ourselves from having to decrypt some data in
>>the cases where we can't access the stack information directly from the
>>API or via debugging running Heat code (where the data isn't encrypted
>>anymore).
>
>Under what circumstances are we "leaking sensitive user information"?
>
>Are you just trying to mitigate a potential attack vector, in the event of
>a bug which leaks data from the DB? If so, is the user-data encrypted in
>the nova DB?
>
>It seems to me that this will only be a worthwhile exercise if the
>sensitive stuff is encrypted everywhere, and many/most use-cases I can
>think of which require sensitive data involve that data ending up in nova
>user|meta-data?
>
>Steve
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
O
Re: [openstack-dev] [Heat]Heat template parameters encryption
Excerpts from Steven Hardy's message of 2014-06-05 02:23:40 -0700:
> On Thu, Jun 05, 2014 at 12:17:07AM +, Randall Burt wrote:
> > On Jun 4, 2014, at 7:05 PM, Clint Byrum
> > wrote:
> >
> > > Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
> > >> On 04/06/14 15:58, Vijendar Komalla wrote:
> > >>> Hi Devs,
> > >>> I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
> > >>> for Heat parameters encryption blueprint
> > >>> https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
> > >>> This quick and dirty implementation encrypts all the parameters on on
> > >>> Stack 'store' and decrypts on on Stack 'load'.
> > >>> Following are couple of improvements I am thinking about;
> > >>> 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
> > >>> all the parameters together as a dictionary [something like
> > >>> crypt.encrypt(json.dumps(param_dictionary))]
> > >>
> > >> Yeah, definitely don't encrypt them individually.
> > >>
> > >>> 2. Just encrypt parameters that were marked as 'hidden', instead of
> > >>> encrypting all parameters
> > >>>
> > >>> I would like to hear your feedback/suggestions.
> > >>
> > >> Just as a heads-up, we will soon need to store the properties of
> > >> resources too, at which point parameters become the least of our
> > >> problems. (In fact, in theory we wouldn't even need to store
> > >> parameters... and probably by the time convergence is completely
> > >> implemented, we won't.) Which is to say that there's almost certainly no
> > >> point in discriminating between hidden and non-hidden parameters.
> > >>
> > >> I'll refrain from commenting on whether the extra security this affords
> > >> is worth the giant pain it causes in debugging, except to say that IMO
> > >> there should be a config option to disable the feature (and if it's
> > >> enabled by default, it should probably be disabled by default in e.g.
> > >> devstack).
> > >
> > > Storing secrets seems like a job for Barbican. That handles the giant
> > > pain problem because in devstack you can just tell Barbican to have an
> > > open read policy.
> > >
> > > I'd rather see good hooks for Barbican than blanket encryption. I've
> > > worked with a few things like this and they are despised and worked
> > > around universally because of the reason Zane has expressed concern about:
> > > debugging gets ridiculous.
> > >
> > > How about this:
> > >
> > > parameters:
> > > secrets:
> > >type: sensitive
> > > resources:
> > > sensitive_deployment:
> > >type: OS::Heat::StructuredDeployment
> > >properties:
> > > config: weverConfig
> > > server: myserver
> > > input_values:
> > >secret_handle: { get_param: secrets }
> > >
> > > The sensitive type would, on the client side, store the value in Barbican,
> > > never in Heat. Instead it would just pass in a handle which the user
> > > can then build policy around. Obviously this implies the user would set
> > > up Barbican's in-instance tools to access the secrets value. But the
> > > idea is, let Heat worry about being high performing and introspectable,
> > > and then let Barbican worry about sensitive things.
> >
> > While certainly ideal, it doesn't solve the current problem since we can't
> > yet guarantee Barbican will even be available in a given release of
> > OpenStack. In the meantime, Heat continues to store sensitive user
> > information unencrypted in its database. Once Barbican is integrated, I'd
> > be all for changing this implementation, but until then, we do need an
> > interim solution. Sure, debugging is a pain and as developers we can
> > certainly grumble, but leaking sensitive user information because we were
> > too fussed to protect data at rest seems worse IMO. Additionally, the
> > solution as described sounds like we're imposing a pretty awkward process
> > on a user to save ourselves from having to decrypt some data in the cases
> > where we can't access the stack information directly from the API or via
> > debugging running Heat code (where the data isn't encrypted anymore).
>
> Under what circumstances are we "leaking sensitive user information"?
>
> Are you just trying to mitigate a potential attack vector, in the event of
> a bug which leaks data from the DB? If so, is the user-data encrypted in
> the nova DB?
>
> It seems to me that this will only be a worthwhile exercise if the
> sensitive stuff is encrypted everywhere, and many/most use-cases I can
> think of which require sensitive data involve that data ending up in nova
> user|meta-data?
I tend to agree Steve. The strategy to move things into a system with
strong policy controls like Barbican will mitigate these risks, as even
compromise of the given secret access information may not yield access
to the actual secrets. Basically, let's help facilitate end-to-end
encryption and access control, not just mitigate one attack vector
because the end
Re: [openstack-dev] [Heat]Heat template parameters encryption
On Thu, Jun 05, 2014 at 12:17:07AM +, Randall Burt wrote:
> On Jun 4, 2014, at 7:05 PM, Clint Byrum
> wrote:
>
> > Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
> >> On 04/06/14 15:58, Vijendar Komalla wrote:
> >>> Hi Devs,
> >>> I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
> >>> for Heat parameters encryption blueprint
> >>> https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
> >>> This quick and dirty implementation encrypts all the parameters on on
> >>> Stack 'store' and decrypts on on Stack 'load'.
> >>> Following are couple of improvements I am thinking about;
> >>> 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
> >>> all the parameters together as a dictionary [something like
> >>> crypt.encrypt(json.dumps(param_dictionary))]
> >>
> >> Yeah, definitely don't encrypt them individually.
> >>
> >>> 2. Just encrypt parameters that were marked as 'hidden', instead of
> >>> encrypting all parameters
> >>>
> >>> I would like to hear your feedback/suggestions.
> >>
> >> Just as a heads-up, we will soon need to store the properties of
> >> resources too, at which point parameters become the least of our
> >> problems. (In fact, in theory we wouldn't even need to store
> >> parameters... and probably by the time convergence is completely
> >> implemented, we won't.) Which is to say that there's almost certainly no
> >> point in discriminating between hidden and non-hidden parameters.
> >>
> >> I'll refrain from commenting on whether the extra security this affords
> >> is worth the giant pain it causes in debugging, except to say that IMO
> >> there should be a config option to disable the feature (and if it's
> >> enabled by default, it should probably be disabled by default in e.g.
> >> devstack).
> >
> > Storing secrets seems like a job for Barbican. That handles the giant
> > pain problem because in devstack you can just tell Barbican to have an
> > open read policy.
> >
> > I'd rather see good hooks for Barbican than blanket encryption. I've
> > worked with a few things like this and they are despised and worked
> > around universally because of the reason Zane has expressed concern about:
> > debugging gets ridiculous.
> >
> > How about this:
> >
> > parameters:
> > secrets:
> >type: sensitive
> > resources:
> > sensitive_deployment:
> >type: OS::Heat::StructuredDeployment
> >properties:
> > config: weverConfig
> > server: myserver
> > input_values:
> >secret_handle: { get_param: secrets }
> >
> > The sensitive type would, on the client side, store the value in Barbican,
> > never in Heat. Instead it would just pass in a handle which the user
> > can then build policy around. Obviously this implies the user would set
> > up Barbican's in-instance tools to access the secrets value. But the
> > idea is, let Heat worry about being high performing and introspectable,
> > and then let Barbican worry about sensitive things.
>
> While certainly ideal, it doesn't solve the current problem since we can't
> yet guarantee Barbican will even be available in a given release of
> OpenStack. In the meantime, Heat continues to store sensitive user
> information unencrypted in its database. Once Barbican is integrated, I'd be
> all for changing this implementation, but until then, we do need an interim
> solution. Sure, debugging is a pain and as developers we can certainly
> grumble, but leaking sensitive user information because we were too fussed to
> protect data at rest seems worse IMO. Additionally, the solution as described
> sounds like we're imposing a pretty awkward process on a user to save
> ourselves from having to decrypt some data in the cases where we can't access
> the stack information directly from the API or via debugging running Heat
> code (where the data isn't encrypted anymore).
Under what circumstances are we "leaking sensitive user information"?
Are you just trying to mitigate a potential attack vector, in the event of
a bug which leaks data from the DB? If so, is the user-data encrypted in
the nova DB?
It seems to me that this will only be a worthwhile exercise if the
sensitive stuff is encrypted everywhere, and many/most use-cases I can
think of which require sensitive data involve that data ending up in nova
user|meta-data?
Steve
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat]Heat template parameters encryption
On Jun 4, 2014, at 7:30 PM, Clint Byrum
wrote:
> Excerpts from Randall Burt's message of 2014-06-04 17:17:07 -0700:
>> On Jun 4, 2014, at 7:05 PM, Clint Byrum
>> wrote:
>>
>>> Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
On 04/06/14 15:58, Vijendar Komalla wrote:
> Hi Devs,
> I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
> for Heat parameters encryption blueprint
> https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
> This quick and dirty implementation encrypts all the parameters on on
> Stack 'store' and decrypts on on Stack 'load'.
> Following are couple of improvements I am thinking about;
> 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
> all the parameters together as a dictionary [something like
> crypt.encrypt(json.dumps(param_dictionary))]
Yeah, definitely don't encrypt them individually.
> 2. Just encrypt parameters that were marked as 'hidden', instead of
> encrypting all parameters
>
> I would like to hear your feedback/suggestions.
Just as a heads-up, we will soon need to store the properties of
resources too, at which point parameters become the least of our
problems. (In fact, in theory we wouldn't even need to store
parameters... and probably by the time convergence is completely
implemented, we won't.) Which is to say that there's almost certainly no
point in discriminating between hidden and non-hidden parameters.
I'll refrain from commenting on whether the extra security this affords
is worth the giant pain it causes in debugging, except to say that IMO
there should be a config option to disable the feature (and if it's
enabled by default, it should probably be disabled by default in e.g.
devstack).
>>>
>>> Storing secrets seems like a job for Barbican. That handles the giant
>>> pain problem because in devstack you can just tell Barbican to have an
>>> open read policy.
>>>
>>> I'd rather see good hooks for Barbican than blanket encryption. I've
>>> worked with a few things like this and they are despised and worked
>>> around universally because of the reason Zane has expressed concern about:
>>> debugging gets ridiculous.
>>>
>>> How about this:
>>>
>>> parameters:
>>> secrets:
>>> type: sensitive
>>> resources:
>>> sensitive_deployment:
>>> type: OS::Heat::StructuredDeployment
>>> properties:
>>> config: weverConfig
>>> server: myserver
>>> input_values:
>>> secret_handle: { get_param: secrets }
>>>
>>> The sensitive type would, on the client side, store the value in Barbican,
>>> never in Heat. Instead it would just pass in a handle which the user
>>> can then build policy around. Obviously this implies the user would set
>>> up Barbican's in-instance tools to access the secrets value. But the
>>> idea is, let Heat worry about being high performing and introspectable,
>>> and then let Barbican worry about sensitive things.
>>
>> While certainly ideal, it doesn't solve the current problem since we can't
>> yet guarantee Barbican will even be available in a given release of
>> OpenStack. In the meantime, Heat continues to store sensitive user
>> information unencrypted in its database. Once Barbican is integrated, I'd be
>> all for changing this implementation, but until then, we do need an interim
>> solution. Sure, debugging is a pain and as developers we can certainly
>> grumble, but leaking sensitive user information because we were too fussed
>> to protect data at rest seems worse IMO. Additionally, the solution as
>> described sounds like we're imposing a pretty awkward process on a user to
>> save ourselves from having to decrypt some data in the cases where we can't
>> access the stack information directly from the API or via debugging running
>> Heat code (where the data isn't encrypted anymore).
>>
>
> I have made that exact, reasoned argument before, and then later seen
> giant swathes of code with things like
>
> if CONF.dont_encrypt:
> ...
>
> The next thing that happens is one by one the production deployments
> eventually end up with dont_encrypt because they can't debug anything
> and they've all had a multi-hour downtime event while they dealt with
> the encrypted database on several levels.
>
> I'm not coddling developers. I'm coddling operators.
>
> It's a different story if you only encrypt the sensitive "leaf" data,
> like passwords, credit card numbers, or personally identifying data. The
> operator can still have some clue what is going on if they can see the
> non-sensitive data.
>
> The design I suggested above isn't really that awkward and will
> be infinitely easier to understand for an operator than encrypting
> everything.
So, you're simply advocating a more granular approach in which we only encrypt
the values for inputs marked as hidden in the interim?
___
Re: [openstack-dev] [Heat]Heat template parameters encryption
Excerpts from Randall Burt's message of 2014-06-04 17:17:07 -0700:
> On Jun 4, 2014, at 7:05 PM, Clint Byrum
> wrote:
>
> > Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
> >> On 04/06/14 15:58, Vijendar Komalla wrote:
> >>> Hi Devs,
> >>> I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
> >>> for Heat parameters encryption blueprint
> >>> https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
> >>> This quick and dirty implementation encrypts all the parameters on on
> >>> Stack 'store' and decrypts on on Stack 'load'.
> >>> Following are couple of improvements I am thinking about;
> >>> 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
> >>> all the parameters together as a dictionary [something like
> >>> crypt.encrypt(json.dumps(param_dictionary))]
> >>
> >> Yeah, definitely don't encrypt them individually.
> >>
> >>> 2. Just encrypt parameters that were marked as 'hidden', instead of
> >>> encrypting all parameters
> >>>
> >>> I would like to hear your feedback/suggestions.
> >>
> >> Just as a heads-up, we will soon need to store the properties of
> >> resources too, at which point parameters become the least of our
> >> problems. (In fact, in theory we wouldn't even need to store
> >> parameters... and probably by the time convergence is completely
> >> implemented, we won't.) Which is to say that there's almost certainly no
> >> point in discriminating between hidden and non-hidden parameters.
> >>
> >> I'll refrain from commenting on whether the extra security this affords
> >> is worth the giant pain it causes in debugging, except to say that IMO
> >> there should be a config option to disable the feature (and if it's
> >> enabled by default, it should probably be disabled by default in e.g.
> >> devstack).
> >
> > Storing secrets seems like a job for Barbican. That handles the giant
> > pain problem because in devstack you can just tell Barbican to have an
> > open read policy.
> >
> > I'd rather see good hooks for Barbican than blanket encryption. I've
> > worked with a few things like this and they are despised and worked
> > around universally because of the reason Zane has expressed concern about:
> > debugging gets ridiculous.
> >
> > How about this:
> >
> > parameters:
> > secrets:
> >type: sensitive
> > resources:
> > sensitive_deployment:
> >type: OS::Heat::StructuredDeployment
> >properties:
> > config: weverConfig
> > server: myserver
> > input_values:
> >secret_handle: { get_param: secrets }
> >
> > The sensitive type would, on the client side, store the value in Barbican,
> > never in Heat. Instead it would just pass in a handle which the user
> > can then build policy around. Obviously this implies the user would set
> > up Barbican's in-instance tools to access the secrets value. But the
> > idea is, let Heat worry about being high performing and introspectable,
> > and then let Barbican worry about sensitive things.
>
> While certainly ideal, it doesn't solve the current problem since we can't
> yet guarantee Barbican will even be available in a given release of
> OpenStack. In the meantime, Heat continues to store sensitive user
> information unencrypted in its database. Once Barbican is integrated, I'd be
> all for changing this implementation, but until then, we do need an interim
> solution. Sure, debugging is a pain and as developers we can certainly
> grumble, but leaking sensitive user information because we were too fussed to
> protect data at rest seems worse IMO. Additionally, the solution as described
> sounds like we're imposing a pretty awkward process on a user to save
> ourselves from having to decrypt some data in the cases where we can't access
> the stack information directly from the API or via debugging running Heat
> code (where the data isn't encrypted anymore).
>
I have made that exact, reasoned argument before, and then later seen
giant swathes of code with things like
if CONF.dont_encrypt:
...
The next thing that happens is one by one the production deployments
eventually end up with dont_encrypt because they can't debug anything
and they've all had a multi-hour downtime event while they dealt with
the encrypted database on several levels.
I'm not coddling developers. I'm coddling operators.
It's a different story if you only encrypt the sensitive "leaf" data,
like passwords, credit card numbers, or personally identifying data. The
operator can still have some clue what is going on if they can see the
non-sensitive data.
The design I suggested above isn't really that awkward and will
be infinitely easier to understand for an operator than encrypting
everything.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat]Heat template parameters encryption
On Jun 4, 2014, at 7:05 PM, Clint Byrum
wrote:
> Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
>> On 04/06/14 15:58, Vijendar Komalla wrote:
>>> Hi Devs,
>>> I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
>>> for Heat parameters encryption blueprint
>>> https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
>>> This quick and dirty implementation encrypts all the parameters on on
>>> Stack 'store' and decrypts on on Stack 'load'.
>>> Following are couple of improvements I am thinking about;
>>> 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
>>> all the parameters together as a dictionary [something like
>>> crypt.encrypt(json.dumps(param_dictionary))]
>>
>> Yeah, definitely don't encrypt them individually.
>>
>>> 2. Just encrypt parameters that were marked as 'hidden', instead of
>>> encrypting all parameters
>>>
>>> I would like to hear your feedback/suggestions.
>>
>> Just as a heads-up, we will soon need to store the properties of
>> resources too, at which point parameters become the least of our
>> problems. (In fact, in theory we wouldn't even need to store
>> parameters... and probably by the time convergence is completely
>> implemented, we won't.) Which is to say that there's almost certainly no
>> point in discriminating between hidden and non-hidden parameters.
>>
>> I'll refrain from commenting on whether the extra security this affords
>> is worth the giant pain it causes in debugging, except to say that IMO
>> there should be a config option to disable the feature (and if it's
>> enabled by default, it should probably be disabled by default in e.g.
>> devstack).
>
> Storing secrets seems like a job for Barbican. That handles the giant
> pain problem because in devstack you can just tell Barbican to have an
> open read policy.
>
> I'd rather see good hooks for Barbican than blanket encryption. I've
> worked with a few things like this and they are despised and worked
> around universally because of the reason Zane has expressed concern about:
> debugging gets ridiculous.
>
> How about this:
>
> parameters:
> secrets:
>type: sensitive
> resources:
> sensitive_deployment:
>type: OS::Heat::StructuredDeployment
>properties:
> config: weverConfig
> server: myserver
> input_values:
>secret_handle: { get_param: secrets }
>
> The sensitive type would, on the client side, store the value in Barbican,
> never in Heat. Instead it would just pass in a handle which the user
> can then build policy around. Obviously this implies the user would set
> up Barbican's in-instance tools to access the secrets value. But the
> idea is, let Heat worry about being high performing and introspectable,
> and then let Barbican worry about sensitive things.
While certainly ideal, it doesn't solve the current problem since we can't yet
guarantee Barbican will even be available in a given release of OpenStack. In
the meantime, Heat continues to store sensitive user information unencrypted in
its database. Once Barbican is integrated, I'd be all for changing this
implementation, but until then, we do need an interim solution. Sure, debugging
is a pain and as developers we can certainly grumble, but leaking sensitive
user information because we were too fussed to protect data at rest seems worse
IMO. Additionally, the solution as described sounds like we're imposing a
pretty awkward process on a user to save ourselves from having to decrypt some
data in the cases where we can't access the stack information directly from the
API or via debugging running Heat code (where the data isn't encrypted anymore).
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat]Heat template parameters encryption
Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
> On 04/06/14 15:58, Vijendar Komalla wrote:
> > Hi Devs,
> > I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
> > for Heat parameters encryption blueprint
> > https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
> > This quick and dirty implementation encrypts all the parameters on on
> > Stack 'store' and decrypts on on Stack 'load'.
> > Following are couple of improvements I am thinking about;
> > 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
> > all the parameters together as a dictionary [something like
> > crypt.encrypt(json.dumps(param_dictionary))]
>
> Yeah, definitely don't encrypt them individually.
>
> > 2. Just encrypt parameters that were marked as 'hidden', instead of
> > encrypting all parameters
> >
> > I would like to hear your feedback/suggestions.
>
> Just as a heads-up, we will soon need to store the properties of
> resources too, at which point parameters become the least of our
> problems. (In fact, in theory we wouldn't even need to store
> parameters... and probably by the time convergence is completely
> implemented, we won't.) Which is to say that there's almost certainly no
> point in discriminating between hidden and non-hidden parameters.
>
> I'll refrain from commenting on whether the extra security this affords
> is worth the giant pain it causes in debugging, except to say that IMO
> there should be a config option to disable the feature (and if it's
> enabled by default, it should probably be disabled by default in e.g.
> devstack).
Storing secrets seems like a job for Barbican. That handles the giant
pain problem because in devstack you can just tell Barbican to have an
open read policy.
I'd rather see good hooks for Barbican than blanket encryption. I've
worked with a few things like this and they are despised and worked
around universally because of the reason Zane has expressed concern about:
debugging gets ridiculous.
How about this:
parameters:
secrets:
type: sensitive
resources:
sensitive_deployment:
type: OS::Heat::StructuredDeployment
properties:
config: weverConfig
server: myserver
input_values:
secret_handle: { get_param: secrets }
The sensitive type would, on the client side, store the value in Barbican,
never in Heat. Instead it would just pass in a handle which the user
can then build policy around. Obviously this implies the user would set
up Barbican's in-instance tools to access the secrets value. But the
idea is, let Heat worry about being high performing and introspectable,
and then let Barbican worry about sensitive things.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat]Heat template parameters encryption
On 04/06/14 15:58, Vijendar Komalla wrote: Hi Devs, I have submitted an WIP review (https://review.openstack.org/#/c/97900/) for Heat parameters encryption blueprint https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters This quick and dirty implementation encrypts all the parameters on on Stack 'store' and decrypts on on Stack 'load'. Following are couple of improvements I am thinking about; 1. Instead of encrypting individual parameters, on Stack 'store' encrypt all the parameters together as a dictionary [something like crypt.encrypt(json.dumps(param_dictionary))] Yeah, definitely don't encrypt them individually. 2. Just encrypt parameters that were marked as 'hidden', instead of encrypting all parameters I would like to hear your feedback/suggestions. Just as a heads-up, we will soon need to store the properties of resources too, at which point parameters become the least of our problems. (In fact, in theory we wouldn't even need to store parameters... and probably by the time convergence is completely implemented, we won't.) Which is to say that there's almost certainly no point in discriminating between hidden and non-hidden parameters. I'll refrain from commenting on whether the extra security this affords is worth the giant pain it causes in debugging, except to say that IMO there should be a config option to disable the feature (and if it's enabled by default, it should probably be disabled by default in e.g. devstack). cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat]Heat template parameters encryption
Hi Devs, I have submitted an WIP review (https://review.openstack.org/#/c/97900/) for Heat parameters encryption blueprint https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters This quick and dirty implementation encrypts all the parameters on on Stack 'store' and decrypts on on Stack 'load'. Following are couple of improvements I am thinking about; 1. Instead of encrypting individual parameters, on Stack 'store' encrypt all the parameters together as a dictionary [something like crypt.encrypt(json.dumps(param_dictionary))] 2. Just encrypt parameters that were marked as 'hidden', instead of encrypting all parameters I would like to hear your feedback/suggestions. Thanks, Vijendar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
