Re: [openstack-dev] 2 Minute tokens
On 10/13/2014 06:21 PM, Preston L. Bannister wrote: Too-short token expiration times are one of my concerns, in my current exercise. Working on a replacement for Nova backup. Basically creating backups jobs, writing the jobs into a queue, with a background worker that reads jobs from the queue. Tokens could expire while the jobs are in the queue (not too likely). Tokens could expire during the execution of a backup (while can be very long running, in some cases). Had not run into mention of trusts before. Is the intent to cover this sort of use-case? Keystone trusts are User to User delegations appropriate for long running tasks. So, yeah, should work for these use cases. (Pulled up what I could find on trusts. Need to chew on this a bit, as it is not immediately clear if this fits.) On Wed, Oct 1, 2014 at 6:53 AM, Adam Young ayo...@redhat.com mailto:ayo...@redhat.com wrote: On 10/01/2014 04:14 AM, Steven Hardy wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What is keeping us from dropping the (scoped) token duration to 5 minutes? If we could keep their lifetime as short as network skew lets us, we would be able to: Get rid of revocation checking. Get rid of persisted tokens. OK, so that assumes we can move back to PKI tokens, but we're working on that. What are the uses that require long lived tokens? Can they be replaced with a better mechanism for long term delegation (OAuth or Keystone trusts) as Heat has done? FWIW I think you're misrepresenting Heat's usage of Trusts here - 2 minute tokens will break Heat just as much as any other service: https://bugs.launchpad.net/heat/+bug/1306294 http://lists.openstack.org/pipermail/openstack-dev/2014-September/045585.html Summary: - Heat uses the request token to process requests (e.g stack create), which may take an arbitrary amount of time (default timeout one hour). - Some use-cases demand timeout of more than one hour (specifically big TripleO deployments), heat breaks in these situations atm, folks are working around it by using long (several hour) token expiry times. - Trusts are only used of asynchronous signalling, e.g Ceilometer signals Heat, we switch to a trust scoped token to process the response to the alarm (e.g launch more instances on behalf of the user for autoscaling) My understanding, ref notes in that bug, is that using Trusts while servicing a request to effectively circumvent token expiry was not legit (or at least yukky and to be avoided). If you think otherwise then please let me know, as that would be the simplest way to fix the bug above (switch to a trust token while doing the long-running create operation). Using trusts to circumvent timeout is OK. There are two issues in tension here: 1. A user needs to be able to maintain control of their own data. 2. We want to limit the attack surface provided by tokens. Since tokens are currently blanket access to the users data, there really is no lessening of control by using trusts in a wider context. I'd argue that using trusts would actually reduce the capability for abuse,if coupled with short lived tokens. With long lived tokens, anyone can reuse the token. With a trust, only the trustee would be able to create a new token. Could we start by identifying the set of operations that are currently timing out due to the one hour token duration and add an optional trustid on those operations? Trusts is not really ideal for this use-case anyway, as it requires the service to have knowledge of the roles to delegate (or that the user provides a pre-created trust), ref bug #1366133. I suppose we could just delegate all the roles we find in the request scope and be done with it, given that bug has been wontfixed. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] 2 Minute tokens
Too-short token expiration times are one of my concerns, in my current exercise. Working on a replacement for Nova backup. Basically creating backups jobs, writing the jobs into a queue, with a background worker that reads jobs from the queue. Tokens could expire while the jobs are in the queue (not too likely). Tokens could expire during the execution of a backup (while can be very long running, in some cases). Had not run into mention of trusts before. Is the intent to cover this sort of use-case? (Pulled up what I could find on trusts. Need to chew on this a bit, as it is not immediately clear if this fits.) On Wed, Oct 1, 2014 at 6:53 AM, Adam Young ayo...@redhat.com wrote: On 10/01/2014 04:14 AM, Steven Hardy wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What is keeping us from dropping the (scoped) token duration to 5 minutes? If we could keep their lifetime as short as network skew lets us, we would be able to: Get rid of revocation checking. Get rid of persisted tokens. OK, so that assumes we can move back to PKI tokens, but we're working on that. What are the uses that require long lived tokens? Can they be replaced with a better mechanism for long term delegation (OAuth or Keystone trusts) as Heat has done? FWIW I think you're misrepresenting Heat's usage of Trusts here - 2 minute tokens will break Heat just as much as any other service: https://bugs.launchpad.net/heat/+bug/1306294 http://lists.openstack.org/pipermail/openstack-dev/2014- September/045585.html Summary: - Heat uses the request token to process requests (e.g stack create), which may take an arbitrary amount of time (default timeout one hour). - Some use-cases demand timeout of more than one hour (specifically big TripleO deployments), heat breaks in these situations atm, folks are working around it by using long (several hour) token expiry times. - Trusts are only used of asynchronous signalling, e.g Ceilometer signals Heat, we switch to a trust scoped token to process the response to the alarm (e.g launch more instances on behalf of the user for autoscaling) My understanding, ref notes in that bug, is that using Trusts while servicing a request to effectively circumvent token expiry was not legit (or at least yukky and to be avoided). If you think otherwise then please let me know, as that would be the simplest way to fix the bug above (switch to a trust token while doing the long-running create operation). Using trusts to circumvent timeout is OK. There are two issues in tension here: 1. A user needs to be able to maintain control of their own data. 2. We want to limit the attack surface provided by tokens. Since tokens are currently blanket access to the users data, there really is no lessening of control by using trusts in a wider context. I'd argue that using trusts would actually reduce the capability for abuse,if coupled with short lived tokens. With long lived tokens, anyone can reuse the token. With a trust, only the trustee would be able to create a new token. Could we start by identifying the set of operations that are currently timing out due to the one hour token duration and add an optional trustid on those operations? Trusts is not really ideal for this use-case anyway, as it requires the service to have knowledge of the roles to delegate (or that the user provides a pre-created trust), ref bug #1366133. I suppose we could just delegate all the roles we find in the request scope and be done with it, given that bug has been wontfixed. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On Wed, Oct 1, 2014 at 3:47 AM, Adam Young ayo...@redhat.com wrote: 1. Identify the roles for the APIs that Cinder is going to be calling on swift based on Swifts policy.json FYI: there is no Swifts policy.json in mainline code, there is one external middleware available that provides it here https://github.com/cloudwatt/swiftpolicy.git. Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What is keeping us from dropping the (scoped) token duration to 5 minutes? If we could keep their lifetime as short as network skew lets us, we would be able to: Get rid of revocation checking. Get rid of persisted tokens. OK, so that assumes we can move back to PKI tokens, but we're working on that. What are the uses that require long lived tokens? Can they be replaced with a better mechanism for long term delegation (OAuth or Keystone trusts) as Heat has done? FWIW I think you're misrepresenting Heat's usage of Trusts here - 2 minute tokens will break Heat just as much as any other service: https://bugs.launchpad.net/heat/+bug/1306294 http://lists.openstack.org/pipermail/openstack-dev/2014-September/045585.html Summary: - Heat uses the request token to process requests (e.g stack create), which may take an arbitrary amount of time (default timeout one hour). - Some use-cases demand timeout of more than one hour (specifically big TripleO deployments), heat breaks in these situations atm, folks are working around it by using long (several hour) token expiry times. - Trusts are only used of asynchronous signalling, e.g Ceilometer signals Heat, we switch to a trust scoped token to process the response to the alarm (e.g launch more instances on behalf of the user for autoscaling) My understanding, ref notes in that bug, is that using Trusts while servicing a request to effectively circumvent token expiry was not legit (or at least yukky and to be avoided). If you think otherwise then please let me know, as that would be the simplest way to fix the bug above (switch to a trust token while doing the long-running create operation). Trusts is not really ideal for this use-case anyway, as it requires the service to have knowledge of the roles to delegate (or that the user provides a pre-created trust), ref bug #1366133. I suppose we could just delegate all the roles we find in the request scope and be done with it, given that bug has been wontfixed. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On 10/01/2014 04:14 AM, Steven Hardy wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What is keeping us from dropping the (scoped) token duration to 5 minutes? If we could keep their lifetime as short as network skew lets us, we would be able to: Get rid of revocation checking. Get rid of persisted tokens. OK, so that assumes we can move back to PKI tokens, but we're working on that. What are the uses that require long lived tokens? Can they be replaced with a better mechanism for long term delegation (OAuth or Keystone trusts) as Heat has done? FWIW I think you're misrepresenting Heat's usage of Trusts here - 2 minute tokens will break Heat just as much as any other service: https://bugs.launchpad.net/heat/+bug/1306294 http://lists.openstack.org/pipermail/openstack-dev/2014-September/045585.html Summary: - Heat uses the request token to process requests (e.g stack create), which may take an arbitrary amount of time (default timeout one hour). - Some use-cases demand timeout of more than one hour (specifically big TripleO deployments), heat breaks in these situations atm, folks are working around it by using long (several hour) token expiry times. - Trusts are only used of asynchronous signalling, e.g Ceilometer signals Heat, we switch to a trust scoped token to process the response to the alarm (e.g launch more instances on behalf of the user for autoscaling) My understanding, ref notes in that bug, is that using Trusts while servicing a request to effectively circumvent token expiry was not legit (or at least yukky and to be avoided). If you think otherwise then please let me know, as that would be the simplest way to fix the bug above (switch to a trust token while doing the long-running create operation). Using trusts to circumvent timeout is OK. There are two issues in tension here: 1. A user needs to be able to maintain control of their own data. 2. We want to limit the attack surface provided by tokens. Since tokens are currently blanket access to the users data, there really is no lessening of control by using trusts in a wider context. I'd argue that using trusts would actually reduce the capability for abuse,if coupled with short lived tokens. With long lived tokens, anyone can reuse the token. With a trust, only the trustee would be able to create a new token. Could we start by identifying the set of operations that are currently timing out due to the one hour token duration and add an optional trustid on those operations? Trusts is not really ideal for this use-case anyway, as it requires the service to have knowledge of the roles to delegate (or that the user provides a pre-created trust), ref bug #1366133. I suppose we could just delegate all the roles we find in the request scope and be done with it, given that bug has been wontfixed. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] 2 Minute tokens
What is keeping us from dropping the (scoped) token duration to 5 minutes? If we could keep their lifetime as short as network skew lets us, we would be able to: Get rid of revocation checking. Get rid of persisted tokens. OK, so that assumes we can move back to PKI tokens, but we're working on that. What are the uses that require long lived tokens? Can they be replaced with a better mechanism for long term delegation (OAuth or Keystone trusts) as Heat has done? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What are the uses that require long lived tokens? Glance has operations which can take a long time, such as uploading and downloading large images. signature.asc Description: Digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On 09/30/2014 10:44 AM, Adam Young wrote: What is keeping us from dropping the (scoped) token duration to 5 minutes? If we could keep their lifetime as short as network skew lets us, we would be able to: Get rid of revocation checking. Get rid of persisted tokens. OK, so that assumes we can move back to PKI tokens, but we're working on that. What are the uses that require long lived tokens? Can they be replaced with a better mechanism for long term delegation (OAuth or Keystone trusts) as Heat has done? I think you will find that most folks just don't know the intracacies of non-UUID tokens in Keystone. I think we'd be open to any options that are reliable, well-documented and don't produce 4K in each HTTP request. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On 09/30/2014 11:37 AM, Adam Young wrote: On 09/30/2014 11:06 AM, Louis Taylor wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What are the uses that require long lived tokens? Glance has operations which can take a long time, such as uploading and downloading large images. Yes, but the token is only authenticated at the start of the operation. Does anything need to happen afterwards? Funny you mention it... :) We were just having this conversation on IRC about Nikesh's issues with some Tempest volume tests and a token expiration problem. So, yes, a Glance upload operation makes a series of HTTP calls in the course of the upload: POST $registry/images -- Creates the queued image record ... upload of chunked body of HTTP request to backend like Swift .. PUT $registry/images/IMAGE_ID -- update image status and checksum So, what seems to be happening here is that the PUT call at the end of uploading the snapshot is using the same token that was created in the keystone client of the tempest test case during the test classes' setUpClass() method, and the test class ends up running for 1 hour, and by the time the PUT call is reached, the token has expired. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On 09/30/2014 11:58 AM, Jay Pipes wrote: On 09/30/2014 11:37 AM, Adam Young wrote: On 09/30/2014 11:06 AM, Louis Taylor wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What are the uses that require long lived tokens? Glance has operations which can take a long time, such as uploading and downloading large images. Yes, but the token is only authenticated at the start of the operation. Does anything need to happen afterwards? Funny you mention it... :) We were just having this conversation on IRC about Nikesh's issues with some Tempest volume tests and a token expiration problem. So, yes, a Glance upload operation makes a series of HTTP calls in the course of the upload: POST $registry/images -- Creates the queued image record ... upload of chunked body of HTTP request to backend like Swift .. PUT $registry/images/IMAGE_ID -- update image status and checksum So, what seems to be happening here is that the PUT call at the end of uploading the snapshot is using the same token that was created in the keystone client of the tempest test case during the test classes' setUpClass() method, and the test class ends up running for 1 hour, and by the time the PUT call is reached, the token has expired. Yes... and there is this whole unresolved dev thread on this - http://lists.openstack.org/pipermail/openstack-dev/2014-September/045567.html -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On 09/30/2014 12:21 PM, Sean Dague wrote: On 09/30/2014 11:58 AM, Jay Pipes wrote: On 09/30/2014 11:37 AM, Adam Young wrote: On 09/30/2014 11:06 AM, Louis Taylor wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What are the uses that require long lived tokens? Glance has operations which can take a long time, such as uploading and downloading large images. Yes, but the token is only authenticated at the start of the operation. Does anything need to happen afterwards? Funny you mention it... :) We were just having this conversation on IRC about Nikesh's issues with some Tempest volume tests and a token expiration problem. So, yes, a Glance upload operation makes a series of HTTP calls in the course of the upload: POST $registry/images -- Creates the queued image record ... upload of chunked body of HTTP request to backend like Swift .. PUT $registry/images/IMAGE_ID -- update image status and checksum So, what seems to be happening here is that the PUT call at the end of uploading the snapshot is using the same token that was created in the keystone client of the tempest test case during the test classes' setUpClass() method, and the test class ends up running for 1 hour, and by the time the PUT call is reached, the token has expired. Yes... and there is this whole unresolved dev thread on this - http://lists.openstack.org/pipermail/openstack-dev/2014-September/045567.html -Sean This is a test case, so the tempest test has enough information to request a new token, it just does not request it? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On 09/30/2014 12:21 PM, Sean Dague wrote: On 09/30/2014 11:58 AM, Jay Pipes wrote: On 09/30/2014 11:37 AM, Adam Young wrote: On 09/30/2014 11:06 AM, Louis Taylor wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What are the uses that require long lived tokens? Glance has operations which can take a long time, such as uploading and downloading large images. Yes, but the token is only authenticated at the start of the operation. Does anything need to happen afterwards? Funny you mention it... :) We were just having this conversation on IRC about Nikesh's issues with some Tempest volume tests and a token expiration problem. So, yes, a Glance upload operation makes a series of HTTP calls in the course of the upload: POST $registry/images -- Creates the queued image record ... upload of chunked body of HTTP request to backend like Swift .. PUT $registry/images/IMAGE_ID -- update image status and checksum So, what seems to be happening here is that the PUT call at the end of uploading the snapshot is using the same token that was created in the keystone client of the tempest test case during the test classes' setUpClass() method, and the test class ends up running for 1 hour, and by the time the PUT call is reached, the token has expired. Yes... and there is this whole unresolved dev thread on this - http://lists.openstack.org/pipermail/openstack-dev/2014-September/045567.html -Sean This is comparable to the HEAT use case that Keystone Trusts were originally designed to solve. If the glance client knows the roles required to perform those operations, it could create the trust up front, with the Glance Service user as the trustee; the trustee execute the trust when it needs the token. Are there other cases besides the glance one that require long lived tokens? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
On Oct 1, 2014 12:37 AM, Adam Young ayo...@redhat.com wrote: On 09/30/2014 12:21 PM, Sean Dague wrote: On 09/30/2014 11:58 AM, Jay Pipes wrote: On 09/30/2014 11:37 AM, Adam Young wrote: On 09/30/2014 11:06 AM, Louis Taylor wrote: On Tue, Sep 30, 2014 at 10:44:51AM -0400, Adam Young wrote: What are the uses that require long lived tokens? Glance has operations which can take a long time, such as uploading and downloading large images. Yes, but the token is only authenticated at the start of the operation. Does anything need to happen afterwards? Funny you mention it... :) We were just having this conversation on IRC about Nikesh's issues with some Tempest volume tests and a token expiration problem. So, yes, a Glance upload operation makes a series of HTTP calls in the course of the upload: POST $registry/images -- Creates the queued image record ... upload of chunked body of HTTP request to backend like Swift .. PUT $registry/images/IMAGE_ID -- update image status and checksum So, what seems to be happening here is that the PUT call at the end of uploading the snapshot is using the same token that was created in the keystone client of the tempest test case during the test classes' setUpClass() method, and the test class ends up running for 1 hour, and by the time the PUT call is reached, the token has expired. Yes... and there is this whole unresolved dev thread on this - http://lists.openstack.org/pipermail/openstack-dev/2014-September/045567.html -Sean This is comparable to the HEAT use case that Keystone Trusts were originally designed to solve. If the glance client knows the roles required to perform those operations, it could create the trust up front, with the Glance Service user as the trustee; the trustee execute the trust when it needs the token. Are there other cases besides the glance one that require long lived tokens? Cinder backups. These do many swift operations over a long period, often hours. They should probably be converted to trusts, but I'd need some guidance do so. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 2 Minute tokens
This is comparable to the HEAT use case that Keystone Trusts were originally designed to solve. If the glance client knows the roles required to perform those operations, it could create the trust up front, with the Glance Service user as the trustee; the trustee execute the trust when it needs the token. Are there other cases besides the glance one that require long lived tokens? Cinder backups. These do many swift operations over a long period, often hours. They should probably be converted to trusts, but I'd need some guidance do so. 1. Identify the roles for the APIs that Cinder is going to be calling on swift based on Swifts policy.json 2. The user Creates trust: this could be done by the cinder client. The create trust call is here: trustee user is the Cinder service user trustor user is the actual human being (or reasonable facimile thereof) that needs this work done. https://github.com/openstack/identity-api/blob/master/v3/src/markdown/identity-api-v3-os-trust-ext.md#create-trust-post-os-trusttrusts impersonation should only be set if the API requires the tokens user_id to reflect the trustor. I know that swift was one of the use cases for impersonation, but I don't think it is required. Expiration date time is optional, but probably appropriate; chose a value that is guaranteed to be long enough for the workflow, but that still limits the window for attack. If this backup routinely takes 12 hours, perhaps 24 hours is appropriate. When the user requests the Cinder backup, they will have to include the trust ID in the request. The cinder endpoint, prior to making a call to Swift, will get a token from Keystone, passing in the trust id as per https://github.com/openstack/identity-api/blob/master/v3/src/markdown/identity-api-v3-os-trust-ext.md#consuming-a-trust-with-post-authtokens ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
