Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use
From: Adam Young [mailto:ayo...@redhat.com] Sent: Thursday, June 20, 2013 6:30 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use On 06/20/2013 04:50 PM, Ali, Haneef wrote: 1) I'm really not sure how that will solve the original issue (Token table size increase). Of course we can have a job to remove the expired token. It is not expiry that is the issue, but revocation. Expirey is handled by the fact that the token is a signed document with a timestamp in it. We don't really need to store expired tokens at all. [Arun] One of the issue is unlimited number of active tokens possible through keystone for same credentials. Which can possibly be turned into DoS attack on cloud services. So we can possibly look into keystone for solution as token generation is one of its key responsibility. Removal of expired tokens is separate aspect which will be needed after some point regardless of token is re-used or not. 2) We really have to think how the other services are using keystone. Keystone createToken volume is going to increase. Fixing one issue going to create another one. Yes it will. But in the past, the load was on Keystone token validate, and PKI has removed that load. Right now, the greater load on Keystone is coming from token create, but that is because token caching is not in place. With proper caching, Keystone would be hit only once for most workloads. It is currently hit for every Remote call. It is not the token generation that is the issue, but the issuing of the tokens that needs to be throttled back. [Arun] We cannot just have solution for happy path/situations. Being available in cloud , there are going to be varying types of clients and cannot just expect that each of them will have caching or will always be able to work with PKI token format (like third party services/applications running *on the cloud*). Throttling of token issuance request will require complex rate-limiting logic because of various input combinations and business rules associated with it. There can be another solution where keystone re-uses active token based on some selection logic and still able to server auth request without rate limiting errors. 1. If I understood correctly swift is using memcache to increase the validateToken performance. What will happen to it? Obviously load to validateToken will also increase. Validate token happens in process with PKI tokens, not via remote call. Memcache just prevents swift from having to make that check more than once per token. Revocation still needs to be checked every time. [Arun] There are issues with PKI token approach as well (lifespan of token, data size limit, role and status changes after token generation). If shorter timespan is used, then essentially we will be increasing createToken requests. 2. In few cases I have seen VM creation taking more than 5 min. ( download image from glance and create vm). Short lived token ( 5 min) will be a real fun in this case. That is what trusts are for. Nova should not be using a bearer token to perform operations on behalf of the user. Nova should be getting a delegated token via a trust to perform those operations. If a vm takes 5 minutes, it should not matter if the tokens time out, as Nova will get a token when it needs it. Bearer tokens are a poor design approach, and we have work going on that will remedy that. [Arun] Not sure how delegated token or current v3 trust/ role model is going to work here as token needs to have user roles (or at least delegated permissions with user's *all* privilege) to work on user behalf. Are we talking about impersonating user by Nova application in some way? In short-lived (non-PKI format), we are just diverting request load from validate token to create token which is relatively expensive operation. We need some smarter mechanism to limit proliferation of tokens as they are essentially user's credentials for a limited time. Thanks Haneef From: Ravi Chunduru [mailto:ravi...@gmail.com] Sent: Thursday, June 20, 2013 11:49 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use +1 On Thu, Jun 20, 2013 at 11:37 AM, Dolph Mathews dolph.math...@gmail.commailto:dolph.math...@gmail.com wrote: On Wed, Jun 19, 2013 at 2:20 PM, Adam Young ayo...@redhat.commailto:ayo...@redhat.com wrote: I really want to go the other way on this: I want token to be very short lived, ideally something like 1 minute, but probably 5 minutes to account for clock skew. I want to get rid of token revocation list checking. I'd like to get away from revocation altogether: tokens are not stored in the backend. If they are ephemeral, we can just check that the token has a valid signature and that the time has not expired. +10 On 06/19/2013 12:59 PM, Ravi Chunduru wrote: Thats still an open item
Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use
1) I’m really not sure how that will solve the original issue (Token table size increase). Of course we can have a job to remove the expired token. 2) We really have to think how the other services are using keystone. Keystone “createToken” volume is going to increase. Fixing one issue going to create another one. 1.If I understood correctly swift is using memcache to increase the validateToken performance. What will happen to it? Obviously load to “validateToken” will also increase. 2. In few cases I have seen VM creation taking more than 5 min. ( download image from glance and create vm). Short lived token ( 5 min) will be a real fun in this case. Thanks Haneef From: Ravi Chunduru [mailto:ravi...@gmail.com] Sent: Thursday, June 20, 2013 11:49 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use +1 On Thu, Jun 20, 2013 at 11:37 AM, Dolph Mathews dolph.math...@gmail.commailto:dolph.math...@gmail.com wrote: On Wed, Jun 19, 2013 at 2:20 PM, Adam Young ayo...@redhat.commailto:ayo...@redhat.com wrote: I really want to go the other way on this: I want token to be very short lived, ideally something like 1 minute, but probably 5 minutes to account for clock skew. I want to get rid of token revocation list checking. I'd like to get away from revocation altogether: tokens are not stored in the backend. If they are ephemeral, we can just check that the token has a valid signature and that the time has not expired. +10 On 06/19/2013 12:59 PM, Ravi Chunduru wrote: Thats still an open item in this thread. Let me summarize once again 1) Use case for keystone not to re-issue same token for same credentials 2) Ratelimit cons and service unavailability 3) Further information on python keyring if not going by keystone re-issue of the tokens. On Wed, Jun 19, 2013 at 9:16 AM, Yee, Guang guang@hp.commailto:guang@hp.com wrote: Just out of curiosity, is there really a use case where user need to request multiple tokens of the same scope, where the only difference are the expiration dates? Guang From: Dolph Mathews [mailto:dolph.math...@gmail.commailto:dolph.math...@gmail.com] Sent: Wednesday, June 19, 2013 7:27 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use On Wed, Jun 19, 2013 at 1:42 AM, Ali, Haneef haneef@hp.commailto:haneef@hp.com wrote: 1) Token Caching is not always going to help. It depends on the application.E.g A user writes a cron job to check the health of swift by listing a predefined container every 1 minute.This will obviously create a token every minute. 2) Also I like to understand how rate limiting is done for v3 tokens. Rate limiting involves source ip + request pattern. In V3 there are so many ways to get the token and the rate limiting becomes too complex Rate limit the number of requests to POST /v2.0/tokens and POST /v3/auth/tokens Just for unscoped token, all the following are equivalent requests. In case of scoped tokens we have even more combinations. Rouge clients can easily mess with rate limiting by mixing request patterns. Also rate limiting across regions may not be possible. a.UserId/Password b. UserName/Password/domainId c. UserName/Password/DomainName Thanks Haneef From: Ravi Chunduru [mailto:ravi...@gmail.commailto:ravi...@gmail.com] Sent: Tuesday, June 18, 2013 11:02 PM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use I agree we need a way to overcome these rogue clients but by rate limiting genuine requests will get effected. Then one would need retries and some times critical operations gets failed. It beats the whole logic of being available. About the keyrings, How do we tackle if a service is using JSON API calls and not python clients? Thanks, -Ravi. On Tue, Jun 18, 2013 at 6:37 PM, Adam Young ayo...@redhat.commailto:ayo...@redhat.com wrote: On 06/18/2013 09:13 PM, Kant, Arun wrote: The issue with having un-managed number of tokens for same credential is that it can be easily exploited. Getting a token is one of initial step (gateway) to get access to services. A rogue client can keep creating unlimited number of tokens and possibly create denial of service attack on services. If there are somewhat limited number of tokens, then cloud provider can possibly use tokenId based rate-limiting approach. Better here to rate limit, then. Extending the expiry to some fixed interval might be okay as that can be considered as continuing user session similar to what is seen when a user keeps browsing an application while logged in. Tokens are resources created by Keystone. No reason to ask to create something new if it is not needed. The caching needs to be done client side. We have ongoing work using python-keyring to support
Re: [openstack-dev] FW: [Keystone][Folsom] Token re-use
On 06/18/2013 09:13 PM, Kant, Arun wrote: The issue with having un-managed number of tokens for same credential is that it can be easily exploited. Getting a token is one of initial step (gateway) to get access to services. A rogue client can keep creating unlimited number of tokens and possibly create denial of service attack on services. If there are somewhat limited number of tokens, then cloud provider can possibly use tokenId based rate-limiting approach. Better here to rate limit, then. Extending the expiry to some fixed interval might be okay as that can be considered as continuing user session similar to what is seen when a user keeps browsing an application while logged in. Tokens are resources created by Keystone. No reason to ask to create something new if it is not needed. The caching needs to be done client side. We have ongoing work using python-keyring to support that. -Arun *From: *Adam Young ayo...@redhat.com mailto:ayo...@redhat.com *Reply-To: *OpenStack Development Mailing List openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org *Date: *Friday, June 14, 2013 3:33 PM *To: *openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org *Subject: *Re: [openstack-dev] [Keystone][Folsom] Token re-use On 06/13/2013 07:58 PM, Ravi Chunduru wrote: Hi, We are having Folsom setup and we find that our token table increases a lot. I understand client can re-use the token but why doesnt keystone reuse the token if client asks it with same credentials.. I would like to know if there is any reason for not doing so. Thanks in advance, -- Ravi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev You can cache the token on the client side and reuse. Tokens have a an expiry, so if you request a new token, you extend the expiry. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
