Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
Comments in-line. -Original Message- From: Joshua Harlow Reply: OpenStack Development Mailing List (not for usage questions) > Date: September 29, 2014 at 21:52:20 To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift? > +1 Lets not continue to expand the usage of persisted tokens :-/ We can look at moving away from persisted tokens, but as of the Kilo cycle there is no plan to eliminate UUID tokens (or PKI/PKIZ tokens for that matter). This means that we will need to continue to support them. If there is a legitimate desire to utilize swift or something similar and make this experience better, it behooves us to consider accepting these improvements. I believe that non-persistent tokens are a great place to strive for, but it won’t meet every deployment need. Moreover, currently PKI tokens (the “persistent-less” capable version) adds significant overhead to each request. While 5k-10k of data doesn’t seem like a lot, when there is a large volume of actions taken it adds up fast. This is putting the onus on the end user to acquire the “authorization” and transmit that same information over and over again to each of the APIs (especially in the swift use-case of many, many, many small object requests). 5-10k for a request consisting of a few hundred bytes is a significant overhead. Asking a service local to your datacenter might be in some cases more efficient and produce less edge traffic. There is a reason why websites often use separate domains (top-level) for their CDNs rather than just sourcing from the primary domain. This is to (especially in the case of large cookies such as are generated by Social Media sites) avoid having to transit the primary user cookie when gathering an asset, especially if that asset is publicly viewable. > We should be trying to move away from such types of persistence and its > associated complexity > IMHO. Most major websites don't need to have tokens that are saved around > regions in there > internal backend databases, just so people can use a REST webservice (or a > website...) > so I would hope that we don't need to (if it works for major websites why > doesn't it work > for us?). As of today, PKI tokens (and even UUID token bodies) convey a lot of information. In fact, the tokens convey all the information (except if the token is revoked) needed to interact with the API endpoint. A lot of the “big” web apps will not encode all of that information into the signed cookie, some of that information can be sourced directly (often things like ACL-like-constructs or RBAC information won’t be in the cookie itself). PKI tokens are also significantly more difficult to deploy (needing certificates, CAs, and ensuring everything is in sync is definitely more work than saying “talk to URL xxx and ask if the token is valid”; you’re trading one set of operational concerns and scaling points for another). Now if we start looking at OAuth and some of the other SSO technology we start getting a little bit closer to what we need to be able to eliminate the backing store with the exception of two critical items: 1) OAuth-type workflow would be a reauthorization for each endpoint (provided they are on separate domains/hosts/etc, which is the case for each of our services). You don’t (for example) Authenticate with UbuntuOne for https://review.openstack.org and without re-authenticating go to the wiki pages and edit the wiki. The wiki, while it uses the same authentication source, same identity source, and is even on an associated host/domain, requires a re-auth to use. If you were to get an SSO login to Keystone and then each endpoint requires another bounce back to Keystone for AuthZ it isn’t really making the REST API any more usable nor making the user experience any better. 2) All of the OpenStack specific permissions are not necessarily conveyed by the Identity Provider (Keystone is not typically the Identity provider in this case), and most Identity Providers do not have the concept of Projects, Domains, or even Roles that are always going to make sense within OpenStack. We’re still back to needing to ask for information from Keystone. > My 2 cents is that we should really think about why this is needed and why we > can't operate > using signed-cookie like mechanisms (after all it works for everyone else). > If cross-region > tokens are a problem, then maybe we should solve the root of the issue > (having a token that > works across regions) so that no replication is needed at all… With PKI tokens (if you share the signing certificate) it is possible to handle cross region support. There are some limitations here (such as possibly needing to replicate the Identity/Assignment information between regions so an Auth
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Mon, Sep 29 2014, Morgan Fainberg wrote: > The big issue you're going to run into is locking. The indexes need to have > a distributed lock that guarantees that each index is read/updated/released > atomically (similar to the SQL transaction). The way memcache and redis > handle this is by trying to add a record that is based on the index record > name and if that "add" fails (already exists) we assume the referenced > record is locked. We automatically timeout the lock after a period of time. For distributed locking, using tooz¹ could solve that issue. ¹ http://tooz.readthedocs.org/en/latest/ -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
+1 Lets not continue to expand the usage of persisted tokens :-/ We should be trying to move away from such types of persistence and its associated complexity IMHO. Most major websites don't need to have tokens that are saved around regions in there internal backend databases, just so people can use a REST webservice (or a website...) so I would hope that we don't need to (if it works for major websites why doesn't it work for us?). My 2 cents is that we should really think about why this is needed and why we can't operate using signed-cookie like mechanisms (after all it works for everyone else). If cross-region tokens are a problem, then maybe we should solve the root of the issue (having a token that works across regions) so that no replication is needed at all... On Sep 29, 2014, at 8:22 PM, Adam Young wrote: > On 09/29/2014 12:12 PM, Jay Pipes wrote: >> Hey Stackers, >> >> So, I had a thought this morning (uh-oh, I know...). >> >> What if we wrote a token driver in Keystone that uses Swift for backend >> storage? >> >> I have long been an advocate of the memcache token driver versus the SQL >> driver for performance reasons. However, the problem with the memcache token >> driver is that if you want to run multiple OpenStack regions, you could >> share the identity data in Keystone using replicated database technology >> (mysql galera/PXC, pgpool II, or even standard mysql master/slave), but each >> region needs to have its own memcache service for tokens. This means that >> tokens are not shared across regions, which means that users have to log in >> separately to each region's dashboard. >> >> I personally considered this a tradeoff worth accepting. But then, today, I >> thought... what about storing tokens in a globally-distributed Swift >> cluster? That would take care of the replication needs automatically, since >> Swift would do the needful. And, add to that, Swift was designed for storing >> lots of small objects, which tokens are... >> >> Thoughts? I think it would be a cool dogfooding effort if nothing else, and >> give users yet another choice in how they handle multi-region tokens. > > Um...I hate all persisted tokens. This takes them to a new level of badness. > > Do we really need this? > > > >> >> Best, >> -jay >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
Excerpts from Adam Young's message of 2014-09-29 20:22:35 -0700: > On 09/29/2014 12:12 PM, Jay Pipes wrote: > > Hey Stackers, > > > > So, I had a thought this morning (uh-oh, I know...). > > > > What if we wrote a token driver in Keystone that uses Swift for > > backend storage? > > > > I have long been an advocate of the memcache token driver versus the > > SQL driver for performance reasons. However, the problem with the > > memcache token driver is that if you want to run multiple OpenStack > > regions, you could share the identity data in Keystone using > > replicated database technology (mysql galera/PXC, pgpool II, or even > > standard mysql master/slave), but each region needs to have its own > > memcache service for tokens. This means that tokens are not shared > > across regions, which means that users have to log in separately to > > each region's dashboard. > > > > I personally considered this a tradeoff worth accepting. But then, > > today, I thought... what about storing tokens in a > > globally-distributed Swift cluster? That would take care of the > > replication needs automatically, since Swift would do the needful. > > And, add to that, Swift was designed for storing lots of small > > objects, which tokens are... > > > > Thoughts? I think it would be a cool dogfooding effort if nothing > > else, and give users yet another choice in how they handle > > multi-region tokens. > > Um...I hate all persisted tokens. This takes them to a new level of > badness. > > Do we really need this? > FWIW I'm 100% with you Adam. I would like to see a world without a token storage problem in Keystone. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On 09/29/2014 12:12 PM, Jay Pipes wrote: Hey Stackers, So, I had a thought this morning (uh-oh, I know...). What if we wrote a token driver in Keystone that uses Swift for backend storage? I have long been an advocate of the memcache token driver versus the SQL driver for performance reasons. However, the problem with the memcache token driver is that if you want to run multiple OpenStack regions, you could share the identity data in Keystone using replicated database technology (mysql galera/PXC, pgpool II, or even standard mysql master/slave), but each region needs to have its own memcache service for tokens. This means that tokens are not shared across regions, which means that users have to log in separately to each region's dashboard. I personally considered this a tradeoff worth accepting. But then, today, I thought... what about storing tokens in a globally-distributed Swift cluster? That would take care of the replication needs automatically, since Swift would do the needful. And, add to that, Swift was designed for storing lots of small objects, which tokens are... Thoughts? I think it would be a cool dogfooding effort if nothing else, and give users yet another choice in how they handle multi-region tokens. Um...I hate all persisted tokens. This takes them to a new level of badness. Do we really need this? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
-Original Message- From: Clint Byrum Reply: OpenStack Development Mailing List (not for usage questions) > Date: September 29, 2014 at 16:17:39 To: openstack-dev > Subject: Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift? > Excerpts from Clay Gerrard's message of 2014-09-29 16:05:14 -0700: > > On Mon, Sep 29, 2014 at 2:53 PM, Chmouel Boudjnah > > wrote: > > > > > > > > > > > eventual consistency will only affect container listing and I don't think > > > there is a need for container listing in that driver. > > > > > > > > well now hold on... > > > > if you're doing an overwrite in the face of server failures you could still > > get a stale read if a server with an old copy comes back into the fray and > > you read before replication sorts it out, or read a old version of a key > > you deleted > > For tokens, there are really only two answers that matter: > > * does ID==X exist? * has ID==X been revoked? > > I think as long as you have a separate container for revocations and > tokens, then resurrections would be fine. The records themselves would > be immutable so edits aren't a problem. This isn’t exactly true. In the case of certain actions (user/project/domain/role delete, user password change, and a number of other ones) you need to be able to find all tokens associated to that entity so they can be revoked (if you’re not using revocation events). This likely means for this type of backend revocation events are a requirement (eliminates the need to list out each token id/provide a way to lookup a token based upon the entity being acted upon) and the ‘enumerated’ token indexes should not be supported. —Morgan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Mon, Sep 29, 2014 at 4:15 PM, Clint Byrum wrote: > It would, however, be bad to get a 404 for something that is otherwise > present.. as that will result in an erroneous failure for the client. > > That almost never happens, but is possible if all the primaries are down*, a system than leans harder on the C a similar failure would be expected to treat a similar impossible question as a failure/error. * It's actually if all the same nodes that answered the previous write are down; there's some trixies with error-limiting and stable handoffs that help with subsequent read-your-writes behavior that actually make it fairly difficult to write data that you can't then read back out unless you basically track where all of the writes go and then shut down *exactly* those nodes and make a read before replication beats you to it. Just shutting down all three primary locations will just write and then read from the same handoff locations, even if the primaries subsequently come back online (unless the primaries have an old copy - but it sounds like that's not going on in your application). Again, all of this has to do with under failure edge cases. A healthy swift system; or even one that's only moderately degraded won't really see much of this. Depending on the deployment latencies may be a concern if you're using this as a "cache" - have you looked at Swauth [1] already? -Clay 1. https://github.com/gholt/swauth ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
Excerpts from Clay Gerrard's message of 2014-09-29 16:05:14 -0700: > On Mon, Sep 29, 2014 at 2:53 PM, Chmouel Boudjnah > wrote: > > > > > > > eventual consistency will only affect container listing and I don't think > > there is a need for container listing in that driver. > > > > > well now hold on... > > if you're doing an overwrite in the face of server failures you could still > get a stale read if a server with an old copy comes back into the fray and > you read before replication sorts it out, or read a old version of a key > you deleted For tokens, there are really only two answers that matter: * does ID==X exist? * has ID==X been revoked? I think as long as you have a separate container for revocations and tokens, then resurrections would be fine. The records themselves would be immutable so edits aren't a problem. It would, however, be bad to get a 404 for something that is otherwise present.. as that will result in an erroneous failure for the client. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On 30/09/2014 01:05, Clay Gerrard wrote: eventual consistency will only affect container listing and I don't think there is a need for container listing in that driver. well now hold on... if you're doing an overwrite in the face of server failures you could still get a stale read if a server with an old copy comes back into the fray and you read before replication sorts it out, or read a old version of a key you deleted yeah sure thanks for clarifying but from my understanding all tokens are new keys/object there is not overwriting going on, Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Mon, Sep 29, 2014 at 2:53 PM, Chmouel Boudjnah wrote: > > > eventual consistency will only affect container listing and I don't think > there is a need for container listing in that driver. > > well now hold on... if you're doing an overwrite in the face of server failures you could still get a stale read if a server with an old copy comes back into the fray and you read before replication sorts it out, or read a old version of a key you deleted -Clay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Mon, Sep 29, 2014 at 11:47 PM, Dmitry Mescheryakov < dmescherya...@mirantis.com> wrote: > As a result of operation #1 the token will be saved into Swift by the > Keystone. But due to eventual consistency it could happen that validation > of token in operation #2 will not see the saved token. Probability depends > on time gap between ops #1 and #2: the smaller the gap, the higher is > probability (less time to sync). Also it depends on Swift installation > size: the bigger is installation, the higher is probability (bigger 'space' > for inconsistency). eventual consistency will only affect container listing and I don't think there is a need for container listing in that driver. Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
Hey Jay, Did you consider Swift's eventual consistency? The general use case for many OpenStack application is: 1. obtain the token from Keystone 2. perform some operation in OpenStack providing token as credentials. As a result of operation #1 the token will be saved into Swift by the Keystone. But due to eventual consistency it could happen that validation of token in operation #2 will not see the saved token. Probability depends on time gap between ops #1 and #2: the smaller the gap, the higher is probability (less time to sync). Also it depends on Swift installation size: the bigger is installation, the higher is probability (bigger 'space' for inconsistency). I believe that I've seen such inconsistency in Rackspace Cloud Files a couple of years ago. We uploaded a file using an application into the Files, but saw it in browser only a couple minutes later. It is my understanding that Ceph exposing Swift API is not affected though, as it is strongly consistent. Thanks, Dmitry 2014-09-29 20:12 GMT+04:00 Jay Pipes : > Hey Stackers, > > So, I had a thought this morning (uh-oh, I know...). > > What if we wrote a token driver in Keystone that uses Swift for backend > storage? > > I have long been an advocate of the memcache token driver versus the SQL > driver for performance reasons. However, the problem with the memcache > token driver is that if you want to run multiple OpenStack regions, you > could share the identity data in Keystone using replicated database > technology (mysql galera/PXC, pgpool II, or even standard mysql > master/slave), but each region needs to have its own memcache service for > tokens. This means that tokens are not shared across regions, which means > that users have to log in separately to each region's dashboard. > > I personally considered this a tradeoff worth accepting. But then, today, > I thought... what about storing tokens in a globally-distributed Swift > cluster? That would take care of the replication needs automatically, since > Swift would do the needful. And, add to that, Swift was designed for > storing lots of small objects, which tokens are... > > Thoughts? I think it would be a cool dogfooding effort if nothing else, > and give users yet another choice in how they handle multi-region tokens. > > Best, > -jay > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Monday, September 29, 2014, Julien Danjou wrote: > On Mon, Sep 29 2014, Jay Pipes wrote: > > > Any objection to me taking up the work? Was there any associated > blueprint for > > it? > > As said on IRC, go ahead. There's no blueprint associated AFAIK. :) > > -- > Julien Danjou > /* Free Software hacker >http://julien.danjou.info */ > There is of course a benefit to having more options for deployers to utilize for the token persistence, especially if we solve complaints about the current token systems in the process. A couple thoughts: I highly recommend making this a dogpile backend (KVS) in keystone. It should help eliminate a bunch of the harder bits of code since it will use the KVS house keeping code for the indexes (same way it works in memcache or the in-memory dict backend). The big issue you're going to run into is locking. The indexes need to have a distributed lock that guarantees that each index is read/updated/released atomically (similar to the SQL transaction). The way memcache and redis handle this is by trying to add a record that is based on the index record name and if that "add" fails (already exists) we assume the referenced record is locked. We automatically timeout the lock after a period of time. I am also curious what the performance profile of swift-as-a-token-backend will look like. You are adding a large amount of overhead to the request (in theory) as the swift call is another http request. And depending on how swift is running, may require more token request work (tokens, especially v2.0 since he token is is in the body must be considered secure data, and therefore should not be directly accessible to the world at large), keeping security concerns in mind. There are some other cool ideas this can lead to, but let's talk about the reality and limitations (and how we solve those issues) for our current use cases before we dive into the next steps. The first thing we will need is (of course) a spec for keystone. Let's try and get the spec proposed soon so we can ensure that we get anything that needs to land in kilo through the door earlier in the cycle. Cheers, Morgan Sent via mobile ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Mon, Sep 29 2014, Jay Pipes wrote: > Any objection to me taking up the work? Was there any associated blueprint for > it? As said on IRC, go ahead. There's no blueprint associated AFAIK. :) -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Mon, Sep 29, 2014 at 11:25 AM, Jay Pipes wrote: > On 09/29/2014 12:15 PM, Julien Danjou wrote: > >> On Mon, Sep 29 2014, Jay Pipes wrote: >> >> What if we wrote a token driver in Keystone that uses Swift for backend >>> storage? >>> >> >> Yay! I already wrote a PoC to that: >> >>https://review.openstack.org/#/c/86016/ >> > > Sweet! :) > > It has been rejected because this patch didn't use the generic approach >> that Keystone tries to use to all the storage backends. Otherwise, I've >> tested it a bit with devstack and it worked fine. >> >> I didn't continue to work on it by lack of time, and because the effort >> to use Swift as a "generic" cache mechanism seemed a bit tricky to me. >> > > Any objection to me taking up the work? Was there any associated blueprint > for it? To the best of my knowledge, I haven't seen a spec or blueprint proposed for a swift token backend. > > Best, > -jay > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On 09/29/2014 12:15 PM, Julien Danjou wrote: On Mon, Sep 29 2014, Jay Pipes wrote: What if we wrote a token driver in Keystone that uses Swift for backend storage? Yay! I already wrote a PoC to that: https://review.openstack.org/#/c/86016/ Sweet! :) It has been rejected because this patch didn't use the generic approach that Keystone tries to use to all the storage backends. Otherwise, I've tested it a bit with devstack and it worked fine. I didn't continue to work on it by lack of time, and because the effort to use Swift as a "generic" cache mechanism seemed a bit tricky to me. Any objection to me taking up the work? Was there any associated blueprint for it? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][swift] Has anybody considered storing tokens in Swift?
On Mon, Sep 29 2014, Jay Pipes wrote: > What if we wrote a token driver in Keystone that uses Swift for backend > storage? Yay! I already wrote a PoC to that: https://review.openstack.org/#/c/86016/ It has been rejected because this patch didn't use the generic approach that Keystone tries to use to all the storage backends. Otherwise, I've tested it a bit with devstack and it worked fine. I didn't continue to work on it by lack of time, and because the effort to use Swift as a "generic" cache mechanism seemed a bit tricky to me. -- Julien Danjou // Free Software hacker // http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev