Re: [Openstack] Keystone, pki tokens and memcache
On 06/17/2013 12:27 AM, Sam Morrison wrote: I'm currently looking into Grizzly and have been having some issues getting PKI tokens to work. If I have memcache as the token backend keystone issues uuid based tokens, if I have sql as the backend then it issues PKI tokens. Does this mean you can't use memcache backend if you want to use PKI tokens? Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp You are making additional configuration changes beyond the Backend. The config options for PKI vs UUID is defined in keystone/common/config.py 'token_format', group='signing', default=PKI The Backend is in the same place: 'driver', group='token', default='keystone.token.backends.sql.Token' So to set UUID tokens, in the config file, [signing] token_format=UUID or explicitly [signing] token_format=PKI ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone, pki tokens and memcache
On 18/06/2013, at 1:18 AM, Adam Young ayo...@redhat.com wrote: On 06/17/2013 12:27 AM, Sam Morrison wrote: I'm currently looking into Grizzly and have been having some issues getting PKI tokens to work. If I have memcache as the token backend keystone issues uuid based tokens, if I have sql as the backend then it issues PKI tokens. Does this mean you can't use memcache backend if you want to use PKI tokens? Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp You are making additional configuration changes beyond the Backend. The config options for PKI vs UUID is defined in keystone/common/config.py 'token_format', group='signing', default=PKI The Backend is in the same place: 'driver', group='token', default='keystone.token.backends.sql.Token' So to set UUID tokens, in the config file, [signing] token_format=UUID or explicitly [signing] token_format=PKI Yeah I had token_format set. I switched back to memcache and it started working, not sure what is going on there. Will see if I can replicate. Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone PKI support
On 09/04/2012 09:36 AM, boden wrote: Hi, I'm trying to better understand the current status of PKI (http://wiki.openstack.org/PKI) and delegated authZ from a folsom perspective. I can see the blueprint targets folsom-rc1, is marked as implemented (https://blueprints.launchpad.net/keystone/+spec/pki) and I've browsed some of the related code dropped into master. However its not clear to me exactly where this PKI support stands as I haven't found any docs on setting up the services to use it, nor am I seeing PKI based tokens used when I run with the latest keystone code. Is it safe to assume PKI will be supported for folsom and we will see some updated docs in due time? If so will there be any known limitations? Thx much ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp The PKI token code is very new. As such, It has not been enabled by default. There are changes that have gone in to the master branch that are not available on Folsom 3 that are necessary to the proper functioning of PKI. I wrote to the Fedora cloud mailing list a quicjk blurb on testing them. http://www.spinics.net/linux/fedora/fedora-cloud/msg01644.html I will write up some more detailed steps for general usage. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
Hi Adam, The blueprint as revised to address Joe's comments looks good to me - nice work. I especially like how the middleware is intended to cache the revocation list for a configurable amount of time - it mirrors how token caching already works. Cheers, Maru On 2012-08-07, at 10:09 AM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? I have written up a blueprint for PKI token revocation. Please provide feedback. https://blueprints.launchpad.net/keystone/+spec/pki-revoke Thanks, Maru ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? I have written up a blueprint for PKI token revocation. Please provide feedback. https://blueprints.launchpad.net/keystone/+spec/pki-revoke Thanks, Maru ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
On 08/02/2012 10:54 PM, Nathanael Burton wrote: Adam, I haven't yet had a chance to review how the new PKI signed tokens is implemented, but what you're describing sounds quite similar to online certificate status protocol (OCSP) but for tokens. Yes, I don't really have new idea here, just reimplementaiton of ideas from other projects. Nate On Aug 2, 2012 10:24 PM, Adam Young ayo...@redhat.com mailto:ayo...@redhat.com wrote: On 08/01/2012 11:05 PM, Maru Newby wrote: Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I realize that I had been thinking about the revocation list as something that needs to be broadcast. This is certainly not the case. A much better approach would be for the Keystone server to have a list of revoked tokens exposed in an URL. Then, as service like Glance or Nova can query the Revocation list on a simple schedule. The time out would be configurable, of course. There is a question about what to do if the keystone server cannot be reached during that interval. Since the current behavior is for authentication to fail, I suppose we would continue doing that, but also wait a random amount of time and then requery the Keystone server. In the future, I would like to make the set of Keystone servers a configurable list, and the policy for revocation checking should be able to vary per server: some Keystone servers in a federated approach might not be accessible. In those cases, it might be necessary for one Keystone server to proxy the revocation list for another server. Let me know if this scheme makes sense to you. If so, we can write it up as an additional blueprint. It should not be that hard to implement. I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25 years. Assuming that PKI ticket lifespan is short enough, revocation should not be required. What will be tricky is to balance the needs of long lived tokens (delayed operations, long running operations) against the needs for reasonable token timeout. PKI Token revocation would look like CRLs in the
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
Hey Maru, I think you're putting too many words in Adam's mouth here. First, Adam didnt assert is wasnt valuable, useful, or nessecary - simply that it wasnt in the first cut and not in the list that we agreed was critically essential to an initial implementation. As you noted, its a complex and somewhat tricky issue to get right. There's always room for more participation to correct the flaws you see in the existing system - the beauty of open source. I would love to see continued work on the signing and revocation work to drive in these features that mean so much to you. I'd be happy to open a blueprint if you can stand behind it, define what you think it required, and commit to the work to implement that revocation mechanism. Implying negative emotions on Adam's part when he's been one driving the implementation and doing the work is simply inappropriate. Please consider the blueprint route, definition of a viable solution, and work to make it happen instead of name calling and asserting how the developers doing the work are screwing up. - joe On Aug 1, 2012, at 8:05 PM, Maru Newby mne...@internap.com wrote: Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25 years. Assuming that PKI ticket lifespan is short enough, revocation should not be required. What will be tricky is to balance the needs of long lived tokens (delayed operations, long running operations) against the needs for reasonable token timeout. PKI Token revocation would look like CRLs in the Certificate world. While they are used, they are clunky. Each time a token gets revoked, a blast message would have to go out to all registered parties informing them of the revocation. Keystone does not yet have a message queue interface, so doing that is prohibitive in the first implementation. Note that users can get disabled, and token chaining will no longer work: you won't be able to use a token to get a new token from Keystone. Thanks, Maru ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help :
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
On 08/02/2012 01:56 AM, Joseph Heck wrote: Hey Maru, I think you're putting too many words in Adam's mouth here. First, Adam didnt assert is wasnt valuable, useful, or nessecary - simply that it wasnt in the first cut and not in the list that we agreed was critically essential to an initial implementation. As you noted, its a complex and somewhat tricky issue to get right. There's always room for more participation to correct the flaws you see in the existing system - the beauty of open source. I would love to see continued work on the signing and revocation work to drive in these features that mean so much to you. I'd be happy to open a blueprint if you can stand behind it, define what you think it required, and commit to the work to implement that revocation mechanism. Implying negative emotions on Adam's part when he's been one driving the implementation and doing the work is simply inappropriate. Please consider the blueprint route, definition of a viable solution, and work to make it happen instead of name calling and asserting how the developers doing the work are screwing up. Thanks for the support Joe. I don't think Maru was being too harsh. So long as he doesn't start calling me Sir as that is always an followed by you are making a scene. - joe On Aug 1, 2012, at 8:05 PM, Maru Newby mne...@internap.com mailto:mne...@internap.com wrote: Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. I think you have valid concerns. Realistically, I think 5 minutes is too short, and for many operations, 24 hours would be the right granularity. However, The timespan of the tokens is configurable, and the policy of the deploying organization should dictate. Remember, this is the administrative interface for virtual machines, and not the applications running in them. Removing someone from access to creating/rebooting/destroying virtual machines is a much more deliberate decision than banning someone from a public forum. Aside from someone getting fired, I am not sure how essential it is that we have rapid revocation of tokens. And firing someone is usually part of the whole escort from the building routine. So, let me put the onus on you: make the argument for rapid revocation of tokens. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
This was a concern for HP as well. This is one of the reasons we were happy to see that signed tokens are currently a deployment option. So, you can continue to use the unsigned model until such a time that revocation can be put into place for the token signing model. Jason From: openstack-bounces+jason.rouault=hp@lists.launchpad.net [mailto:openstack-bounces+jason.rouault=hp@lists.launchpad.net] On Behalf Of Maru Newby Sent: Wednesday, August 01, 2012 7:20 PM To: openstack@lists.launchpad.net (openstack@lists.launchpad.net) Subject: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? Thanks, Maru smime.p7s Description: S/MIME cryptographic signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
On 08/01/2012 11:05 PM, Maru Newby wrote: Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I realize that I had been thinking about the revocation list as something that needs to be broadcast. This is certainly not the case. A much better approach would be for the Keystone server to have a list of revoked tokens exposed in an URL. Then, as service like Glance or Nova can query the Revocation list on a simple schedule. The time out would be configurable, of course. There is a question about what to do if the keystone server cannot be reached during that interval. Since the current behavior is for authentication to fail, I suppose we would continue doing that, but also wait a random amount of time and then requery the Keystone server. In the future, I would like to make the set of Keystone servers a configurable list, and the policy for revocation checking should be able to vary per server: some Keystone servers in a federated approach might not be accessible. In those cases, it might be necessary for one Keystone server to proxy the revocation list for another server. Let me know if this scheme makes sense to you. If so, we can write it up as an additional blueprint. It should not be that hard to implement. I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25 years. Assuming that PKI ticket lifespan is short enough, revocation should not be required. What will be tricky is to balance the needs of long lived tokens (delayed operations, long running operations) against the needs for reasonable token timeout. PKI Token revocation would look like CRLs in the Certificate world. While they are used, they are clunky. Each time a token gets revoked, a blast message would have to go out to all registered parties informing them of the revocation. Keystone does not yet have a message queue interface, so doing that is prohibitive in the first implementation. Note that users can get disabled, and token chaining will no longer work: you won't be able to use a token to get a new token from Keystone. Thanks, Maru ___ Mailing list:https://launchpad.net/~openstack Post to :openstack@lists.launchpad.net Unsubscribe :https://launchpad.net/~openstack More help
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
Hi Adam, I was thinking along the same lines - the revocation list could be accessed via a simple url. It wouldn't even have to be hosted by Keystone, necessarily. For larger clusters where performance might become an issue, what about generating to a static file as needed that is made available via any of the usual web server suspects? As to whether the keystone server cannot be reached, that could be configurable. Some deployments might prefer permissive failure, others restrictive failure. I can see the case for both options. +1, also, to the set of Keystone servers being a configurable list, with differential policies for revocation checking. As to a justification for revocation, my use-case is more Swift (and integrated CDN) than Nova. A rogue user being able to manipulate VMs is one thing, but being able to expose potentially private data to a really wide audience is another. I would rate the damage potential of an object storage compromise as easily as great as application-level compromise. I would be happy to participate in creating and implementing these ideas. How can I help? Thanks, Maru On 2012-08-02, at 10:24 PM, Adam Young wrote: On 08/01/2012 11:05 PM, Maru Newby wrote: Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I realize that I had been thinking about the revocation list as something that needs to be broadcast. This is certainly not the case. A much better approach would be for the Keystone server to have a list of revoked tokens exposed in an URL. Then, as service like Glance or Nova can query the Revocation list on a simple schedule. The time out would be configurable, of course. There is a question about what to do if the keystone server cannot be reached during that interval. Since the current behavior is for authentication to fail, I suppose we would continue doing that, but also wait a random amount of time and then requery the Keystone server. In the future, I would like to make the set of Keystone servers a configurable list, and the policy for revocation checking should be able to vary per server: some Keystone servers in a federated approach might not be accessible. In those cases, it might be necessary for one Keystone server to proxy the revocation list for another server. Let me know if this scheme makes sense to you. If so, we can write it up as an additional blueprint. It should not be that hard to implement. I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
Hi Adam, I apologize if I came across as disrespectful. I was becoming frustrated that what I perceived as a valid concern was seemingly being ignored, but I recognize that there is no excuse for addressing you in a manner that I would not myself wish to be treated. I will do better going forward. Thanks, Maru ps: Thank you for the reminder, Joe! On 2012-08-02, at 1:56 AM, Joseph Heck wrote: Hey Maru, I think you're putting too many words in Adam's mouth here. First, Adam didnt assert is wasnt valuable, useful, or nessecary - simply that it wasnt in the first cut and not in the list that we agreed was critically essential to an initial implementation. As you noted, its a complex and somewhat tricky issue to get right. There's always room for more participation to correct the flaws you see in the existing system - the beauty of open source. I would love to see continued work on the signing and revocation work to drive in these features that mean so much to you. I'd be happy to open a blueprint if you can stand behind it, define what you think it required, and commit to the work to implement that revocation mechanism. Implying negative emotions on Adam's part when he's been one driving the implementation and doing the work is simply inappropriate. Please consider the blueprint route, definition of a viable solution, and work to make it happen instead of name calling and asserting how the developers doing the work are screwing up. - joe On Aug 1, 2012, at 8:05 PM, Maru Newby mne...@internap.com wrote: Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25 years. Assuming that PKI ticket lifespan is short enough, revocation should not be required. What will be tricky is to balance the needs of long lived tokens (delayed operations, long running operations) against the needs for reasonable token timeout. PKI Token revocation would look like CRLs in the Certificate world. While they are used, they are clunky. Each time a token gets revoked, a blast message would have to go out to all registered parties informing them of the revocation. Keystone does not yet have a message queue interface, so doing that is
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
Adam, I haven't yet had a chance to review how the new PKI signed tokens is implemented, but what you're describing sounds quite similar to online certificate status protocol (OCSP) but for tokens. Nate On Aug 2, 2012 10:24 PM, Adam Young ayo...@redhat.com wrote: On 08/01/2012 11:05 PM, Maru Newby wrote: Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I realize that I had been thinking about the revocation list as something that needs to be broadcast. This is certainly not the case. A much better approach would be for the Keystone server to have a list of revoked tokens exposed in an URL. Then, as service like Glance or Nova can query the Revocation list on a simple schedule. The time out would be configurable, of course. There is a question about what to do if the keystone server cannot be reached during that interval. Since the current behavior is for authentication to fail, I suppose we would continue doing that, but also wait a random amount of time and then requery the Keystone server. In the future, I would like to make the set of Keystone servers a configurable list, and the policy for revocation checking should be able to vary per server: some Keystone servers in a federated approach might not be accessible. In those cases, it might be necessary for one Keystone server to proxy the revocation list for another server. Let me know if this scheme makes sense to you. If so, we can write it up as an additional blueprint. It should not be that hard to implement. I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25 years. Assuming that PKI ticket lifespan is short enough, revocation should not be required. What will be tricky is to balance the needs of long lived tokens (delayed operations, long running operations) against the needs for reasonable token timeout. PKI Token revocation would look like CRLs in the Certificate world. While they are used, they are clunky. Each time a token gets revoked, a blast message would have to go out to all registered parties informing them of the revocation. Keystone does not yet have a message queue interface, so doing that is prohibitive in the first implementation. Note that users can get disabled, and token chaining will no longer work: you won't be able to use a
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25 years. Assuming that PKI ticket lifespan is short enough, revocation should not be required. What will be tricky is to balance the needs of long lived tokens (delayed operations, long running operations) against the needs for reasonable token timeout. PKI Token revocation would look like CRLs in the Certificate world. While they are used, they are clunky. Each time a token gets revoked, a blast message would have to go out to all registered parties informing them of the revocation. Keystone does not yet have a message queue interface, so doing that is prohibitive in the first implementation. Note that users can get disabled, and token chaining will no longer work: you won't be able to use a token to get a new token from Keystone. Thanks, Maru ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone: 'PKI Signed Tokens' lack support for revocation
Hi Adam, I apologize if my questions were answered before. I wasn't aware that what I perceive as a very serious security concern was openly discussed. The arguments against revocation support, as you've described them, seem to be: - it's complicated/messy/expensive to implement and/or execute - Kerberos doesn't need it, so why would we? I'm not sure why either of these arguments would justify the potential security hole that a lack of revocation represents, but I suppose a 'short enough' token lifespan could minimize that hole. But how short a span are you suggesting as being acceptable? The delay between when a user's access permissions change (whether roles, password or even account deactivation) and when the ticket reflects that change is my concern. The default in Keystone has been 24h, which is clearly too long. Something on the order of 5 minutes would be ideal, but then ticket issuance could become the bottleneck. Validity that's much longer could be a real problem, though. Maybe not at the cloud administration level, but for a given project I can imagine someone being fired and their access being revoked. How long is an acceptable period for that ticket to still be valid? How much damage could be done by someone who should no longer have access to an account if their access cannot be revoked, by anyone, at all? I'm hearing that you, as the implementer of this feature, don't consider the lack of revocation to be an issue. What am I missing? Is support for revocation so repugnant that the potential security hole is preferable? I can see that from a developer's perspective, but I don't understand why someone deploying Keystone wouldn't avoid PKI tokens until revocation support became available. Thanks, Maru On 2012-08-01, at 9:47 PM, Adam Young wrote: On 08/01/2012 09:19 PM, Maru Newby wrote: I see that support for PKI Signed Tokens has been added to Keystone without support for token revocation. I tried to raise this issue on the bug report: https://bugs.launchpad.net/keystone/+bug/1003962/comments/4 And the review: https://review.openstack.org/#/c/7754/ I'm curious as to whether anybody shares my concern and if there is a specific reason why nobody responded to my question as to why revocation is not required for this new token scheme. Anybody? It was discussed back when I wrote the Blueprint. While it is possible to do revocations with PKI, it is expensive and requires a lot of extra checking. Revocation is a policy decision, and the assumption is that people that are going to use PKI tokens are comfortable with out revocation. Kerberos service tickets have the same limitation, and Kerberos has been in deployment that way for close to 25 years. Assuming that PKI ticket lifespan is short enough, revocation should not be required. What will be tricky is to balance the needs of long lived tokens (delayed operations, long running operations) against the needs for reasonable token timeout. PKI Token revocation would look like CRLs in the Certificate world. While they are used, they are clunky. Each time a token gets revoked, a blast message would have to go out to all registered parties informing them of the revocation. Keystone does not yet have a message queue interface, so doing that is prohibitive in the first implementation. Note that users can get disabled, and token chaining will no longer work: you won't be able to use a token to get a new token from Keystone. Thanks, Maru ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Keystone] PKI
Fully agreed. Academic and Research sites have extensive X.509 infrastructure that we would not wish to duplicate. Are you only looking at user certificates or are host certificates in the scope too ? Tim From: openstack-bounces+tim.bell=cern...@lists.launchpad.net [mailto:openstack-bounces+tim.bell=cern...@lists.launchpad.net] On Behalf Of Adam Young Sent: 16 May 2012 03:10 To: openstack@lists.launchpad.net Subject: Re: [Openstack] [Keystone] PKI Well, the PKI pieces are the same regardless of the CA and certificate issuing pieces. All we will need to do is to use a signing key to sign a document. So EJBCA or Dogtag will work equally as well. If people already have a CA infrastructure, they should be able to leverage that, too. On 05/15/2012 04:47 PM, Thor Wolpert wrote: If you're open to levarging other OSS projects, http://www.ejbca.org/architecture.html us a great one to look at, assuming you need a PKI implementation available. I believe it is at least worth a look. On Tue, May 15, 2012 at 1:30 PM, Razique Mahroua razique.mahr...@gmail.com wrote: great topic :) mailto:he...@mac.com Joseph Heck 15 mai 2012 21:06 Coming out of the Keystone meeting from today (http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-me eting.2012-05-15-18.02.html), I thought it worth mentioning that adam young has been doing some tremendous lifting in terms of looking at adding in PKI support to Keystone. The writeup and details are on the OpenStack wiki at http://wiki.openstack.org/PKI I rather suspect there's a lot of interest in this topic, so I wanted to make sure the broader community knew about the effort, what we were thinking, and were we are. If you're interested in discussing, the keystone meeting is on Tuesday mornings at 18:00 UTC -joe ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp -- Nuage Co - Razique Mahroua razique.mahr...@gmail.com ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp smime.p7s Description: S/MIME cryptographic signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Keystone] PKI
great topic :) Joseph Heck 15 mai 2012 21:06Coming out of the Keystone meeting from today (http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-15-18.02.html), I thought it worth mentioning that adam young has been doing some tremendous lifting in terms of looking at adding in PKI support to Keystone. The writeup and details are on the OpenStack wiki at http://wiki.openstack.org/PKII rather suspect there's a lot of interest in this topic, so I wanted to make sure the broader community knew about the effort, what we were thinking, and were we are. If you're interested in discussing, the keystone meeting is on Tuesday mornings at 18:00 UTC-joe___Mailing list: https://launchpad.net/~openstackPost to : openstack@lists.launchpad.netUnsubscribe : https://launchpad.net/~openstackMore help : https://help.launchpad.net/ListHelp-- Nuage Co - Razique Mahroua razique.mahr...@gmail.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Keystone] PKI
If you're open to levarging other OSS projects, http://www.ejbca.org/architecture.html us a great one to look at, assuming you need a PKI implementation available. I believe it is at least worth a look. On Tue, May 15, 2012 at 1:30 PM, Razique Mahroua razique.mahr...@gmail.comwrote: great topic :) Joseph Heck he...@mac.com 15 mai 2012 21:06 Coming out of the Keystone meeting from today ( http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-15-18.02.html), I thought it worth mentioning that adam young has been doing some tremendous lifting in terms of looking at adding in PKI support to Keystone. The writeup and details are on the OpenStack wiki at http://wiki.openstack.org/PKI I rather suspect there's a lot of interest in this topic, so I wanted to make sure the broader community knew about the effort, what we were thinking, and were we are. If you're interested in discussing, the keystone meeting is on Tuesday mornings at 18:00 UTC -joe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Nuage Co - Razique Mahroua razique.mahr...@gmail.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Keystone] PKI
Well, the PKI pieces are the same regardless of the CA and certificate issuing pieces. All we will need to do is to use a signing key to sign a document. So EJBCA or Dogtag will work equally as well. If people already have a CA infrastructure, they should be able to leverage that, too. On 05/15/2012 04:47 PM, Thor Wolpert wrote: If you're open to levarging other OSS projects, http://www.ejbca.org/architecture.html us a great one to look at, assuming you need a PKI implementation available. I believe it is at least worth a look. On Tue, May 15, 2012 at 1:30 PM, Razique Mahroua razique.mahr...@gmail.com mailto:razique.mahr...@gmail.com wrote: great topic :) Joseph Heck mailto:he...@mac.com 15 mai 2012 21:06 Coming out of the Keystone meeting from today (http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-15-18.02.html), I thought it worth mentioning that adam young has been doing some tremendous lifting in terms of looking at adding in PKI support to Keystone. The writeup and details are on the OpenStack wiki at http://wiki.openstack.org/PKI I rather suspect there's a lot of interest in this topic, so I wanted to make sure the broader community knew about the effort, what we were thinking, and were we are. If you're interested in discussing, the keystone meeting is on Tuesday mornings at 18:00 UTC -joe ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp -- Nuage Co - Razique Mahroua razique.mahr...@gmail.com mailto:razique.mahr...@gmail.com ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Keystone] PKI
Hi Adam,
Can you please clarify the following in PKI blueprint?
1) Do you assume that roles won't be changed after getToken and before
validateToken?
!--
if the token contains just the following data :
- {username: admiyo,tenant: Fedora,expires: 2359:05May2012, roles:
[admin,editor]}
This message is then encrypted with Keystones private key. Any service that
has Keystones public key can decrypt the message. Since it is decrypted
with the public key, it had to be encrypted by Keystone, and is therefore
valid. The Keystone Certificate only has to be distributed once to each
service, and can be fetched on demand.
--
What is keystone private key? Do you mean user private key?
!--When a user is created in Keystone, they will be given a
one-time-password that they will then use to establish a key-pair. Only the
Public Key will be stored on the Keystone server, the Private key will only
be stored on the end users system. The public key will be signed by the
certificate authority (X509) and then stored in the Keystone system. From
this point on, when authenticating to Keystone, the user will use the
client certificate.
--
1) Why do we need to store users client cert in keystone system? BTW what
do you mean by keystone system? Is it keystone server? or any system
like swift/nova which uses keystone to authenticate
Thanks
On Tue, May 15, 2012 at 6:09 PM, Adam Young ayo...@redhat.com wrote:
Well, the PKI pieces are the same regardless of the CA and certificate
issuing pieces. All we will need to do is to use a signing key to sign a
document. So EJBCA or Dogtag will work equally as well. If people already
have a CA infrastructure, they should be able to leverage that, too.
On 05/15/2012 04:47 PM, Thor Wolpert wrote:
If you're open to levarging other OSS projects,
http://www.ejbca.org/architecture.html us a great one to look at,
assuming you need a PKI implementation available.
I believe it is at least worth a look.
On Tue, May 15, 2012 at 1:30 PM, Razique Mahroua
razique.mahr...@gmail.com wrote:
great topic :)
Joseph Heck he...@mac.com
15 mai 2012 21:06
Coming out of the Keystone meeting from today (
http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-15-18.02.html),
I thought it worth mentioning that adam young has been doing some
tremendous lifting in terms of looking at adding in PKI support to
Keystone. The writeup and details are on the OpenStack wiki at
http://wiki.openstack.org/PKI
I rather suspect there's a lot of interest in this topic, so I wanted to
make sure the broader community knew about the effort, what we were
thinking, and were we are.
If you're interested in discussing, the keystone meeting is on Tuesday
mornings at 18:00 UTC
-joe
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
--
Nuage Co - Razique Mahroua
razique.mahr...@gmail.com
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
