Re: [openstack-dev] [Keystone] Store quotas in Keystone
I just tried to read the full spec for this blueprint https://blueprints.launchpad.net/keystone/+spec/store-quota-data https://wiki.openstack.org/wiki/KeystoneCentralizedQuotaManagement And nothing explains why this blueprint is needed or what it is trying to accomplish, all it has is a design for keystone. Both the introduction and user stories sections just say 'TBD'. How can we have proper discussion of this blueprint without that information? best, Joe sent on the go On Dec 3, 2013 7:35 PM, Joe Gordon joe.gord...@gmail.com wrote: On Dec 3, 2013 6:49 PM, John Dickinson m...@not.mn wrote: On Dec 3, 2013, at 8:05 AM, Jay Pipes jaypi...@gmail.com wrote: On 12/03/2013 10:04 AM, John Dickinson wrote: How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. From reading below, it does not look like a centralized lookup is what the design is. A push-change strategy is what is described, where the quota numbers themselves are stored in a canonical location in Keystone, but when those numbers are changed, Keystone would send a notification of that change to subscribing services such as Swift, which would presumably have one or more levels of caching for things like account and container quotas... Yes, I get that, and there are already methods in Swift to support that. The trick, though, is either (1) storing all the canonical info in Keystone and scaling that or (2) storing some boiled down version, if possible, and fanning that out to all of the resources in Swift. Both are difficult and require storing the information in the central Keystone store. If I remember correctly the motivation for using keystone for quotas is so there is one easy place to set quotas across all projects. Why not hide this complexity with the unified client instead? That has been the answer we have been using for pulling out assorted proxy APIs in nova (nova image-list volume-list) etc. Best, -jay --John On Dec 3, 2013, at 6:53 AM, Oleg Gelbukh ogelb...@mirantis.com wrote: Chmouel, We reviewed the design of this feature at the summit with CERN and HP teams. Centralized quota storage in Keystone is an anticipated feature, but there are concerns about adding quota enforcement logic for every service to Keystone. The agreed solution is to add quota numbers storage to Keystone, and add mechanism that will notify services about change to the quota. Service, in turn, will update quota cache and apply the new quota value according to its own enforcement rules. More detailed capture of the discussion on etherpad: https://etherpad.openstack.org/p/CentralizedQuotas Re this particular change, we plan to reuse this API extension code, but extended to support domain-level quota as well. -- Best regards, Oleg Gelbukh Mirantis Labs On Mon, Dec 2, 2013 at 5:39 PM, Chmouel Boudjnah chmo...@enovance.com wrote: Hello, I was wondering what was the status of Keystone being the central place across all OpenStack projects for quotas. There is already an implementation from Dmitry here : https://review.openstack.org/#/c/40568/ but hasn't seen activities since october waiting for icehouse development to be started and a few bits to be cleaned and added (i.e: the sqlite migration). It would be great if we can get this rekicked to get that for icehouse-2. Thanks, Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Store quotas in Keystone
Chmouel, We reviewed the design of this feature at the summit with CERN and HP teams. Centralized quota storage in Keystone is an anticipated feature, but there are concerns about adding quota enforcement logic for every service to Keystone. The agreed solution is to add quota numbers storage to Keystone, and add mechanism that will notify services about change to the quota. Service, in turn, will update quota cache and apply the new quota value according to its own enforcement rules. More detailed capture of the discussion on etherpad: https://etherpad.openstack.org/p/CentralizedQuotas Re this particular change, we plan to reuse this API extension code, but extended to support domain-level quota as well. -- Best regards, Oleg Gelbukh Mirantis Labs On Mon, Dec 2, 2013 at 5:39 PM, Chmouel Boudjnah chmo...@enovance.comwrote: Hello, I was wondering what was the status of Keystone being the central place across all OpenStack projects for quotas. There is already an implementation from Dmitry here : https://review.openstack.org/#/c/40568/ but hasn't seen activities since october waiting for icehouse development to be started and a few bits to be cleaned and added (i.e: the sqlite migration). It would be great if we can get this rekicked to get that for icehouse-2. Thanks, Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Store quotas in Keystone
How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. --John On Dec 3, 2013, at 6:53 AM, Oleg Gelbukh ogelb...@mirantis.com wrote: Chmouel, We reviewed the design of this feature at the summit with CERN and HP teams. Centralized quota storage in Keystone is an anticipated feature, but there are concerns about adding quota enforcement logic for every service to Keystone. The agreed solution is to add quota numbers storage to Keystone, and add mechanism that will notify services about change to the quota. Service, in turn, will update quota cache and apply the new quota value according to its own enforcement rules. More detailed capture of the discussion on etherpad: https://etherpad.openstack.org/p/CentralizedQuotas Re this particular change, we plan to reuse this API extension code, but extended to support domain-level quota as well. -- Best regards, Oleg Gelbukh Mirantis Labs On Mon, Dec 2, 2013 at 5:39 PM, Chmouel Boudjnah chmo...@enovance.com wrote: Hello, I was wondering what was the status of Keystone being the central place across all OpenStack projects for quotas. There is already an implementation from Dmitry here : https://review.openstack.org/#/c/40568/ but hasn't seen activities since october waiting for icehouse development to be started and a few bits to be cleaned and added (i.e: the sqlite migration). It would be great if we can get this rekicked to get that for icehouse-2. Thanks, Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Store quotas in Keystone
Sorry to jump into this late and all, but I am curious. Why not borrow the concept of flavors from Nova and apply them to quotas? While it is open to interpretation and I most certainly could be wrong, the why of flavors is that you want to plan. If you know that you have flavors that are 1/4, 1/8, 1/16, 1/32, and so on the size of your standard host node then you know that if you plan for each node that 4 of the 1/4th flavor tenants in storage and CPU and 32 times 4 (or 128) possible IP addresses ... then you know your worst case infrastructure requirements. As a cloud operator, you know how many maximum IP you need and how many compute nodes you need. So in the terms of quotas, why not borrow the same concept and allow for the creation of a quota system that then allows a cloud operator to plan. That should limit the total number of quotas and make the problem space smaller and easier to deal with right? Or have I missed a lot of the conversation and should I run out and do some reading? Pointers would be welcome. # Shawn Hartsock - Original Message - From: John Dickinson m...@not.mn To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Cc: Chmouel Boudjnah chmo...@chmouel.com Sent: Tuesday, December 3, 2013 10:04:47 AM Subject: Re: [openstack-dev] [Keystone] Store quotas in Keystone How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. --John On Dec 3, 2013, at 6:53 AM, Oleg Gelbukh ogelb...@mirantis.com wrote: Chmouel, We reviewed the design of this feature at the summit with CERN and HP teams. Centralized quota storage in Keystone is an anticipated feature, but there are concerns about adding quota enforcement logic for every service to Keystone. The agreed solution is to add quota numbers storage to Keystone, and add mechanism that will notify services about change to the quota. Service, in turn, will update quota cache and apply the new quota value according to its own enforcement rules. More detailed capture of the discussion on etherpad: https://urldefense.proofpoint.com/v1/url?u=https://etherpad.openstack.org/p/CentralizedQuotask=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=U9jd8i1QXRWQEdLI1XfrWPjXsJaoGrk8w31ffdfY7Wk%3D%0Am=tzBzbXgJCXEysjYVOTpv38Q4gcaAQ%2Fi8DgWNbal6W78%3D%0As=10a3de3d5f5eeea63349193030efc95dfcd127f9275b8244b0fe02ce5e3da2ab Re this particular change, we plan to reuse this API extension code, but extended to support domain-level quota as well. -- Best regards, Oleg Gelbukh Mirantis Labs On Mon, Dec 2, 2013 at 5:39 PM, Chmouel Boudjnah chmo...@enovance.com wrote: Hello, I was wondering what was the status of Keystone being the central place across all OpenStack projects for quotas. There is already an implementation from Dmitry here : https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/%23/c/40568/k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=U9jd8i1QXRWQEdLI1XfrWPjXsJaoGrk8w31ffdfY7Wk%3D%0Am=tzBzbXgJCXEysjYVOTpv38Q4gcaAQ%2Fi8DgWNbal6W78%3D%0As=783921e458225eec7098c9755b96d5f2c45a8c744d6eec76ee30cb7a583d522b but hasn't seen activities since october waiting for icehouse development to be started and a few bits to be cleaned and added (i.e: the sqlite migration). It would be great if we can get this rekicked to get that for icehouse-2. Thanks, Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=U9jd8i1QXRWQEdLI1XfrWPjXsJaoGrk8w31ffdfY7Wk%3D%0Am=tzBzbXgJCXEysjYVOTpv38Q4gcaAQ%2Fi8DgWNbal6W78%3D%0As=a18fc7f57ed9d5071ef1aa8893069f260a9b40f75c2c405be351d7271a1b1794 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=U9jd8i1QXRWQEdLI1XfrWPjXsJaoGrk8w31ffdfY7Wk%3D%0Am=tzBzbXgJCXEysjYVOTpv38Q4gcaAQ%2Fi8DgWNbal6W78%3D%0As=a18fc7f57ed9d5071ef1aa8893069f260a9b40f75c2c405be351d7271a1b1794 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=U9jd8i1QXRWQEdLI1XfrWPjXsJaoGrk8w31ffdfY7Wk%3D%0Am=tzBzbXgJCXEysjYVOTpv38Q4gcaAQ%2Fi8DgWNbal6W78%3D%0As=a18fc7f57ed9d5071ef1aa8893069f260a9b40f75c2c405be351d7271a1b1794
Re: [openstack-dev] [Keystone] Store quotas in Keystone
On 12/03/2013 10:04 AM, John Dickinson wrote: How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. From reading below, it does not look like a centralized lookup is what the design is. A push-change strategy is what is described, where the quota numbers themselves are stored in a canonical location in Keystone, but when those numbers are changed, Keystone would send a notification of that change to subscribing services such as Swift, which would presumably have one or more levels of caching for things like account and container quotas... Best, -jay --John On Dec 3, 2013, at 6:53 AM, Oleg Gelbukh ogelb...@mirantis.com wrote: Chmouel, We reviewed the design of this feature at the summit with CERN and HP teams. Centralized quota storage in Keystone is an anticipated feature, but there are concerns about adding quota enforcement logic for every service to Keystone. The agreed solution is to add quota numbers storage to Keystone, and add mechanism that will notify services about change to the quota. Service, in turn, will update quota cache and apply the new quota value according to its own enforcement rules. More detailed capture of the discussion on etherpad: https://etherpad.openstack.org/p/CentralizedQuotas Re this particular change, we plan to reuse this API extension code, but extended to support domain-level quota as well. -- Best regards, Oleg Gelbukh Mirantis Labs On Mon, Dec 2, 2013 at 5:39 PM, Chmouel Boudjnah chmo...@enovance.com wrote: Hello, I was wondering what was the status of Keystone being the central place across all OpenStack projects for quotas. There is already an implementation from Dmitry here : https://review.openstack.org/#/c/40568/ but hasn't seen activities since october waiting for icehouse development to be started and a few bits to be cleaned and added (i.e: the sqlite migration). It would be great if we can get this rekicked to get that for icehouse-2. Thanks, Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Store quotas in Keystone
On Dec 3, 2013, at 8:05 AM, Jay Pipes jaypi...@gmail.com wrote: On 12/03/2013 10:04 AM, John Dickinson wrote: How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. From reading below, it does not look like a centralized lookup is what the design is. A push-change strategy is what is described, where the quota numbers themselves are stored in a canonical location in Keystone, but when those numbers are changed, Keystone would send a notification of that change to subscribing services such as Swift, which would presumably have one or more levels of caching for things like account and container quotas... Yes, I get that, and there are already methods in Swift to support that. The trick, though, is either (1) storing all the canonical info in Keystone and scaling that or (2) storing some boiled down version, if possible, and fanning that out to all of the resources in Swift. Both are difficult and require storing the information in the central Keystone store. Best, -jay --John On Dec 3, 2013, at 6:53 AM, Oleg Gelbukh ogelb...@mirantis.com wrote: Chmouel, We reviewed the design of this feature at the summit with CERN and HP teams. Centralized quota storage in Keystone is an anticipated feature, but there are concerns about adding quota enforcement logic for every service to Keystone. The agreed solution is to add quota numbers storage to Keystone, and add mechanism that will notify services about change to the quota. Service, in turn, will update quota cache and apply the new quota value according to its own enforcement rules. More detailed capture of the discussion on etherpad: https://etherpad.openstack.org/p/CentralizedQuotas Re this particular change, we plan to reuse this API extension code, but extended to support domain-level quota as well. -- Best regards, Oleg Gelbukh Mirantis Labs On Mon, Dec 2, 2013 at 5:39 PM, Chmouel Boudjnah chmo...@enovance.com wrote: Hello, I was wondering what was the status of Keystone being the central place across all OpenStack projects for quotas. There is already an implementation from Dmitry here : https://review.openstack.org/#/c/40568/ but hasn't seen activities since october waiting for icehouse development to be started and a few bits to be cleaned and added (i.e: the sqlite migration). It would be great if we can get this rekicked to get that for icehouse-2. Thanks, Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Store quotas in Keystone
On 12/03/2013 11:40 AM, John Dickinson wrote: On Dec 3, 2013, at 8:05 AM, Jay Pipes jaypi...@gmail.com wrote: On 12/03/2013 10:04 AM, John Dickinson wrote: How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. From reading below, it does not look like a centralized lookup is what the design is. A push-change strategy is what is described, where the quota numbers themselves are stored in a canonical location in Keystone, but when those numbers are changed, Keystone would send a notification of that change to subscribing services such as Swift, which would presumably have one or more levels of caching for things like account and container quotas... Yes, I get that, and there are already methods in Swift to support that. The trick, though, is either (1) storing all the canonical info in Keystone and scaling that or (2) storing some boiled down version, if possible, and fanning that out to all of the resources in Swift. Both are difficult and require storing the information in the central Keystone store. The storage driver for quotas in Keystone could use something like Cassandra as its data store, leaving the Keystone endpoint stateless and only responsible for relaying the update message to subscribers. Each type of thing Keystone manages -- identity, token, catalog, etc -- can have a different storage driver. Adding a new storage driver for Cassandra and its ilk would be pretty trivial. That way Keystone folks can focus on the job at hand (notifying subscribers of updates to quotas) and Cassandra developers can focus on scaling data storage and retrieval. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Store quotas in Keystone
On Dec 3, 2013 6:49 PM, John Dickinson m...@not.mn wrote: On Dec 3, 2013, at 8:05 AM, Jay Pipes jaypi...@gmail.com wrote: On 12/03/2013 10:04 AM, John Dickinson wrote: How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. From reading below, it does not look like a centralized lookup is what the design is. A push-change strategy is what is described, where the quota numbers themselves are stored in a canonical location in Keystone, but when those numbers are changed, Keystone would send a notification of that change to subscribing services such as Swift, which would presumably have one or more levels of caching for things like account and container quotas... Yes, I get that, and there are already methods in Swift to support that. The trick, though, is either (1) storing all the canonical info in Keystone and scaling that or (2) storing some boiled down version, if possible, and fanning that out to all of the resources in Swift. Both are difficult and require storing the information in the central Keystone store. If I remember correctly the motivation for using keystone for quotas is so there is one easy place to set quotas across all projects. Why not hide this complexity with the unified client instead? That has been the answer we have been using for pulling out assorted proxy APIs in nova (nova image-list volume-list) etc. Best, -jay --John On Dec 3, 2013, at 6:53 AM, Oleg Gelbukh ogelb...@mirantis.com wrote: Chmouel, We reviewed the design of this feature at the summit with CERN and HP teams. Centralized quota storage in Keystone is an anticipated feature, but there are concerns about adding quota enforcement logic for every service to Keystone. The agreed solution is to add quota numbers storage to Keystone, and add mechanism that will notify services about change to the quota. Service, in turn, will update quota cache and apply the new quota value according to its own enforcement rules. More detailed capture of the discussion on etherpad: https://etherpad.openstack.org/p/CentralizedQuotas Re this particular change, we plan to reuse this API extension code, but extended to support domain-level quota as well. -- Best regards, Oleg Gelbukh Mirantis Labs On Mon, Dec 2, 2013 at 5:39 PM, Chmouel Boudjnah chmo...@enovance.com wrote: Hello, I was wondering what was the status of Keystone being the central place across all OpenStack projects for quotas. There is already an implementation from Dmitry here : https://review.openstack.org/#/c/40568/ but hasn't seen activities since october waiting for icehouse development to be started and a few bits to be cleaned and added (i.e: the sqlite migration). It would be great if we can get this rekicked to get that for icehouse-2. Thanks, Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Store quotas in Keystone
John Dickinson wrote: How are you proposing that this integrate with Swift's account and container quotas (especially since there may be hundreds of thousands of accounts and millions (billions?) of containers in a single Swift cluster)? A centralized lookup for quotas doesn't really seem to be a scalable solution. FWIW this was added for discussion at next project/release coordination meeting (Dec 10, 21:00 UTC)... to give time for everyone to catch up with what this actually is. -- Thierry Carrez (ttx) signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
