Re: [openstack-dev] Quota management and enforcement across projects
Hi, Interesting thread. I’ve just started looking at Boson myself to manage quotas across multiple regions. I think one of the cases when having quota management as a separate service is justified, is a multi-region case [1]. 1) Some deployments can share Keystone, so there’s a need to synchronise resource usage across multiple OpenStacks and enforce quota in a distributed manner. Architecturally this will mean that there will be one Boson instance with the “quota management” and “admin” roles exposing REST API endpoint and talking REST to other client specific Boson instances. This use case can be an example of why it can be reasonable to have a separate service for managing quotas, reservations, usages. 2) As I understand it, the intention of Boson is to synchronise usage, not own this information. The “Usage synchronisation” paragraph in the wiki [3] describes one possible approach: “…Boson will keep freshness information on the usage data; should it determine that the usage information it has is not fresh enough, it will reject reservation creation requests with a special response code, which tells the service to send fresh usage information for certain resources along with resending the reservation creation requests”. Further, the service can reject reservation creation requests in case of communication failure, tracker unavailability, etc. 3) Have you looked at Blazar [3] as a possible reservation mechanism instead of implementing a new reservation interface in Boson? Does it at all make sense to use it in the context of quota management? Regards, Dimitri [1] http://dc636.4shared.com/download/Z6O_jJSGba/multiregion_arch.png?lgfp=3000 [2]https://wiki.openstack.org/wiki/Boson#Usage_Synchronization [3] https://wiki.openstack.org/wiki/Blazar From: Salvatore Orlando sorla...@nicira.commailto:sorla...@nicira.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday 15 January 2015 02:22 To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Quota management and enforcement across projects I'm resurrecting this thread to provide an update on this effort. I have been looking at Boson [1] as a base for starting developing a service for managing quotas and reservations [2]. Boson's model would satisfy most of the requirements for this service, and implementing additional requirements, such as hierarchical multi-tenancy should be quite easy (at least from the high-level design perspective). In the rest of this post I'm going to refer to this service both as Boson and quota service. Without going into too many technical details (which I am happy to discuss separately), the quota service will need to provide the 4 interfaces depicted in [3]. 1) The admin interface has the main purpose of keeping track of the services which use boson, registering resources to track, and managing their lifecycle. 2) The quota mgmt interface does pretty much what the quota extension does for many openstack project - manage resource limits per project/users, configure quota classes, etc. 3) The reservation interface handles the request/commit/cancel process supported by many Openstack projects 4) the usage interface provides abstraction to access the resource usage tracker and feed information to it. These interfaces are then implemented by the Boson object model, depicted in [4]. This object model is not very different from the one originally proposed for Boson [5] The proposed object model simplifies a bit the original one, merging the reservation and request concepts, as well as doing without the SpecificResource concept (at least for the moment). However, the proposed object model adds the Quota class concept and introduces the possibility of having child-parent relationships between Resources and User Info. The former will allow for applying quota to resources which are scoped within a parent resource (e.g.: static routes per logical router), whereas the latter should enable the hierarchical multi-tenancy use case. Keeping in mind the interfaces discussed in [3], the component diagram [6] can be devised. There is a distinct component for each interface - plus one component for DB management. Most of the interactions are, of course, with the DB manager. Component design should be done in a way that the various components are as independent as possible. There are some interactions among components, but they can likely be replaced with interactions with the DB manager component. The Boson quota service therefore represents a centralized endpoint for managing quotas, tracking resource usage, and performing resource reservation. Conceptually, this is all good; however, would it scale from an architectural perspective? The main problem with this approach
Re: [openstack-dev] Quota management and enforcement across projects
I'm resurrecting this thread to provide an update on this effort. I have been looking at Boson [1] as a base for starting developing a service for managing quotas and reservations [2]. Boson's model would satisfy most of the requirements for this service, and implementing additional requirements, such as hierarchical multi-tenancy should be quite easy (at least from the high-level design perspective). In the rest of this post I'm going to refer to this service both as Boson and quota service. Without going into too many technical details (which I am happy to discuss separately), the quota service will need to provide the 4 interfaces depicted in [3]. 1) The admin interface has the main purpose of keeping track of the services which use boson, registering resources to track, and managing their lifecycle. 2) The quota mgmt interface does pretty much what the quota extension does for many openstack project - manage resource limits per project/users, configure quota classes, etc. 3) The reservation interface handles the request/commit/cancel process supported by many Openstack projects 4) the usage interface provides abstraction to access the resource usage tracker and feed information to it. These interfaces are then implemented by the Boson object model, depicted in [4]. This object model is not very different from the one originally proposed for Boson [5] The proposed object model simplifies a bit the original one, merging the reservation and request concepts, as well as doing without the SpecificResource concept (at least for the moment). However, the proposed object model adds the Quota class concept and introduces the possibility of having child-parent relationships between Resources and User Info. The former will allow for applying quota to resources which are scoped within a parent resource (e.g.: static routes per logical router), whereas the latter should enable the hierarchical multi-tenancy use case. Keeping in mind the interfaces discussed in [3], the component diagram [6] can be devised. There is a distinct component for each interface - plus one component for DB management. Most of the interactions are, of course, with the DB manager. Component design should be done in a way that the various components are as independent as possible. There are some interactions among components, but they can likely be replaced with interactions with the DB manager component. The Boson quota service therefore represents a centralized endpoint for managing quotas, tracking resource usage, and performing resource reservation. Conceptually, this is all good; however, would it scale from an architectural perspective? The main problem with this approach indeed is that the quota service itself can become a huge bottleneck and add more latency to resource processing. This problem can probably be solved looking at the above mentioned interfaces as roles, and expecting Boson to scale horizontally via deployment of several instances of the service which can implement different roles, some of which can be dedicated to specific services. For instance, let's consider the example architecture depicted in [7]. Here there are 3 boson instances. One instance has the quota management and admin role and takes care of registering resources for enabled services, and setting/retrieving resource limits. It might expose its own REST API endpoint over HTTP to this aim. The other two instances have the reservation and usage roles, and communicate with client applications over AMQP. However one instance will listen only to neutron messages and the other only to nova messages (nova and neutron have been used merely as an example, the same applies to any project). This architecture will enable some sort of horizontal scale out as every project will then have its own instance for tracking resources and managing reservations. However, there will still be need for some communication across the various Boson instances. For instance in order to make a reservation, current resource limits should be fetched interacting with the management component or from the database. This overhead can be reduced considerably by introducing caches, which would result particularly effective for data such as resource limits which do not tend to change a lot over time. Finally, it is also worth noting that in theory nothing prevents us from running Boson's management interface as a Keystone 3rd party component and therefore running its API on the Keystone endpoint. Summarising, I hope I managed to convey the essentials of where I think Boson should be place in the OpenStack ecosystem, and how would it work from a conceptual and architectural perspective. I am aware that this is not exhaustive at al, but is probably the best that can be achieved in a mailing list post. Nevertheless, there are still some fundamental questions that I am trying to answer in a convincing way. 1) would anyone ever deploy a service aimed exclusively at managing quotas and
Re: [openstack-dev] Quota management and enforcement across projects
On 11/17/2014 05:49 PM, Salvatore Orlando wrote: Hi all, I am resuming this thread following the session we had at the summit in Paris (etherpad here [1]) While there was some sort of consensus regarding what this library should do, and how it should do it, the session ended with some open questions which we need to address before finalising the specification. There was a rather large consensus that the only viable architecture would be one where the quota management library owns resource usage data. However, this raises further concerns around: - ownership of resource usage records. As the quota library owns usage data, it becomes the authoritative source of truth for it. This could be problematic in some projects, particularly nova, where the resource tracker currently own usage data. Well, just to be clear... the resource tracker in Nova owns the usage records for the compute node, not the usage records for a tenant or user (the quota driver and DB tables own that data). Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
Le 18/11/2014 20:05, Doug Hellmann a écrit : On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote: I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. I'll also point out that it was largely because of the storage needs that I chose to propose Boson[1] as a separate app, rather than as a library. Further, the dimensions over which quota-covered resources needed to be tracked seemed to me to be complicated enough that it would be better to define a new app and make it support that one domain well, which is why I didn't propose it as something to add to Keystone. Consider: nova has quotas that are applied by user, other quotas that are applied by tenant, and even some quotas on what could be considered sub-resources—a limit on the number of security group rules per security group, for instance. My current feeling is that, if we can figure out a way to make the quota problem into an acceptable library, that will work; it would probably have to maintain its own database separate from the client app and have features for automatically managing the schema, since we couldn't necessarily rely on the client app to invoke the proper juju there. If, on the other hand, that ends up failing, then the best route is probably to begin by developing a separate app, like Boson, as a PoC; then, after we have some idea of just how difficult it is to actually solve the problem, we can evaluate whether it makes sense to actually fold it into a service like Keystone, or whether it should stand on its own. (Personally, I think Boson should be created and should stand on its own, but I also envision using it for purposes outside of OpenStack…) Thanks for mentioning Boson again. I’m embarrassed that I completely forgot about the fact that you mentioned this at the summit. I’ll have to look at the proposal more closely before I comment in any detail, but I take it as a good sign that we’re coming back around to the idea of solving this with an app instead of a library. I assume I'm really late in the thread so I can just sit and give +1 to this direction : IMHO, quotas need to managed thanks to a CRUD interface which implies to get an app, as it sounds unreasonable to extend each consumer app API. That said, back to Blazar, I just would like to emphasize that Blazar is not trying to address the quota enforcement level, but rather provide a centralized endpoint for managing reservations. Consequently, Blazar can also be considered as a consumer of this quota system, whatever it's in a library or on a separate REST API. Last thing, I don't think that a quota application necessarly means that quotas enforcement should be managed thanks to external calls to this app. I can rather see an external system able to set for each project a local view of what should be enforced locally. If operators don't want to deploy that quota management project, it's up to them to address the hetergenous setups for each project. My 2 cts (too), -Sylvain Doug Just my $.02… [1] https://wiki.openstack.org/wiki/Boson -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
All I can say at the moment is that Usage and Quota management is a crappy thing to do in OpenStack. Every service has it's own way of doing it both in clients and api's. +n+ for making a effort in standardizing this thing in a way that could be alike across projects.. 2014-11-19 14:33 GMT+01:00 Sylvain Bauza sba...@redhat.com: Le 18/11/2014 20:05, Doug Hellmann a écrit : On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote: I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. I'll also point out that it was largely because of the storage needs that I chose to propose Boson[1] as a separate app, rather than as a library. Further, the dimensions over which quota-covered resources needed to be tracked seemed to me to be complicated enough that it would be better to define a new app and make it support that one domain well, which is why I didn't propose it as something to add to Keystone. Consider: nova has quotas that are applied by user, other quotas that are applied by tenant, and even some quotas on what could be considered sub-resources—a limit on the number of security group rules per security group, for instance. My current feeling is that, if we can figure out a way to make the quota problem into an acceptable library, that will work; it would probably have to maintain its own database separate from the client app and have features for automatically managing the schema, since we couldn't necessarily rely on the client app to invoke the proper juju there. If, on the other hand, that ends up failing, then the best route is probably to begin by developing a separate app, like Boson, as a PoC; then, after we have some idea of just how difficult it is to actually solve the problem, we can evaluate whether it makes sense to actually fold it into a service like Keystone, or whether it should stand on its own. (Personally, I think Boson should be created and should stand on its own, but I also envision using it for purposes outside of OpenStack…) Thanks for mentioning Boson again. I’m embarrassed that I completely forgot about the fact that you mentioned this at the summit. I’ll have to look at the proposal more closely before I comment in any detail, but I take it as a good sign that we’re coming back around to the idea of solving this with an app instead of a library. I assume I'm really late in the thread so I can just sit and give +1 to this direction : IMHO, quotas need to managed thanks to a CRUD interface which implies to get an app, as it sounds unreasonable to extend each consumer app API. That said, back to Blazar, I just would like to emphasize that Blazar is not trying to address the quota enforcement level, but rather provide a centralized endpoint for managing reservations. Consequently, Blazar can also be considered as a consumer of this quota system, whatever it's in a library or on a separate REST API. Last thing, I don't think that a quota application necessarly means that quotas enforcement should be managed thanks to external calls to this app. I can rather see an external system able to set for each project a local view of what should be enforced locally. If operators don't want to deploy that quota management project, it's up to them to address the hetergenous setups for each project. My 2 cts (too), -Sylvain Doug Just my $.02… [1] https://wiki.openstack.org/wiki/Boson -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
On Nov 19, 2014, at 8:33 AM, Sylvain Bauza sba...@redhat.com wrote: Le 18/11/2014 20:05, Doug Hellmann a écrit : On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote: I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. I'll also point out that it was largely because of the storage needs that I chose to propose Boson[1] as a separate app, rather than as a library. Further, the dimensions over which quota-covered resources needed to be tracked seemed to me to be complicated enough that it would be better to define a new app and make it support that one domain well, which is why I didn't propose it as something to add to Keystone. Consider: nova has quotas that are applied by user, other quotas that are applied by tenant, and even some quotas on what could be considered sub-resources—a limit on the number of security group rules per security group, for instance. My current feeling is that, if we can figure out a way to make the quota problem into an acceptable library, that will work; it would probably have to maintain its own database separate from the client app and have features for automatically managing the schema, since we couldn't necessarily rely on the client app to invoke the proper juju there. If, on the other hand, that ends up failing, then the best route is probably to begin by developing a separate app, like Boson, as a PoC; then, after we have some idea of just how difficult it is to actually solve the problem, we can evaluate whether it makes sense to actually fold it into a service like Keystone, or whether it should stand on its own. (Personally, I think Boson should be created and should stand on its own, but I also envision using it for purposes outside of OpenStack…) Thanks for mentioning Boson again. I’m embarrassed that I completely forgot about the fact that you mentioned this at the summit. I’ll have to look at the proposal more closely before I comment in any detail, but I take it as a good sign that we’re coming back around to the idea of solving this with an app instead of a library. I assume I'm really late in the thread so I can just sit and give +1 to this direction : IMHO, quotas need to managed thanks to a CRUD interface which implies to get an app, as it sounds unreasonable to extend each consumer app API. That said, back to Blazar, I just would like to emphasize that Blazar is not trying to address the quota enforcement level, but rather provide a centralized endpoint for managing reservations. Consequently, Blazar can also be considered as a consumer of this quota system, whatever it's in a library or on a separate REST API. Last thing, I don't think that a quota application necessarly means that quotas enforcement should be managed thanks to external calls to this app. I can rather see an external system able to set for each project a local view of what should be enforced locally. If operators don't want to deploy that quota management project, it's up to them to address the hetergenous setups for each project. I’m not sure what this means. You want the new service to be optional? How would apps written against the service find and manage quota data if the service isn’t there? Doug My 2 cts (too), -Sylvain Doug Just my $.02… [1] https://wiki.openstack.org/wiki/Boson -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list
Re: [openstack-dev] Quota management and enforcement across projects
Le 19/11/2014 15:06, Doug Hellmann a écrit : On Nov 19, 2014, at 8:33 AM, Sylvain Bauza sba...@redhat.com wrote: Le 18/11/2014 20:05, Doug Hellmann a écrit : On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote: I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. I'll also point out that it was largely because of the storage needs that I chose to propose Boson[1] as a separate app, rather than as a library. Further, the dimensions over which quota-covered resources needed to be tracked seemed to me to be complicated enough that it would be better to define a new app and make it support that one domain well, which is why I didn't propose it as something to add to Keystone. Consider: nova has quotas that are applied by user, other quotas that are applied by tenant, and even some quotas on what could be considered sub-resources—a limit on the number of security group rules per security group, for instance. My current feeling is that, if we can figure out a way to make the quota problem into an acceptable library, that will work; it would probably have to maintain its own database separate from the client app and have features for automatically managing the schema, since we couldn't necessarily rely on the client app to invoke the proper juju there. If, on the other hand, that ends up failing, then the best route is probably to begin by developing a separate app, like Boson, as a PoC; then, after we have some idea of just how difficult it is to actually solve the problem, we can evaluate whether it makes sense to actually fold it into a service like Keystone, or whether it should stand on its own. (Personally, I think Boson should be created and should stand on its own, but I also envision using it for purposes outside of OpenStack…) Thanks for mentioning Boson again. I’m embarrassed that I completely forgot about the fact that you mentioned this at the summit. I’ll have to look at the proposal more closely before I comment in any detail, but I take it as a good sign that we’re coming back around to the idea of solving this with an app instead of a library. I assume I'm really late in the thread so I can just sit and give +1 to this direction : IMHO, quotas need to managed thanks to a CRUD interface which implies to get an app, as it sounds unreasonable to extend each consumer app API. That said, back to Blazar, I just would like to emphasize that Blazar is not trying to address the quota enforcement level, but rather provide a centralized endpoint for managing reservations. Consequently, Blazar can also be considered as a consumer of this quota system, whatever it's in a library or on a separate REST API. Last thing, I don't think that a quota application necessarly means that quotas enforcement should be managed thanks to external calls to this app. I can rather see an external system able to set for each project a local view of what should be enforced locally. If operators don't want to deploy that quota management project, it's up to them to address the hetergenous setups for each project. I’m not sure what this means. You want the new service to be optional? How would apps written against the service find and manage quota data if the service isn’t there? My bad. Let me rephrase it. I'm seeing this service as providing added value for managing quotas by ensuring consistency across all projects. But as I said, I'm also thinking that the quota enforcement has still to be done at the customer project level. So, I can imagine a client (or a Facade if you prefer) providing quota resources to the customer app which could be either fetched (thru some caching) from the service, or directly taken from the existing quota DB. In order to do that, I could imagine those steps : #1 : customer app makes use of oslo.quota for managing its own quota resources #2 : the external app
Re: [openstack-dev] Quota management and enforcement across projects
On Nov 19, 2014, at 9:51 AM, Sylvain Bauza sba...@redhat.com wrote: Le 19/11/2014 15:06, Doug Hellmann a écrit : On Nov 19, 2014, at 8:33 AM, Sylvain Bauza sba...@redhat.com wrote: Le 18/11/2014 20:05, Doug Hellmann a écrit : On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote: I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. I'll also point out that it was largely because of the storage needs that I chose to propose Boson[1] as a separate app, rather than as a library. Further, the dimensions over which quota-covered resources needed to be tracked seemed to me to be complicated enough that it would be better to define a new app and make it support that one domain well, which is why I didn't propose it as something to add to Keystone. Consider: nova has quotas that are applied by user, other quotas that are applied by tenant, and even some quotas on what could be considered sub-resources—a limit on the number of security group rules per security group, for instance. My current feeling is that, if we can figure out a way to make the quota problem into an acceptable library, that will work; it would probably have to maintain its own database separate from the client app and have features for automatically managing the schema, since we couldn't necessarily rely on the client app to invoke the proper juju there. If, on the other hand, that ends up failing, then the best route is probably to begin by developing a separate app, like Boson, as a PoC; then, after we have some idea of just how difficult it is to actually solve the problem, we can evaluate whether it makes sense to actually fold it into a service like Keystone, or whether it should stand on its own. (Personally, I think Boson should be created and should stand on its own, but I also envision using it for purposes outside of OpenStack…) Thanks for mentioning Boson again. I’m embarrassed that I completely forgot about the fact that you mentioned this at the summit. I’ll have to look at the proposal more closely before I comment in any detail, but I take it as a good sign that we’re coming back around to the idea of solving this with an app instead of a library. I assume I'm really late in the thread so I can just sit and give +1 to this direction : IMHO, quotas need to managed thanks to a CRUD interface which implies to get an app, as it sounds unreasonable to extend each consumer app API. That said, back to Blazar, I just would like to emphasize that Blazar is not trying to address the quota enforcement level, but rather provide a centralized endpoint for managing reservations. Consequently, Blazar can also be considered as a consumer of this quota system, whatever it's in a library or on a separate REST API. Last thing, I don't think that a quota application necessarly means that quotas enforcement should be managed thanks to external calls to this app. I can rather see an external system able to set for each project a local view of what should be enforced locally. If operators don't want to deploy that quota management project, it's up to them to address the hetergenous setups for each project. I’m not sure what this means. You want the new service to be optional? How would apps written against the service find and manage quota data if the service isn’t there? My bad. Let me rephrase it. I'm seeing this service as providing added value for managing quotas by ensuring consistency across all projects. But as I said, I'm also thinking that the quota enforcement has still to be done at the customer project level. Oh, yes, that is true. I envision the API for the new service having a call that means “try to consume X units of a given quota” and that it would return information about whether that can be done. The apps would have to define what
Re: [openstack-dev] Quota management and enforcement across projects
On 20 November 2014 05:25, openstack-dev-requ...@lists.openstack.org wrote: -- Message: 24 Date: Wed, 19 Nov 2014 10:57:17 -0500 From: Doug Hellmann d...@doughellmann.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Quota management and enforcement across projects Message-ID: 13f4f7a1-d4ec-4d14-a163-d477a4fd9...@doughellmann.com Content-Type: text/plain; charset=windows-1252 On Nov 19, 2014, at 9:51 AM, Sylvain Bauza sba...@redhat.com wrote: My bad. Let me rephrase it. I'm seeing this service as providing added value for managing quotas by ensuring consistency across all projects. But as I said, I'm also thinking that the quota enforcement has still to be done at the customer project level. Oh, yes, that is true. I envision the API for the new service having a call that means ?try to consume X units of a given quota? and that it would return information about whether that can be done. The apps would have to define what quotas they care about, and make the appropriate calls. For actions initiated directly through core OpenStack service APIs (Nova, Cinder, Neutron, etc - anything using Keystone policy), shouldn't quota-enforcement be handled by Keystone? To me this is just a subset of authz, and OpenStack already has a well established service for such decisions. It sounds like the idea here is to provide something generic that could be used outside of OpenStack? I worry that might be premature scope creep that detracts from the outcome. -- Cheers, ~Blairo ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
On Thu, 2014-11-20 at 10:16 +1100, Blair Bethwaite wrote: For actions initiated directly through core OpenStack service APIs (Nova, Cinder, Neutron, etc - anything using Keystone policy), shouldn't quota-enforcement be handled by Keystone? To me this is just a subset of authz, and OpenStack already has a well established service for such decisions. If you look a little earlier in the thread, you will find a post from me where I point out just how complicated quota management actually is. I suggest that it should be developed as a proof-of-concept as a separate service; from there, we can see whether it makes sense to roll it into Keystone or maintain it as a separate thing. -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
Apparently, like everything Openstack-y we have a gathered a good crowd of people with different opinions, more or less different, more or less strong. My only strong opinion is that any ocean-boiling attempt should be carefully avoided, and any proposed approach should add as little as possible in terms of delays and complexity to the current process. Anyway, I am glad nobody said so far QaaS. If you're thinking to do so, please don't. Please. The reason for the now almost-abandoned proposal for a library was to have a simple solution for allowing every project to enforce quotas in the same ways. As Kevin said, managing quotas for resources and their sequence attributes (like the sec group rules for instance), can become pretty messy soon. As of today this problem is avoided since quota management logic is baked in every project's business logic. The library proposal also shamefully avoided this problem - by focusing on enforcement only. For enforcement, as Doug said, this is about answering the question can I consume X units of a given resource?. In a distributed, non-locking, world, this is not really obvious. A Boson-like proposal would solve this problem by providing a centralised enforcement point. In this case the only possible approach is one in which such centralised endpoint does both enforcement and management. While I too think that we need a centralised quota management endpoint, be it folded in Keystone or not, I still need to be convinced that this is the right architectural decision for enforcing quotas. First, can I consume X unit of a given resource implies the external service should be aware of current resource usage. Arguably it's the consumer service (eg: nova, cinder) that owns this information, not the quota service. The quota service can however be made aware of this in several ways: - notifications. In this case the service will also become a consumer of a telemetry service like ceilometer or stacktach. I'm not sure we want to add that dependency - the API call itself might ask for reserving a given amount resource and communicate at the same time resource usage. This sounds better, but why would it be better than the dual approach, in which the consumer service makes the reservation using quota values fetched by an external service. Between quota and usage, the latter seem to be the more dynamic ones; quotas can also be easily cached. - usage info might be updated when reservations are committed. This looks smart, but requirers more thinking for dealing with failure scenarios where the failure occurs once resources are reserved but before they're committed and there is no way to know which ones were actually and which ones not. Second, calling an external service is not cheap. Especially since those calls might be done several times for completing a single API request. Think about POST /servers with all the ramifications into glance, neutron, and cinder. In the simplest case we'd need two calls - one to reserve a resource (or a set of resources), and one to either confirm or cancel the reservation. This alone adds two round trips for each API call. Could this scale well? To some extent it reminds me of the mayhem that was the nova/neutron library before caching for keystone tokens was fixed. Somebody mentioned an analogy with AuthZ. For some reason, perhaps different from the reasons of who originally brought the analogy, I also believe that this is a decent model to follow. Quotas for each resource, together with the resource structure, might be stored by a centralised service. This is pretty much inline with Boson's goals, if I understand correctly. It will expose tenant-facing REST APIs for quota management and will finally solve the problem of having a standardized way for handling quotas, regardless of whether it stand as its own service or is folded into Keystone, operators should be happier. And there's nothing more satisfying that the face of a happy operator... provided we also do not break backward compatibility! Enforcement is something that in my opinion should happen locally, just like we do authZ locally using information provided by Keystone upon authentication. The issue Doug correctly pointed out however is that quota enforcement is a too complex to be implemented in a library, like, for instance, oslo.policy. So, assuming there might consensus on doing quota enforcement locally in the consumer application, what should be the 'thing' that enforces policy if it can't be a library? Could it be a quota agent which communicates with quota server, caches quota info? If yes, APIs like those for managing reservation, over which transport should be invoked? IPC, AMQP, REST? I need to go back to the drawing board to see if this can possibly work, but in the meanwhile your feedback is more than welcome. Salvatore On 20 November 2014 00:39, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Thu, 2014-11-20 at 10:16 +1100, Blair Bethwaite wrote: For
Re: [openstack-dev] Quota management and enforcement across projects
+1 There are lots of concurrent requests to the quota management service if it's shared for projects, especially if it's shared for multi-regions (KeyStone can be global service for multi-regions), also latency will affect the end user experience. POC is good idea to verify the concept. Best Regards Chaoyi Huang ( Joe Huang ) -Original Message- From: Kevin L. Mitchell [mailto:kevin.mitch...@rackspace.com] Sent: Thursday, November 20, 2014 7:40 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Quota management and enforcement across projects On Thu, 2014-11-20 at 10:16 +1100, Blair Bethwaite wrote: For actions initiated directly through core OpenStack service APIs (Nova, Cinder, Neutron, etc - anything using Keystone policy), shouldn't quota-enforcement be handled by Keystone? To me this is just a subset of authz, and OpenStack already has a well established service for such decisions. If you look a little earlier in the thread, you will find a post from me where I point out just how complicated quota management actually is. I suggest that it should be developed as a proof-of-concept as a separate service; from there, we can see whether it makes sense to roll it into Keystone or maintain it as a separate thing. -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote: I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. I'll also point out that it was largely because of the storage needs that I chose to propose Boson[1] as a separate app, rather than as a library. Further, the dimensions over which quota-covered resources needed to be tracked seemed to me to be complicated enough that it would be better to define a new app and make it support that one domain well, which is why I didn't propose it as something to add to Keystone. Consider: nova has quotas that are applied by user, other quotas that are applied by tenant, and even some quotas on what could be considered sub-resources—a limit on the number of security group rules per security group, for instance. My current feeling is that, if we can figure out a way to make the quota problem into an acceptable library, that will work; it would probably have to maintain its own database separate from the client app and have features for automatically managing the schema, since we couldn't necessarily rely on the client app to invoke the proper juju there. If, on the other hand, that ends up failing, then the best route is probably to begin by developing a separate app, like Boson, as a PoC; then, after we have some idea of just how difficult it is to actually solve the problem, we can evaluate whether it makes sense to actually fold it into a service like Keystone, or whether it should stand on its own. (Personally, I think Boson should be created and should stand on its own, but I also envision using it for purposes outside of OpenStack…) Thanks for mentioning Boson again. I’m embarrassed that I completely forgot about the fact that you mentioned this at the summit. I’ll have to look at the proposal more closely before I comment in any detail, but I take it as a good sign that we’re coming back around to the idea of solving this with an app instead of a library. Doug Just my $.02… [1] https://wiki.openstack.org/wiki/Boson -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
Hi all, I am resuming this thread following the session we had at the summit in Paris (etherpad here [1]) While there was some sort of consensus regarding what this library should do, and how it should do it, the session ended with some open questions which we need to address before finalising the specification. There was a rather large consensus that the only viable architecture would be one where the quota management library owns resource usage data. However, this raises further concerns around: - ownership of resource usage records. As the quota library owns usage data, it becomes the authoritative source of truth for it. This could be problematic in some projects, particularly nova, where the resource tracker currently own usage data. - inserting quota-related db migrations in the consumer project timeline. We'll need to solve this for both sqlalchemy.migrate and alembic (I seem to recall Doug has a solution for this) While the proposed interface for quota enforcing was generally well received, another open question is whether this library should also be responsible for quota management. Each project currently exposes its own extension for setting quotas for tenants, and this is rather bad from an usability perspective. whether this should be responsibility of the quota library, handle by a separate quota service, or handled through keystone is debatable. Personally I think the library should not get in the business of defining APIs. However, it can define the business logic for managing tenant quotas which can then be leveraged by the projects which will adopt it. Either way, I reckon that we might be better off by limiting our scope to quota enforcement for the first iteration of this library. The last, and possibly most important question, is whether this library belongs to the oslo-incubator or should live into its own repository. I would personally prefer the oslo incubator because its process makes adoption by OpenStack projects fairly simple. On the other hand, I also understand the concerns around a library that comes with its own data models and has the potential of becoming something bigger than a simple library. I would like to defer an answer to this question to the oslo core team, which are surely a lot more experienced than me on the matter of the incubator. To this aim, the proposed specification [2] has been temporarily put in WIP. A few more notes: 1) We agreed that we won't be constrained about saving any part of the current module, since it's not used by any project. We will however reuse whatever can be reused (mostly for not wasting time by writing the same thing again) 2) As concerns data backends to support, we agreed that we would start with the simplest possible solution. A single SQLalchemy-backed data source. Potentially we won't even have a framework for plugging different kind of data sources. However if contributors step in with support for multiple backends, we will consider it. 3) The idea of having a quota engine leveraging data sources was quite welcomed. Ideally it's not that different from the engine/driver model in the current module, but there will be a cleaner separation of responsibilities. Salvatore [1] https://etherpad.openstack.org/p/kilo-oslo-common-quota-library [2] https://review.openstack.org/#/c/132127/ On 15 October 2014 20:54, Doug Hellmann d...@doughellmann.com wrote: We can make it support whatever we want! :-) The existing code almost certainly predates domain support in keystone, so I think this is a case of needing to catch up not a conscious decision to leave it out. Doug On Oct 15, 2014, at 12:15 PM, Ajaya Agrawal ajku@gmail.com wrote: Hi, Would the new library support quota on domain level also? As it stands in oslo-incubator it only does quota enforcement on project level only. The use case for this is quota enforcement across multiple projects. For e.g. as a cloud provider, I would like my customer to create only #X volumes across all his projects. -Ajaya Cheers, Ajaya On Wed, Oct 15, 2014 at 7:04 PM, Doug Hellmann d...@doughellmann.com wrote: Sigh. Because I typed the wrong command. Thanks for pointing that out. I don’t see any instances of “quota” in openstack-common.conf files: $ grep quota */openstack-common.conf or in any projects under openstack/“: $ ls */*/openstack/common/quota.py ls: cannot access */*/openstack/common/quota.py: No such file or directory I don’t know where manila’s copy came from, but if it has been copied from the incubator by hand and then changed we should fix that up. Doug On Oct 15, 2014, at 5:28 AM, Valeriy Ponomaryov vponomar...@mirantis.com wrote: But why policy is being discussed on quota thread? On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov vponomar...@mirantis.com wrote: Manila project does use policy common code from incubator. Our small wrapper for it:
Re: [openstack-dev] Quota management and enforcement across projects
On Nov 17, 2014, at 5:49 PM, Salvatore Orlando sorla...@nicira.com wrote: Hi all, I am resuming this thread following the session we had at the summit in Paris (etherpad here [1]) While there was some sort of consensus regarding what this library should do, and how it should do it, the session ended with some open questions which we need to address before finalising the specification. There was a rather large consensus that the only viable architecture would be one where the quota management library owns resource usage data. However, this raises further concerns around: - ownership of resource usage records. As the quota library owns usage data, it becomes the authoritative source of truth for it. This could be problematic in some projects, particularly nova, where the resource tracker currently own usage data. - inserting quota-related db migrations in the consumer project timeline. We'll need to solve this for both sqlalchemy.migrate and alembic (I seem to recall Doug has a solution for this) While the proposed interface for quota enforcing was generally well received, another open question is whether this library should also be responsible for quota management. Each project currently exposes its own extension for setting quotas for tenants, and this is rather bad from an usability perspective. whether this should be responsibility of the quota library, handle by a separate quota service, or handled through keystone is debatable. Personally I think the library should not get in the business of defining APIs. However, it can define the business logic for managing tenant quotas which can then be leveraged by the projects which will adopt it. Either way, I reckon that we might be better off by limiting our scope to quota enforcement for the first iteration of this library. The last, and possibly most important question, is whether this library belongs to the oslo-incubator or should live into its own repository. I would personally prefer the oslo incubator because its process makes adoption by OpenStack projects fairly simple. On the other hand, I also understand the concerns around a library that comes with its own data models and has the potential of becoming something bigger than a simple library. I would like to defer an answer to this question to the oslo core team, which are surely a lot more experienced than me on the matter of the incubator. I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. Doug To this aim, the proposed specification [2] has been temporarily put in WIP. A few more notes: 1) We agreed that we won't be constrained about saving any part of the current module, since it's not used by any project. We will however reuse whatever can be reused (mostly for not wasting time by writing the same thing again) 2) As concerns data backends to support, we agreed that we would start with the simplest possible solution. A single SQLalchemy-backed data source. Potentially we won't even have a framework for plugging different kind of data sources. However if contributors step in with support for multiple backends, we will consider it. 3) The idea of having a quota engine leveraging data sources was quite welcomed. Ideally it's not that different from the engine/driver model in the current module, but there will be a cleaner separation of responsibilities. Salvatore [1] https://etherpad.openstack.org/p/kilo-oslo-common-quota-library [2] https://review.openstack.org/#/c/132127/ On 15 October 2014 20:54, Doug Hellmann d...@doughellmann.com wrote: We can make it support whatever we want! :-) The existing code almost certainly predates domain support in keystone, so I think this is a case of needing to catch up not a conscious decision to leave it out. Doug On Oct 15, 2014, at 12:15 PM, Ajaya Agrawal ajku@gmail.com wrote: Hi, Would the new
Re: [openstack-dev] Quota management and enforcement across projects
On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote: I’ve spent a bit of time thinking about the resource ownership issue. The challenge there is we don’t currently have any libraries that define tables in the schema of an application. I think that’s a good pattern to maintain, since it avoids introducing a lot of tricky issues like how to manage migrations for the library, how to ensure they are run by the application, etc. The fact that this common quota thing needs to store some data in a schema that it controls says to me that it is really an app and not a library. Making the quota manager an app solves the API definition issue, too, since we can describe a generic way to configure quotas and other applications can then use that API to define specific rules using the quota manager’s API. I don’t know if we need a new application or if it would make sense to, as with policy, add quota management features to keystone. A single well-defined app has some appeal, but there’s also a certain amount of extra ramp-up time needed to go that route that we wouldn’t need if we added the features directly to keystone. I'll also point out that it was largely because of the storage needs that I chose to propose Boson[1] as a separate app, rather than as a library. Further, the dimensions over which quota-covered resources needed to be tracked seemed to me to be complicated enough that it would be better to define a new app and make it support that one domain well, which is why I didn't propose it as something to add to Keystone. Consider: nova has quotas that are applied by user, other quotas that are applied by tenant, and even some quotas on what could be considered sub-resources—a limit on the number of security group rules per security group, for instance. My current feeling is that, if we can figure out a way to make the quota problem into an acceptable library, that will work; it would probably have to maintain its own database separate from the client app and have features for automatically managing the schema, since we couldn't necessarily rely on the client app to invoke the proper juju there. If, on the other hand, that ends up failing, then the best route is probably to begin by developing a separate app, like Boson, as a PoC; then, after we have some idea of just how difficult it is to actually solve the problem, we can evaluate whether it makes sense to actually fold it into a service like Keystone, or whether it should stand on its own. (Personally, I think Boson should be created and should stand on its own, but I also envision using it for purposes outside of OpenStack…) Just my $.02… [1] https://wiki.openstack.org/wiki/Boson -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
Manila project does use policy common code from incubator. Our small wrapper for it: https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com wrote: Doug, I totally agree with your findings on the policy module. Neutron already has some customizations there and we already have a few contributors working on syncing it back with oslo-incubator during the Kilo release cycle. However, my query was about the quota module. From what I gather it seems not a lot of projects use it: $ find . -name openstack-common.conf | xargs grep quota $ Salvatore On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote: On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote: Hi Doug, do you know if the existing quota oslo-incubator module has already some active consumers? In the meanwhile I've pushed a spec to neutron-specs for improving quota management there [1] It looks like a lot of projects are syncing the module: $ grep policy */openstack-common.conf barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy ceilometer/openstack-common.conf:module=policy cinder/openstack-common.conf:module=policy designate/openstack-common.conf:module=policy gantt/openstack-common.conf:module=policy glance/openstack-common.conf:module=policy heat/openstack-common.conf:module=policy horizon/openstack-common.conf:module=policy ironic/openstack-common.conf:module=policy keystone/openstack-common.conf:module=policy manila/openstack-common.conf:module=policy neutron/openstack-common.conf:module=policy nova/openstack-common.conf:module=policy trove/openstack-common.conf:module=policy tuskar/openstack-common.conf:module=policy I’m not sure how many are actively using it, but I wouldn’t expect them to copy it in if they weren’t using it at all. Now, I can either work on the oslo-incubator module and leverage it in Neutron, or develop the quota module in Neutron, and move it to oslo-incubator once we validate it with Neutron. The latter approach seems easier from a workflow perspective - as it avoid the intermediate steps of moving code from oslo-incubator to neutron. On the other hand it will delay adoption in oslo-incubator. The policy module is up for graduation this cycle. It may end up in its own library, to allow us to build a review team for the code more easily than if we put it in with some of the other semi-related modules like the server code. We’re still working that out [1], and if you expect to make a lot of incompatible changes we should delay graduation to make that simpler. Either way, since we have so many consumers, I think it would be easier to have the work happen in Oslo somewhere so we can ensure those changes are useful to and usable by all of the existing consumers. Doug [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals What's your opinion? Regards, Salvatore [1] https://review.openstack.org/#/c/128318/ On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote: On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote: Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py — dims If someone wants to drive creating a useful library during kilo, please consider adding the topic to the etherpad we’re using to plan summit sessions and then come participate in the Oslo meeting this Friday 16:00 UTC. https://etherpad.openstack.org/p/kilo-oslo-summit-topics Doug On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. This is what I had in mind as well. A simple library for quota enforcement which can be used regardless of where and how you do it, which might depend on the application business logic, the WSGI framework in use, or other factors. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have
Re: [openstack-dev] Quota management and enforcement across projects
But why policy is being discussed on quota thread? On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov vponomar...@mirantis.com wrote: Manila project does use policy common code from incubator. Our small wrapper for it: https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com wrote: Doug, I totally agree with your findings on the policy module. Neutron already has some customizations there and we already have a few contributors working on syncing it back with oslo-incubator during the Kilo release cycle. However, my query was about the quota module. From what I gather it seems not a lot of projects use it: $ find . -name openstack-common.conf | xargs grep quota $ Salvatore On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote: On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote: Hi Doug, do you know if the existing quota oslo-incubator module has already some active consumers? In the meanwhile I've pushed a spec to neutron-specs for improving quota management there [1] It looks like a lot of projects are syncing the module: $ grep policy */openstack-common.conf barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy ceilometer/openstack-common.conf:module=policy cinder/openstack-common.conf:module=policy designate/openstack-common.conf:module=policy gantt/openstack-common.conf:module=policy glance/openstack-common.conf:module=policy heat/openstack-common.conf:module=policy horizon/openstack-common.conf:module=policy ironic/openstack-common.conf:module=policy keystone/openstack-common.conf:module=policy manila/openstack-common.conf:module=policy neutron/openstack-common.conf:module=policy nova/openstack-common.conf:module=policy trove/openstack-common.conf:module=policy tuskar/openstack-common.conf:module=policy I’m not sure how many are actively using it, but I wouldn’t expect them to copy it in if they weren’t using it at all. Now, I can either work on the oslo-incubator module and leverage it in Neutron, or develop the quota module in Neutron, and move it to oslo-incubator once we validate it with Neutron. The latter approach seems easier from a workflow perspective - as it avoid the intermediate steps of moving code from oslo-incubator to neutron. On the other hand it will delay adoption in oslo-incubator. The policy module is up for graduation this cycle. It may end up in its own library, to allow us to build a review team for the code more easily than if we put it in with some of the other semi-related modules like the server code. We’re still working that out [1], and if you expect to make a lot of incompatible changes we should delay graduation to make that simpler. Either way, since we have so many consumers, I think it would be easier to have the work happen in Oslo somewhere so we can ensure those changes are useful to and usable by all of the existing consumers. Doug [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals What's your opinion? Regards, Salvatore [1] https://review.openstack.org/#/c/128318/ On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote: On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote: Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py — dims If someone wants to drive creating a useful library during kilo, please consider adding the topic to the etherpad we’re using to plan summit sessions and then come participate in the Oslo meeting this Friday 16:00 UTC. https://etherpad.openstack.org/p/kilo-oslo-summit-topics Doug On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. This is what I had in mind as well. A simple library for quota enforcement which can be used regardless of where and how you do it, which might depend on the application business logic, the WSGI framework in use, or other factors. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from
Re: [openstack-dev] Quota management and enforcement across projects
Sigh. Because I typed the wrong command. Thanks for pointing that out. I don’t see any instances of “quota” in openstack-common.conf files: $ grep quota */openstack-common.conf or in any projects under openstack/“: $ ls */*/openstack/common/quota.py ls: cannot access */*/openstack/common/quota.py: No such file or directory I don’t know where manila’s copy came from, but if it has been copied from the incubator by hand and then changed we should fix that up. Doug On Oct 15, 2014, at 5:28 AM, Valeriy Ponomaryov vponomar...@mirantis.com wrote: But why policy is being discussed on quota thread? On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov vponomar...@mirantis.com wrote: Manila project does use policy common code from incubator. Our small wrapper for it: https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com wrote: Doug, I totally agree with your findings on the policy module. Neutron already has some customizations there and we already have a few contributors working on syncing it back with oslo-incubator during the Kilo release cycle. However, my query was about the quota module. From what I gather it seems not a lot of projects use it: $ find . -name openstack-common.conf | xargs grep quota $ Salvatore On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote: On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote: Hi Doug, do you know if the existing quota oslo-incubator module has already some active consumers? In the meanwhile I've pushed a spec to neutron-specs for improving quota management there [1] It looks like a lot of projects are syncing the module: $ grep policy */openstack-common.conf barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy ceilometer/openstack-common.conf:module=policy cinder/openstack-common.conf:module=policy designate/openstack-common.conf:module=policy gantt/openstack-common.conf:module=policy glance/openstack-common.conf:module=policy heat/openstack-common.conf:module=policy horizon/openstack-common.conf:module=policy ironic/openstack-common.conf:module=policy keystone/openstack-common.conf:module=policy manila/openstack-common.conf:module=policy neutron/openstack-common.conf:module=policy nova/openstack-common.conf:module=policy trove/openstack-common.conf:module=policy tuskar/openstack-common.conf:module=policy I’m not sure how many are actively using it, but I wouldn’t expect them to copy it in if they weren’t using it at all. Now, I can either work on the oslo-incubator module and leverage it in Neutron, or develop the quota module in Neutron, and move it to oslo-incubator once we validate it with Neutron. The latter approach seems easier from a workflow perspective - as it avoid the intermediate steps of moving code from oslo-incubator to neutron. On the other hand it will delay adoption in oslo-incubator. The policy module is up for graduation this cycle. It may end up in its own library, to allow us to build a review team for the code more easily than if we put it in with some of the other semi-related modules like the server code. We’re still working that out [1], and if you expect to make a lot of incompatible changes we should delay graduation to make that simpler. Either way, since we have so many consumers, I think it would be easier to have the work happen in Oslo somewhere so we can ensure those changes are useful to and usable by all of the existing consumers. Doug [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals What's your opinion? Regards, Salvatore [1] https://review.openstack.org/#/c/128318/ On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote: On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote: Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py — dims If someone wants to drive creating a useful library during kilo, please consider adding the topic to the etherpad we’re using to plan summit sessions and then come participate in the Oslo meeting this Friday 16:00 UTC. https://etherpad.openstack.org/p/kilo-oslo-summit-topics Doug On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but
Re: [openstack-dev] Quota management and enforcement across projects
Hi, Would the new library support quota on domain level also? As it stands in oslo-incubator it only does quota enforcement on project level only. The use case for this is quota enforcement across multiple projects. For e.g. as a cloud provider, I would like my customer to create only #X volumes across all his projects. -Ajaya Cheers, Ajaya On Wed, Oct 15, 2014 at 7:04 PM, Doug Hellmann d...@doughellmann.com wrote: Sigh. Because I typed the wrong command. Thanks for pointing that out. I don’t see any instances of “quota” in openstack-common.conf files: $ grep quota */openstack-common.conf or in any projects under openstack/“: $ ls */*/openstack/common/quota.py ls: cannot access */*/openstack/common/quota.py: No such file or directory I don’t know where manila’s copy came from, but if it has been copied from the incubator by hand and then changed we should fix that up. Doug On Oct 15, 2014, at 5:28 AM, Valeriy Ponomaryov vponomar...@mirantis.com wrote: But why policy is being discussed on quota thread? On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov vponomar...@mirantis.com wrote: Manila project does use policy common code from incubator. Our small wrapper for it: https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com wrote: Doug, I totally agree with your findings on the policy module. Neutron already has some customizations there and we already have a few contributors working on syncing it back with oslo-incubator during the Kilo release cycle. However, my query was about the quota module. From what I gather it seems not a lot of projects use it: $ find . -name openstack-common.conf | xargs grep quota $ Salvatore On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote: On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote: Hi Doug, do you know if the existing quota oslo-incubator module has already some active consumers? In the meanwhile I've pushed a spec to neutron-specs for improving quota management there [1] It looks like a lot of projects are syncing the module: $ grep policy */openstack-common.conf barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy ceilometer/openstack-common.conf:module=policy cinder/openstack-common.conf:module=policy designate/openstack-common.conf:module=policy gantt/openstack-common.conf:module=policy glance/openstack-common.conf:module=policy heat/openstack-common.conf:module=policy horizon/openstack-common.conf:module=policy ironic/openstack-common.conf:module=policy keystone/openstack-common.conf:module=policy manila/openstack-common.conf:module=policy neutron/openstack-common.conf:module=policy nova/openstack-common.conf:module=policy trove/openstack-common.conf:module=policy tuskar/openstack-common.conf:module=policy I’m not sure how many are actively using it, but I wouldn’t expect them to copy it in if they weren’t using it at all. Now, I can either work on the oslo-incubator module and leverage it in Neutron, or develop the quota module in Neutron, and move it to oslo-incubator once we validate it with Neutron. The latter approach seems easier from a workflow perspective - as it avoid the intermediate steps of moving code from oslo-incubator to neutron. On the other hand it will delay adoption in oslo-incubator. The policy module is up for graduation this cycle. It may end up in its own library, to allow us to build a review team for the code more easily than if we put it in with some of the other semi-related modules like the server code. We’re still working that out [1], and if you expect to make a lot of incompatible changes we should delay graduation to make that simpler. Either way, since we have so many consumers, I think it would be easier to have the work happen in Oslo somewhere so we can ensure those changes are useful to and usable by all of the existing consumers. Doug [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals What's your opinion? Regards, Salvatore [1] https://review.openstack.org/#/c/128318/ On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote: On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote: Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py — dims If someone wants to drive creating a useful library during kilo, please consider adding the topic to the etherpad we’re using to plan summit sessions and then come participate in the Oslo meeting this Friday 16:00 UTC. https://etherpad.openstack.org/p/kilo-oslo-summit-topics Doug On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com
Re: [openstack-dev] Quota management and enforcement across projects
Hi Doug, do you know if the existing quota oslo-incubator module has already some active consumers? In the meanwhile I've pushed a spec to neutron-specs for improving quota management there [1] Now, I can either work on the oslo-incubator module and leverage it in Neutron, or develop the quota module in Neutron, and move it to oslo-incubator once we validate it with Neutron. The latter approach seems easier from a workflow perspective - as it avoid the intermediate steps of moving code from oslo-incubator to neutron. On the other hand it will delay adoption in oslo-incubator. What's your opinion? Regards, Salvatore [1] https://review.openstack.org/#/c/128318/ On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote: On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote: Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py — dims If someone wants to drive creating a useful library during kilo, please consider adding the topic to the etherpad we’re using to plan summit sessions and then come participate in the Oslo meeting this Friday 16:00 UTC. https://etherpad.openstack.org/p/kilo-oslo-summit-topics Doug On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. This is what I had in mind as well. A simple library for quota enforcement which can be used regardless of where and how you do it, which might depend on the application business logic, the WSGI framework in use, or other factors. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have the resources needed and spec(s) in progress / info on etherpads (similar to how the multitenancy stuff was handled at the last summit) as early as possible. Why not centralize quota management via the python-openstackclient, what is the benefit of getting keystone involved? Providing this through the openstack client in my opinion has the disadvantage that users which either use the REST API direct or write their own clients won't leverage it. I don't think it's a reasonable assumption that everybody will use python-openstackclient, is it? Said that, storing quotas in keystone poses a further challenge to the scalability of the system, which we shall perhaps address by using appropriate caching strategies and leveraging keystone notifications. Until we get that, I think that the openstack client will be the best way of getting a unified quota management experience. Salvatore Cheers, Morgan Sent via mobile On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com wrote: Thanks Vish, this seems a very reasonable first step as well - and since most projects would be enforcing quotas in the same way, the shared library would be the logical next step. After all this is quite the same thing we do with authZ. Duncan is expressing valid concerns which in my opinion can be addressed with an appropriate design - and a decent implementation. Salvatore On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com wrote: The proposal in the past was to keep quota enforcement local, but to put the resource limits into keystone. This seems like an obvious first step to me. Then a shared library for enforcing quotas with decent performance should be next. The quota calls in nova are extremely inefficient right now and it will only get worse when we try to add hierarchical projects and quotas. Vish On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote: Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote:
Re: [openstack-dev] Quota management and enforcement across projects
On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote: Hi Doug, do you know if the existing quota oslo-incubator module has already some active consumers? In the meanwhile I've pushed a spec to neutron-specs for improving quota management there [1] It looks like a lot of projects are syncing the module: $ grep policy */openstack-common.conf barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy ceilometer/openstack-common.conf:module=policy cinder/openstack-common.conf:module=policy designate/openstack-common.conf:module=policy gantt/openstack-common.conf:module=policy glance/openstack-common.conf:module=policy heat/openstack-common.conf:module=policy horizon/openstack-common.conf:module=policy ironic/openstack-common.conf:module=policy keystone/openstack-common.conf:module=policy manila/openstack-common.conf:module=policy neutron/openstack-common.conf:module=policy nova/openstack-common.conf:module=policy trove/openstack-common.conf:module=policy tuskar/openstack-common.conf:module=policy I’m not sure how many are actively using it, but I wouldn’t expect them to copy it in if they weren’t using it at all. Now, I can either work on the oslo-incubator module and leverage it in Neutron, or develop the quota module in Neutron, and move it to oslo-incubator once we validate it with Neutron. The latter approach seems easier from a workflow perspective - as it avoid the intermediate steps of moving code from oslo-incubator to neutron. On the other hand it will delay adoption in oslo-incubator. The policy module is up for graduation this cycle. It may end up in its own library, to allow us to build a review team for the code more easily than if we put it in with some of the other semi-related modules like the server code. We’re still working that out [1], and if you expect to make a lot of incompatible changes we should delay graduation to make that simpler. Either way, since we have so many consumers, I think it would be easier to have the work happen in Oslo somewhere so we can ensure those changes are useful to and usable by all of the existing consumers. Doug [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals What's your opinion? Regards, Salvatore [1] https://review.openstack.org/#/c/128318/ On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote: On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote: Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py — dims If someone wants to drive creating a useful library during kilo, please consider adding the topic to the etherpad we’re using to plan summit sessions and then come participate in the Oslo meeting this Friday 16:00 UTC. https://etherpad.openstack.org/p/kilo-oslo-summit-topics Doug On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. This is what I had in mind as well. A simple library for quota enforcement which can be used regardless of where and how you do it, which might depend on the application business logic, the WSGI framework in use, or other factors. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have the resources needed and spec(s) in progress / info on etherpads (similar to how the multitenancy stuff was handled at the last summit) as early as possible. Why not centralize quota management via the python-openstackclient, what is the benefit of getting keystone involved? Providing this through the openstack client in my opinion has the disadvantage that users which either use the REST API direct or write their own clients won't leverage it. I don't think it's a reasonable assumption that everybody will use python-openstackclient, is it? Said that, storing quotas in keystone poses a further challenge to the scalability of the system, which we shall perhaps address by using appropriate caching strategies and
Re: [openstack-dev] Quota management and enforcement across projects
Doug, I totally agree with your findings on the policy module. Neutron already has some customizations there and we already have a few contributors working on syncing it back with oslo-incubator during the Kilo release cycle. However, my query was about the quota module. From what I gather it seems not a lot of projects use it: $ find . -name openstack-common.conf | xargs grep quota $ Salvatore On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote: On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote: Hi Doug, do you know if the existing quota oslo-incubator module has already some active consumers? In the meanwhile I've pushed a spec to neutron-specs for improving quota management there [1] It looks like a lot of projects are syncing the module: $ grep policy */openstack-common.conf barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy ceilometer/openstack-common.conf:module=policy cinder/openstack-common.conf:module=policy designate/openstack-common.conf:module=policy gantt/openstack-common.conf:module=policy glance/openstack-common.conf:module=policy heat/openstack-common.conf:module=policy horizon/openstack-common.conf:module=policy ironic/openstack-common.conf:module=policy keystone/openstack-common.conf:module=policy manila/openstack-common.conf:module=policy neutron/openstack-common.conf:module=policy nova/openstack-common.conf:module=policy trove/openstack-common.conf:module=policy tuskar/openstack-common.conf:module=policy I’m not sure how many are actively using it, but I wouldn’t expect them to copy it in if they weren’t using it at all. Now, I can either work on the oslo-incubator module and leverage it in Neutron, or develop the quota module in Neutron, and move it to oslo-incubator once we validate it with Neutron. The latter approach seems easier from a workflow perspective - as it avoid the intermediate steps of moving code from oslo-incubator to neutron. On the other hand it will delay adoption in oslo-incubator. The policy module is up for graduation this cycle. It may end up in its own library, to allow us to build a review team for the code more easily than if we put it in with some of the other semi-related modules like the server code. We’re still working that out [1], and if you expect to make a lot of incompatible changes we should delay graduation to make that simpler. Either way, since we have so many consumers, I think it would be easier to have the work happen in Oslo somewhere so we can ensure those changes are useful to and usable by all of the existing consumers. Doug [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals What's your opinion? Regards, Salvatore [1] https://review.openstack.org/#/c/128318/ On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote: On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote: Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py — dims If someone wants to drive creating a useful library during kilo, please consider adding the topic to the etherpad we’re using to plan summit sessions and then come participate in the Oslo meeting this Friday 16:00 UTC. https://etherpad.openstack.org/p/kilo-oslo-summit-topics Doug On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. This is what I had in mind as well. A simple library for quota enforcement which can be used regardless of where and how you do it, which might depend on the application business logic, the WSGI framework in use, or other factors. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have the resources needed and spec(s) in progress / info on etherpads (similar to how the multitenancy stuff was handled at the last summit) as early as possible. Why not centralize quota management via the python-openstackclient, what is the benefit of getting
Re: [openstack-dev] Quota management and enforcement across projects
On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. This is what I had in mind as well. A simple library for quota enforcement which can be used regardless of where and how you do it, which might depend on the application business logic, the WSGI framework in use, or other factors. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have the resources needed and spec(s) in progress / info on etherpads (similar to how the multitenancy stuff was handled at the last summit) as early as possible. Why not centralize quota management via the python-openstackclient, what is the benefit of getting keystone involved? Providing this through the openstack client in my opinion has the disadvantage that users which either use the REST API direct or write their own clients won't leverage it. I don't think it's a reasonable assumption that everybody will use python-openstackclient, is it? Said that, storing quotas in keystone poses a further challenge to the scalability of the system, which we shall perhaps address by using appropriate caching strategies and leveraging keystone notifications. Until we get that, I think that the openstack client will be the best way of getting a unified quota management experience. Salvatore Cheers, Morgan Sent via mobile On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com wrote: Thanks Vish, this seems a very reasonable first step as well - and since most projects would be enforcing quotas in the same way, the shared library would be the logical next step. After all this is quite the same thing we do with authZ. Duncan is expressing valid concerns which in my opinion can be addressed with an appropriate design - and a decent implementation. Salvatore On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com wrote: The proposal in the past was to keep quota enforcement local, but to put the resource limits into keystone. This seems like an obvious first step to me. Then a shared library for enforcing quotas with decent performance should be next. The quota calls in nova are extremely inefficient right now and it will only get worse when we try to add hierarchical projects and quotas. Vish On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote: Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote: Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions about quota management, for two reasons: 1) despite developing and using openstack on a daily basis I'm still confused by quotas 2) I've found a race condition in neutron quotas and the fix is not trivial. So, rather than start coding
Re: [openstack-dev] Quota management and enforcement across projects
Salvatore, Joe, We do have this at the moment: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py -- dims On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote: On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote: On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. This is what I had in mind as well. A simple library for quota enforcement which can be used regardless of where and how you do it, which might depend on the application business logic, the WSGI framework in use, or other factors. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have the resources needed and spec(s) in progress / info on etherpads (similar to how the multitenancy stuff was handled at the last summit) as early as possible. Why not centralize quota management via the python-openstackclient, what is the benefit of getting keystone involved? Providing this through the openstack client in my opinion has the disadvantage that users which either use the REST API direct or write their own clients won't leverage it. I don't think it's a reasonable assumption that everybody will use python-openstackclient, is it? Said that, storing quotas in keystone poses a further challenge to the scalability of the system, which we shall perhaps address by using appropriate caching strategies and leveraging keystone notifications. Until we get that, I think that the openstack client will be the best way of getting a unified quota management experience. Salvatore Cheers, Morgan Sent via mobile On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com wrote: Thanks Vish, this seems a very reasonable first step as well - and since most projects would be enforcing quotas in the same way, the shared library would be the logical next step. After all this is quite the same thing we do with authZ. Duncan is expressing valid concerns which in my opinion can be addressed with an appropriate design - and a decent implementation. Salvatore On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com wrote: The proposal in the past was to keep quota enforcement local, but to put the resource limits into keystone. This seems like an obvious first step to me. Then a shared library for enforcing quotas with decent performance should be next. The quota calls in nova are extremely inefficient right now and it will only get worse when we try to add hierarchical projects and quotas. Vish On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote: Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote: Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions
Re: [openstack-dev] Quota management and enforcement across projects
On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com wrote: Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). It sounds like maybe we should have an oslo library for quotas? Somewhere where we can share the code,but keep the operations local to each service. If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have the resources needed and spec(s) in progress / info on etherpads (similar to how the multitenancy stuff was handled at the last summit) as early as possible. Why not centralize quota management via the python-openstackclient, what is the benefit of getting keystone involved? Cheers, Morgan Sent via mobile On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com wrote: Thanks Vish, this seems a very reasonable first step as well - and since most projects would be enforcing quotas in the same way, the shared library would be the logical next step. After all this is quite the same thing we do with authZ. Duncan is expressing valid concerns which in my opinion can be addressed with an appropriate design - and a decent implementation. Salvatore On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com wrote: The proposal in the past was to keep quota enforcement local, but to put the resource limits into keystone. This seems like an obvious first step to me. Then a shared library for enforcing quotas with decent performance should be next. The quota calls in nova are extremely inefficient right now and it will only get worse when we try to add hierarchical projects and quotas. Vish On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote: Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote: Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions about quota management, for two reasons: 1) despite developing and using openstack on a daily basis I'm still confused by quotas 2) I've found a race condition in neutron quotas and the fix is not trivial. So, rather than start coding right away, it might probably make more sense to ask the community if there is already a known better approach to quota management - and obviously enforcement. Thanks in advance, Salvatore [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata [2] https://wiki.openstack.org/wiki/Blazar [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
[openstack-dev] Quota management and enforcement across projects
Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions about quota management, for two reasons: 1) despite developing and using openstack on a daily basis I'm still confused by quotas 2) I've found a race condition in neutron quotas and the fix is not trivial. So, rather than start coding right away, it might probably make more sense to ask the community if there is already a known better approach to quota management - and obviously enforcement. Thanks in advance, Salvatore [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata [2] https://wiki.openstack.org/wiki/Blazar [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. As an end-user: +1 For me it totally makes sense to put the quotas and access together. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Totally agree, I'd rather not run more components running then strictly necessary. It is already quite a lot of work to test all components each time we do upgrade's. Adding complexity / more moving parts is not on the top of my list unless strictly necessary. Cheers, Robert van Leeuwen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote: Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions about quota management, for two reasons: 1) despite developing and using openstack on a daily basis I'm still confused by quotas 2) I've found a race condition in neutron quotas and the fix is not trivial. So, rather than start coding right away, it might probably make more sense to ask the community if there is already a known better approach to quota management - and obviously enforcement. Thanks in advance, Salvatore [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata [2] https://wiki.openstack.org/wiki/Blazar [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
The proposal in the past was to keep quota enforcement local, but to put the resource limits into keystone. This seems like an obvious first step to me. Then a shared library for enforcing quotas with decent performance should be next. The quota calls in nova are extremely inefficient right now and it will only get worse when we try to add hierarchical projects and quotas. Vish On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote: Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote: Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions about quota management, for two reasons: 1) despite developing and using openstack on a daily basis I'm still confused by quotas 2) I've found a race condition in neutron quotas and the fix is not trivial. So, rather than start coding right away, it might probably make more sense to ask the community if there is already a known better approach to quota management - and obviously enforcement. Thanks in advance, Salvatore [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata [2] https://wiki.openstack.org/wiki/Blazar [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
Thanks Vish, this seems a very reasonable first step as well - and since most projects would be enforcing quotas in the same way, the shared library would be the logical next step. After all this is quite the same thing we do with authZ. Duncan is expressing valid concerns which in my opinion can be addressed with an appropriate design - and a decent implementation. Salvatore On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com wrote: The proposal in the past was to keep quota enforcement local, but to put the resource limits into keystone. This seems like an obvious first step to me. Then a shared library for enforcing quotas with decent performance should be next. The quota calls in nova are extremely inefficient right now and it will only get worse when we try to add hierarchical projects and quotas. Vish On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote: Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote: Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions about quota management, for two reasons: 1) despite developing and using openstack on a daily basis I'm still confused by quotas 2) I've found a race condition in neutron quotas and the fix is not trivial. So, rather than start coding right away, it might probably make more sense to ask the community if there is already a known better approach to quota management - and obviously enforcement. Thanks in advance, Salvatore [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata [2] https://wiki.openstack.org/wiki/Blazar [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota management and enforcement across projects
Keeping the enforcement local (same way policy works today) helps limit the fragility, big +1 there. I also agree with Vish, we need a uniform way to talk about quota enforcement similar to how we have a uniform policy language / enforcement model (yes I know it's not perfect, but it's far closer to uniform than quota management is). If there is still interest of placing quota in keystone, let's talk about how that will work and what will be needed from Keystone . The previous attempt didn't get much traction and stalled out early in implementation. If we want to revisit this lets make sure we have the resources needed and spec(s) in progress / info on etherpads (similar to how the multitenancy stuff was handled at the last summit) as early as possible. Cheers, Morgan Sent via mobile On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com wrote: Thanks Vish, this seems a very reasonable first step as well - and since most projects would be enforcing quotas in the same way, the shared library would be the logical next step. After all this is quite the same thing we do with authZ. Duncan is expressing valid concerns which in my opinion can be addressed with an appropriate design - and a decent implementation. Salvatore On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com javascript:_e(%7B%7D,'cvml','vishvana...@gmail.com'); wrote: The proposal in the past was to keep quota enforcement local, but to put the resource limits into keystone. This seems like an obvious first step to me. Then a shared library for enforcing quotas with decent performance should be next. The quota calls in nova are extremely inefficient right now and it will only get worse when we try to add hierarchical projects and quotas. Vish On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com javascript:_e(%7B%7D,'cvml','duncan.tho...@gmail.com'); wrote: Taking quota out of the service / adding remote calls for quota management is going to make things fragile - you've somehow got to deal with the cases where your quota manager is slow, goes away, hiccups, drops connections etc. You'll also need some way of reconciling actual usage against quota usage periodically, to detect problems. On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com javascript:_e(%7B%7D,'cvml','sorla...@nicira.com'); wrote: Hi, Quota management is currently one of those things where every openstack project does its own thing. While quotas are obviously managed in a similar way for each project, there are subtle differences which ultimately result in lack of usability. I recall that in the past there have been several calls for unifying quota management. The blueprint [1] for instance, hints at the possibility of storing quotas in keystone. On the other hand, the blazar project [2, 3] seems to aim at solving this problem for good enabling resource reservation and therefore potentially freeing openstack projects from managing and enforcing quotas. While Blazar is definetely a good thing to have, I'm not entirely sure we want to make it a required component for every deployment. Perhaps single projects should still be able to enforce quota. On the other hand, at least on paper, the idea of making Keystone THE endpoint for managing quotas, and then letting the various project enforce them, sounds promising - is there any reason for which this blueprint is stalled to the point that it seems forgotten now? I'm coming to the mailing list with these random questions about quota management, for two reasons: 1) despite developing and using openstack on a daily basis I'm still confused by quotas 2) I've found a race condition in neutron quotas and the fix is not trivial. So, rather than start coding right away, it might probably make more sense to ask the community if there is already a known better approach to quota management - and obviously enforcement. Thanks in advance, Salvatore [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata [2] https://wiki.openstack.org/wiki/Blazar [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org'); http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org'); http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org');
Re: [openstack-dev] Quota Management
Glad to see this, i will be glad to contribute on it if the project could move on.. On Apr 4, 2014, at 10:01, Cazzolato, Sergio J sergio.j.cazzol...@intel.com wrote: Glad to see that, for sure I'll participate of this session. Thanks -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Thursday, April 03, 2014 7:21 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Quota Management On Thu, 2014-04-03 at 14:41 -0500, Kevin L. Mitchell wrote: On Thu, 2014-04-03 at 19:16 +, Cazzolato, Sergio J wrote: Jay, thanks for taking ownership on this idea, we are really interested to contribute to this, so what do you think are the next steps to move on? Perhaps a summit session on quota management would be in order? Done: http://summit.openstack.org/cfp/details/221 Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On 03/04/14 23:20, Jay Pipes wrote: On Thu, 2014-04-03 at 14:41 -0500, Kevin L. Mitchell wrote: On Thu, 2014-04-03 at 19:16 +, Cazzolato, Sergio J wrote: Jay, thanks for taking ownership on this idea, we are really interested to contribute to this, so what do you think are the next steps to move on? Perhaps a summit session on quota management would be in order? Done: http://summit.openstack.org/cfp/details/221 Thank you for proposing the session, I'm hopeful having this in the new cross-project track will have a positive impact on the discussion. I'm under the impression that this comes back regularly as a session topic but keeps hitting barriers when it comes to actual implementation (perhaps because important stakeholders were missing from the session before). I'd like to bring up the cross-project discussion from last time this was discussed in December [1] as a reference, since the same questions/objections will likely come back again. One of the main issues was that this shouldn't live in Keystone, which could be resolved by using Boson, but the rest shows a reluctance from the projects to delegate quota management, and uncertainty around the use cases. Oslo was also mentioned as a possible place to help with improving the consistency. I'd love a more consistent way to handle and manage quotas across multiple projects as this would help Horizon too, for very similar reasons than are mentioned here. Thanks, Julie [1] http://eavesdrop.openstack.org/meetings/project/2013/project.2013-12-10-21.02.log.html from 21:10 Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Quota Management
Hi All, I'd like to know your thoughts regarding Quota Management... I've been contributing to this topic for icehouse and noticed some issues and discussions around its implementation like code is duplicated, synch problems with database, not having an homogeneous logic, etc... so I was thinking that maybe a centralized implementation could be a solution for this... As far as I know there was a discussion during the last summit and the decision was to use Keystone for a Centralized Quota Management solution but I don't have the details on that discussion... Also I was looking at Boson (https://wiki.openstack.org/wiki/Boson) that seems to be a nice solution for this and also addresses the scenario where Nova is deployed in a multi-cell manner and some other interesting things. Sergio ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On Thu, 2014-04-03 at 15:02 +, Cazzolato, Sergio J wrote: Hi All, I’d like to know your thoughts regarding Quota Management… I’ve been contributing to this topic for icehouse and noticed some issues and discussions around its implementation like code is duplicated, synch problems with database, not having an homogeneous logic, etc… so I was thinking that maybe a centralized implementation could be a solution for this… As far as I know there was a discussion during the last summit and the decision was to use Keystone for a Centralized Quota Management solution but I don’t have the details on that discussion… Also I was looking at Boson (https://wiki.openstack.org/wiki/Boson) that seems to be a nice solution for this and also addresses the scenario where Nova is deployed in a multi-cell manner and some other interesting things. Boson does indeed look interesting, but who is working on it, if anyone at this point? I agree that having a centralized quota management system makes sense, in order to make the handling of quotas and reservations consistent across projects as well as to deal with global quota management properly (i.e. quotas that span multiple cells/AZs) Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On Thu, Apr 3, 2014 at 11:22 AM, Jay Pipes jaypi...@gmail.com wrote: On Thu, 2014-04-03 at 15:02 +, Cazzolato, Sergio J wrote: Hi All, I’d like to know your thoughts regarding Quota Management… I’ve been contributing to this topic for icehouse and noticed some issues and discussions around its implementation like code is duplicated, synch problems with database, not having an homogeneous logic, etc… so I was thinking that maybe a centralized implementation could be a solution for this… As far as I know there was a discussion during the last summit and the decision was to use Keystone for a Centralized Quota Management solution but I don’t have the details on that discussion… Also I was looking at Boson (https://wiki.openstack.org/wiki/Boson) that seems to be a nice solution for this and also addresses the scenario where Nova is deployed in a multi-cell manner and some other interesting things. Boson does indeed look interesting, but who is working on it, if anyone at this point? I agree that having a centralized quota management system makes sense, in order to make the handling of quotas and reservations consistent across projects as well as to deal with global quota management properly (i.e. quotas that span multiple cells/AZs) ++ but it looks to be an abandoned effort. There's a couple forks, but not with any recent activity: https://github.com/klmitch/boson (correct me if I'm just looking in the wrong place!) Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On Thu, 2014-04-03 at 09:22 -0700, Jay Pipes wrote: Boson does indeed look interesting, but who is working on it, if anyone at this point? I agree that having a centralized quota management system makes sense, in order to make the handling of quotas and reservations consistent across projects as well as to deal with global quota management properly (i.e. quotas that span multiple cells/AZs) At present, no one is really working on Boson. The CERN folks were interested in it, but I wasn't able to free up the time I would have needed to actually develop on it, partly because of lack of interest from the rest of the community at the time. I'd say that, currently, Boson is facing the chicken-and-egg problem—no one works on it because no one is working on it. -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On Thu, 2014-04-03 at 12:13 -0500, Kevin L. Mitchell wrote: On Thu, 2014-04-03 at 09:22 -0700, Jay Pipes wrote: Boson does indeed look interesting, but who is working on it, if anyone at this point? I agree that having a centralized quota management system makes sense, in order to make the handling of quotas and reservations consistent across projects as well as to deal with global quota management properly (i.e. quotas that span multiple cells/AZs) At present, no one is really working on Boson. The CERN folks were interested in it, but I wasn't able to free up the time I would have needed to actually develop on it, partly because of lack of interest from the rest of the community at the time. I'd say that, currently, Boson is facing the chicken-and-egg problem—no one works on it because no one is working on it. I would personally be interested in picking up the work. Kevin, any issues with me moving the code on your GitHub repo into stackforge and linking up the upstream CI harness? All the best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On Thu, 2014-04-03 at 10:54 -0700, Jay Pipes wrote: On Thu, 2014-04-03 at 12:13 -0500, Kevin L. Mitchell wrote: On Thu, 2014-04-03 at 09:22 -0700, Jay Pipes wrote: Boson does indeed look interesting, but who is working on it, if anyone at this point? I agree that having a centralized quota management system makes sense, in order to make the handling of quotas and reservations consistent across projects as well as to deal with global quota management properly (i.e. quotas that span multiple cells/AZs) At present, no one is really working on Boson. The CERN folks were interested in it, but I wasn't able to free up the time I would have needed to actually develop on it, partly because of lack of interest from the rest of the community at the time. I'd say that, currently, Boson is facing the chicken-and-egg problem—no one works on it because no one is working on it. I would personally be interested in picking up the work. Kevin, any issues with me moving the code on your GitHub repo into stackforge and linking up the upstream CI harness? None at all; please do so! :) -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
Jay, thanks for taking ownership on this idea, we are really interested to contribute to this, so what do you think are the next steps to move on? Please let me know whatever you need to accelerate on this. Sergio Cazzolato -Original Message- From: Kevin L. Mitchell [mailto:kevin.mitch...@rackspace.com] Sent: Thursday, April 03, 2014 3:13 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Quota Management On Thu, 2014-04-03 at 10:54 -0700, Jay Pipes wrote: On Thu, 2014-04-03 at 12:13 -0500, Kevin L. Mitchell wrote: On Thu, 2014-04-03 at 09:22 -0700, Jay Pipes wrote: Boson does indeed look interesting, but who is working on it, if anyone at this point? I agree that having a centralized quota management system makes sense, in order to make the handling of quotas and reservations consistent across projects as well as to deal with global quota management properly (i.e. quotas that span multiple cells/AZs) At present, no one is really working on Boson. The CERN folks were interested in it, but I wasn't able to free up the time I would have needed to actually develop on it, partly because of lack of interest from the rest of the community at the time. I'd say that, currently, Boson is facing the chicken-and-egg problem—no one works on it because no one is working on it. I would personally be interested in picking up the work. Kevin, any issues with me moving the code on your GitHub repo into stackforge and linking up the upstream CI harness? None at all; please do so! :) -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On Thu, 2014-04-03 at 19:16 +, Cazzolato, Sergio J wrote: Jay, thanks for taking ownership on this idea, we are really interested to contribute to this, so what do you think are the next steps to move on? Perhaps a summit session on quota management would be in order? -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
+1 -Original Message- From: Kevin L. Mitchell [mailto:kevin.mitch...@rackspace.com] Sent: Thursday, April 03, 2014 4:42 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Quota Management On Thu, 2014-04-03 at 19:16 +, Cazzolato, Sergio J wrote: Jay, thanks for taking ownership on this idea, we are really interested to contribute to this, so what do you think are the next steps to move on? Perhaps a summit session on quota management would be in order? -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
On Thu, 2014-04-03 at 14:41 -0500, Kevin L. Mitchell wrote: On Thu, 2014-04-03 at 19:16 +, Cazzolato, Sergio J wrote: Jay, thanks for taking ownership on this idea, we are really interested to contribute to this, so what do you think are the next steps to move on? Perhaps a summit session on quota management would be in order? Done: http://summit.openstack.org/cfp/details/221 Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Quota Management
Glad to see that, for sure I'll participate of this session. Thanks -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Thursday, April 03, 2014 7:21 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Quota Management On Thu, 2014-04-03 at 14:41 -0500, Kevin L. Mitchell wrote: On Thu, 2014-04-03 at 19:16 +, Cazzolato, Sergio J wrote: Jay, thanks for taking ownership on this idea, we are really interested to contribute to this, so what do you think are the next steps to move on? Perhaps a summit session on quota management would be in order? Done: http://summit.openstack.org/cfp/details/221 Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
