Re: [openstack-dev] Quota management and enforcement across projects

2015-03-24 Thread Dimitri Mazmanov
Hi,
Interesting  thread. I’ve just started looking at Boson myself to manage quotas 
across multiple regions. I think one of the cases when having quota management 
as a separate service is justified, is a multi-region case [1].

1) Some deployments can share Keystone, so there’s a need to synchronise 
resource usage across multiple OpenStacks and enforce quota in a distributed 
manner. Architecturally this will mean that there will be one Boson instance 
with the “quota management” and “admin” roles exposing REST API endpoint and 
talking REST to other client specific Boson instances. This use case can be an 
example of why it can be reasonable to have a separate service for managing 
quotas, reservations, usages.

2) As I understand it, the intention of Boson is to synchronise usage, not own 
this information. The “Usage synchronisation” paragraph in the wiki [3] 
describes one possible approach:
“…Boson will keep freshness information on the usage data; should it determine 
that the usage information it has is not fresh enough, it will reject 
reservation creation requests with a special response code, which tells the 
service to send fresh usage information for certain resources along with 
resending the reservation creation requests”.
Further, the service can reject reservation creation requests in case of 
communication failure, tracker unavailability, etc.

3) Have you looked at Blazar [3] as a possible reservation mechanism instead of 
implementing a new reservation interface in Boson? Does it at all make sense to 
use it in the context of quota management?

Regards,
Dimitri

[1] http://dc636.4shared.com/download/Z6O_jJSGba/multiregion_arch.png?lgfp=3000
[2]https://wiki.openstack.org/wiki/Boson#Usage_Synchronization
[3] https://wiki.openstack.org/wiki/Blazar

From: Salvatore Orlando sorla...@nicira.commailto:sorla...@nicira.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday 15 January 2015 02:22
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] Quota management and enforcement across projects

I'm resurrecting this thread to provide an update on this effort.

I have been looking at Boson [1] as a base for starting developing a service 
for managing quotas and reservations [2].
Boson's model would satisfy most of the requirements for this service, and 
implementing additional requirements, such as hierarchical multi-tenancy should 
be quite easy (at least from the high-level design perspective). In the rest of 
this post I'm going to refer to this service both as Boson and quota 
service.

Without going into too many technical details (which I am happy to discuss 
separately), the quota service will need to provide the 4 interfaces depicted 
in [3].
1) The admin interface has the main purpose of keeping track of the services 
which use boson, registering resources to track, and managing their lifecycle.
2) The quota mgmt interface does pretty much what the quota extension does 
for many openstack project - manage resource limits per project/users, 
configure quota classes, etc.
3) The reservation interface handles the request/commit/cancel process 
supported by many Openstack projects
4) the usage interface provides abstraction to access the resource usage 
tracker and feed information to it.

These interfaces are then implemented by the Boson object model, depicted in 
[4].
This object model is not very different from the one originally proposed for 
Boson [5]
The proposed object model simplifies a bit the original one, merging the 
reservation and request concepts, as well as doing without the 
SpecificResource concept (at least for the moment). However, the proposed 
object model adds the Quota class concept and introduces the possibility of 
having child-parent relationships between Resources and User Info. The former 
will allow for applying quota to resources which are scoped within a parent 
resource (e.g.: static routes per logical router), whereas the latter should 
enable the hierarchical multi-tenancy use case.

Keeping in mind the interfaces discussed in [3], the component diagram [6] can 
be devised. There is a distinct component for each interface - plus one 
component for DB management. Most of the interactions are, of course, with the 
DB manager. Component design should be done in a way that the various 
components are as independent as possible. There are some interactions among 
components, but they can likely be replaced with interactions with the DB 
manager component.

The Boson quota service therefore represents a centralized endpoint for 
managing quotas, tracking resource usage, and performing resource reservation.
Conceptually, this is all good; however, would it scale from an architectural 
perspective? The main problem with this approach

Re: [openstack-dev] Quota management and enforcement across projects

2015-01-14 Thread Salvatore Orlando
I'm resurrecting this thread to provide an update on this effort.

I have been looking at Boson [1] as a base for starting developing a
service for managing quotas and reservations [2].
Boson's model would satisfy most of the requirements for this service, and
implementing additional requirements, such as hierarchical multi-tenancy
should be quite easy (at least from the high-level design perspective). In
the rest of this post I'm going to refer to this service both as Boson
and quota service.

Without going into too many technical details (which I am happy to discuss
separately), the quota service will need to provide the 4 interfaces
depicted in [3].
1) The admin interface has the main purpose of keeping track of the
services which use boson, registering resources to track, and managing
their lifecycle.
2) The quota mgmt interface does pretty much what the quota extension
does for many openstack project - manage resource limits per project/users,
configure quota classes, etc.
3) The reservation interface handles the request/commit/cancel process
supported by many Openstack projects
4) the usage interface provides abstraction to access the resource usage
tracker and feed information to it.

These interfaces are then implemented by the Boson object model, depicted
in [4].
This object model is not very different from the one originally proposed
for Boson [5]
The proposed object model simplifies a bit the original one, merging the
reservation and request concepts, as well as doing without the
SpecificResource concept (at least for the moment). However, the proposed
object model adds the Quota class concept and introduces the possibility
of having child-parent relationships between Resources and User Info. The
former will allow for applying quota to resources which are scoped within a
parent resource (e.g.: static routes per logical router), whereas the
latter should enable the hierarchical multi-tenancy use case.

Keeping in mind the interfaces discussed in [3], the component diagram [6]
can be devised. There is a distinct component for each interface - plus one
component for DB management. Most of the interactions are, of course, with
the DB manager. Component design should be done in a way that the various
components are as independent as possible. There are some interactions
among components, but they can likely be replaced with interactions with
the DB manager component.

The Boson quota service therefore represents a centralized endpoint for
managing quotas, tracking resource usage, and performing resource
reservation.
Conceptually, this is all good; however, would it scale from an
architectural perspective? The main problem with this approach indeed is
that the quota service itself can become a huge bottleneck and add more
latency to resource processing. This problem can probably be solved looking
at the above mentioned interfaces as roles, and expecting Boson to scale
horizontally via deployment of several instances of the service which can
implement different roles, some of which can be dedicated to specific
services.

For instance, let's consider the example architecture depicted in [7]. Here
there are 3 boson instances.
One instance has the quota management and admin role and takes care of
registering resources for enabled services, and setting/retrieving resource
limits. It might expose its own REST API endpoint over HTTP to this aim.
The other two instances have the reservation and usage roles, and
communicate with client applications over AMQP. However one instance will
listen only to neutron messages and the other only to nova messages
(nova and neutron have been used merely as an example, the same applies to
any project). This architecture will enable some sort of horizontal scale
out as every project will then have its own instance for tracking resources
and managing reservations.
However, there will still be need for some communication across the various
Boson instances. For instance in order to make a reservation, current
resource limits should be fetched interacting with the management component
or from the database. This overhead can be reduced considerably by
introducing caches, which would result particularly effective for data such
as resource limits which do not tend to change a lot over time.
Finally, it is also worth noting that in theory nothing prevents us from
running Boson's management interface as a Keystone 3rd party component and
therefore running its API on the Keystone endpoint.

Summarising, I hope I managed to convey the essentials of where I think
Boson should be place in the OpenStack ecosystem, and how would it work
from a conceptual and architectural perspective. I am aware that this is
not exhaustive at al, but is probably the best that can be achieved in a
mailing list post.

Nevertheless, there are still some fundamental questions that I am trying
to answer in a convincing way.

1) would anyone ever deploy a service aimed exclusively at managing quotas
and 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-28 Thread Jay Pipes

On 11/17/2014 05:49 PM, Salvatore Orlando wrote:

Hi all,

I am resuming this thread following the session we had at the summit in
Paris (etherpad here [1])

While there was some sort of consensus regarding what this library
should do, and how it should do it, the session ended with some open
questions which we need to address before finalising the specification.

There was a rather large consensus that the only viable architecture
would be one where the quota management library owns resource usage
data. However, this raises further concerns around:
- ownership of resource usage records. As the quota library owns usage
data, it becomes the authoritative source of truth for it. This could be
problematic in some projects, particularly nova, where the resource
tracker currently own usage data.


Well, just to be clear... the resource tracker in Nova owns the usage 
records for the compute node, not the usage records for a tenant or user 
(the quota driver and DB tables own that data).


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Sylvain Bauza


Le 18/11/2014 20:05, Doug Hellmann a écrit :

On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com 
wrote:


On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote:

I’ve spent a bit of time thinking about the resource ownership issue.
The challenge there is we don’t currently have any libraries that
define tables in the schema of an application. I think that’s a good
pattern to maintain, since it avoids introducing a lot of tricky
issues like how to manage migrations for the library, how to ensure
they are run by the application, etc. The fact that this common quota
thing needs to store some data in a schema that it controls says to me
that it is really an app and not a library. Making the quota manager
an app solves the API definition issue, too, since we can describe a
generic way to configure quotas and other applications can then use
that API to define specific rules using the quota manager’s API.

I don’t know if we need a new application or if it would make sense
to, as with policy, add quota management features to keystone. A
single well-defined app has some appeal, but there’s also a certain
amount of extra ramp-up time needed to go that route that we wouldn’t
need if we added the features directly to keystone.

I'll also point out that it was largely because of the storage needs
that I chose to propose Boson[1] as a separate app, rather than as a
library.  Further, the dimensions over which quota-covered resources
needed to be tracked seemed to me to be complicated enough that it would
be better to define a new app and make it support that one domain well,
which is why I didn't propose it as something to add to Keystone.
Consider: nova has quotas that are applied by user, other quotas that
are applied by tenant, and even some quotas on what could be considered
sub-resources—a limit on the number of security group rules per security
group, for instance.

My current feeling is that, if we can figure out a way to make the quota
problem into an acceptable library, that will work; it would probably
have to maintain its own database separate from the client app and have
features for automatically managing the schema, since we couldn't
necessarily rely on the client app to invoke the proper juju there.  If,
on the other hand, that ends up failing, then the best route is probably
to begin by developing a separate app, like Boson, as a PoC; then, after
we have some idea of just how difficult it is to actually solve the
problem, we can evaluate whether it makes sense to actually fold it into
a service like Keystone, or whether it should stand on its own.

(Personally, I think Boson should be created and should stand on its
own, but I also envision using it for purposes outside of OpenStack…)

Thanks for mentioning Boson again. I’m embarrassed that I completely forgot 
about the fact that you mentioned this at the summit.

I’ll have to look at the proposal more closely before I comment in any detail, 
but I take it as a good sign that we’re coming back around to the idea of 
solving this with an app instead of a library.


I assume I'm really late in the thread so I can just sit and give +1 to 
this direction : IMHO, quotas need to managed thanks to a CRUD interface 
which implies to get an app, as it sounds unreasonable to extend each 
consumer app API.


That said, back to Blazar, I just would like to emphasize that Blazar is 
not trying to address the quota enforcement level, but rather provide a 
centralized endpoint for managing reservations.
Consequently, Blazar can also be considered as a consumer of this quota 
system, whatever it's in a library or on a separate REST API.


Last thing, I don't think that a quota application necessarly means that 
quotas enforcement should be managed thanks to external calls to this 
app. I can rather see an external system able to set for each project a 
local view of what should be enforced locally. If operators don't want 
to deploy that quota management project, it's up to them to address the 
hetergenous setups for each project.


My 2 cts (too),
-Sylvain


Doug


Just my $.02…

[1] https://wiki.openstack.org/wiki/Boson
--
Kevin L. Mitchell kevin.mitch...@rackspace.com
Rackspace


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Endre Karlson
All I can say at the moment is that Usage and Quota management is a crappy
thing to do in OpenStack. Every service has it's own way of doing it both
in clients and api's. +n+ for making a effort in standardizing this thing
in a way that could be alike across projects..

2014-11-19 14:33 GMT+01:00 Sylvain Bauza sba...@redhat.com:


 Le 18/11/2014 20:05, Doug Hellmann a écrit :

  On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell 
 kevin.mitch...@rackspace.com wrote:

  On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote:

 I’ve spent a bit of time thinking about the resource ownership issue.
 The challenge there is we don’t currently have any libraries that
 define tables in the schema of an application. I think that’s a good
 pattern to maintain, since it avoids introducing a lot of tricky
 issues like how to manage migrations for the library, how to ensure
 they are run by the application, etc. The fact that this common quota
 thing needs to store some data in a schema that it controls says to me
 that it is really an app and not a library. Making the quota manager
 an app solves the API definition issue, too, since we can describe a
 generic way to configure quotas and other applications can then use
 that API to define specific rules using the quota manager’s API.

 I don’t know if we need a new application or if it would make sense
 to, as with policy, add quota management features to keystone. A
 single well-defined app has some appeal, but there’s also a certain
 amount of extra ramp-up time needed to go that route that we wouldn’t
 need if we added the features directly to keystone.

 I'll also point out that it was largely because of the storage needs
 that I chose to propose Boson[1] as a separate app, rather than as a
 library.  Further, the dimensions over which quota-covered resources
 needed to be tracked seemed to me to be complicated enough that it would
 be better to define a new app and make it support that one domain well,
 which is why I didn't propose it as something to add to Keystone.
 Consider: nova has quotas that are applied by user, other quotas that
 are applied by tenant, and even some quotas on what could be considered
 sub-resources—a limit on the number of security group rules per security
 group, for instance.

 My current feeling is that, if we can figure out a way to make the quota
 problem into an acceptable library, that will work; it would probably
 have to maintain its own database separate from the client app and have
 features for automatically managing the schema, since we couldn't
 necessarily rely on the client app to invoke the proper juju there.  If,
 on the other hand, that ends up failing, then the best route is probably
 to begin by developing a separate app, like Boson, as a PoC; then, after
 we have some idea of just how difficult it is to actually solve the
 problem, we can evaluate whether it makes sense to actually fold it into
 a service like Keystone, or whether it should stand on its own.

 (Personally, I think Boson should be created and should stand on its
 own, but I also envision using it for purposes outside of OpenStack…)

 Thanks for mentioning Boson again. I’m embarrassed that I completely
 forgot about the fact that you mentioned this at the summit.

 I’ll have to look at the proposal more closely before I comment in any
 detail, but I take it as a good sign that we’re coming back around to the
 idea of solving this with an app instead of a library.


 I assume I'm really late in the thread so I can just sit and give +1 to
 this direction : IMHO, quotas need to managed thanks to a CRUD interface
 which implies to get an app, as it sounds unreasonable to extend each
 consumer app API.

 That said, back to Blazar, I just would like to emphasize that Blazar is
 not trying to address the quota enforcement level, but rather provide a
 centralized endpoint for managing reservations.
 Consequently, Blazar can also be considered as a consumer of this quota
 system, whatever it's in a library or on a separate REST API.

 Last thing, I don't think that a quota application necessarly means that
 quotas enforcement should be managed thanks to external calls to this app.
 I can rather see an external system able to set for each project a local
 view of what should be enforced locally. If operators don't want to deploy
 that quota management project, it's up to them to address the hetergenous
 setups for each project.

 My 2 cts (too),
 -Sylvain


  Doug

  Just my $.02…

 [1] https://wiki.openstack.org/wiki/Boson
 --
 Kevin L. Mitchell kevin.mitch...@rackspace.com
 Rackspace


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Doug Hellmann

On Nov 19, 2014, at 8:33 AM, Sylvain Bauza sba...@redhat.com wrote:

 
 Le 18/11/2014 20:05, Doug Hellmann a écrit :
 On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell 
 kevin.mitch...@rackspace.com wrote:
 
 On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote:
 I’ve spent a bit of time thinking about the resource ownership issue.
 The challenge there is we don’t currently have any libraries that
 define tables in the schema of an application. I think that’s a good
 pattern to maintain, since it avoids introducing a lot of tricky
 issues like how to manage migrations for the library, how to ensure
 they are run by the application, etc. The fact that this common quota
 thing needs to store some data in a schema that it controls says to me
 that it is really an app and not a library. Making the quota manager
 an app solves the API definition issue, too, since we can describe a
 generic way to configure quotas and other applications can then use
 that API to define specific rules using the quota manager’s API.
 
 I don’t know if we need a new application or if it would make sense
 to, as with policy, add quota management features to keystone. A
 single well-defined app has some appeal, but there’s also a certain
 amount of extra ramp-up time needed to go that route that we wouldn’t
 need if we added the features directly to keystone.
 I'll also point out that it was largely because of the storage needs
 that I chose to propose Boson[1] as a separate app, rather than as a
 library.  Further, the dimensions over which quota-covered resources
 needed to be tracked seemed to me to be complicated enough that it would
 be better to define a new app and make it support that one domain well,
 which is why I didn't propose it as something to add to Keystone.
 Consider: nova has quotas that are applied by user, other quotas that
 are applied by tenant, and even some quotas on what could be considered
 sub-resources—a limit on the number of security group rules per security
 group, for instance.
 
 My current feeling is that, if we can figure out a way to make the quota
 problem into an acceptable library, that will work; it would probably
 have to maintain its own database separate from the client app and have
 features for automatically managing the schema, since we couldn't
 necessarily rely on the client app to invoke the proper juju there.  If,
 on the other hand, that ends up failing, then the best route is probably
 to begin by developing a separate app, like Boson, as a PoC; then, after
 we have some idea of just how difficult it is to actually solve the
 problem, we can evaluate whether it makes sense to actually fold it into
 a service like Keystone, or whether it should stand on its own.
 
 (Personally, I think Boson should be created and should stand on its
 own, but I also envision using it for purposes outside of OpenStack…)
 Thanks for mentioning Boson again. I’m embarrassed that I completely forgot 
 about the fact that you mentioned this at the summit.
 
 I’ll have to look at the proposal more closely before I comment in any 
 detail, but I take it as a good sign that we’re coming back around to the 
 idea of solving this with an app instead of a library.
 
 I assume I'm really late in the thread so I can just sit and give +1 to this 
 direction : IMHO, quotas need to managed thanks to a CRUD interface which 
 implies to get an app, as it sounds unreasonable to extend each consumer app 
 API.
 
 That said, back to Blazar, I just would like to emphasize that Blazar is not 
 trying to address the quota enforcement level, but rather provide a 
 centralized endpoint for managing reservations.
 Consequently, Blazar can also be considered as a consumer of this quota 
 system, whatever it's in a library or on a separate REST API.
 
 Last thing, I don't think that a quota application necessarly means that 
 quotas enforcement should be managed thanks to external calls to this app. I 
 can rather see an external system able to set for each project a local view 
 of what should be enforced locally. If operators don't want to deploy that 
 quota management project, it's up to them to address the hetergenous setups 
 for each project.

I’m not sure what this means. You want the new service to be optional? How 
would apps written against the service find and manage quota data if the 
service isn’t there?

Doug

 
 My 2 cts (too),
 -Sylvain
 
 Doug
 
 Just my $.02…
 
 [1] https://wiki.openstack.org/wiki/Boson
 -- 
 Kevin L. Mitchell kevin.mitch...@rackspace.com
 Rackspace
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Sylvain Bauza


Le 19/11/2014 15:06, Doug Hellmann a écrit :

On Nov 19, 2014, at 8:33 AM, Sylvain Bauza sba...@redhat.com wrote:


Le 18/11/2014 20:05, Doug Hellmann a écrit :

On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com 
wrote:


On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote:

I’ve spent a bit of time thinking about the resource ownership issue.
The challenge there is we don’t currently have any libraries that
define tables in the schema of an application. I think that’s a good
pattern to maintain, since it avoids introducing a lot of tricky
issues like how to manage migrations for the library, how to ensure
they are run by the application, etc. The fact that this common quota
thing needs to store some data in a schema that it controls says to me
that it is really an app and not a library. Making the quota manager
an app solves the API definition issue, too, since we can describe a
generic way to configure quotas and other applications can then use
that API to define specific rules using the quota manager’s API.

I don’t know if we need a new application or if it would make sense
to, as with policy, add quota management features to keystone. A
single well-defined app has some appeal, but there’s also a certain
amount of extra ramp-up time needed to go that route that we wouldn’t
need if we added the features directly to keystone.

I'll also point out that it was largely because of the storage needs
that I chose to propose Boson[1] as a separate app, rather than as a
library.  Further, the dimensions over which quota-covered resources
needed to be tracked seemed to me to be complicated enough that it would
be better to define a new app and make it support that one domain well,
which is why I didn't propose it as something to add to Keystone.
Consider: nova has quotas that are applied by user, other quotas that
are applied by tenant, and even some quotas on what could be considered
sub-resources—a limit on the number of security group rules per security
group, for instance.

My current feeling is that, if we can figure out a way to make the quota
problem into an acceptable library, that will work; it would probably
have to maintain its own database separate from the client app and have
features for automatically managing the schema, since we couldn't
necessarily rely on the client app to invoke the proper juju there.  If,
on the other hand, that ends up failing, then the best route is probably
to begin by developing a separate app, like Boson, as a PoC; then, after
we have some idea of just how difficult it is to actually solve the
problem, we can evaluate whether it makes sense to actually fold it into
a service like Keystone, or whether it should stand on its own.

(Personally, I think Boson should be created and should stand on its
own, but I also envision using it for purposes outside of OpenStack…)

Thanks for mentioning Boson again. I’m embarrassed that I completely forgot 
about the fact that you mentioned this at the summit.

I’ll have to look at the proposal more closely before I comment in any detail, 
but I take it as a good sign that we’re coming back around to the idea of 
solving this with an app instead of a library.

I assume I'm really late in the thread so I can just sit and give +1 to this 
direction : IMHO, quotas need to managed thanks to a CRUD interface which 
implies to get an app, as it sounds unreasonable to extend each consumer app 
API.

That said, back to Blazar, I just would like to emphasize that Blazar is not 
trying to address the quota enforcement level, but rather provide a centralized 
endpoint for managing reservations.
Consequently, Blazar can also be considered as a consumer of this quota system, 
whatever it's in a library or on a separate REST API.

Last thing, I don't think that a quota application necessarly means that quotas 
enforcement should be managed thanks to external calls to this app. I can 
rather see an external system able to set for each project a local view of what 
should be enforced locally. If operators don't want to deploy that quota 
management project, it's up to them to address the hetergenous setups for each 
project.

I’m not sure what this means. You want the new service to be optional? How 
would apps written against the service find and manage quota data if the 
service isn’t there?


My bad. Let me rephrase it. I'm seeing this service as providing added 
value for managing quotas by ensuring consistency across all projects. 
But as I said, I'm also thinking that the quota enforcement has still to 
be done at the customer project level.


So, I can imagine a client (or a Facade if you prefer) providing quota 
resources to the customer app which could be either fetched (thru some 
caching) from the service, or directly taken from the existing quota DB.


In order to do that, I could imagine those steps :
 #1 : customer app makes use of oslo.quota for managing its own quota 
resources
 #2 : the external app 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Doug Hellmann

On Nov 19, 2014, at 9:51 AM, Sylvain Bauza sba...@redhat.com wrote:

 
 Le 19/11/2014 15:06, Doug Hellmann a écrit :
 On Nov 19, 2014, at 8:33 AM, Sylvain Bauza sba...@redhat.com wrote:
 
 Le 18/11/2014 20:05, Doug Hellmann a écrit :
 On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell 
 kevin.mitch...@rackspace.com wrote:
 
 On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote:
 I’ve spent a bit of time thinking about the resource ownership issue.
 The challenge there is we don’t currently have any libraries that
 define tables in the schema of an application. I think that’s a good
 pattern to maintain, since it avoids introducing a lot of tricky
 issues like how to manage migrations for the library, how to ensure
 they are run by the application, etc. The fact that this common quota
 thing needs to store some data in a schema that it controls says to me
 that it is really an app and not a library. Making the quota manager
 an app solves the API definition issue, too, since we can describe a
 generic way to configure quotas and other applications can then use
 that API to define specific rules using the quota manager’s API.
 
 I don’t know if we need a new application or if it would make sense
 to, as with policy, add quota management features to keystone. A
 single well-defined app has some appeal, but there’s also a certain
 amount of extra ramp-up time needed to go that route that we wouldn’t
 need if we added the features directly to keystone.
 I'll also point out that it was largely because of the storage needs
 that I chose to propose Boson[1] as a separate app, rather than as a
 library.  Further, the dimensions over which quota-covered resources
 needed to be tracked seemed to me to be complicated enough that it would
 be better to define a new app and make it support that one domain well,
 which is why I didn't propose it as something to add to Keystone.
 Consider: nova has quotas that are applied by user, other quotas that
 are applied by tenant, and even some quotas on what could be considered
 sub-resources—a limit on the number of security group rules per security
 group, for instance.
 
 My current feeling is that, if we can figure out a way to make the quota
 problem into an acceptable library, that will work; it would probably
 have to maintain its own database separate from the client app and have
 features for automatically managing the schema, since we couldn't
 necessarily rely on the client app to invoke the proper juju there.  If,
 on the other hand, that ends up failing, then the best route is probably
 to begin by developing a separate app, like Boson, as a PoC; then, after
 we have some idea of just how difficult it is to actually solve the
 problem, we can evaluate whether it makes sense to actually fold it into
 a service like Keystone, or whether it should stand on its own.
 
 (Personally, I think Boson should be created and should stand on its
 own, but I also envision using it for purposes outside of OpenStack…)
 Thanks for mentioning Boson again. I’m embarrassed that I completely 
 forgot about the fact that you mentioned this at the summit.
 
 I’ll have to look at the proposal more closely before I comment in any 
 detail, but I take it as a good sign that we’re coming back around to the 
 idea of solving this with an app instead of a library.
 I assume I'm really late in the thread so I can just sit and give +1 to 
 this direction : IMHO, quotas need to managed thanks to a CRUD interface 
 which implies to get an app, as it sounds unreasonable to extend each 
 consumer app API.
 
 That said, back to Blazar, I just would like to emphasize that Blazar is 
 not trying to address the quota enforcement level, but rather provide a 
 centralized endpoint for managing reservations.
 Consequently, Blazar can also be considered as a consumer of this quota 
 system, whatever it's in a library or on a separate REST API.
 
 Last thing, I don't think that a quota application necessarly means that 
 quotas enforcement should be managed thanks to external calls to this app. 
 I can rather see an external system able to set for each project a local 
 view of what should be enforced locally. If operators don't want to deploy 
 that quota management project, it's up to them to address the hetergenous 
 setups for each project.
 I’m not sure what this means. You want the new service to be optional? How 
 would apps written against the service find and manage quota data if the 
 service isn’t there?
 
 My bad. Let me rephrase it. I'm seeing this service as providing added value 
 for managing quotas by ensuring consistency across all projects. But as I 
 said, I'm also thinking that the quota enforcement has still to be done at 
 the customer project level.

Oh, yes, that is true. I envision the API for the new service having a call 
that means “try to consume X units of a given quota” and that it would return 
information about whether that can be done. The apps would have to define what 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Blair Bethwaite
On 20 November 2014 05:25,  openstack-dev-requ...@lists.openstack.org wrote:
 --

 Message: 24
 Date: Wed, 19 Nov 2014 10:57:17 -0500
 From: Doug Hellmann d...@doughellmann.com
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] Quota management and enforcement across
 projects
 Message-ID: 13f4f7a1-d4ec-4d14-a163-d477a4fd9...@doughellmann.com
 Content-Type: text/plain; charset=windows-1252


 On Nov 19, 2014, at 9:51 AM, Sylvain Bauza sba...@redhat.com wrote:
 My bad. Let me rephrase it. I'm seeing this service as providing added value 
 for managing quotas by ensuring consistency across all projects. But as I 
 said, I'm also thinking that the quota enforcement has still to be done at 
 the customer project level.

 Oh, yes, that is true. I envision the API for the new service having a call 
 that means ?try to consume X units of a given quota? and that it would return 
 information about whether that can be done. The apps would have to define 
 what quotas they care about, and make the appropriate calls.

For actions initiated directly through core OpenStack service APIs
(Nova, Cinder, Neutron, etc - anything using Keystone policy),
shouldn't quota-enforcement be handled by Keystone? To me this is just
a subset of authz, and OpenStack already has a well established
service for such decisions.

It sounds like the idea here is to provide something generic that
could be used outside of OpenStack? I worry that might be premature
scope creep that detracts from the outcome.

-- 
Cheers,
~Blairo

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Kevin L. Mitchell
On Thu, 2014-11-20 at 10:16 +1100, Blair Bethwaite wrote:
 For actions initiated directly through core OpenStack service APIs
 (Nova, Cinder, Neutron, etc - anything using Keystone policy),
 shouldn't quota-enforcement be handled by Keystone? To me this is just
 a subset of authz, and OpenStack already has a well established
 service for such decisions.

If you look a little earlier in the thread, you will find a post from me
where I point out just how complicated quota management actually is.  I
suggest that it should be developed as a proof-of-concept as a separate
service; from there, we can see whether it makes sense to roll it into
Keystone or maintain it as a separate thing.
-- 
Kevin L. Mitchell kevin.mitch...@rackspace.com
Rackspace


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread Salvatore Orlando
Apparently, like everything Openstack-y we have a gathered a good crowd of
people with different opinions, more or less different, more or less strong.
My only strong opinion is that any ocean-boiling attempt should be
carefully avoided, and any proposed approach should add as little as
possible in terms of delays and complexity to the current process.
Anyway, I am glad nobody said so far QaaS. If you're thinking to do so,
please don't. Please.

The reason for the now almost-abandoned proposal for a library was to have
a simple solution for allowing every project to enforce quotas in the same
ways.
As Kevin said, managing quotas for resources and their sequence attributes
(like the sec group rules for instance), can become pretty messy soon. As
of today this problem is avoided since quota management logic is baked in
every project's business logic. The library proposal also shamefully
avoided this problem - by focusing on enforcement only.

For enforcement, as Doug said, this is about answering the question
can I consume
X units of a given resource?. In a distributed, non-locking, world, this
is not really obvious. A Boson-like proposal would solve this problem by
providing a centralised enforcement point. In this case the only possible
approach is one in which such centralised endpoint does both enforcement
and management.
While I too think that we need a centralised quota management endpoint, be
it folded in Keystone or not, I still need to be convinced that this is the
right architectural decision for enforcing quotas.

First, can I consume X unit of a given resource implies the external
service should be aware of current resource usage. Arguably it's the
consumer service (eg: nova, cinder) that owns this information, not the
quota service. The quota service can however be made aware of this in
several ways:
- notifications. In this case the service will also become a consumer of a
telemetry service like ceilometer or stacktach. I'm not sure we want to add
that dependency
- the API call itself might ask for reserving a given amount resource and
communicate at the same time resource usage. This sounds better, but why
would it be better than the dual approach, in which the consumer service
makes the reservation using quota values fetched by an external service.
Between quota and usage, the latter seem to be the more dynamic ones;
quotas can also be easily cached.
- usage info might be updated when reservations are committed. This looks
smart, but requirers more thinking for dealing with failure scenarios where
the failure occurs once resources are reserved but before they're committed
and there is no way to know which ones were actually and which ones not.

Second, calling an external service is not cheap. Especially since those
calls might be done several times for completing a single API request.
Think about POST /servers with all the ramifications into glance, neutron,
and cinder. In the simplest case we'd need two calls - one to reserve a
resource (or a set of resources), and one to either confirm or cancel the
reservation. This alone adds two round trips for each API call. Could this
scale well? To some extent it reminds me of the mayhem that was the
nova/neutron library before caching for keystone tokens was fixed.

Somebody mentioned an analogy with AuthZ. For some reason, perhaps
different from the reasons of who originally brought the analogy, I also
believe that this is a decent model to follow.
Quotas for each resource, together with the resource structure, might be
stored by a centralised service. This is pretty much inline with Boson's
goals, if I understand correctly. It will expose tenant-facing REST APIs
for quota management and will finally solve the problem of having a
standardized way for handling quotas, regardless of whether it stand as its
own service or is folded into Keystone, operators should be happier. And
there's nothing more satisfying that the face of a happy operator...
provided we also do not break backward compatibility!

Enforcement is something that in my opinion should happen locally, just
like we do authZ locally using information provided by Keystone upon
authentication. The issue Doug correctly pointed out however is that quota
enforcement is a too complex to be implemented in a library, like, for
instance, oslo.policy.
So, assuming there might consensus on doing quota enforcement locally in
the consumer application, what should be the 'thing' that enforces policy
if it can't be a library? Could it be a quota agent which communicates with
quota server, caches quota info? If yes, APIs like those for managing
reservation, over which transport should be invoked? IPC, AMQP, REST? I
need to go back to the drawing board to see if this can possibly work, but
in the meanwhile your feedback is more than welcome.

Salvatore

On 20 November 2014 00:39, Kevin L. Mitchell kevin.mitch...@rackspace.com
wrote:

 On Thu, 2014-11-20 at 10:16 +1100, Blair Bethwaite wrote:
  For 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-19 Thread joehuang
+1

There are lots of concurrent requests to the quota management service if it's 
shared for projects, especially if it's shared for multi-regions (KeyStone can 
be global service for multi-regions), also latency will affect the end user 
experience. POC is good idea to verify the concept.

Best Regards
Chaoyi Huang ( Joe Huang )

-Original Message-
From: Kevin L. Mitchell [mailto:kevin.mitch...@rackspace.com] 
Sent: Thursday, November 20, 2014 7:40 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] Quota management and enforcement across projects

On Thu, 2014-11-20 at 10:16 +1100, Blair Bethwaite wrote:
 For actions initiated directly through core OpenStack service APIs 
 (Nova, Cinder, Neutron, etc - anything using Keystone policy), 
 shouldn't quota-enforcement be handled by Keystone? To me this is just 
 a subset of authz, and OpenStack already has a well established 
 service for such decisions.

If you look a little earlier in the thread, you will find a post from me where 
I point out just how complicated quota management actually is.  I suggest that 
it should be developed as a proof-of-concept as a separate service; from there, 
we can see whether it makes sense to roll it into Keystone or maintain it as a 
separate thing.
--
Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-11-18 Thread Doug Hellmann

On Nov 17, 2014, at 7:18 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com 
wrote:

 On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote:
 I’ve spent a bit of time thinking about the resource ownership issue.
 The challenge there is we don’t currently have any libraries that
 define tables in the schema of an application. I think that’s a good
 pattern to maintain, since it avoids introducing a lot of tricky
 issues like how to manage migrations for the library, how to ensure
 they are run by the application, etc. The fact that this common quota
 thing needs to store some data in a schema that it controls says to me
 that it is really an app and not a library. Making the quota manager
 an app solves the API definition issue, too, since we can describe a
 generic way to configure quotas and other applications can then use
 that API to define specific rules using the quota manager’s API.
 
 I don’t know if we need a new application or if it would make sense
 to, as with policy, add quota management features to keystone. A
 single well-defined app has some appeal, but there’s also a certain
 amount of extra ramp-up time needed to go that route that we wouldn’t
 need if we added the features directly to keystone.
 
 I'll also point out that it was largely because of the storage needs
 that I chose to propose Boson[1] as a separate app, rather than as a
 library.  Further, the dimensions over which quota-covered resources
 needed to be tracked seemed to me to be complicated enough that it would
 be better to define a new app and make it support that one domain well,
 which is why I didn't propose it as something to add to Keystone.
 Consider: nova has quotas that are applied by user, other quotas that
 are applied by tenant, and even some quotas on what could be considered
 sub-resources—a limit on the number of security group rules per security
 group, for instance.
 
 My current feeling is that, if we can figure out a way to make the quota
 problem into an acceptable library, that will work; it would probably
 have to maintain its own database separate from the client app and have
 features for automatically managing the schema, since we couldn't
 necessarily rely on the client app to invoke the proper juju there.  If,
 on the other hand, that ends up failing, then the best route is probably
 to begin by developing a separate app, like Boson, as a PoC; then, after
 we have some idea of just how difficult it is to actually solve the
 problem, we can evaluate whether it makes sense to actually fold it into
 a service like Keystone, or whether it should stand on its own.
 
 (Personally, I think Boson should be created and should stand on its
 own, but I also envision using it for purposes outside of OpenStack…)

Thanks for mentioning Boson again. I’m embarrassed that I completely forgot 
about the fact that you mentioned this at the summit.

I’ll have to look at the proposal more closely before I comment in any detail, 
but I take it as a good sign that we’re coming back around to the idea of 
solving this with an app instead of a library.

Doug

 
 Just my $.02…
 
 [1] https://wiki.openstack.org/wiki/Boson
 -- 
 Kevin L. Mitchell kevin.mitch...@rackspace.com
 Rackspace
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-11-17 Thread Salvatore Orlando
Hi all,

I am resuming this thread following the session we had at the summit in
Paris (etherpad here [1])

While there was some sort of consensus regarding what this library should
do, and how it should do it, the session ended with some open questions
which we need to address before finalising the specification.

There was a rather large consensus that the only viable architecture would
be one where the quota management library owns resource usage data.
However, this raises further concerns around:
- ownership of resource usage records. As the quota library owns usage
data, it becomes the authoritative source of truth for it. This could be
problematic in some projects, particularly nova, where the resource tracker
currently own usage data.
- inserting quota-related db migrations in the consumer project timeline.
We'll need to solve this for both sqlalchemy.migrate and alembic (I seem to
recall Doug has a solution for this)

While the proposed interface for quota enforcing was generally well
received, another open question is whether this library should also be
responsible for quota management. Each project currently exposes its own
extension for setting quotas for tenants, and this is rather bad from an
usability perspective. whether this should be responsibility of the quota
library, handle by a separate quota service, or handled through keystone
is debatable. Personally I think the library should not get in the business
of defining APIs. However, it can define the business logic for managing
tenant quotas which can then be leveraged by the projects which will adopt
it. Either way, I reckon that we might be better off by limiting our scope
to quota enforcement for the first iteration of this library.

The last, and possibly most important question, is whether this library
belongs to the oslo-incubator or should live into its own repository. I
would personally prefer the oslo incubator because its process makes
adoption by OpenStack projects fairly simple. On the other hand, I also
understand the concerns around a library that comes with its own data
models and has the potential of becoming something bigger than a simple
library. I would like to defer an answer to this question to the oslo core
team, which are surely a lot more experienced than me on the matter of the
incubator.

To this aim, the proposed specification [2] has been temporarily put in WIP.

A few more notes:

1) We agreed that we won't be constrained about saving any part of the
current module, since it's not used by any project. We will however reuse
whatever can be reused (mostly for not wasting time by writing the same
thing again)

2) As concerns data backends to support, we agreed that we would start with
the simplest possible solution. A single SQLalchemy-backed data source.
Potentially we won't even have a framework for plugging different kind of
data sources. However if contributors step in with support for multiple
backends, we will consider it.

3) The idea of having a quota engine leveraging data sources was quite
welcomed. Ideally it's not that different from the engine/driver model in
the current module, but there will be a cleaner separation of
responsibilities.

Salvatore

[1] https://etherpad.openstack.org/p/kilo-oslo-common-quota-library
[2] https://review.openstack.org/#/c/132127/

On 15 October 2014 20:54, Doug Hellmann d...@doughellmann.com wrote:

 We can make it support whatever we want! :-)

 The existing code almost certainly predates domain support in keystone, so
 I think this is a case of needing to catch up not a conscious decision to
 leave it out.

 Doug


 On Oct 15, 2014, at 12:15 PM, Ajaya Agrawal ajku@gmail.com wrote:

 Hi,

 Would the new library support quota on domain level also? As it stands in
 oslo-incubator it only does quota enforcement on project level only. The
 use case for this is quota enforcement across multiple projects. For e.g.
 as a cloud provider, I would like my customer to create only #X volumes
 across all his projects.

 -Ajaya

 Cheers,
 Ajaya

 On Wed, Oct 15, 2014 at 7:04 PM, Doug Hellmann d...@doughellmann.com
 wrote:

 Sigh. Because I typed the wrong command. Thanks for pointing that out.

 I don’t see any instances of “quota” in openstack-common.conf files:

 $ grep quota */openstack-common.conf

 or in any projects under openstack/“:

 $ ls */*/openstack/common/quota.py
 ls: cannot access */*/openstack/common/quota.py: No such file or directory

 I don’t know where manila’s copy came from, but if it has been copied
 from the incubator by hand and then changed we should fix that up.

 Doug

 On Oct 15, 2014, at 5:28 AM, Valeriy Ponomaryov vponomar...@mirantis.com
 wrote:

 But why policy is being discussed on quota thread?

 On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov 
 vponomar...@mirantis.com wrote:

 Manila project does use policy common code from incubator.

 Our small wrapper for it:
 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-17 Thread Doug Hellmann

On Nov 17, 2014, at 5:49 PM, Salvatore Orlando sorla...@nicira.com wrote:

 Hi all,
 
 I am resuming this thread following the session we had at the summit in Paris 
 (etherpad here [1])
 
 While there was some sort of consensus regarding what this library should do, 
 and how it should do it, the session ended with some open questions which we 
 need to address before finalising the specification.
 
 There was a rather large consensus that the only viable architecture would be 
 one where the quota management library owns resource usage data. However, 
 this raises further concerns around:
 - ownership of resource usage records. As the quota library owns usage data, 
 it becomes the authoritative source of truth for it. This could be 
 problematic in some projects, particularly nova, where the resource tracker 
 currently own usage data.
 - inserting quota-related db migrations in the consumer project timeline. 
 We'll need to solve this for both sqlalchemy.migrate and alembic (I seem to 
 recall Doug has a solution for this)
 
 While the proposed interface for quota enforcing was generally well received, 
 another open question is whether this library should also be responsible for 
 quota management. Each project currently exposes its own extension for 
 setting quotas for tenants, and this is rather bad from an usability 
 perspective. whether this should be responsibility of the quota library, 
 handle by a separate quota service, or handled through keystone is 
 debatable. Personally I think the library should not get in the business of 
 defining APIs. However, it can define the business logic for managing tenant 
 quotas which can then be leveraged by the projects which will adopt it. 
 Either way, I reckon that we might be better off by limiting our scope to 
 quota enforcement for the first iteration of this library.
 
 The last, and possibly most important question, is whether this library 
 belongs to the oslo-incubator or should live into its own repository. I would 
 personally prefer the oslo incubator because its process makes adoption by 
 OpenStack projects fairly simple. On the other hand, I also understand the 
 concerns around a library that comes with its own data models and has the 
 potential of becoming something bigger than a simple library. I would like to 
 defer an answer to this question to the oslo core team, which are surely a 
 lot more experienced than me on the matter of the incubator.

I’ve spent a bit of time thinking about the resource ownership issue. The 
challenge there is we don’t currently have any libraries that define tables in 
the schema of an application. I think that’s a good pattern to maintain, since 
it avoids introducing a lot of tricky issues like how to manage migrations for 
the library, how to ensure they are run by the application, etc. The fact that 
this common quota thing needs to store some data in a schema that it controls 
says to me that it is really an app and not a library. Making the quota manager 
an app solves the API definition issue, too, since we can describe a generic 
way to configure quotas and other applications can then use that API to define 
specific rules using the quota manager’s API.

I don’t know if we need a new application or if it would make sense to, as with 
policy, add quota management features to keystone. A single well-defined app 
has some appeal, but there’s also a certain amount of extra ramp-up time needed 
to go that route that we wouldn’t need if we added the features directly to 
keystone.

Doug

 
 To this aim, the proposed specification [2] has been temporarily put in WIP.
 
 A few more notes:
 
 1) We agreed that we won't be constrained about saving any part of the 
 current module, since it's not used by any project. We will however reuse 
 whatever can be reused (mostly for not wasting time by writing the same thing 
 again)
 
 2) As concerns data backends to support, we agreed that we would start with 
 the simplest possible solution. A single SQLalchemy-backed data source. 
 Potentially we won't even have a framework for plugging different kind of 
 data sources. However if contributors step in with support for multiple 
 backends, we will consider it. 
 
 3) The idea of having a quota engine leveraging data sources was quite 
 welcomed. Ideally it's not that different from the engine/driver model in the 
 current module, but there will be a cleaner separation of responsibilities.
 
 Salvatore
 
 [1] https://etherpad.openstack.org/p/kilo-oslo-common-quota-library
 [2] https://review.openstack.org/#/c/132127/
 
 On 15 October 2014 20:54, Doug Hellmann d...@doughellmann.com wrote:
 We can make it support whatever we want! :-)
 
 The existing code almost certainly predates domain support in keystone, so I 
 think this is a case of needing to catch up not a conscious decision to leave 
 it out.
 
 Doug
 
 
 On Oct 15, 2014, at 12:15 PM, Ajaya Agrawal ajku@gmail.com wrote:
 
 Hi,
 
 Would the new 

Re: [openstack-dev] Quota management and enforcement across projects

2014-11-17 Thread Kevin L. Mitchell
On Mon, 2014-11-17 at 18:48 -0500, Doug Hellmann wrote:
 I’ve spent a bit of time thinking about the resource ownership issue.
 The challenge there is we don’t currently have any libraries that
 define tables in the schema of an application. I think that’s a good
 pattern to maintain, since it avoids introducing a lot of tricky
 issues like how to manage migrations for the library, how to ensure
 they are run by the application, etc. The fact that this common quota
 thing needs to store some data in a schema that it controls says to me
 that it is really an app and not a library. Making the quota manager
 an app solves the API definition issue, too, since we can describe a
 generic way to configure quotas and other applications can then use
 that API to define specific rules using the quota manager’s API.
 
 I don’t know if we need a new application or if it would make sense
 to, as with policy, add quota management features to keystone. A
 single well-defined app has some appeal, but there’s also a certain
 amount of extra ramp-up time needed to go that route that we wouldn’t
 need if we added the features directly to keystone.

I'll also point out that it was largely because of the storage needs
that I chose to propose Boson[1] as a separate app, rather than as a
library.  Further, the dimensions over which quota-covered resources
needed to be tracked seemed to me to be complicated enough that it would
be better to define a new app and make it support that one domain well,
which is why I didn't propose it as something to add to Keystone.
Consider: nova has quotas that are applied by user, other quotas that
are applied by tenant, and even some quotas on what could be considered
sub-resources—a limit on the number of security group rules per security
group, for instance.

My current feeling is that, if we can figure out a way to make the quota
problem into an acceptable library, that will work; it would probably
have to maintain its own database separate from the client app and have
features for automatically managing the schema, since we couldn't
necessarily rely on the client app to invoke the proper juju there.  If,
on the other hand, that ends up failing, then the best route is probably
to begin by developing a separate app, like Boson, as a PoC; then, after
we have some idea of just how difficult it is to actually solve the
problem, we can evaluate whether it makes sense to actually fold it into
a service like Keystone, or whether it should stand on its own.

(Personally, I think Boson should be created and should stand on its
own, but I also envision using it for purposes outside of OpenStack…)

Just my $.02…

[1] https://wiki.openstack.org/wiki/Boson
-- 
Kevin L. Mitchell kevin.mitch...@rackspace.com
Rackspace


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-10-15 Thread Valeriy Ponomaryov
Manila project does use policy common code from incubator.

Our small wrapper for it:
https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py


On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com
wrote:

 Doug,

 I totally agree with your findings on the policy module.
 Neutron already has some customizations there and we already have a few
 contributors working on syncing it back with oslo-incubator during the Kilo
 release cycle.

 However, my query was about the quota module.
 From what I gather it seems not a lot of projects use it:

 $ find . -name openstack-common.conf | xargs grep quota
 $

 Salvatore

 On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com
 wrote:

 Hi Doug,

 do you know if the existing quota oslo-incubator module has already some
 active consumers?
 In the meanwhile I've pushed a spec to neutron-specs for improving quota
 management there [1]


 It looks like a lot of projects are syncing the module:

 $ grep policy */openstack-common.conf

 barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy
 ceilometer/openstack-common.conf:module=policy
 cinder/openstack-common.conf:module=policy
 designate/openstack-common.conf:module=policy
 gantt/openstack-common.conf:module=policy
 glance/openstack-common.conf:module=policy
 heat/openstack-common.conf:module=policy
 horizon/openstack-common.conf:module=policy
 ironic/openstack-common.conf:module=policy
 keystone/openstack-common.conf:module=policy
 manila/openstack-common.conf:module=policy
 neutron/openstack-common.conf:module=policy
 nova/openstack-common.conf:module=policy
 trove/openstack-common.conf:module=policy
 tuskar/openstack-common.conf:module=policy

 I’m not sure how many are actively using it, but I wouldn’t expect them
 to copy it in if they weren’t using it at all.


 Now, I can either work on the oslo-incubator module and leverage it in
 Neutron, or develop the quota module in Neutron, and move it to
 oslo-incubator once we validate it with Neutron. The latter approach seems
 easier from a workflow perspective - as it avoid the intermediate steps of
 moving code from oslo-incubator to neutron. On the other hand it will delay
 adoption in oslo-incubator.


 The policy module is up for graduation this cycle. It may end up in its
 own library, to allow us to build a review team for the code more easily
 than if we put it in with some of the other semi-related modules like the
 server code. We’re still working that out [1], and if you expect to make a
 lot of incompatible changes we should delay graduation to make that simpler.

 Either way, since we have so many consumers, I think it would be easier
 to have the work happen in Oslo somewhere so we can ensure those changes
 are useful to and usable by all of the existing consumers.

 Doug

 [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals


 What's your opinion?

 Regards,
 Salvatore

 [1] https://review.openstack.org/#/c/128318/

 On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote:

  Salvatore, Joe,
 
  We do have this at the moment:
 
 
 https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
 
  — dims

 If someone wants to drive creating a useful library during kilo, please
 consider adding the topic to the etherpad we’re using to plan summit
 sessions and then come participate in the Oslo meeting this Friday 16:00
 UTC.

 https://etherpad.openstack.org/p/kilo-oslo-summit-topics

 Doug

 
  On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com
 wrote:
 
  On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:
 
 
  On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
  morgan.fainb...@gmail.com wrote:
 
  Keeping the enforcement local (same way policy works today) helps
 limit
  the fragility, big +1 there.
 
  I also agree with Vish, we need a uniform way to talk about quota
  enforcement similar to how we have a uniform policy language /
 enforcement
  model (yes I know it's not perfect, but it's far closer to uniform
 than
  quota management is).
 
 
  It sounds like maybe we should have an oslo library for quotas?
 Somewhere
  where we can share the code,but keep the operations local to each
 service.
 
 
  This is what I had in mind as well. A simple library for quota
 enforcement
  which can be used regardless of where and how you do it, which might
 depend
  on the application business logic, the WSGI framework in use, or other
  factors.
 
 
 
 
  If there is still interest of placing quota in keystone, let's talk
 about
  how that will work and what will be needed from Keystone . The
 previous
  attempt didn't get much traction and stalled out early in
 implementation. If
  we want to revisit this lets make sure we have 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-15 Thread Valeriy Ponomaryov
But why policy is being discussed on quota thread?

On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov 
vponomar...@mirantis.com wrote:

 Manila project does use policy common code from incubator.

 Our small wrapper for it:
 https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py


 On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com
 wrote:

 Doug,

 I totally agree with your findings on the policy module.
 Neutron already has some customizations there and we already have a few
 contributors working on syncing it back with oslo-incubator during the Kilo
 release cycle.

 However, my query was about the quota module.
 From what I gather it seems not a lot of projects use it:

 $ find . -name openstack-common.conf | xargs grep quota
 $

 Salvatore

 On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com
 wrote:

 Hi Doug,

 do you know if the existing quota oslo-incubator module has already some
 active consumers?
 In the meanwhile I've pushed a spec to neutron-specs for improving quota
 management there [1]


 It looks like a lot of projects are syncing the module:

 $ grep policy */openstack-common.conf

 barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy
 ceilometer/openstack-common.conf:module=policy
 cinder/openstack-common.conf:module=policy
 designate/openstack-common.conf:module=policy
 gantt/openstack-common.conf:module=policy
 glance/openstack-common.conf:module=policy
 heat/openstack-common.conf:module=policy
 horizon/openstack-common.conf:module=policy
 ironic/openstack-common.conf:module=policy
 keystone/openstack-common.conf:module=policy
 manila/openstack-common.conf:module=policy
 neutron/openstack-common.conf:module=policy
 nova/openstack-common.conf:module=policy
 trove/openstack-common.conf:module=policy
 tuskar/openstack-common.conf:module=policy

 I’m not sure how many are actively using it, but I wouldn’t expect them
 to copy it in if they weren’t using it at all.


 Now, I can either work on the oslo-incubator module and leverage it in
 Neutron, or develop the quota module in Neutron, and move it to
 oslo-incubator once we validate it with Neutron. The latter approach seems
 easier from a workflow perspective - as it avoid the intermediate steps of
 moving code from oslo-incubator to neutron. On the other hand it will delay
 adoption in oslo-incubator.


 The policy module is up for graduation this cycle. It may end up in its
 own library, to allow us to build a review team for the code more easily
 than if we put it in with some of the other semi-related modules like the
 server code. We’re still working that out [1], and if you expect to make a
 lot of incompatible changes we should delay graduation to make that simpler.

 Either way, since we have so many consumers, I think it would be easier
 to have the work happen in Oslo somewhere so we can ensure those changes
 are useful to and usable by all of the existing consumers.

 Doug

 [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals


 What's your opinion?

 Regards,
 Salvatore

 [1] https://review.openstack.org/#/c/128318/

 On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote:

  Salvatore, Joe,
 
  We do have this at the moment:
 
 
 https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
 
  — dims

 If someone wants to drive creating a useful library during kilo, please
 consider adding the topic to the etherpad we’re using to plan summit
 sessions and then come participate in the Oslo meeting this Friday 16:00
 UTC.

 https://etherpad.openstack.org/p/kilo-oslo-summit-topics

 Doug

 
  On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando 
 sorla...@nicira.com wrote:
 
  On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:
 
 
  On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
  morgan.fainb...@gmail.com wrote:
 
  Keeping the enforcement local (same way policy works today) helps
 limit
  the fragility, big +1 there.
 
  I also agree with Vish, we need a uniform way to talk about quota
  enforcement similar to how we have a uniform policy language /
 enforcement
  model (yes I know it's not perfect, but it's far closer to uniform
 than
  quota management is).
 
 
  It sounds like maybe we should have an oslo library for quotas?
 Somewhere
  where we can share the code,but keep the operations local to each
 service.
 
 
  This is what I had in mind as well. A simple library for quota
 enforcement
  which can be used regardless of where and how you do it, which might
 depend
  on the application business logic, the WSGI framework in use, or
 other
  factors.
 
 
 
 
  If there is still interest of placing quota in keystone, let's
 talk about
  how that will work and what will be needed from 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-15 Thread Doug Hellmann
Sigh. Because I typed the wrong command. Thanks for pointing that out.

I don’t see any instances of “quota” in openstack-common.conf files:

$ grep quota */openstack-common.conf

or in any projects under openstack/“:

$ ls */*/openstack/common/quota.py
ls: cannot access */*/openstack/common/quota.py: No such file or directory

I don’t know where manila’s copy came from, but if it has been copied from the 
incubator by hand and then changed we should fix that up.

Doug

On Oct 15, 2014, at 5:28 AM, Valeriy Ponomaryov vponomar...@mirantis.com 
wrote:

 But why policy is being discussed on quota thread?
 
 On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov 
 vponomar...@mirantis.com wrote:
 Manila project does use policy common code from incubator.
 
 Our small wrapper for it: 
 https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py
 
 
 On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com 
 wrote:
 Doug,
 
 I totally agree with your findings on the policy module.
 Neutron already has some customizations there and we already have a few 
 contributors working on syncing it back with oslo-incubator during the Kilo 
 release cycle.
 
 However, my query was about the quota module.
 From what I gather it seems not a lot of projects use it:
 
 $ find . -name openstack-common.conf | xargs grep quota
 $
 
 Salvatore
 
 On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote:
 
 On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote:
 
 Hi Doug,
 
 do you know if the existing quota oslo-incubator module has already some 
 active consumers?
 In the meanwhile I've pushed a spec to neutron-specs for improving quota 
 management there [1]
 
 It looks like a lot of projects are syncing the module:
 
 $ grep policy */openstack-common.conf
 barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy
 ceilometer/openstack-common.conf:module=policy
 cinder/openstack-common.conf:module=policy
 designate/openstack-common.conf:module=policy
 gantt/openstack-common.conf:module=policy
 glance/openstack-common.conf:module=policy
 heat/openstack-common.conf:module=policy
 horizon/openstack-common.conf:module=policy
 ironic/openstack-common.conf:module=policy
 keystone/openstack-common.conf:module=policy
 manila/openstack-common.conf:module=policy
 neutron/openstack-common.conf:module=policy
 nova/openstack-common.conf:module=policy
 trove/openstack-common.conf:module=policy
 tuskar/openstack-common.conf:module=policy
 
 I’m not sure how many are actively using it, but I wouldn’t expect them to 
 copy it in if they weren’t using it at all.
 
 
 Now, I can either work on the oslo-incubator module and leverage it in 
 Neutron, or develop the quota module in Neutron, and move it to 
 oslo-incubator once we validate it with Neutron. The latter approach seems 
 easier from a workflow perspective - as it avoid the intermediate steps of 
 moving code from oslo-incubator to neutron. On the other hand it will delay 
 adoption in oslo-incubator.
 
 The policy module is up for graduation this cycle. It may end up in its own 
 library, to allow us to build a review team for the code more easily than if 
 we put it in with some of the other semi-related modules like the server 
 code. We’re still working that out [1], and if you expect to make a lot of 
 incompatible changes we should delay graduation to make that simpler.
 
 Either way, since we have so many consumers, I think it would be easier to 
 have the work happen in Oslo somewhere so we can ensure those changes are 
 useful to and usable by all of the existing consumers.
 
 Doug
 
 [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals
 
 
 What's your opinion?
 
 Regards,
 Salvatore
 
 [1] https://review.openstack.org/#/c/128318/
 
 On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote:
 
 On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote:
 
  Salvatore, Joe,
 
  We do have this at the moment:
 
  https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
 
  — dims
 
 If someone wants to drive creating a useful library during kilo, please 
 consider adding the topic to the etherpad we’re using to plan summit 
 sessions and then come participate in the Oslo meeting this Friday 16:00 UTC.
 
 https://etherpad.openstack.org/p/kilo-oslo-summit-topics
 
 Doug
 
 
  On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com 
  wrote:
 
  On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:
 
 
  On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
  morgan.fainb...@gmail.com wrote:
 
  Keeping the enforcement local (same way policy works today) helps limit
  the fragility, big +1 there.
 
  I also agree with Vish, we need a uniform way to talk about quota
  enforcement similar to how we have a uniform policy language / 
  enforcement
  model (yes I know it's not perfect, but 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-15 Thread Ajaya Agrawal
Hi,

Would the new library support quota on domain level also? As it stands in
oslo-incubator it only does quota enforcement on project level only. The
use case for this is quota enforcement across multiple projects. For e.g.
as a cloud provider, I would like my customer to create only #X volumes
across all his projects.

-Ajaya

Cheers,
Ajaya

On Wed, Oct 15, 2014 at 7:04 PM, Doug Hellmann d...@doughellmann.com
wrote:

 Sigh. Because I typed the wrong command. Thanks for pointing that out.

 I don’t see any instances of “quota” in openstack-common.conf files:

 $ grep quota */openstack-common.conf

 or in any projects under openstack/“:

 $ ls */*/openstack/common/quota.py
 ls: cannot access */*/openstack/common/quota.py: No such file or directory

 I don’t know where manila’s copy came from, but if it has been copied from
 the incubator by hand and then changed we should fix that up.

 Doug

 On Oct 15, 2014, at 5:28 AM, Valeriy Ponomaryov vponomar...@mirantis.com
 wrote:

 But why policy is being discussed on quota thread?

 On Wed, Oct 15, 2014 at 11:55 AM, Valeriy Ponomaryov 
 vponomar...@mirantis.com wrote:

 Manila project does use policy common code from incubator.

 Our small wrapper for it:
 https://github.com/openstack/manila/blob/8203c51081680a7a9dba30ae02d7c43d6e18a124/manila/policy.py


 On Wed, Oct 15, 2014 at 2:41 AM, Salvatore Orlando sorla...@nicira.com
 wrote:

 Doug,

 I totally agree with your findings on the policy module.
 Neutron already has some customizations there and we already have a
 few contributors working on syncing it back with oslo-incubator during the
 Kilo release cycle.

 However, my query was about the quota module.
 From what I gather it seems not a lot of projects use it:

 $ find . -name openstack-common.conf | xargs grep quota
 $

 Salvatore

 On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com
 wrote:

 Hi Doug,

 do you know if the existing quota oslo-incubator module has already
 some active consumers?
 In the meanwhile I've pushed a spec to neutron-specs for improving
 quota management there [1]


 It looks like a lot of projects are syncing the module:

 $ grep policy */openstack-common.conf

 barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy
 ceilometer/openstack-common.conf:module=policy
 cinder/openstack-common.conf:module=policy
 designate/openstack-common.conf:module=policy
 gantt/openstack-common.conf:module=policy
 glance/openstack-common.conf:module=policy
 heat/openstack-common.conf:module=policy
 horizon/openstack-common.conf:module=policy
 ironic/openstack-common.conf:module=policy
 keystone/openstack-common.conf:module=policy
 manila/openstack-common.conf:module=policy
 neutron/openstack-common.conf:module=policy
 nova/openstack-common.conf:module=policy
 trove/openstack-common.conf:module=policy
 tuskar/openstack-common.conf:module=policy

 I’m not sure how many are actively using it, but I wouldn’t expect them
 to copy it in if they weren’t using it at all.


 Now, I can either work on the oslo-incubator module and leverage it in
 Neutron, or develop the quota module in Neutron, and move it to
 oslo-incubator once we validate it with Neutron. The latter approach seems
 easier from a workflow perspective - as it avoid the intermediate steps of
 moving code from oslo-incubator to neutron. On the other hand it will delay
 adoption in oslo-incubator.


 The policy module is up for graduation this cycle. It may end up in its
 own library, to allow us to build a review team for the code more easily
 than if we put it in with some of the other semi-related modules like the
 server code. We’re still working that out [1], and if you expect to make a
 lot of incompatible changes we should delay graduation to make that 
 simpler.

 Either way, since we have so many consumers, I think it would be easier
 to have the work happen in Oslo somewhere so we can ensure those changes
 are useful to and usable by all of the existing consumers.

 Doug

 [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals


 What's your opinion?

 Regards,
 Salvatore

 [1] https://review.openstack.org/#/c/128318/

 On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com
 wrote:

  Salvatore, Joe,
 
  We do have this at the moment:
 
 
 https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
 
  — dims

 If someone wants to drive creating a useful library during kilo,
 please consider adding the topic to the etherpad we’re using to plan 
 summit
 sessions and then come participate in the Oslo meeting this Friday 16:00
 UTC.

 https://etherpad.openstack.org/p/kilo-oslo-summit-topics

 Doug

 
  On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando 
 sorla...@nicira.com wrote:
 
  On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-14 Thread Salvatore Orlando
Hi Doug,

do you know if the existing quota oslo-incubator module has already some
active consumers?
In the meanwhile I've pushed a spec to neutron-specs for improving quota
management there [1]

Now, I can either work on the oslo-incubator module and leverage it in
Neutron, or develop the quota module in Neutron, and move it to
oslo-incubator once we validate it with Neutron. The latter approach seems
easier from a workflow perspective - as it avoid the intermediate steps of
moving code from oslo-incubator to neutron. On the other hand it will delay
adoption in oslo-incubator.

What's your opinion?

Regards,
Salvatore

[1] https://review.openstack.org/#/c/128318/

On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote:

  Salvatore, Joe,
 
  We do have this at the moment:
 
 
 https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
 
  — dims

 If someone wants to drive creating a useful library during kilo, please
 consider adding the topic to the etherpad we’re using to plan summit
 sessions and then come participate in the Oslo meeting this Friday 16:00
 UTC.

 https://etherpad.openstack.org/p/kilo-oslo-summit-topics

 Doug

 
  On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com
 wrote:
 
  On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:
 
 
  On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
  morgan.fainb...@gmail.com wrote:
 
  Keeping the enforcement local (same way policy works today) helps
 limit
  the fragility, big +1 there.
 
  I also agree with Vish, we need a uniform way to talk about quota
  enforcement similar to how we have a uniform policy language /
 enforcement
  model (yes I know it's not perfect, but it's far closer to uniform
 than
  quota management is).
 
 
  It sounds like maybe we should have an oslo library for quotas?
 Somewhere
  where we can share the code,but keep the operations local to each
 service.
 
 
  This is what I had in mind as well. A simple library for quota
 enforcement
  which can be used regardless of where and how you do it, which might
 depend
  on the application business logic, the WSGI framework in use, or other
  factors.
 
 
 
 
  If there is still interest of placing quota in keystone, let's talk
 about
  how that will work and what will be needed from Keystone . The
 previous
  attempt didn't get much traction and stalled out early in
 implementation. If
  we want to revisit this lets make sure we have the resources needed
 and
  spec(s) in progress / info on etherpads (similar to how the
 multitenancy
  stuff was handled at the last summit) as early as possible.
 
 
  Why not centralize quota management via the python-openstackclient,
 what
  is the benefit of getting keystone involved?
 
 
  Providing this through the openstack client in my opinion has the
  disadvantage that users which either use the REST API direct or write
 their
  own clients won't leverage it. I don't think it's a reasonable
 assumption
  that everybody will use python-openstackclient, is it?
 
  Said that, storing quotas in keystone poses a further challenge to the
  scalability of the system, which we shall perhaps address by using
  appropriate caching strategies and leveraging keystone notifications.
 Until
  we get that, I think that the openstack client will be the best way of
  getting a unified quota management experience.
 
  Salvatore
 
 
 
  Cheers,
  Morgan
 
  Sent via mobile
 
 
  On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com
  wrote:
 
  Thanks Vish,
 
  this seems a very reasonable first step as well - and since most
  projects would be enforcing quotas in the same way, the shared
 library would
  be the logical next step.
  After all this is quite the same thing we do with authZ.
 
  Duncan is expressing valid concerns which in my opinion can be
 addressed
  with an appropriate design - and a decent implementation.
 
  Salvatore
 
  On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com
  wrote:
 
  The proposal in the past was to keep quota enforcement local, but to
  put the resource limits into keystone. This seems like an obvious
 first
  step to me. Then a shared library for enforcing quotas with decent
  performance should be next. The quota calls in nova are extremely
  inefficient right now and it will only get worse when we try to add
  hierarchical projects and quotas.
 
  Vish
 
  On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com
  wrote:
 
  Taking quota out of the service / adding remote calls for quota
  management is going to make things fragile - you've somehow got to
  deal with the cases where your quota manager is slow, goes away,
  hiccups, drops connections etc. You'll also need some way of
  reconciling actual usage against quota usage periodically, to
 detect
  problems.
 
  On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com
  wrote:
 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-14 Thread Doug Hellmann

On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com wrote:

 Hi Doug,
 
 do you know if the existing quota oslo-incubator module has already some 
 active consumers?
 In the meanwhile I've pushed a spec to neutron-specs for improving quota 
 management there [1]

It looks like a lot of projects are syncing the module:

$ grep policy */openstack-common.conf
barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy
ceilometer/openstack-common.conf:module=policy
cinder/openstack-common.conf:module=policy
designate/openstack-common.conf:module=policy
gantt/openstack-common.conf:module=policy
glance/openstack-common.conf:module=policy
heat/openstack-common.conf:module=policy
horizon/openstack-common.conf:module=policy
ironic/openstack-common.conf:module=policy
keystone/openstack-common.conf:module=policy
manila/openstack-common.conf:module=policy
neutron/openstack-common.conf:module=policy
nova/openstack-common.conf:module=policy
trove/openstack-common.conf:module=policy
tuskar/openstack-common.conf:module=policy

I’m not sure how many are actively using it, but I wouldn’t expect them to copy 
it in if they weren’t using it at all.

 
 Now, I can either work on the oslo-incubator module and leverage it in 
 Neutron, or develop the quota module in Neutron, and move it to 
 oslo-incubator once we validate it with Neutron. The latter approach seems 
 easier from a workflow perspective - as it avoid the intermediate steps of 
 moving code from oslo-incubator to neutron. On the other hand it will delay 
 adoption in oslo-incubator.

The policy module is up for graduation this cycle. It may end up in its own 
library, to allow us to build a review team for the code more easily than if we 
put it in with some of the other semi-related modules like the server code. 
We’re still working that out [1], and if you expect to make a lot of 
incompatible changes we should delay graduation to make that simpler.

Either way, since we have so many consumers, I think it would be easier to have 
the work happen in Oslo somewhere so we can ensure those changes are useful to 
and usable by all of the existing consumers.

Doug

[1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals

 
 What's your opinion?
 
 Regards,
 Salvatore
 
 [1] https://review.openstack.org/#/c/128318/
 
 On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote:
 
 On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote:
 
  Salvatore, Joe,
 
  We do have this at the moment:
 
  https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
 
  — dims
 
 If someone wants to drive creating a useful library during kilo, please 
 consider adding the topic to the etherpad we’re using to plan summit sessions 
 and then come participate in the Oslo meeting this Friday 16:00 UTC.
 
 https://etherpad.openstack.org/p/kilo-oslo-summit-topics
 
 Doug
 
 
  On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com 
  wrote:
 
  On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:
 
 
  On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
  morgan.fainb...@gmail.com wrote:
 
  Keeping the enforcement local (same way policy works today) helps limit
  the fragility, big +1 there.
 
  I also agree with Vish, we need a uniform way to talk about quota
  enforcement similar to how we have a uniform policy language / 
  enforcement
  model (yes I know it's not perfect, but it's far closer to uniform than
  quota management is).
 
 
  It sounds like maybe we should have an oslo library for quotas? Somewhere
  where we can share the code,but keep the operations local to each service.
 
 
  This is what I had in mind as well. A simple library for quota enforcement
  which can be used regardless of where and how you do it, which might depend
  on the application business logic, the WSGI framework in use, or other
  factors.
 
 
 
 
  If there is still interest of placing quota in keystone, let's talk about
  how that will work and what will be needed from Keystone . The previous
  attempt didn't get much traction and stalled out early in 
  implementation. If
  we want to revisit this lets make sure we have the resources needed and
  spec(s) in progress / info on etherpads (similar to how the multitenancy
  stuff was handled at the last summit) as early as possible.
 
 
  Why not centralize quota management via the python-openstackclient, what
  is the benefit of getting keystone involved?
 
 
  Providing this through the openstack client in my opinion has the
  disadvantage that users which either use the REST API direct or write their
  own clients won't leverage it. I don't think it's a reasonable assumption
  that everybody will use python-openstackclient, is it?
 
  Said that, storing quotas in keystone poses a further challenge to the
  scalability of the system, which we shall perhaps address by using
  appropriate caching strategies and 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-14 Thread Salvatore Orlando
Doug,

I totally agree with your findings on the policy module.
Neutron already has some customizations there and we already have a few
contributors working on syncing it back with oslo-incubator during the Kilo
release cycle.

However, my query was about the quota module.
From what I gather it seems not a lot of projects use it:

$ find . -name openstack-common.conf | xargs grep quota
$

Salvatore

On 15 October 2014 00:34, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 14, 2014, at 12:31 PM, Salvatore Orlando sorla...@nicira.com
 wrote:

 Hi Doug,

 do you know if the existing quota oslo-incubator module has already some
 active consumers?
 In the meanwhile I've pushed a spec to neutron-specs for improving quota
 management there [1]


 It looks like a lot of projects are syncing the module:

 $ grep policy */openstack-common.conf

 barbican/openstack-common.conf:modules=gettextutils,jsonutils,log,local,timeutils,importutils,policy
 ceilometer/openstack-common.conf:module=policy
 cinder/openstack-common.conf:module=policy
 designate/openstack-common.conf:module=policy
 gantt/openstack-common.conf:module=policy
 glance/openstack-common.conf:module=policy
 heat/openstack-common.conf:module=policy
 horizon/openstack-common.conf:module=policy
 ironic/openstack-common.conf:module=policy
 keystone/openstack-common.conf:module=policy
 manila/openstack-common.conf:module=policy
 neutron/openstack-common.conf:module=policy
 nova/openstack-common.conf:module=policy
 trove/openstack-common.conf:module=policy
 tuskar/openstack-common.conf:module=policy

 I’m not sure how many are actively using it, but I wouldn’t expect them to
 copy it in if they weren’t using it at all.


 Now, I can either work on the oslo-incubator module and leverage it in
 Neutron, or develop the quota module in Neutron, and move it to
 oslo-incubator once we validate it with Neutron. The latter approach seems
 easier from a workflow perspective - as it avoid the intermediate steps of
 moving code from oslo-incubator to neutron. On the other hand it will delay
 adoption in oslo-incubator.


 The policy module is up for graduation this cycle. It may end up in its
 own library, to allow us to build a review team for the code more easily
 than if we put it in with some of the other semi-related modules like the
 server code. We’re still working that out [1], and if you expect to make a
 lot of incompatible changes we should delay graduation to make that simpler.

 Either way, since we have so many consumers, I think it would be easier to
 have the work happen in Oslo somewhere so we can ensure those changes are
 useful to and usable by all of the existing consumers.

 Doug

 [1] https://etherpad.openstack.org/p/kilo-oslo-library-proposals


 What's your opinion?

 Regards,
 Salvatore

 [1] https://review.openstack.org/#/c/128318/

 On 8 October 2014 18:52, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 8, 2014, at 7:03 AM, Davanum Srinivas dava...@gmail.com wrote:

  Salvatore, Joe,
 
  We do have this at the moment:
 
 
 https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py
 
  — dims

 If someone wants to drive creating a useful library during kilo, please
 consider adding the topic to the etherpad we’re using to plan summit
 sessions and then come participate in the Oslo meeting this Friday 16:00
 UTC.

 https://etherpad.openstack.org/p/kilo-oslo-summit-topics

 Doug

 
  On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com
 wrote:
 
  On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:
 
 
  On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
  morgan.fainb...@gmail.com wrote:
 
  Keeping the enforcement local (same way policy works today) helps
 limit
  the fragility, big +1 there.
 
  I also agree with Vish, we need a uniform way to talk about quota
  enforcement similar to how we have a uniform policy language /
 enforcement
  model (yes I know it's not perfect, but it's far closer to uniform
 than
  quota management is).
 
 
  It sounds like maybe we should have an oslo library for quotas?
 Somewhere
  where we can share the code,but keep the operations local to each
 service.
 
 
  This is what I had in mind as well. A simple library for quota
 enforcement
  which can be used regardless of where and how you do it, which might
 depend
  on the application business logic, the WSGI framework in use, or other
  factors.
 
 
 
 
  If there is still interest of placing quota in keystone, let's talk
 about
  how that will work and what will be needed from Keystone . The
 previous
  attempt didn't get much traction and stalled out early in
 implementation. If
  we want to revisit this lets make sure we have the resources needed
 and
  spec(s) in progress / info on etherpads (similar to how the
 multitenancy
  stuff was handled at the last summit) as early as possible.
 
 
  Why not centralize quota management via the python-openstackclient,
 what
  is the benefit of getting 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-08 Thread Salvatore Orlando
On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:


 On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg 
 morgan.fainb...@gmail.com wrote:

 Keeping the enforcement local (same way policy works today) helps limit
 the fragility, big +1 there.

 I also agree with Vish, we need a uniform way to talk about quota
 enforcement similar to how we have a uniform policy language / enforcement
 model (yes I know it's not perfect, but it's far closer to uniform than
 quota management is).


 It sounds like maybe we should have an oslo library for quotas? Somewhere
 where we can share the code,but keep the operations local to each service.


This is what I had in mind as well. A simple library for quota enforcement
which can be used regardless of where and how you do it, which might depend
on the application business logic, the WSGI framework in use, or other
factors.




 If there is still interest of placing quota in keystone, let's talk about
 how that will work and what will be needed from Keystone . The previous
 attempt didn't get much traction and stalled out early in implementation.
 If we want to revisit this lets make sure we have the resources needed and
 spec(s) in progress / info on etherpads (similar to how the multitenancy
 stuff was handled at the last summit) as early as possible.


 Why not centralize quota management via the python-openstackclient, what
 is the benefit of getting keystone involved?


Providing this through the openstack client in my opinion has the
disadvantage that users which either use the REST API direct or write their
own clients won't leverage it. I don't think it's a reasonable assumption
that everybody will use python-openstackclient, is it?

Said that, storing quotas in keystone poses a further challenge to the
scalability of the system, which we shall perhaps address by using
appropriate caching strategies and leveraging keystone notifications. Until
we get that, I think that the openstack client will be the best way of
getting a unified quota management experience.

Salvatore



 Cheers,
 Morgan

 Sent via mobile


 On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com
 wrote:

 Thanks Vish,

 this seems a very reasonable first step as well - and since most
 projects would be enforcing quotas in the same way, the shared library
 would be the logical next step.
 After all this is quite the same thing we do with authZ.

 Duncan is expressing valid concerns which in my opinion can be addressed
 with an appropriate design - and a decent implementation.

 Salvatore

 On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com
 wrote:

 The proposal in the past was to keep quota enforcement local, but to
 put the resource limits into keystone. This seems like an obvious first
 step to me. Then a shared library for enforcing quotas with decent
 performance should be next. The quota calls in nova are extremely
 inefficient right now and it will only get worse when we try to add
 hierarchical projects and quotas.

 Vish

 On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com
 wrote:

  Taking quota out of the service / adding remote calls for quota
  management is going to make things fragile - you've somehow got to
  deal with the cases where your quota manager is slow, goes away,
  hiccups, drops connections etc. You'll also need some way of
  reconciling actual usage against quota usage periodically, to detect
  problems.
 
  On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com
 wrote:
  Hi,
 
  Quota management is currently one of those things where every
 openstack
  project does its own thing. While quotas are obviously managed in a
 similar
  way for each project, there are subtle differences which ultimately
 result
  in lack of usability.
 
  I recall that in the past there have been several calls for unifying
 quota
  management. The blueprint [1] for instance, hints at the possibility
 of
  storing quotas in keystone.
  On the other hand, the blazar project [2, 3] seems to aim at solving
 this
  problem for good enabling resource reservation and therefore
 potentially
  freeing openstack projects from managing and enforcing quotas.
 
  While Blazar is definetely a good thing to have, I'm not entirely
 sure we
  want to make it a required component for every deployment. Perhaps
 single
  projects should still be able to enforce quota. On the other hand,
 at least
  on paper, the idea of making Keystone THE endpoint for managing
 quotas,
  and then letting the various project enforce them, sounds promising
 - is
  there any reason for which this blueprint is stalled to the point
 that it
  seems forgotten now?
 
  I'm coming to the mailing list with these random questions about
 quota
  management, for two reasons:
  1) despite developing and using openstack on a daily basis I'm still
  confused by quotas
  2) I've found a race condition in neutron quotas and the fix is not
 trivial.
  So, rather than start coding 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-08 Thread Davanum Srinivas
Salvatore, Joe,

We do have this at the moment:

https://github.com/openstack/oslo-incubator/blob/master/openstack/common/quota.py

-- dims

On Wed, Oct 8, 2014 at 2:29 AM, Salvatore Orlando sorla...@nicira.com wrote:

 On 8 October 2014 04:13, Joe Gordon joe.gord...@gmail.com wrote:


 On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg
 morgan.fainb...@gmail.com wrote:

 Keeping the enforcement local (same way policy works today) helps limit
 the fragility, big +1 there.

 I also agree with Vish, we need a uniform way to talk about quota
 enforcement similar to how we have a uniform policy language / enforcement
 model (yes I know it's not perfect, but it's far closer to uniform than
 quota management is).


 It sounds like maybe we should have an oslo library for quotas? Somewhere
 where we can share the code,but keep the operations local to each service.


 This is what I had in mind as well. A simple library for quota enforcement
 which can be used regardless of where and how you do it, which might depend
 on the application business logic, the WSGI framework in use, or other
 factors.




 If there is still interest of placing quota in keystone, let's talk about
 how that will work and what will be needed from Keystone . The previous
 attempt didn't get much traction and stalled out early in implementation. If
 we want to revisit this lets make sure we have the resources needed and
 spec(s) in progress / info on etherpads (similar to how the multitenancy
 stuff was handled at the last summit) as early as possible.


 Why not centralize quota management via the python-openstackclient, what
 is the benefit of getting keystone involved?


 Providing this through the openstack client in my opinion has the
 disadvantage that users which either use the REST API direct or write their
 own clients won't leverage it. I don't think it's a reasonable assumption
 that everybody will use python-openstackclient, is it?

 Said that, storing quotas in keystone poses a further challenge to the
 scalability of the system, which we shall perhaps address by using
 appropriate caching strategies and leveraging keystone notifications. Until
 we get that, I think that the openstack client will be the best way of
 getting a unified quota management experience.

 Salvatore



 Cheers,
 Morgan

 Sent via mobile


 On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com
 wrote:

 Thanks Vish,

 this seems a very reasonable first step as well - and since most
 projects would be enforcing quotas in the same way, the shared library 
 would
 be the logical next step.
 After all this is quite the same thing we do with authZ.

 Duncan is expressing valid concerns which in my opinion can be addressed
 with an appropriate design - and a decent implementation.

 Salvatore

 On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com
 wrote:

 The proposal in the past was to keep quota enforcement local, but to
 put the resource limits into keystone. This seems like an obvious first
 step to me. Then a shared library for enforcing quotas with decent
 performance should be next. The quota calls in nova are extremely
 inefficient right now and it will only get worse when we try to add
 hierarchical projects and quotas.

 Vish

 On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com
 wrote:

  Taking quota out of the service / adding remote calls for quota
  management is going to make things fragile - you've somehow got to
  deal with the cases where your quota manager is slow, goes away,
  hiccups, drops connections etc. You'll also need some way of
  reconciling actual usage against quota usage periodically, to detect
  problems.
 
  On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com
  wrote:
  Hi,
 
  Quota management is currently one of those things where every
  openstack
  project does its own thing. While quotas are obviously managed in a
  similar
  way for each project, there are subtle differences which ultimately
  result
  in lack of usability.
 
  I recall that in the past there have been several calls for unifying
  quota
  management. The blueprint [1] for instance, hints at the possibility
  of
  storing quotas in keystone.
  On the other hand, the blazar project [2, 3] seems to aim at solving
  this
  problem for good enabling resource reservation and therefore
  potentially
  freeing openstack projects from managing and enforcing quotas.
 
  While Blazar is definetely a good thing to have, I'm not entirely
  sure we
  want to make it a required component for every deployment. Perhaps
  single
  projects should still be able to enforce quota. On the other hand,
  at least
  on paper, the idea of making Keystone THE endpoint for managing
  quotas,
  and then letting the various project enforce them, sounds promising
  - is
  there any reason for which this blueprint is stalled to the point
  that it
  seems forgotten now?
 
  I'm coming to the mailing list with these random questions 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-07 Thread Joe Gordon
On Fri, Oct 3, 2014 at 10:47 AM, Morgan Fainberg morgan.fainb...@gmail.com
wrote:

 Keeping the enforcement local (same way policy works today) helps limit
 the fragility, big +1 there.

 I also agree with Vish, we need a uniform way to talk about quota
 enforcement similar to how we have a uniform policy language / enforcement
 model (yes I know it's not perfect, but it's far closer to uniform than
 quota management is).


It sounds like maybe we should have an oslo library for quotas? Somewhere
where we can share the code,but keep the operations local to each service.



 If there is still interest of placing quota in keystone, let's talk about
 how that will work and what will be needed from Keystone . The previous
 attempt didn't get much traction and stalled out early in implementation.
 If we want to revisit this lets make sure we have the resources needed and
 spec(s) in progress / info on etherpads (similar to how the multitenancy
 stuff was handled at the last summit) as early as possible.


Why not centralize quota management via the python-openstackclient, what is
the benefit of getting keystone involved?



 Cheers,
 Morgan

 Sent via mobile


 On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com wrote:

 Thanks Vish,

 this seems a very reasonable first step as well - and since most projects
 would be enforcing quotas in the same way, the shared library would be the
 logical next step.
 After all this is quite the same thing we do with authZ.

 Duncan is expressing valid concerns which in my opinion can be addressed
 with an appropriate design - and a decent implementation.

 Salvatore

 On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com
 wrote:

 The proposal in the past was to keep quota enforcement local, but to
 put the resource limits into keystone. This seems like an obvious first
 step to me. Then a shared library for enforcing quotas with decent
 performance should be next. The quota calls in nova are extremely
 inefficient right now and it will only get worse when we try to add
 hierarchical projects and quotas.

 Vish

 On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com
 wrote:

  Taking quota out of the service / adding remote calls for quota
  management is going to make things fragile - you've somehow got to
  deal with the cases where your quota manager is slow, goes away,
  hiccups, drops connections etc. You'll also need some way of
  reconciling actual usage against quota usage periodically, to detect
  problems.
 
  On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com
 wrote:
  Hi,
 
  Quota management is currently one of those things where every
 openstack
  project does its own thing. While quotas are obviously managed in a
 similar
  way for each project, there are subtle differences which ultimately
 result
  in lack of usability.
 
  I recall that in the past there have been several calls for unifying
 quota
  management. The blueprint [1] for instance, hints at the possibility
 of
  storing quotas in keystone.
  On the other hand, the blazar project [2, 3] seems to aim at solving
 this
  problem for good enabling resource reservation and therefore
 potentially
  freeing openstack projects from managing and enforcing quotas.
 
  While Blazar is definetely a good thing to have, I'm not entirely
 sure we
  want to make it a required component for every deployment. Perhaps
 single
  projects should still be able to enforce quota. On the other hand, at
 least
  on paper, the idea of making Keystone THE endpoint for managing
 quotas,
  and then letting the various project enforce them, sounds promising -
 is
  there any reason for which this blueprint is stalled to the point
 that it
  seems forgotten now?
 
  I'm coming to the mailing list with these random questions about quota
  management, for two reasons:
  1) despite developing and using openstack on a daily basis I'm still
  confused by quotas
  2) I've found a race condition in neutron quotas and the fix is not
 trivial.
  So, rather than start coding right away, it might probably make more
 sense
  to ask the community if there is already a known better approach to
 quota
  management - and obviously enforcement.
 
  Thanks in advance,
  Salvatore
 
  [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata
  [2] https://wiki.openstack.org/wiki/Blazar
  [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
  --
  Duncan Thomas
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 

Re: [openstack-dev] Quota management and enforcement across projects

2014-10-03 Thread Robert van Leeuwen
 I recall that in the past there have been several calls for unifying quota 
 management.
 The blueprint [1] for instance, hints at the possibility of storing quotas in 
 keystone.
As an end-user: +1
For me it totally makes sense to put the quotas and access together.

 On the other hand, the blazar project [2, 3] seems to aim at solving this 
 problem for good
 enabling resource reservation and therefore potentially freeing openstack 
 projects from managing and enforcing quotas.
 While Blazar is definetely a good thing to have, I'm not entirely sure we 
 want to make it a required component for every deployment.

Totally agree, I'd rather not run more components running then strictly 
necessary.
It is already quite a lot of work to test all components each time we do 
upgrade's.
Adding complexity / more moving parts is not on the top of my list unless 
strictly necessary.

Cheers,
Robert van Leeuwen


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-10-03 Thread Duncan Thomas
Taking quota out of the service / adding remote calls for quota
management is going to make things fragile - you've somehow got to
deal with the cases where your quota manager is slow, goes away,
hiccups, drops connections etc. You'll also need some way of
reconciling actual usage against quota usage periodically, to detect
problems.

On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote:
 Hi,

 Quota management is currently one of those things where every openstack
 project does its own thing. While quotas are obviously managed in a similar
 way for each project, there are subtle differences which ultimately result
 in lack of usability.

 I recall that in the past there have been several calls for unifying quota
 management. The blueprint [1] for instance, hints at the possibility of
 storing quotas in keystone.
 On the other hand, the blazar project [2, 3] seems to aim at solving this
 problem for good enabling resource reservation and therefore potentially
 freeing openstack projects from managing and enforcing quotas.

 While Blazar is definetely a good thing to have, I'm not entirely sure we
 want to make it a required component for every deployment. Perhaps single
 projects should still be able to enforce quota. On the other hand, at least
 on paper, the idea of making Keystone THE endpoint for managing quotas,
 and then letting the various project enforce them, sounds promising - is
 there any reason for which this blueprint is stalled to the point that it
 seems forgotten now?

 I'm coming to the mailing list with these random questions about quota
 management, for two reasons:
 1) despite developing and using openstack on a daily basis I'm still
 confused by quotas
 2) I've found a race condition in neutron quotas and the fix is not trivial.
 So, rather than start coding right away, it might probably make more sense
 to ask the community if there is already a known better approach to quota
 management - and obviously enforcement.

 Thanks in advance,
 Salvatore

 [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata
 [2] https://wiki.openstack.org/wiki/Blazar
 [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Duncan Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-10-03 Thread Vishvananda Ishaya
The proposal in the past was to keep quota enforcement local, but to
put the resource limits into keystone. This seems like an obvious first
step to me. Then a shared library for enforcing quotas with decent
performance should be next. The quota calls in nova are extremely
inefficient right now and it will only get worse when we try to add
hierarchical projects and quotas.

Vish

On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote:

 Taking quota out of the service / adding remote calls for quota
 management is going to make things fragile - you've somehow got to
 deal with the cases where your quota manager is slow, goes away,
 hiccups, drops connections etc. You'll also need some way of
 reconciling actual usage against quota usage periodically, to detect
 problems.
 
 On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote:
 Hi,
 
 Quota management is currently one of those things where every openstack
 project does its own thing. While quotas are obviously managed in a similar
 way for each project, there are subtle differences which ultimately result
 in lack of usability.
 
 I recall that in the past there have been several calls for unifying quota
 management. The blueprint [1] for instance, hints at the possibility of
 storing quotas in keystone.
 On the other hand, the blazar project [2, 3] seems to aim at solving this
 problem for good enabling resource reservation and therefore potentially
 freeing openstack projects from managing and enforcing quotas.
 
 While Blazar is definetely a good thing to have, I'm not entirely sure we
 want to make it a required component for every deployment. Perhaps single
 projects should still be able to enforce quota. On the other hand, at least
 on paper, the idea of making Keystone THE endpoint for managing quotas,
 and then letting the various project enforce them, sounds promising - is
 there any reason for which this blueprint is stalled to the point that it
 seems forgotten now?
 
 I'm coming to the mailing list with these random questions about quota
 management, for two reasons:
 1) despite developing and using openstack on a daily basis I'm still
 confused by quotas
 2) I've found a race condition in neutron quotas and the fix is not trivial.
 So, rather than start coding right away, it might probably make more sense
 to ask the community if there is already a known better approach to quota
 management - and obviously enforcement.
 
 Thanks in advance,
 Salvatore
 
 [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata
 [2] https://wiki.openstack.org/wiki/Blazar
 [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 -- 
 Duncan Thomas
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-10-03 Thread Salvatore Orlando
Thanks Vish,

this seems a very reasonable first step as well - and since most projects
would be enforcing quotas in the same way, the shared library would be the
logical next step.
After all this is quite the same thing we do with authZ.

Duncan is expressing valid concerns which in my opinion can be addressed
with an appropriate design - and a decent implementation.

Salvatore

On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com wrote:

 The proposal in the past was to keep quota enforcement local, but to
 put the resource limits into keystone. This seems like an obvious first
 step to me. Then a shared library for enforcing quotas with decent
 performance should be next. The quota calls in nova are extremely
 inefficient right now and it will only get worse when we try to add
 hierarchical projects and quotas.

 Vish

 On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com wrote:

  Taking quota out of the service / adding remote calls for quota
  management is going to make things fragile - you've somehow got to
  deal with the cases where your quota manager is slow, goes away,
  hiccups, drops connections etc. You'll also need some way of
  reconciling actual usage against quota usage periodically, to detect
  problems.
 
  On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com wrote:
  Hi,
 
  Quota management is currently one of those things where every openstack
  project does its own thing. While quotas are obviously managed in a
 similar
  way for each project, there are subtle differences which ultimately
 result
  in lack of usability.
 
  I recall that in the past there have been several calls for unifying
 quota
  management. The blueprint [1] for instance, hints at the possibility of
  storing quotas in keystone.
  On the other hand, the blazar project [2, 3] seems to aim at solving
 this
  problem for good enabling resource reservation and therefore potentially
  freeing openstack projects from managing and enforcing quotas.
 
  While Blazar is definetely a good thing to have, I'm not entirely sure
 we
  want to make it a required component for every deployment. Perhaps
 single
  projects should still be able to enforce quota. On the other hand, at
 least
  on paper, the idea of making Keystone THE endpoint for managing
 quotas,
  and then letting the various project enforce them, sounds promising - is
  there any reason for which this blueprint is stalled to the point that
 it
  seems forgotten now?
 
  I'm coming to the mailing list with these random questions about quota
  management, for two reasons:
  1) despite developing and using openstack on a daily basis I'm still
  confused by quotas
  2) I've found a race condition in neutron quotas and the fix is not
 trivial.
  So, rather than start coding right away, it might probably make more
 sense
  to ask the community if there is already a known better approach to
 quota
  management - and obviously enforcement.
 
  Thanks in advance,
  Salvatore
 
  [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata
  [2] https://wiki.openstack.org/wiki/Blazar
  [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
  --
  Duncan Thomas
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Quota management and enforcement across projects

2014-10-03 Thread Morgan Fainberg
Keeping the enforcement local (same way policy works today) helps limit the
fragility, big +1 there.

I also agree with Vish, we need a uniform way to talk about quota
enforcement similar to how we have a uniform policy language / enforcement
model (yes I know it's not perfect, but it's far closer to uniform than
quota management is).

If there is still interest of placing quota in keystone, let's talk about
how that will work and what will be needed from Keystone . The previous
attempt didn't get much traction and stalled out early in implementation.
If we want to revisit this lets make sure we have the resources needed and
spec(s) in progress / info on etherpads (similar to how the multitenancy
stuff was handled at the last summit) as early as possible.

Cheers,
Morgan

Sent via mobile

On Friday, October 3, 2014, Salvatore Orlando sorla...@nicira.com wrote:

 Thanks Vish,

 this seems a very reasonable first step as well - and since most projects
 would be enforcing quotas in the same way, the shared library would be the
 logical next step.
 After all this is quite the same thing we do with authZ.

 Duncan is expressing valid concerns which in my opinion can be addressed
 with an appropriate design - and a decent implementation.

 Salvatore

 On 3 October 2014 18:25, Vishvananda Ishaya vishvana...@gmail.com
 javascript:_e(%7B%7D,'cvml','vishvana...@gmail.com'); wrote:

 The proposal in the past was to keep quota enforcement local, but to
 put the resource limits into keystone. This seems like an obvious first
 step to me. Then a shared library for enforcing quotas with decent
 performance should be next. The quota calls in nova are extremely
 inefficient right now and it will only get worse when we try to add
 hierarchical projects and quotas.

 Vish

 On Oct 3, 2014, at 7:53 AM, Duncan Thomas duncan.tho...@gmail.com
 javascript:_e(%7B%7D,'cvml','duncan.tho...@gmail.com'); wrote:

  Taking quota out of the service / adding remote calls for quota
  management is going to make things fragile - you've somehow got to
  deal with the cases where your quota manager is slow, goes away,
  hiccups, drops connections etc. You'll also need some way of
  reconciling actual usage against quota usage periodically, to detect
  problems.
 
  On 3 October 2014 15:03, Salvatore Orlando sorla...@nicira.com
 javascript:_e(%7B%7D,'cvml','sorla...@nicira.com'); wrote:
  Hi,
 
  Quota management is currently one of those things where every openstack
  project does its own thing. While quotas are obviously managed in a
 similar
  way for each project, there are subtle differences which ultimately
 result
  in lack of usability.
 
  I recall that in the past there have been several calls for unifying
 quota
  management. The blueprint [1] for instance, hints at the possibility of
  storing quotas in keystone.
  On the other hand, the blazar project [2, 3] seems to aim at solving
 this
  problem for good enabling resource reservation and therefore
 potentially
  freeing openstack projects from managing and enforcing quotas.
 
  While Blazar is definetely a good thing to have, I'm not entirely sure
 we
  want to make it a required component for every deployment. Perhaps
 single
  projects should still be able to enforce quota. On the other hand, at
 least
  on paper, the idea of making Keystone THE endpoint for managing
 quotas,
  and then letting the various project enforce them, sounds promising -
 is
  there any reason for which this blueprint is stalled to the point that
 it
  seems forgotten now?
 
  I'm coming to the mailing list with these random questions about quota
  management, for two reasons:
  1) despite developing and using openstack on a daily basis I'm still
  confused by quotas
  2) I've found a race condition in neutron quotas and the fix is not
 trivial.
  So, rather than start coding right away, it might probably make more
 sense
  to ask the community if there is already a known better approach to
 quota
  management - and obviously enforcement.
 
  Thanks in advance,
  Salvatore
 
  [1] https://blueprints.launchpad.net/keystone/+spec/service-metadata
  [2] https://wiki.openstack.org/wiki/Blazar
  [3] https://review.openstack.org/#/q/project:stackforge/blazar,n,z
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
 javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org');
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
  --
  Duncan Thomas
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
 javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org');
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 javascript:_e(%7B%7D,'cvml','OpenStack-dev@lists.openstack.org');