Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Vilobh Meshram]

Thanks everyone for being there at the Design Summit talk on Cross project
Quotas.

Points from the discussion are captured in this etherpad[1] for reference.

-Vilobh
[1] https://etherpad.openstack.org/p/newton-quota-library

On Tue, Apr 26, 2016 at 9:33 AM, Jay Pipes  wrote:

> On 04/25/2016 02:05 PM, Joshua Harlow wrote:
>
>> Is the generation stuff going to be exposed outside of the AP?
>>
>
> No, wasn't planning on it.
>
> I'm sort of hoping not(?), because a service (if one everyone wanted to
>> create it) for say zookeeper (or etcd or consul...) could use its
>> built-in generation equivalent (every znode has a version that u can use
>> to do equivalent things).
>>
>> Thus it's like the gist I posted earlier @
>>
>> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
>>
>> So might be nice to not expose such a thing outside of the db-layer.
>>
>> Amrith Kumar wrote:
>>
>>> On Sat, 2016-04-23 at 21:41 +, Amrith Kumar wrote:
>>>
 Ok to beer and high bandwidth. FYI Jay the distributed high perf db we
 did a couple of years ago is now open source. Just saying. Mysql plug
 compatible 

>>>
>>> -amrith


 --
 Amrith Kumar
 amr...@tesora.com


  Original message 
 From: Jay Pipes
 Date: 04/23/2016 4:10 PM (GMT-05:00)
 To: Amrith Kumar,
 openstack-dev@lists.openstack.org
 Cc: vilob...@yahoo-inc.com, nik.koma...@gmail.com, Ed Leafe
 
 Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
 Management Library proposal


 Looking forward to arriving in Austin so that I can buy you a beer,
 Amrith, and have a high-bandwidth conversation about how you're
 wrong. :P

>>>
>>>
>>> Jay and I chatted and it took a long time to come to an agreement
>>> because we weren't able to find any beer.
>>>
>>> Here's what I think we've agreed about. The library will store data in
>>> two tables;
>>>
>>> 1. the detail table which stores the individual claims and the resource
>>> class.
>>> 2. a generation table which stores the resource class and a generation.
>>>
>>> When a claim is received, the requestor performs the following
>>> operations.
>>>
>>> begin
>>>
>>>  select sum(detail.claims) as total_claims,
>>>  generation.resource as resource,
>>>  generation.generation last_generation
>>>  from detail, generation
>>>  where detail.resource = generation.resource
>>>  and generation.resource =
>>>  group by generation.generation, generation.resource
>>>
>>>  if total_claims + this_claim<  limit
>>>  insert into detail values (this_claim, resource)
>>>
>>>  update generation
>>>  set generation = generation + 1
>>>  where generation = last_generation
>>>
>>>  if @@rowcount = 1
>>>  -- all good
>>>  commit
>>>  else
>>>  rollback
>>>  -- try again
>>>
>>>
>>>
>>> There will be some bootstrapping that will be required for the situation
>>> where there are no detail records for a given resource and so on but I
>>> think we can figure that out easily. The easiest way I can think of
>>> doing that is to lose the join and do both the queries (one against the
>>> detail and one against the generation table within the same
>>> transaction).
>>>
>>> Using the generation table update as the locking mechanism that prevents
>>> multiple requestors from making concurrent claims.
>>>
>>> So long as people don't go and try and read these tables and change
>>> tables outside of the methods that the library provides, we can
>>> guarantee that this is al safe and will not oversubscribe.
>>>
>>> -amrith
>>>
>>> Comments inline.

 On 04/23/2016 11:25 AM, Amrith Kumar wrote:

> On Sat, 2016-04-23 at 10:26 -0400, Andrew Laski wrote:
>
>> On Fri, Apr 22, 2016, at 09:57 PM, Tim Bell wrote:
>>
>> I have reservations on f and g.
>>>
>>>
>>> On f., We have had a number of discussions in the past about
>>> centralising quota (e.g. Boson) and the project teams of the other
>>> components wanted to keep the quota contents ‘close’. This can
>>> always be reviewed further with them but I would hope for at least
>>>
>> a

> standard schema structure of tables in each project for the
>>>
>> handling

> of quota.
>>>
>>>
>>> On g., aren’t all projects now nested projects ? If we have the
>>> complexity of handling nested projects sorted out in the common
>>> library, is there a reason why a project would not want to support
>>> nested projects ?
>>>
>>>
>>> One other issue is how to do reconcilliation, each project needs
>>>
>> to

> 

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Duncan Thomas]

On 23 Apr 2016 14:26, "Jay Pipes"  wrote:

> On 04/23/2016 03:18 PM, Mike Perez wrote:

>> How about extending a volume? A volume is a resource and can be extended
in
>> Cinder today.
>
>
> Yep, understood :) I recognize some resource amounts can be modified for
some resource classes. How about *shrinking* a volume. Is that supported?

Currently, no, but saying we can never implement it because the quota
library will never support it is the tail wagging the dog in a big way.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Ed Leafe]

On Apr 23, 2016, at 3:10 PM, Jay Pipes  wrote:
> 
> Looking forward to arriving in Austin so that I can buy you a beer, Amrith, 
> and have a high-bandwidth conversation about how you're wrong. :P

Beer is a great addition to any conversation!

-- Ed

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Edward Leafe]

On Apr 23, 2016, at 3:10 PM, Jay Pipes > wrote:

> BTW, note to Ed Leafe... unless your distributed data store supports 
> transactional semantics, you can't use a distributed data store for these 
> types of solutions. Instead, you will need to write a whole bunch of code 
> that does post-auditing of claims and quotas and a system that accepts that 
> oversubscription and out-of-sync quota limits and usages is a fact of life. 

Yes, that was one of the things that I liked about Cassandra: 
http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0 


For instance, in the scheduler, making a claim would simply fail if another 
process had changed any of the resources in question, as the transaction would 
not be allowed.

> Not to mention needing to implement JOINs in Python.

Heh, JOIN in a column family database is not needed. You do have to think about 
data a lot differently, and that means unlearning everything you know about 
normalization. As a long-time SQL DB admin, it took my brain quite some time to 
be able to understand the different approach.

-- Ed Leafe__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Mike Perez]

On 15:25 Apr 23, Jay Pipes wrote:
> On 04/23/2016 03:18 PM, Mike Perez wrote:
> >On 14:54 Apr 18, Jay Pipes wrote:
> >>On 04/16/2016 05:51 PM, Amrith Kumar wrote:
> >>> - update_resource(id or resource, newsize)
> >>
> >>Resizing resources is a bad idea, IMHO. Resources are easier to deal with
> >>when they are considered of immutable size and simple (i.e. not complex or
> >>nested). I think the problem here is in the definition of resource classes
> >>improperly.
> >>
> >>For example, a "cluster" is not a resource. It is a collection of resources
> >>of type node. "Resizing" a cluster is a misnomer, because you aren't
> >>resizing a resource at all. Instead, you are creating or destroying
> >>resources inside the cluster (i.e. joining or leaving cluster nodes).
> >>
> >>BTW, this is also why the "resize instance" API in Nova is such a giant pain
> >>in the ass. It's attempting to "modify" the instance "resource" when the
> >>instance isn't really the resource at all. The VCPU, RAM_MB, DISK_GB, and
> >>PCI devices are the actual resources. The instance is a convenient way to
> >>tie those resources together, and doing a "resize" of the instance behind
> >>the scenes actually performs a *move* operation, which isn't a *change* of
> >>the original resources. Rather, it is a creation of a new set of resources
> >>(of the new amounts) and a deletion of the old set of resources.
> >
> >How about extending a volume? A volume is a resource and can be extended in
> >Cinder today.
> 
> Yep, understood :) I recognize some resource amounts can be modified for
> some resource classes. How about *shrinking* a volume. Is that supported?

Nope, too tricky at this time.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Tom Barron]

On 04/23/2016 03:25 PM, Jay Pipes wrote:
> On 04/23/2016 03:18 PM, Mike Perez wrote:
>> On 14:54 Apr 18, Jay Pipes wrote:
>>> On 04/16/2016 05:51 PM, Amrith Kumar wrote:
  - update_resource(id or resource, newsize)
>>>
>>> Resizing resources is a bad idea, IMHO. Resources are easier to deal
>>> with
>>> when they are considered of immutable size and simple (i.e. not
>>> complex or
>>> nested). I think the problem here is in the definition of resource
>>> classes
>>> improperly.
>>>
>>> For example, a "cluster" is not a resource. It is a collection of
>>> resources
>>> of type node. "Resizing" a cluster is a misnomer, because you aren't
>>> resizing a resource at all. Instead, you are creating or destroying
>>> resources inside the cluster (i.e. joining or leaving cluster nodes).
>>>
>>> BTW, this is also why the "resize instance" API in Nova is such a
>>> giant pain
>>> in the ass. It's attempting to "modify" the instance "resource" when the
>>> instance isn't really the resource at all. The VCPU, RAM_MB, DISK_GB,
>>> and
>>> PCI devices are the actual resources. The instance is a convenient
>>> way to

-- Tom
>>> tie those resources together, and doing a "resize" of the instance
>>> behind
>>> the scenes actually performs a *move* operation, which isn't a
>>> *change* of
>>> the original resources. Rather, it is a creation of a new set of
>>> resources
>>> (of the new amounts) and a deletion of the old set of resources.
>>
>> How about extending a volume? A volume is a resource and can be
>> extended in
>> Cinder today.
> 
> Yep, understood :) I recognize some resource amounts can be modified for
> some resource classes. How about *shrinking* a volume. Is that supported?

manila has apis for both extending and shrinking shares.

FWIW I very much like the notion that we should be able to check actual
up-to-date resource usage, use a single source of truth, and reduce the
amount of races that have to be handled.  The previous one-sentence
paragraph provides a data-point and is not intended as an objection.

-- Tom

> 
> Best,
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Amrith Kumar]

On Sat, 2016-04-23 at 15:25 -0400, Jay Pipes wrote:
> On 04/23/2016 03:18 PM, Mike Perez wrote:
> > On 14:54 Apr 18, Jay Pipes wrote:
> >> On 04/16/2016 05:51 PM, Amrith Kumar wrote:
> >>>  - update_resource(id or resource, newsize)
> >>
> >> Resizing resources is a bad idea, IMHO. Resources are easier to deal with
> >> when they are considered of immutable size and simple (i.e. not complex or
> >> nested). I think the problem here is in the definition of resource classes
> >> improperly.
> >>
> >> For example, a "cluster" is not a resource. It is a collection of resources
> >> of type node. "Resizing" a cluster is a misnomer, because you aren't
> >> resizing a resource at all. Instead, you are creating or destroying
> >> resources inside the cluster (i.e. joining or leaving cluster nodes).
> >>
> >> BTW, this is also why the "resize instance" API in Nova is such a giant 
> >> pain
> >> in the ass. It's attempting to "modify" the instance "resource" when the
> >> instance isn't really the resource at all. The VCPU, RAM_MB, DISK_GB, and
> >> PCI devices are the actual resources. The instance is a convenient way to
> >> tie those resources together, and doing a "resize" of the instance behind
> >> the scenes actually performs a *move* operation, which isn't a *change* of
> >> the original resources. Rather, it is a creation of a new set of resources
> >> (of the new amounts) and a deletion of the old set of resources.
> >
> > How about extending a volume? A volume is a resource and can be extended in
> > Cinder today.
> 
> Yep, understood :) I recognize some resource amounts can be modified for 
> some resource classes. How about *shrinking* a volume. Is that supported?
> 

[amrith] It isn't today and every other week or so (most recently this
past Monday in Percona Live) I get a request for this capability :)

Not saying it is easy (I know it is hard) but honestly answering the
question regarding support.

> Best,
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: This is a digitally signed message part
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Jay Pipes]

On 04/23/2016 03:18 PM, Mike Perez wrote:

On 14:54 Apr 18, Jay Pipes wrote:

On 04/16/2016 05:51 PM, Amrith Kumar wrote:

 - update_resource(id or resource, newsize)


Resizing resources is a bad idea, IMHO. Resources are easier to deal with
when they are considered of immutable size and simple (i.e. not complex or
nested). I think the problem here is in the definition of resource classes
improperly.

For example, a "cluster" is not a resource. It is a collection of resources
of type node. "Resizing" a cluster is a misnomer, because you aren't
resizing a resource at all. Instead, you are creating or destroying
resources inside the cluster (i.e. joining or leaving cluster nodes).

BTW, this is also why the "resize instance" API in Nova is such a giant pain
in the ass. It's attempting to "modify" the instance "resource" when the
instance isn't really the resource at all. The VCPU, RAM_MB, DISK_GB, and
PCI devices are the actual resources. The instance is a convenient way to
tie those resources together, and doing a "resize" of the instance behind
the scenes actually performs a *move* operation, which isn't a *change* of
the original resources. Rather, it is a creation of a new set of resources
(of the new amounts) and a deletion of the old set of resources.


How about extending a volume? A volume is a resource and can be extended in
Cinder today.


Yep, understood :) I recognize some resource amounts can be modified for 
some resource classes. How about *shrinking* a volume. Is that supported?


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Mike Perez]

On 14:54 Apr 18, Jay Pipes wrote:
> On 04/16/2016 05:51 PM, Amrith Kumar wrote:
> > - update_resource(id or resource, newsize)
> 
> Resizing resources is a bad idea, IMHO. Resources are easier to deal with
> when they are considered of immutable size and simple (i.e. not complex or
> nested). I think the problem here is in the definition of resource classes
> improperly.
> 
> For example, a "cluster" is not a resource. It is a collection of resources
> of type node. "Resizing" a cluster is a misnomer, because you aren't
> resizing a resource at all. Instead, you are creating or destroying
> resources inside the cluster (i.e. joining or leaving cluster nodes).
> 
> BTW, this is also why the "resize instance" API in Nova is such a giant pain
> in the ass. It's attempting to "modify" the instance "resource" when the
> instance isn't really the resource at all. The VCPU, RAM_MB, DISK_GB, and
> PCI devices are the actual resources. The instance is a convenient way to
> tie those resources together, and doing a "resize" of the instance behind
> the scenes actually performs a *move* operation, which isn't a *change* of
> the original resources. Rather, it is a creation of a new set of resources
> (of the new amounts) and a deletion of the old set of resources.

How about extending a volume? A volume is a resource and can be extended in
Cinder today.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Jay Pipes]

On 04/22/2016 09:57 PM, Tim Bell wrote:

I have reservations on f and g.


Probably not the *best* choice of words to describe your concerns, Tim :P

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Amrith Kumar]

 to that. It is sorely needed. My way of trying to go there complicated the 
API too much.

Let me know if I can help in any way, but you already knew that.

Thanks,

-amrith

> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: Friday, April 22, 2016 11:19 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
> Management Library proposal
> 
> On 04/22/2016 07:51 AM, Amrith Kumar wrote:
> > If it doesn't make it harder, I would like to see if we can make the
> > quota API take care of the ordering of requests. i.e. if the quota API
> > is an extension of Jay's example and accepts some data structure
> > (dict?) with all the claims that a project wants to make for some
> > operation, and then proceeds to make those claims for the project in
> > the consistent order, I think it would be of some value.
> 
> Amrith, above you hit upon a critical missing piece: a *consistent*,
> *versioned* way of representing the set of resources and capabilities for
> a particular request.
> 
> Each service project has its own different expectations for the format of
> these requested resources and capabilities. I would love to get to a
> single, oslo.versioned_objects way of describing the request for some
> resources and capabilities.
> 
> I'm working on a spec that proposes standardizing the qualitative side of
> the request (the capabilities). In Nova, we're getting close to
> standardizing the quantitative side of the request (resources).
> Hopefully before too long I'll have a spec up that explains my vision of
> this unified request representation.
> 
> Best,
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Jay Pipes]

On 04/22/2016 07:51 AM, Amrith Kumar wrote:

If it doesn’t make it harder, I would like to see if we can make the
quota API take care of the ordering of requests. i.e. if the quota API
is an extension of Jay’s example and accepts some data structure (dict?)
with all the claims that a project wants to make for some operation, and
then proceeds to make those claims for the project in the consistent
order, I think it would be of some value.


Amrith, above you hit upon a critical missing piece: a *consistent*, 
*versioned* way of representing the set of resources and capabilities 
for a particular request.


Each service project has its own different expectations for the format 
of these requested resources and capabilities. I would love to get to a 
single, oslo.versioned_objects way of describing the request for some 
resources and capabilities.


I'm working on a spec that proposes standardizing the qualitative side 
of the request (the capabilities). In Nova, we're getting close to 
standardizing the quantitative side of the request (resources). 
Hopefully before too long I'll have a spec up that explains my vision of 
this unified request representation.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Vilobh Meshram]

I strongly agree with Jay on the points related to "no reservation" ,
keeping the interface simple and the role for Delimiter (impose limits on
resource consumption and enforce quotas).

The point to keep user quota, tenant quotas in Keystone sounds interestring
and would need support from Keystone team. We have a Cross project session
planned [1] and will definitely bring that up in that session.

The main thought with which Delimiter was formed was to enforce resource
quota in transaction safe manner and do it in a cross-project conducive
manner and it still holds true. Delimiters mission is to impose limits on
resource consumption and enforce quotas in transaction safe manner. Few key
aspects of Delimiter are :-

a. Delimiter will be a new Library and not a Service. Details covered in
spec.

b. Delimiter's role will be to impose limits on resource consumption.

c. Delimiter will not be responsible for rate limiting.

d. Delimiter will not maintain data for the resources. Respective projects
will take care of keeping, maintaining data for the resources and resource
consumption.

e. Delimiter will not have the concept of "reservations". Delimiter will
read or update the "actual" resource tables and will not rely on the
"cached" tables. At present, the quota infrastructure in Nova, Cinder and
other projects have tables such as reservations, quota_usage, etc which are
used as "cached tables" to track re

f. Delimiter will fetch the information for project quota, user quota from
a centralized place, say Keystone, or if that doesn't materialize will
fetch default quota values from respective service. This information will
be cached since it gets updated rarely but read many times.

g. Delimiter will take into consideration whether the project is a Flat or
Nested and will make the calculations of allocated, available resources.
Nested means project namespace is hierarchical and Flat means project
namespace is not hierarchical.

-Vilobh

[1] https://www.openstack.org/summit/austin-2016/summit-schedule/events/9492

On Thu, Apr 21, 2016 at 11:08 PM, Joshua Harlow 
wrote:

> Since people will be on a plane soon,
>
> I threw this together as a example of a quota engine (the zookeeper code
> does even work, and yes it provides transactional semantics due to the nice
> abilities of zookeeper znode versions[1] and its inherent consistency
> model, yippe).
>
> https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85
>
> Someone else can fill in the db quota engine with a similar/equivalent api
> if they so dare, ha. Or even feel to say the gist/api above is crap, cause
> that's ok to, lol.
>
> [1]
> https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Data+Access
>
>
> Amrith Kumar wrote:
>
>> Inline below ... thread is too long, will catch you in Austin.
>>
>> -Original Message-
>>> From: Jay Pipes [mailto:jaypi...@gmail.com]
>>> Sent: Thursday, April 21, 2016 8:08 PM
>>> To: openstack-dev@lists.openstack.org
>>> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
>>> Management Library proposal
>>>
>>> Hmm, where do I start... I think I will just cut to the two primary
>>> disagreements I have. And I will top-post because this email is way too
>>> big.
>>>
>>> 1) On serializable isolation level.
>>>
>>> No, you don't need it at all to prevent races in claiming. Just use a
>>> compare-and-update with retries strategy. Proof is here:
>>>
>>> https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-
>>> L142
>>>
>>> Works great and prevents multiple writers from oversubscribing any
>>> resource without relying on any particular isolation level at all.
>>>
>>> The `generation` field in the inventories table is what allows multiple
>>> writers to ensure a consistent view of the data without needing to rely
>>> on
>>> heavy lock-based semantics and/or RDBMS-specific isolation levels.
>>>
>>>
>> [amrith] this works for what it is doing, we can definitely do this. This
>> will work at any isolation level, yes. I didn't want to go this route
>> because it is going to still require an insert into another table recording
>> what the actual 'thing' is that is claiming the resource and that insert is
>> going to be in a different transaction and managing those two transactions
>> was what I wanted to avoid. I was hoping to avoid having two tables
>> tracking claims, one showing the currently claimed quota and another
>> holding the things that claimed that quota. Have to think again whether
>> that is possible.
>>
>> 2) On reservations.
>>>
>>> The reason I don't believe reservations are necessary to be in a quota
>>> library is because reservations add a concept of a time to a claim of
>>> some
>>> resource. You reserve some resource to be claimed at some point in the
>>> future and release those resources at a point further in time.
>>>
>>> Quota checking doesn't look at what the state of some system will be at
>>> some point in the future. 

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Joshua Harlow]

Since people will be on a plane soon,

I threw this together as a example of a quota engine (the zookeeper code 
does even work, and yes it provides transactional semantics due to the 
nice abilities of zookeeper znode versions[1] and its inherent 
consistency model, yippe).


https://gist.github.com/harlowja/e7175c2d76e020a82ae94467a1441d85

Someone else can fill in the db quota engine with a similar/equivalent 
api if they so dare, ha. Or even feel to say the gist/api above is crap, 
cause that's ok to, lol.


[1] 
https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Data+Access


Amrith Kumar wrote:

Inline below ... thread is too long, will catch you in Austin.


-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Thursday, April 21, 2016 8:08 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
Management Library proposal

Hmm, where do I start... I think I will just cut to the two primary
disagreements I have. And I will top-post because this email is way too
big.

1) On serializable isolation level.

No, you don't need it at all to prevent races in claiming. Just use a
compare-and-update with retries strategy. Proof is here:

https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-
L142

Works great and prevents multiple writers from oversubscribing any
resource without relying on any particular isolation level at all.

The `generation` field in the inventories table is what allows multiple
writers to ensure a consistent view of the data without needing to rely on
heavy lock-based semantics and/or RDBMS-specific isolation levels.



[amrith] this works for what it is doing, we can definitely do this. This will 
work at any isolation level, yes. I didn't want to go this route because it is 
going to still require an insert into another table recording what the actual 
'thing' is that is claiming the resource and that insert is going to be in a 
different transaction and managing those two transactions was what I wanted to 
avoid. I was hoping to avoid having two tables tracking claims, one showing the 
currently claimed quota and another holding the things that claimed that quota. 
Have to think again whether that is possible.


2) On reservations.

The reason I don't believe reservations are necessary to be in a quota
library is because reservations add a concept of a time to a claim of some
resource. You reserve some resource to be claimed at some point in the
future and release those resources at a point further in time.

Quota checking doesn't look at what the state of some system will be at
some point in the future. It simply returns whether the system *right
now* can handle a request *right now* to claim a set of resources.

If you want reservation semantics for some resource, that's totally cool,
but IMHO, a reservation service should live outside of the service that is
actually responsible for providing resources to a consumer.
Merging right-now quota checks and future-based reservations into the same
library just complicates things unnecessarily IMHO.



[amrith] extension of the above ...


3) On resizes.

Look, I recognize some users see some value in resizing their resources.
That's fine. I personally think expand operations are fine, and that
shrink operations are really the operations that should be prohibited in
the API. But, whatever, I'm fine with resizing of requested resource
amounts. My big point is if you don't have a separate table that stores
quota_usages and instead only have a single table that stores the actual
resource usage records, you don't have to do *any* quota check operations
at all upon deletion of a resource. For modifying resource amounts (i.e. a
resize) you merely need to change the calculation of requested resource
amounts to account for the already-consumed usage amount.

Bottom line for me: I really won't support any proposal for a complex
library that takes the resource claim process out of the hands of the
services that own those resources. The simpler the interface of this
library, the better.



[amrith] my proposal would not but this email thread has got too long. Yes, 
simpler interface, will catch you in Austin.


Best,
-jay

On 04/19/2016 09:59 PM, Amrith Kumar wrote:

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Monday, April 18, 2016 2:54 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] More on the topic of DELIMITER, the
Quota Management Library proposal

On 04/16/2016 05:51 PM, Amrith Kumar wrote:

If we therefore assume that this will be a Quota Management Library,
it is safe to assume  that quotas are going to be managed on a
per-project basis, where participating projects will use this library.
I believe that it stands to reason that any data persistence will
have to be in a location decided by the individual project.

Depends on what you mean by "any data persistence". If you 

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Amrith Kumar]

Inline below ... thread is too long, will catch you in Austin.

> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: Thursday, April 21, 2016 8:08 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
> Management Library proposal
> 
> Hmm, where do I start... I think I will just cut to the two primary
> disagreements I have. And I will top-post because this email is way too
> big.
> 
> 1) On serializable isolation level.
> 
> No, you don't need it at all to prevent races in claiming. Just use a
> compare-and-update with retries strategy. Proof is here:
> 
> https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-
> L142
> 
> Works great and prevents multiple writers from oversubscribing any
> resource without relying on any particular isolation level at all.
> 
> The `generation` field in the inventories table is what allows multiple
> writers to ensure a consistent view of the data without needing to rely on
> heavy lock-based semantics and/or RDBMS-specific isolation levels.
> 

[amrith] this works for what it is doing, we can definitely do this. This will 
work at any isolation level, yes. I didn't want to go this route because it is 
going to still require an insert into another table recording what the actual 
'thing' is that is claiming the resource and that insert is going to be in a 
different transaction and managing those two transactions was what I wanted to 
avoid. I was hoping to avoid having two tables tracking claims, one showing the 
currently claimed quota and another holding the things that claimed that quota. 
Have to think again whether that is possible.

> 2) On reservations.
> 
> The reason I don't believe reservations are necessary to be in a quota
> library is because reservations add a concept of a time to a claim of some
> resource. You reserve some resource to be claimed at some point in the
> future and release those resources at a point further in time.
> 
> Quota checking doesn't look at what the state of some system will be at
> some point in the future. It simply returns whether the system *right
> now* can handle a request *right now* to claim a set of resources.
> 
> If you want reservation semantics for some resource, that's totally cool,
> but IMHO, a reservation service should live outside of the service that is
> actually responsible for providing resources to a consumer.
> Merging right-now quota checks and future-based reservations into the same
> library just complicates things unnecessarily IMHO.
> 

[amrith] extension of the above ...

> 3) On resizes.
> 
> Look, I recognize some users see some value in resizing their resources.
> That's fine. I personally think expand operations are fine, and that
> shrink operations are really the operations that should be prohibited in
> the API. But, whatever, I'm fine with resizing of requested resource
> amounts. My big point is if you don't have a separate table that stores
> quota_usages and instead only have a single table that stores the actual
> resource usage records, you don't have to do *any* quota check operations
> at all upon deletion of a resource. For modifying resource amounts (i.e. a
> resize) you merely need to change the calculation of requested resource
> amounts to account for the already-consumed usage amount.
> 
> Bottom line for me: I really won't support any proposal for a complex
> library that takes the resource claim process out of the hands of the
> services that own those resources. The simpler the interface of this
> library, the better.
> 

[amrith] my proposal would not but this email thread has got too long. Yes, 
simpler interface, will catch you in Austin.

> Best,
> -jay
> 
> On 04/19/2016 09:59 PM, Amrith Kumar wrote:
> >> -Original Message-
> >> From: Jay Pipes [mailto:jaypi...@gmail.com]
> >> Sent: Monday, April 18, 2016 2:54 PM
> >> To: openstack-dev@lists.openstack.org
> >> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the
> >> Quota Management Library proposal
> >>
> >> On 04/16/2016 05:51 PM, Amrith Kumar wrote:
> >>> If we therefore assume that this will be a Quota Management Library,
> >>> it is safe to assume  that quotas are going to be managed on a
> >>> per-project basis, where participating projects will use this library.
> >>> I believe that it stands to reason that any data persistence will
> >>> have to be in a location decided by the individual project.
> >>
> >> Depends on what you mean by "any data persistence". If you are
> >> referring to the storage of quota values (per user, per tenant,
> >> global, etc) I think that should be done by the Keystone service.
> >> This data is essentially an attribute of the user or the tenant or the
> service endpoint itself (i.e.
> >> global defaults). This data also rarely changes and logically belongs
> >> to the service that manages users, tenants, and service endpoints:
> Keystone.
> >>
> >> If you 

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Monty Taylor]

On 04/21/2016 07:07 PM, Jay Pipes wrote:

Hmm, where do I start... I think I will just cut to the two primary
disagreements I have. And I will top-post because this email is way too
big.

1) On serializable isolation level.

No, you don't need it at all to prevent races in claiming. Just use a
compare-and-update with retries strategy. Proof is here:

https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-L142


Works great and prevents multiple writers from oversubscribing any
resource without relying on any particular isolation level at all.

The `generation` field in the inventories table is what allows multiple
writers to ensure a consistent view of the data without needing to rely
on heavy lock-based semantics and/or RDBMS-specific isolation levels.

2) On reservations.

The reason I don't believe reservations are necessary to be in a quota
library is because reservations add a concept of a time to a claim of
some resource. You reserve some resource to be claimed at some point in
the future and release those resources at a point further in time.

Quota checking doesn't look at what the state of some system will be at
some point in the future. It simply returns whether the system *right
now* can handle a request *right now* to claim a set of resources.

If you want reservation semantics for some resource, that's totally
cool, but IMHO, a reservation service should live outside of the service
that is actually responsible for providing resources to a consumer.
Merging right-now quota checks and future-based reservations into the
same library just complicates things unnecessarily IMHO.

3) On resizes.

Look, I recognize some users see some value in resizing their resources.
That's fine. I personally think expand operations are fine, and that
shrink operations are really the operations that should be prohibited in
the API. But, whatever, I'm fine with resizing of requested resource
amounts. My big point is if you don't have a separate table that stores
quota_usages and instead only have a single table that stores the actual
resource usage records, you don't have to do *any* quota check
operations at all upon deletion of a resource. For modifying resource
amounts (i.e. a resize) you merely need to change the calculation of
requested resource amounts to account for the already-consumed usage
amount.

Bottom line for me: I really won't support any proposal for a complex
library that takes the resource claim process out of the hands of the
services that own those resources. The simpler the interface of this
library, the better.


I agree with every word that Jay has written here. I especially agree 
with point 1, and in fact have been in favor of that approach over the 
current system of table locks in the nova quota code for about as long 
as there has been nova quota code. But all three points are spot on.



On 04/19/2016 09:59 PM, Amrith Kumar wrote:

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Monday, April 18, 2016 2:54 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
Management Library proposal

On 04/16/2016 05:51 PM, Amrith Kumar wrote:

If we therefore assume that this will be a Quota Management Library,
it is safe to assume  that quotas are going to be managed on a
per-project basis, where participating projects will use this library.
I believe that it stands to reason that any data persistence will have
to be in a location decided by the individual project.


Depends on what you mean by "any data persistence". If you are referring
to the storage of quota values (per user, per tenant, global, etc) I
think
that should be done by the Keystone service. This data is essentially an
attribute of the user or the tenant or the service endpoint itself (i.e.
global defaults). This data also rarely changes and logically belongs to
the service that manages users, tenants, and service endpoints:
Keystone.

If you are referring to the storage of resource usage records, yes, each
service project should own that data (and frankly, I don't see a need to
persist any quota usage data at all, as I mentioned in a previous
reply to
Attila).



[amrith] You make a distinction that I had made implicitly, and it is
important
to highlight it. Thanks for pointing it out. Yes, I meant both of the
above, and as stipulated. Global defaults in keystone (somehow, TBD) and
usage records, on a per-service basis.



That may not be a very interesting statement but the corollary is, I
think, a very significant statement; it cannot be assumed that the
quota management information for all participating projects is in the
same database.


It cannot be assumed that this information is even in a database at
all...



[amrith] I don't follow. If the service in question is to be scalable,
I think it
stands to reason that there must be some mechanism by which instances of
the service can share usage records (as you refer to them, 

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Jay Pipes]

Hmm, where do I start... I think I will just cut to the two primary 
disagreements I have. And I will top-post because this email is way too big.


1) On serializable isolation level.

No, you don't need it at all to prevent races in claiming. Just use a 
compare-and-update with retries strategy. Proof is here:


https://github.com/jaypipes/placement-bench/blob/master/placement.py#L97-L142

Works great and prevents multiple writers from oversubscribing any 
resource without relying on any particular isolation level at all.


The `generation` field in the inventories table is what allows multiple 
writers to ensure a consistent view of the data without needing to rely 
on heavy lock-based semantics and/or RDBMS-specific isolation levels.


2) On reservations.

The reason I don't believe reservations are necessary to be in a quota 
library is because reservations add a concept of a time to a claim of 
some resource. You reserve some resource to be claimed at some point in 
the future and release those resources at a point further in time.


Quota checking doesn't look at what the state of some system will be at 
some point in the future. It simply returns whether the system *right 
now* can handle a request *right now* to claim a set of resources.


If you want reservation semantics for some resource, that's totally 
cool, but IMHO, a reservation service should live outside of the service 
that is actually responsible for providing resources to a consumer. 
Merging right-now quota checks and future-based reservations into the 
same library just complicates things unnecessarily IMHO.


3) On resizes.

Look, I recognize some users see some value in resizing their resources. 
That's fine. I personally think expand operations are fine, and that 
shrink operations are really the operations that should be prohibited in 
the API. But, whatever, I'm fine with resizing of requested resource 
amounts. My big point is if you don't have a separate table that stores 
quota_usages and instead only have a single table that stores the actual 
resource usage records, you don't have to do *any* quota check 
operations at all upon deletion of a resource. For modifying resource 
amounts (i.e. a resize) you merely need to change the calculation of 
requested resource amounts to account for the already-consumed usage amount.


Bottom line for me: I really won't support any proposal for a complex 
library that takes the resource claim process out of the hands of the 
services that own those resources. The simpler the interface of this 
library, the better.


Best,
-jay

On 04/19/2016 09:59 PM, Amrith Kumar wrote:

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Monday, April 18, 2016 2:54 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
Management Library proposal

On 04/16/2016 05:51 PM, Amrith Kumar wrote:

If we therefore assume that this will be a Quota Management Library,
it is safe to assume  that quotas are going to be managed on a
per-project basis, where participating projects will use this library.
I believe that it stands to reason that any data persistence will have
to be in a location decided by the individual project.


Depends on what you mean by "any data persistence". If you are referring
to the storage of quota values (per user, per tenant, global, etc) I think
that should be done by the Keystone service. This data is essentially an
attribute of the user or the tenant or the service endpoint itself (i.e.
global defaults). This data also rarely changes and logically belongs to
the service that manages users, tenants, and service endpoints: Keystone.

If you are referring to the storage of resource usage records, yes, each
service project should own that data (and frankly, I don't see a need to
persist any quota usage data at all, as I mentioned in a previous reply to
Attila).



[amrith] You make a distinction that I had made implicitly, and it is important
to highlight it. Thanks for pointing it out. Yes, I meant both of the
above, and as stipulated. Global defaults in keystone (somehow, TBD) and
usage records, on a per-service basis.



That may not be a very interesting statement but the corollary is, I
think, a very significant statement; it cannot be assumed that the
quota management information for all participating projects is in the
same database.


It cannot be assumed that this information is even in a database at all...



[amrith] I don't follow. If the service in question is to be scalable, I think 
it
stands to reason that there must be some mechanism by which instances of
the service can share usage records (as you refer to them, and I like
that term). I think it stands to reason that there must be some
database, no?


A hypothetical service consuming the Delimiter library provides
requesters with some widgets, and wishes to track the widgets that it
has provisioned both on a per-user basis, and on the 

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Amrith Kumar]

Jay, thanks for the detailed comments. Detailed responses follow.

-amrith

> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: Monday, April 18, 2016 2:54 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] More on the topic of DELIMITER, the Quota
> Management Library proposal
> 
> On 04/16/2016 05:51 PM, Amrith Kumar wrote:
> > If we therefore assume that this will be a Quota Management Library,
> > it is safe to assume  that quotas are going to be managed on a
> > per-project basis, where participating projects will use this library.
> > I believe that it stands to reason that any data persistence will have
> > to be in a location decided by the individual project.
> 
> Depends on what you mean by "any data persistence". If you are referring
> to the storage of quota values (per user, per tenant, global, etc) I think
> that should be done by the Keystone service. This data is essentially an
> attribute of the user or the tenant or the service endpoint itself (i.e.
> global defaults). This data also rarely changes and logically belongs to
> the service that manages users, tenants, and service endpoints: Keystone.
> 
> If you are referring to the storage of resource usage records, yes, each
> service project should own that data (and frankly, I don't see a need to
> persist any quota usage data at all, as I mentioned in a previous reply to
> Attila).
> 

[amrith] You make a distinction that I had made implicitly, and it is important
to highlight it. Thanks for pointing it out. Yes, I meant both of the
above, and as stipulated. Global defaults in keystone (somehow, TBD) and
usage records, on a per-service basis.


> > That may not be a very interesting statement but the corollary is, I
> > think, a very significant statement; it cannot be assumed that the
> > quota management information for all participating projects is in the
> > same database.
> 
> It cannot be assumed that this information is even in a database at all...
> 

[amrith] I don't follow. If the service in question is to be scalable, I think 
it
stands to reason that there must be some mechanism by which instances of
the service can share usage records (as you refer to them, and I like
that term). I think it stands to reason that there must be some
database, no?

> > A hypothetical service consuming the Delimiter library provides
> > requesters with some widgets, and wishes to track the widgets that it
> > has provisioned both on a per-user basis, and on the whole. It should
> > therefore multi-tenant and able to track the widgets on a per tenant
> > basis and if required impose limits on the number of widgets that a
> > tenant may consume at a time, during a course of a period of time, and
> > so on.
> 
> No, this last part is absolutely not what I think quota management should
> be about.
> 
> Rate limiting -- i.e. how many requests a particular user can make of an
> API in a given period of time -- should *not* be handled by OpenStack API
> services, IMHO. It is the responsibility of the deployer to handle this
> using off-the-shelf rate-limiting solutions (open source or proprietary).
> 
> Quotas should only be about the hard limit of different types of resources
> that a user or group of users can consume at a given time.
> 
[amrith] OK, good point. Agreed as stipulated.

> > Such a hypothetical service may also consume resources from other
> > services that it wishes to track, and impose limits on.
> 
> Yes, absolutely agreed.
> 
> > It is also understood as Jay Pipes points out in [4] that the actual
> > process of provisioning widgets could be time consuming and it is
> > ill-advised to hold a database transaction of any kind open for that
> > duration of time. Ensuring that a user does not exceed some limit on
> > the number of concurrent widgets that he or she may create therefore
> > requires some mechanism to track in-flight requests for widgets. I
> > view these as "intent" but not yet materialized.
> 
> It has nothing to do with the amount of concurrent widgets that a user can
> create. It's just about the total number of some resource that may be
> consumed by that user.
> 
> As for an "intent", I don't believe tracking intent is the right way to go
> at all. As I've mentioned before, the major problem in Nova's quota system
> is that there are two tables storing resource usage records: the
> *actual* resource usage tables (the allocations table in the new resource-
> providers modeling and the instance_extra, pci_devices and instances table
> in the legacy modeling) and the *quota usage* tables (quota_usages and
> reservations tables). The quota_usages table does not need to exist at
> all, and neither does the reservations table. Don't do intent-based
> consumption. Instead, just consume (claim) by writing a record for the
> resource class consumed on a provider into the actual resource usages
> table and then "check quotas" by querying the *actual* resource usages and
> 

Re: [openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Jay Pipes]

On 04/16/2016 05:51 PM, Amrith Kumar wrote:

If we therefore assume that this will be a Quota Management Library, it
is safe to assume  that quotas are going to be managed on a per-project
basis, where participating projects will use this library. I believe
that it stands to reason that any data persistence will have to be in a
location decided by the individual project.


Depends on what you mean by "any data persistence". If you are referring 
to the storage of quota values (per user, per tenant, global, etc) I 
think that should be done by the Keystone service. This data is 
essentially an attribute of the user or the tenant or the service 
endpoint itself (i.e. global defaults). This data also rarely changes 
and logically belongs to the service that manages users, tenants, and 
service endpoints: Keystone.


If you are referring to the storage of resource usage records, yes, each 
service project should own that data (and frankly, I don't see a need to 
persist any quota usage data at all, as I mentioned in a previous reply 
to Attila).



That may not be a very interesting statement but the corollary is, I
think, a very significant statement; it cannot be assumed that the
quota management information for all participating projects is in the
same database.


It cannot be assumed that this information is even in a database at all...


A hypothetical service consuming the Delimiter library provides
requesters with some widgets, and wishes to track the widgets that it
has provisioned both on a per-user basis, and on the whole. It should
therefore multi-tenant and able to track the widgets on a per tenant
basis and if required impose limits on the number of widgets that a
tenant may consume at a time, during a course of a period of time, and
so on.


No, this last part is absolutely not what I think quota management 
should be about.


Rate limiting -- i.e. how many requests a particular user can make of an 
API in a given period of time -- should *not* be handled by OpenStack 
API services, IMHO. It is the responsibility of the deployer to handle 
this using off-the-shelf rate-limiting solutions (open source or 
proprietary).


Quotas should only be about the hard limit of different types of 
resources that a user or group of users can consume at a given time.



Such a hypothetical service may also consume resources from other
services that it wishes to track, and impose limits on.


Yes, absolutely agreed.


It is also understood as Jay Pipes points out in [4] that the actual
process of provisioning widgets could be time consuming and it is
ill-advised to hold a database transaction of any kind open for that
duration of time. Ensuring that a user does not exceed some limit on the
number of concurrent widgets that he or she may create therefore
requires some mechanism to track in-flight requests for widgets. I view
these as “intent” but not yet materialized.


It has nothing to do with the amount of concurrent widgets that a user 
can create. It's just about the total number of some resource that may 
be consumed by that user.


As for an "intent", I don't believe tracking intent is the right way to 
go at all. As I've mentioned before, the major problem in Nova's quota 
system is that there are two tables storing resource usage records: the 
*actual* resource usage tables (the allocations table in the new 
resource-providers modeling and the instance_extra, pci_devices and 
instances table in the legacy modeling) and the *quota usage* tables 
(quota_usages and reservations tables). The quota_usages table does not 
need to exist at all, and neither does the reservations table. Don't do 
intent-based consumption. Instead, just consume (claim) by writing a 
record for the resource class consumed on a provider into the actual 
resource usages table and then "check quotas" by querying the *actual* 
resource usages and comparing the SUM(used) values, grouped by resource 
class, against the appropriate quota limits for the user. The 
introduction of the quota_usages and reservations tables to cache usage 
records is the primary reason for the race problems in the Nova (and 
other) quota system because every time you introduce a caching system 
for highly-volatile data (like usage records) you introduce complexity 
into the write path and the need to track the same thing across multiple 
writes to different tables needlessly.



Looking up at this whole infrastructure from the perspective of the
database, I think we should require that the database must not be
required to operate in any isolation mode higher than READ-COMMITTED;
more about that later (i.e. requiring a database run either serializable
or repeatable read is a show stopper).


This is an implementation detail is not relevant to the discussion about 
what the interface of a quota library would look like.



In general therefore, I believe that the hypothetical service processing
requests for widgets would have to handle three kinds of 

[openstack-dev] More on the topic of DELIMITER, the Quota Management Library proposal

[Amrith Kumar]

First, let me thank Vilobh, Nikhil, Duncan and others who have been
working on this quota management library/service issue. Recently there
have been a number of threads [1], [2], [3] on the subject of quotas and
quota management. This email is partially in response to [3] but since
it is too long for a review comment, and since I believe that it is a
discussion more suited for the ML than the review itself,  I’m posting
it to the ML. I will reference it in the review.

I think I can say that there is consensus that if a project is setup to
handle quotas in a standardized way, it should be a library and not a
service. To be clear, I am not implying that there is consensus that
such a project should exist, nor that if such a capability existed that
everyone was on board to use it, merely that several have raised
legitimate objections to a service, and that the community in general
seems to believe that the service approach is misguided.

If we therefore assume that this will be a Quota Management Library, it
is safe to assume  that quotas are going to be managed on a per-project
basis, where participating projects will use this library. I believe
that it stands to reason that any data persistence will have to be in a
location decided by the individual project. That may not be a very
interesting statement but the corollary is, I think, a very significant
statement; it cannot be assumed that the quota management information
for all participating projects is in the same database.

A hypothetical service consuming the Delimiter library provides
requesters with some widgets, and wishes to track the widgets that it
has provisioned both on a per-user basis, and on the whole. It should
therefore multi-tenant and able to track the widgets on a per tenant
basis and if required impose limits on the number of widgets that a
tenant may consume at a time, during a course of a period of time, and
so on. Such a hypothetical service may also consume resources from other
services that it wishes to track, and impose limits on.

It is also understood as Jay Pipes points out in [4] that the actual
process of provisioning widgets could be time consuming and it is
ill-advised to hold a database transaction of any kind open for that
duration of time. Ensuring that a user does not exceed some limit on the
number of concurrent widgets that he or she may create therefore
requires some mechanism to track in-flight requests for widgets. I view
these as “intent” but not yet materialized.

Looking up at this whole infrastructure from the perspective of the
database, I think we should require that the database must not be
required to operate in any isolation mode higher than READ-COMMITTED;
more about that later (i.e. requiring a database run either serializable
or repeatable read is a show stopper).

In general therefore, I believe that the hypothetical service processing
requests for widgets would have to handle three kinds of operations,
provision, modify, and destroy. The names are, I believe,
self-explanatory.

Without loss of generality, one can say that all three of them must
validate that the operation does not violate some limit (no more than X
widgets, no fewer than X widgets, rates, and so on). Assuming that the
service provisions resources from other services, it is also conceivable
that limits be imposed on the quantum of those services consumed. In
practice, I can imagine a service like Trove using the Delimiter project
to perform all of these kinds of limit checks; I’m not suggesting that
it does this today, nor that there is an immediate plan to implement all
of them, just that these all seem like good uses a Quota Management
capability.

- User may not have more than 25 database instances at a time
- User may not have more than 4 clusters at a time
- User may not consume more than 3TB of SSD storage at a time
- User may not launch more than 10 huge instances at a time
- User may not launch more than 3 clusters an hour
- No more than 500 copies of Oracle may be run at a time

While Nova would be the service that limits the number of instances a
user can have at a time, the ability for a service to limit this further
should not be underestimated.

In turn, should Nova and Cinder also use the same Quota Management
Library, they may each impose limitations like:

- User may not launch more than 20 huge instances at a time
- User may not launch more than 3 instances in a minute
- User may not consume more than 15TB of SSD at a time
- User may not have more than 30 volumes at a time

Again, I’m not implying that either Nova or Cinder should provide these
capabilities.

With this in mind, I believe that the minimal set of operations that
Delimiter should provide are:

- define_resource(name, max, min, user_max, user_min, …)
- update_resource_limits(name, user, user_max, user_min, …)
- reserve_resource(name, user, size,