[openstack-dev] [nova][gate] configurable quota threshold warnings?

2014-10-06 Thread Matt Riedemann

I'm floating this in case the idea has come up before.

I've spent most of my day so far digging through logs and code for gate 
race bug 1353962 [1] where an admin tenant is going over-quota when 
trying to allocate fixed IPs for a test instance (the test setup creates 
2 instances and more test instances are created in some of the test cases).


I'm going to push up some patches to nova to make the logs more useful 
when we hit this, i.e. log the quota limit and current usage for the 
given resource (fixed_ips in this case).


But the idea that keeps coming up when I'm digging into stuff like this 
is it'd be nice to know when we're nearing over-quota so we can walk 
that back and figure out what was going on prior to us falling over the 
cliff. This isn't a novel idea, but I haven't heard it discussed before.


What I'm thinking is basically add a config option for setting a 
threshold value which would dump warnings when you're over that 
threshold.  This could be configured per resource, e.g. I only care 
about fixed_ips, instances, etc.  It could be configured per tenant 
(maybe we only care about this for the admin tenant?).


My use case would be specifically related to the bug in question, so I 
want to set the threshold at let's say 80%.  With a default fixed_ips 
quota of 10, when I've gotten over 8 reserved for the admin tenant, I 
start logging warnings that I'm nearing over-quota.  When we go 
over-quota on a request, we could then see what operations/requests were 
going on before this that tipped us over.


Thoughts?

[1] https://bugs.launchpad.net/nova/+bug/1353962

--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][gate] configurable quota threshold warnings?

2014-10-06 Thread Michael Still
Logging like that is useful for admins, but I wonder if we should talk
about notifications as well. That would allow people to build systems which
warned the user as well that they are nearing quota.

Michael

On Tue, Oct 7, 2014 at 2:33 AM, Matt Riedemann mrie...@linux.vnet.ibm.com
wrote:

 I'm floating this in case the idea has come up before.

 I've spent most of my day so far digging through logs and code for gate
 race bug 1353962 [1] where an admin tenant is going over-quota when trying
 to allocate fixed IPs for a test instance (the test setup creates 2
 instances and more test instances are created in some of the test cases).

 I'm going to push up some patches to nova to make the logs more useful
 when we hit this, i.e. log the quota limit and current usage for the given
 resource (fixed_ips in this case).

 But the idea that keeps coming up when I'm digging into stuff like this is
 it'd be nice to know when we're nearing over-quota so we can walk that back
 and figure out what was going on prior to us falling over the cliff. This
 isn't a novel idea, but I haven't heard it discussed before.

 What I'm thinking is basically add a config option for setting a threshold
 value which would dump warnings when you're over that threshold.  This
 could be configured per resource, e.g. I only care about fixed_ips,
 instances, etc.  It could be configured per tenant (maybe we only care
 about this for the admin tenant?).

 My use case would be specifically related to the bug in question, so I
 want to set the threshold at let's say 80%.  With a default fixed_ips quota
 of 10, when I've gotten over 8 reserved for the admin tenant, I start
 logging warnings that I'm nearing over-quota.  When we go over-quota on a
 request, we could then see what operations/requests were going on before
 this that tipped us over.

 Thoughts?

 [1] https://bugs.launchpad.net/nova/+bug/1353962

 --

 Thanks,

 Matt Riedemann


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Rackspace Australia
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev