Re: [Openstack] [Metering] schema and counter definitions

2012-05-10 Thread Loic Dachary
On 05/09/2012 11:11 PM, Doug Hellmann wrote: On Wed, May 9, 2012 at 3:07 PM, Tomasz Paszkowski ss7...@gmail.com mailto:ss7...@gmail.com wrote: On Wed, May 9, 2012 at 8:02 PM, Doug Hellmann doug.hellm...@dreamhost.com mailto:doug.hellm...@dreamhost.com wrote: Nice!

Re: [Openstack] [Metering] schema and counter definitions

2012-05-09 Thread Tomasz Paszkowski
Here is the simplified version of my code (without ampq support, counter stored directly to mysql db). https://github.com/ss7pro/rescnt Code is started from main.py which is constantly collecting counters from libvirt and storing them in a mysql database. On Mon, May 7, 2012 at 9:25 PM, Tomasz

Re: [Openstack] [Metering] schema and counter definitions

2012-05-09 Thread Doug Hellmann
On Wed, May 9, 2012 at 12:42 PM, Tomasz Paszkowski ss7...@gmail.com wrote: Here is the simplified version of my code (without ampq support, counter stored directly to mysql db). https://github.com/ss7pro/rescnt Code is started from main.py which is constantly collecting counters from

Re: [Openstack] [Metering] schema and counter definitions

2012-05-09 Thread Tomasz Paszkowski
On Wed, May 9, 2012 at 8:02 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Nice! For production code I think we are going to want to separate collection from storage, aren't we? We don't want each compute node to require access to the database server (that's an issue with nova that

Re: [Openstack] [Metering] schema and counter definitions

2012-05-09 Thread Doug Hellmann
On Wed, May 9, 2012 at 3:07 PM, Tomasz Paszkowski ss7...@gmail.com wrote: On Wed, May 9, 2012 at 8:02 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Nice! For production code I think we are going to want to separate collection from storage, aren't we? We don't want each compute

Re: [Openstack] [Metering] schema and counter definitions

2012-05-09 Thread Tomasz Paszkowski
I agree. Do you have any plans how to coordinate our efforts ? On Wed, May 9, 2012 at 11:11 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Wed, May 9, 2012 at 3:07 PM, Tomasz Paszkowski ss7...@gmail.com wrote: On Wed, May 9, 2012 at 8:02 PM, Doug Hellmann

Re: [Openstack] [Metering] schema and counter definitions

2012-05-07 Thread Tomasz Paszkowski
On Mon, May 7, 2012 at 5:21 PM, Loic Dachary l...@enovance.com wrote: Hi Tomasz, Hi I could not agree more and this is the reason why I/O shows in the list of meters shown in http://wiki.openstack.org/EfficientMetering (c5) disk IO in megabyte per second has a high impact on the service

Re: [Openstack] [Metering] schema and counter definitions

2012-05-06 Thread Tomasz Paszkowski
Hi, I'd like to share my thoughts on metering openstack resources usage. Except those data available from SystemUsageData and those mentioned in blueprints some of the cloud providers charge for I/O on disk drives (to prevent users from dd if=/dev/zero nosense and to teach them properly

Re: [Openstack] [Metering] schema and counter definitions

2012-05-04 Thread Thierry Carrez
Robert Collins wrote: On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services) whit.tur...@hp.com wrote: Hi - I think a flexible aggregation scheme is needed; the levels of aggregation available should be definable in the meter independent of the sources of usage data themselves. If

Re: [Openstack] [Metering] schema and counter definitions

2012-05-04 Thread nick . barcet
-Original Message- From: Thierry Carrez thie...@openstack.org To: openstack@lists.launchpad.net Sent: Fri, 04 May 2012 2:50 Subject: Re: [Openstack] [Metering] schema and counter definitions Robert Collins wrote: On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services) whit.tur

Re: [Openstack] [Metering] schema and counter definitions

2012-05-04 Thread Nick Barcet
On 05/04/2012 02:50 AM, Thierry Carrez wrote: Robert Collins wrote: On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services) whit.tur...@hp.com wrote: Hi - I think a flexible aggregation scheme is needed; the levels of aggregation available should be definable in the meter independent of

Re: [Openstack] [Metering] schema and counter definitions

2012-05-04 Thread Loic Dachary
On 05/04/2012 11:50 AM, Thierry Carrez wrote: Robert Collins wrote: On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services) whit.tur...@hp.com wrote: Hi - I think a flexible aggregation scheme is needed; the levels of aggregation available should be definable in the meter independent of

Re: [Openstack] [Metering] schema and counter definitions

2012-05-03 Thread Turner, Whit (Cloud Services)
Dachary Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [Metering] schema and counter definitions On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary l...@enovance.com wrote: On 04/30/2012 03:49 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary l

Re: [Openstack] [Metering] schema and counter definitions

2012-05-03 Thread Robert Collins
On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services) whit.tur...@hp.com wrote: Hi - I think a flexible aggregation scheme is needed; the levels of aggregation available should be definable in the meter independent of the sources of usage data themselves. If invoices need to be very

Re: [Openstack] [Metering] schema and counter definitions

2012-05-02 Thread Loic Dachary
On 05/02/2012 07:19 AM, Mark McLoughlin wrote: Hey, On Tue, 2012-05-01 at 23:05 +0200, Loic Dachary wrote: On 05/01/2012 06:13 PM, Mark McLoughlin wrote: Hi Loic, On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote: - I agree that we don't want to go too far with aggregation and lose

Re: [Openstack] [Metering] schema and counter definitions

2012-05-02 Thread Loic Dachary
On 05/02/2012 07:39 AM, Mark McLoughlin wrote: On Tue, 2012-05-01 at 23:05 +0200, Loic Dachary wrote: On 05/01/2012 06:13 PM, Mark McLoughlin wrote: Hi Loic, On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote: To prepare for the next meeting ( thursday 3rd, may 2012

Re: [Openstack] [Metering] schema and counter definitions

2012-05-02 Thread Mark McLoughlin
On Wed, 2012-05-02 at 10:08 +0200, Loic Dachary wrote: My impression is that the notifications system is intended to cover all billable usage in at least Nova and Glance. It's also my understanding. Regarding swift, how would you suggest we approach the problem ? I see two possible courses:

Re: [Openstack] [Metering] schema and counter definitions

2012-05-02 Thread David Kranz
There was a swift talk at the design summit that is related to (a): http://etherpad.openstack.org/FolsomSwiftStatsd. There is a good summary in the referenced blog post: http://www.swiftstack.com/blog/2012/04/11/swift-monitoring-with-statsd/ -David On 5/2/2012 4:19 AM, Mark McLoughlin wrote:

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Loic Dachary
On 04/30/2012 11:39 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary l...@enovance.com mailto:l...@enovance.com wrote: On 04/30/2012 08:03 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary l...@enovance.com mailto:l...@enovance.com

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Nick Barcet
On 05/01/2012 02:23 AM, Loic Dachary wrote: On 04/30/2012 11:39 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary l...@enovance.com mailto:l...@enovance.com wrote: On 04/30/2012 08:03 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Loic Dachary
On 05/01/2012 04:38 PM, Nick Barcet wrote: On 05/01/2012 02:23 AM, Loic Dachary wrote: On 04/30/2012 11:39 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary l...@enovance.com mailto:l...@enovance.com wrote: On 04/30/2012 08:03 PM, Doug Hellmann wrote: On Mon,

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Doug Hellmann
On Tue, May 1, 2012 at 10:38 AM, Nick Barcet nick.bar...@canonical.comwrote: On 05/01/2012 02:23 AM, Loic Dachary wrote: On 04/30/2012 11:39 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary l...@enovance.com mailto:l...@enovance.com wrote: On 04/30/2012

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Doug Hellmann
On Tue, May 1, 2012 at 11:49 AM, Doug Hellmann doug.hellm...@dreamhost.comwrote: On Tue, May 1, 2012 at 10:38 AM, Nick Barcet nick.bar...@canonical.comwrote: On 05/01/2012 02:23 AM, Loic Dachary wrote: On 04/30/2012 11:39 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 3:43 PM, Loic

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Mark McLoughlin
Hi Loic, On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote: To prepare for the next meeting ( thursday 3rd, may 2012 http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and reorganized the Metering blueprint so that it ( hopefully ) incorporates all the information

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Andrew Clay Shafer
I'm glad to see people championing the effort to implement metering. Is there someway to refocus the enthusiasm for solving the metering problem into engineering a general solution in OpenStack? I'm just going to apologize in advance, but I don't think this project is headed in the right

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Loic Dachary
On 05/01/2012 05:49 PM, Doug Hellmann wrote: On Tue, May 1, 2012 at 10:38 AM, Nick Barcet nick.bar...@canonical.com mailto:nick.bar...@canonical.com wrote: On 05/01/2012 02:23 AM, Loic Dachary wrote: On 04/30/2012 11:39 PM, Doug Hellmann wrote: On Mon, Apr 30,

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Loic Dachary
On 05/01/2012 06:13 PM, Mark McLoughlin wrote: Hi Loic, On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote: To prepare for the next meeting ( thursday 3rd, may 2012 http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and reorganized the Metering blueprint so that it (

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Mark McLoughlin
Hey, On Tue, 2012-05-01 at 23:05 +0200, Loic Dachary wrote: On 05/01/2012 06:13 PM, Mark McLoughlin wrote: Hi Loic, On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote: - I agree that we don't want to go too far with aggregation and lose useful data like which instances have

Re: [Openstack] [Metering] schema and counter definitions

2012-05-01 Thread Mark McLoughlin
On Tue, 2012-05-01 at 23:05 +0200, Loic Dachary wrote: On 05/01/2012 06:13 PM, Mark McLoughlin wrote: Hi Loic, On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote: To prepare for the next meeting ( thursday 3rd, may 2012 http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned

[Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Loic Dachary
Hi, To prepare for the next meeting ( thursday 3rd, may 2012 http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and reorganized the Metering blueprint so that it ( hopefully ) incorporates all the information temporarily stored in the etherpad (

Re: [Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Loic Dachary
On 04/30/2012 12:15 PM, Loic Dachary wrote: We could start a discussion from the content of the following sections: http://wiki.openstack.org/EfficientMetering#Counters I think the rationale of the counter aggregation needs to be explained. My understanding is that the metering system will be

Re: [Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Doug Hellmann
On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary l...@enovance.com wrote: On 04/30/2012 12:15 PM, Loic Dachary wrote: We could start a discussion from the content of the following sections: http://wiki.openstack.org/EfficientMetering#Counters I think the rationale of the counter aggregation

Re: [Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Loic Dachary
On 04/30/2012 03:49 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary l...@enovance.com mailto:l...@enovance.com wrote: On 04/30/2012 12:15 PM, Loic Dachary wrote: We could start a discussion from the content of the following sections:

Re: [Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Doug Hellmann
On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary l...@enovance.com wrote: On 04/30/2012 03:49 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary l...@enovance.com wrote: On 04/30/2012 12:15 PM, Loic Dachary wrote: We could start a discussion from the content of the

Re: [Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Loic Dachary
On 04/30/2012 08:03 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary l...@enovance.com mailto:l...@enovance.com wrote: On 04/30/2012 03:49 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary l...@enovance.com mailto:l...@enovance.com

Re: [Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Joshua Harlow
Agreed, I would get as much low-level data as possible and let other systems combine that as they want to form whatever billing model they choose. On 4/30/12 6:49 AM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary l...@enovance.com wrote: On

Re: [Openstack] [Metering] schema and counter definitions

2012-04-30 Thread Doug Hellmann
On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary l...@enovance.com wrote: On 04/30/2012 08:03 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary l...@enovance.com wrote: On 04/30/2012 03:49 PM, Doug Hellmann wrote: On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary