Re: [Openstack] [metering] ceilometer dev docs on readthedocs.org

[Loic Dachary]

On 07/03/2012 07:46 PM, Doug Hellmann wrote:
> I've set up the ceilometer development documentation build on RTD at 
> http://ceilometer.readthedocs.org/en/latest/index.html
>
Hi,

I've updated https://launchpad.net/ceilometer to list this link.

Cheers


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] delta vs. cumulative meters

[Loic Dachary]

On 05/29/2012 07:58 PM, Doug Hellmann wrote:
>
>
> On Tue, May 29, 2012 at 12:30 PM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
> On 05/29/2012 05:42 PM, Doug Hellmann wrote:
> > IIRC, the meters discussed in the wiki [1] are supposed to show "delta" 
> values (usage since the last time an event was generated), although the 
> "Alternate Gauge Design" section discusses cumulative meters instead. The 
> libvirt pollsters we have now produce cumulative data, and it might be 
> complicated and error-prone to change them to compute the deltas internally 
> before generating events. On the other hand, some of the other counters may 
> be more complicated to generate as cumulative values.
> >
> > Rather than taking an either/or approach, I propose we document for 
> each counter whether it uses the delta or cumulative approach. The API server 
> can ask the meter plugin for that piece of information when calculating the 
> aggregate value so that the caller does not have to keep track of the rules 
> for each meter.
> >
> I tend to prefer using a gauge and I agree that documenting what was 
> preferred (gauge or delta) for a given meter / counter is enough.
>
>
> Just to be clear, by "gauge" do you mean the cumulative form of the counter?
>  
Yes, I do ;-)

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] delta vs. cumulative meters

[Loic Dachary]

On 05/29/2012 05:42 PM, Doug Hellmann wrote:
> IIRC, the meters discussed in the wiki [1] are supposed to show "delta" 
> values (usage since the last time an event was generated), although the 
> "Alternate Gauge Design" section discusses cumulative meters instead. The 
> libvirt pollsters we have now produce cumulative data, and it might be 
> complicated and error-prone to change them to compute the deltas internally 
> before generating events. On the other hand, some of the other counters may 
> be more complicated to generate as cumulative values.
>
> Rather than taking an either/or approach, I propose we document for each 
> counter whether it uses the delta or cumulative approach. The API server can 
> ask the meter plugin for that piece of information when calculating the 
> aggregate value so that the caller does not have to keep track of the rules 
> for each meter.
>
I tend to prefer using a gauge and I agree that documenting what was preferred 
(gauge or delta) for a given meter / counter is enough.

My 2ct ;-)

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] Do we need an API and storage?

[Loic Dachary]

On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
>
> I'm now of the opinion that we exclude storage and API from the metering
> project scope. Let's just focus on defining a metering message format,
> bus, and maybe a client-library to make it easy to write metering consumers.
>
> That way we avoid building a lot of things that we only _think will be
> useful_ for potential billing integration. Only writing/delivering such
> an integration component would prove that we built at least something
> that is useful.
>
>

Hi,

I'm a little reluctant to question the whole approach because I'm engaged in it 
:-) It's scary to contemplate the idea of throwing away some of the work done 
but I welcome the challenge. Better lose a few days work than keep going in the 
wrong direction.

Are you proposing that such a library would then be integrated in whatever 
billing system someone has in mind ? For instance

Dough https://github.com/lzyeval/dough
trystack.org billing https://github.com/trystack/dash_billing
nova-billing https://github.com/griddynamics/nova-billing

Would depend on this library and rely on its API to collect what they need. The 
same would be done by proprietary billing systems ?

Regarding the relevance of the metrics exposed by the API, I made sure they 
match the need of the eNovance sales. I'm quite sure Nicolas checked for 
Canonical and Doug did the same regarding the needs of Dreamhost. I'm confident 
that the information is actualy useful, at least in these practical cases.

Getting rid of the storage imposes a constraint on the billing system : it must 
make 100% sure that once a message is consumed it will be reliably archived. It 
also adds a constraint on the chosen bus : it must be able to retain all 
messages for as long as a consumer needs, which may be days or weeks. Or it 
adds a constraint on the billing system which must make 100% sure it will 
consume all relevant messages the bus at all times before they expire.

I have no strong feelings about exposing a bus for everyone to use instead of a 
REST API. I tend to prefer the REST API because it is an established standard 
for OpenStack. Could you expand on why a bus would be significantly better than 
a REST API in this specific case ?

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] resources metadata

[Loic Dachary]

On 05/16/2012 09:32 AM, Nick Barcet wrote:
>> On Tue, May 15, 2012 at 10:26 AM, Doug Hellmann
>> mailto:doug.hellm...@dreamhost.com>> wrote:
>>
>>
>>
>> On Tue, May 15, 2012 at 8:21 AM, Julien Danjou
>> mailto:julien.dan...@enovance.com>> wrote:
>>
>> On Tue, May 15 2012, Loic Dachary wrote:
>>
>> > On 05/15/2012 12:05 PM, Julien Danjou wrote:
>> >>
>> >> OTOH I find the metadata proposal in another table too much
>> >> complicated. Why not storing what metadata in the
>> meter.payload field
>> >> in the same table (e.g. as a JSON string)?
>> > I would be much simpler to store the metadata in the
>> resource_id field
>> > which could be renamed into resource field.
>>
>> That'd be even more radical.
>>
>>
>> I like it because it would simplify the messaging. We can leave the
>> storage optimization question to the daemon that stores the data.
>>
>> > Instead of resource_id=134123 we could have resource={ 'id':
>> 134123,
>> > 'name': 'foobar', 'flavor': 'm1.small' etc.. } There would be
>> no need
>> > for versioning, format, separate table, etc. etc. The only
>> convention
>> > would be that it's a hash with at least one field : the id of the
>> > resource. The rest is metadata.
>> >
>> > It will use a lot of disk space with highly redundant information.
>>
>> Ok, so the current proposal is just early optimization, as I
>> understood.
>>
>> If you want to optimize the storage, why not use resource_id as a
>> foreign key to the metatable table which would contains unique
>> records
>> of metadata?
>>
>> That would allow to store identical metadata once (and therefore
>> optimize space) and will be much simpler. There would not be any
>> need of
>> version, timestamp, or whatever on metadata.
> I fully second Julien's analysis regarding the storage of metadata in a
> separate table.  We are overly complicating the schema for the sole
> purpose of resource optimization.  In fact, the current metadata table
> seems to be defining 5 fields where only one constitutes some valid
> content we would like to extract at some point, the rest existing only
> for relational or versioning purposes.  A bit of an overkill, it seems.
>
> I would not recommend however to overcharge a field that can be used as
> a key value for searches (resource_id) to store additional data, as it
> would block us to search on that key easily.  Why mot just extend the
> meter schema to add a metadata field, which can use the proposed JSON
> format?
>
It makes sense and I updated the wiki accordingly:

http://wiki.openstack.org/EfficientMetering?action=diff&rev2=81&rev1=80

What do you think ?

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] resources metadata

[Loic Dachary]

On 05/15/2012 12:05 PM, Julien Danjou wrote:
>
> OTOH I find the metadata proposal in another table too much
> complicated. Why not storing what metadata in the meter.payload field
> in the same table (e.g. as a JSON string)?
I would be much simpler to store the metadata in the resource_id field which 
could be renamed into resource field.
Instead of resource_id=134123 we could have resource={ 'id': 134123, 'name': 
'foobar', 'flavor': 'm1.small' etc.. }
There would be no need for versioning, format, separate table, etc. etc. The 
only convention would be that it's a hash with at least one field : the id of 
the resource. The rest is metadata.

It will use a lot of disk space with highly redundant information.

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] API Extensibility

[Loic Dachary]

On 05/10/2012 05:54 PM, Doug Hellmann wrote:
>
>
> On Thu, May 10, 2012 at 9:22 AM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
> > Another item that we need to discuss is extensibility of this API.
>
> Hi,
>
> Here is a proposal, which we could discuss further during the meeting.
>
> GET extension=¶m1=foo¶m2=bar
>
> The API looks up /usr/share/ceilometer/extensions/.py and loads it. 
> The  module defines a query function that takes the following arguments:
>
>
> Andrew Bogott is doing some work with a standardized plugin mechanism for 
> Nova which will eventually be put in the common lib for all of the projects. 
> We should look at his work and use it, rather than inventing something else. 
> I think it will eventually use setuptools entrypoints, which eliminates the 
> need to worry about search paths.
I agree.
>
> Why would the extension be a query parameter, rather than a URL component? 
> That is, why wouldn't the extension just add new endpoints that could be 
> queried directly using their own API? Maybe I don't understand the types of 
> extensions you are thinking of.
>  
No specific reason, we can just forget about it. I'm grateful jaypipes 
suggested it during the last meeting, that will save us time.

Cheers
>
>
> * QUERY_STRING (i.e. extension=¶m1=foo¶m2=bar )
>
> * a handler to the storage
> * a pointer to the configuration (assuming there is a /etc/ceilometer.ini 
> file, for instance)
>
> The query function would return the result. For instance { 'in': 20001, 
> 'out': 489324 } if asked for aggregated network usage.
>
> Multiple extensions directories could be specified and searched, allowing 
> a mixture of extensions provided in ceilometer and custom extensions to 
> address specific needs or to mature an new extension.
>
> The primary benefit of defining extensions in this way is to avoid 
> complex conventions for aggregations or other advanced operations. If the API 
> was to impose a syntax or conventions to say "sum this field and this one and 
> display the result ordered in this way and grouped by this field and this 
> one", it would be redundant with the query language of the underlying data. 
> For instance, if using mongodb, it would be difficult to expose all the 
> features provided by http://www.mongodb.org/display/DOCS/Aggregation or 
> http://www.mongodb.org/display/DOCS/MapReduce
>
> Cheers
>
> --
> Loïc Dachary Chief Research Officer
> // eNovance labs   http://labs.enovance.com
> // ✉ l...@enovance.com <mailto:l...@enovance.com>  ☎ +33 1 49 70 99 82 
> 
>
>
> ___
> Mailing list: https://launchpad.net/~openstack 
> <https://launchpad.net/%7Eopenstack>
> Post to : openstack@lists.launchpad.net 
> <mailto:openstack@lists.launchpad.net>
> Unsubscribe : https://launchpad.net/~openstack 
> <https://launchpad.net/%7Eopenstack>
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] resources metadata

[Loic Dachary]

On 05/14/2012 04:15 PM, Doug Hellmann wrote:
>
>
> On Fri, May 11, 2012 at 3:55 PM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
>
> > - The interesting metadata for a resource may depend on the type of
> > resource. Do we need separate "tables" for that or can we normalize
> > somehow?
> > - How do we map a resource to the correct version of its metadata at
> > any given time? Timestamps seem brittle.
> > - Do we need to reflect the metadata in the aggregation API?
> >
> Hi,
>
> I started a new thread for the "metadata" topic. I suspect it deserves 
> it. Although I was reluctant to acknowledge that the metadate should be 
> stored by the metering, yesterday's meeting made me realize that it was 
> mandatory. The compelling reason ( for me ;-) is that it would make it much 
> more difficult to implement a billing system if the metering does not provide 
> a simple way to extract metadata and display it in a human readable way (or 
> meaningfull to accountants ?) .
>
> I see two separate questions :
>
> a) how to store and query metadata ?
> b) what is the semantic of metadata for a given resource ?
>
> My hunch is that there will never be a definitive answer to b) and that 
> the best we can do is to provide a format and leave the semantic to the 
> documentation of the metering system, explaining the metadata of a resource.
>
> Regarding the storage of the metadata, the metering could listen / poll 
> events creating / updating / deleting a given resource and store a history 
> log indexed by the resource id. Something like:
>
> { meter_type: TTT,
> resource_id: RRR,
> metadata: [{ version: ,
> timestamp: TIME1,
> payload: PAYLOAD1 },
> { version: ,
> timestamp: TIME3,
> payload: PAYLOAD2 }]
> }
>
> With PPP being the resource dependant metadata that depends on the type 
> of the resource. And the metadata array being an ordered list of the 
> successive states of the resource over time. The VVV version accounting for 
> changes in the format of the payload.
>
> The query would be :
>
> GET /resource///
>
> and it would return PAYLOAD1 if TIME2 is in the range [TIME1,TIME3[
>
> I'm not sure why you think "timestamp is brittle". Maybe I'm missing 
> something.
>
>
> Each set of metering data will need to be associated with the appropriate 
> metadata from the resource at the time the metering information was 
> collected. The rate of change of metadata and metering events are different, 
> though, so the timestamps of the metadata records are unlikely to match 
> exactly with the values in the metering records. Depending on the clock 
> resolution, it would be possible to have metadata changes and meter data with 
> the same timestamp, resulting in an incorrect association.
Indeed, good point.
>
> We can work around that by maintaining proper foreign key references using 
> the metadata version field as you describe in the schema above (so the 
> resource id and metadata version value point to the correct metadata record). 
> It will make recording the metering data less efficient because we will need 
> to determine the current version for the resource metadata, but we can 
> optimize that eventually through indexes and caching.
>
> Aggregation will also need to take the metadata version into account, so 
> everywhere in the list of queries we say "by resource_id" we need to change 
> that to "by resource_id and version".
I added the idea of a format version for when the payload format changes and 
tried to write down a description of the metadata storage matching this thread 
in the wiki.

http://wiki.openstack.org/EfficientMetering?action=diff&rev2=80&rev1=78

What do you think ?

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] need +2 reviews/approvals

[Loic Dachary]

On 05/14/2012 05:05 PM, Doug Hellmann wrote:
> Julien has done a good bit of work bootstrapping the server implementation, 
> but we need to have more +2 reviews (and approvals) of the code [1] before it 
> will make it into the public repository. Are we holding off on doing those 
> reviews on purpose until after the series of design meetings is complete?
>
> Doug
>
> [1] 
> https://review.stackforge.org/#/q/status:open+project:stackforge/ceilometer,n,z
>
I reviewed positively but I'm not allowed to +2 (only +1 ).

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] public API design

[Loic Dachary]

On 05/11/2012 10:21 PM, Doug Hellmann wrote:
>
>
> On Fri, May 11, 2012 at 3:40 PM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
> On 05/11/2012 05:55 PM, Doug Hellmann wrote:
>
>
> While thinking about this use case I also had a simple idea to optimize 
> the storage of events and added it to the wiki
> http://wiki.openstack.org/EfficientMetering?action=diff&rev2=77&rev1=76 
> <http://wiki.openstack.org/EfficientMetering?action=diff&rev2=77&rev1=76>
>
>
> It will probably be just as efficient to handle that during aggregate 
> calculation one time, rather than trying to do on-the-fly aggregation.
>
I agree and changed the description accordingly:
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=78&rev1=77

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] licensing

[Loic Dachary]

On 05/12/2012 01:30 AM, Doug Hellmann wrote:
>
>
> On Fri, May 11, 2012 at 5:27 PM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
> On 05/11/2012 10:01 PM, Doug Hellmann wrote:
> > I was very surprised to see the change to license ceilometer as AGPL 
> [1]. Why are we not using the same Apache v2 license that all other OpenStack 
> projects are using?
> >
> > Doug
> >
> > [1] https://review.stackforge.org/#/c/29/
> >
> >
> Hi,
>
> Of course it will be Apache v2 when it is incubated, otherwise it will be 
> rejected. eNovance policy is to publish server side code under AGPLv3+, hence 
> the change. This should have been discussed before though. If this raises any 
> issues please let me know and I'll make sure they are resolved.
>
>
> As I pointed out privately, it seems like it will be easier to start with the 
> license we know we will need later than to change the license after we have a 
> bunch of contributions. I appreciate the rationale behind a "default to 
> copyleft" policy, but the community already has a standard for licensing and 
> that should take precedence here.
>
This is reasonable indeed and we will publish all code to ceilometer under 
Apache from now on. Thanks for raising this issue.

Cheers



-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] licensing

[Loic Dachary]

On 05/11/2012 10:01 PM, Doug Hellmann wrote:
> I was very surprised to see the change to license ceilometer as AGPL [1]. Why 
> are we not using the same Apache v2 license that all other OpenStack projects 
> are using?
>
> Doug
>
> [1] https://review.stackforge.org/#/c/29/
>
>
Hi,

Of course it will be Apache v2 when it is incubated, otherwise it will be 
rejected. eNovance policy is to publish server side code under AGPLv3+, hence 
the change. This should have been discussed before though. If this raises any 
issues please let me know and I'll make sure they are resolved.

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] resources metadata (was: public API design)

[Loic Dachary]

> - The interesting metadata for a resource may depend on the type of
> resource. Do we need separate "tables" for that or can we normalize
> somehow?
> - How do we map a resource to the correct version of its metadata at
> any given time? Timestamps seem brittle.
> - Do we need to reflect the metadata in the aggregation API?
>
Hi,

I started a new thread for the "metadata" topic. I suspect it deserves it. 
Although I was reluctant to acknowledge that the metadate should be stored by 
the metering, yesterday's meeting made me realize that it was mandatory. The 
compelling reason ( for me ;-) is that it would make it much more difficult to 
implement a billing system if the metering does not provide a simple way to 
extract metadata and display it in a human readable way (or meaningfull to 
accountants ?) .

I see two separate questions :

a) how to store and query metadata ?
b) what is the semantic of metadata for a given resource ?

My hunch is that there will never be a definitive answer to b) and that the 
best we can do is to provide a format and leave the semantic to the 
documentation of the metering system, explaining the metadata of a resource.

Regarding the storage of the metadata, the metering could listen / poll events 
creating / updating / deleting a given resource and store a history log indexed 
by the resource id. Something like:

{ meter_type: TTT,
resource_id: RRR,
metadata: [{ version: ,
timestamp: TIME1,
payload: PAYLOAD1 },
{ version: ,
timestamp: TIME3,
payload: PAYLOAD2 }]
}

With PPP being the resource dependant metadata that depends on the type of the 
resource. And the metadata array being an ordered list of the successive states 
of the resource over time. The VVV version accounting for changes in the format 
of the payload.

The query would be :

GET /resource///

and it would return PAYLOAD1 if TIME2 is in the range [TIME1,TIME3[

I'm not sure why you think "timestamp is brittle". Maybe I'm missing something.

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [metering] public API design

[Loic Dachary]

On 05/11/2012 05:55 PM, Doug Hellmann wrote:
> During yesterday's meeting we discussed the API proposal at
> http://wiki.openstack.org/EfficientMetering#API and came up with a few
> missing items and other points for discussion. We should try to work out
> those details on the list before the meeting next week.
>
> The original proposal included these API calls:
>
> GET list components
> GET list components meters (argument : name of the component)
> GET list [user_id|project_id|source]
> GET list of meter_type
> GET list of events per [user_id|project_id|source] ( allow to specify user_id 
> or project_id or both )
> GET sum of (meter_volume, meter_duration) for meter_type and 
> [user_id|project_id|source]
>
> They can be broken down into three groups:
>
> - Discover the sorts of things the server can provide:
>
> - list the components providing metering data
> - list the meters for a given component
> - list the known users
> - list the known projects
> - list the known sources
> - list the types of meters known
>
> - Fetch raw event data, without aggregation:
>
> - per user
> - per project
> - per source
> - per user and project
>
> - Produce aggregate views of the data:
>
> - sum "volume" field for meter type over a time period
> - per user
> - per project
> - per source
> - sum "duration" field for meter type over a period of time
> - per user
> - per project
> - per source
I updated the wiki with these three groups, it is a good starting point for 
discussion.
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=75&rev1=74
>
> We agreed that all of the queries related to metering data would need
> to have parameters to set the start and end times with the start time
> inclusive and the end time exclusive (i.e., start <= timestamp < end).
>
> Callers also are likely to want to filter the raw data queries based
> on the meter type, so we need to add that as an optional argument
> there. The aggregate values have the meter type as a required argument
> (because it does not make sense to aggregate data from separate meters
> together).
>
> There are a few other queries missing from that list. The items below
> are based on the meeting notes and my own thoughts from yesterday,
> please add to the list if I have left anything out (or suggest why we
> should not implement something suggested here, of course):
>
> - list discrete events that may not have a duration (instance
> creation, IP allocation, etc.)
> - list raw event data for a resource ("what happened with a specific
> instance?")
> - aggregate event data per meter type for a resource over a period of
> time ("what costs are related to this instance?")
> - sum volume for meter type over a time period for a specific resource
> ("how much total bandwidth was used by a VIF?")
> - sum duration for meter type over a time period for a specific
> resource ("how long did an instance run?")
> - metadata for resources (such as location of instances)
> - aggregating averages in addition to sums
>
I've added this list as a note in the API chapter.
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=76&rev1=75
> Some of these items may be better handled in the consumer of this API
> (the system that actually bills the user). I am curious to know how
> others are handling auditing or other billing detail reporting for
> users.
I will be in a meeting with the participants of the 
http://www.opencloudware.org/ project early next week and I'll ask them. Maybe 
Daniel Dyer could share a practical experience with us.

> In order to support the discrete events, we need to capture
> them. Perhaps those could be implemented as additional counter
> types. Thoughts?
Are there resources that should be billed (henced metered) if their lifespan is 
shorter than the polling / publishing event period ? I mean that if someone 
creates a VM and immediately destroys it, does it really matter. While writing 
this statement, I though of a way to abuse a system that does not account for 
discrete events like "created a VM" but only account for "this VM has run for 
at least X minutes" :-) One could write a software to launch short lived VM at 
specific times and get computing power for free.

Do discrete event need to be separate ? Back to the example above, someone 
creates a VM : that does not create a record in the ceilometer storage. The VM 
is destroyed a few seconds later : it creates an event, as if it was polled / 
published to measure the uptime of the VM. The c1 meter would show the VM has 
been up for 15 seconds since it was created. If the c1 meter is generated from 
an exist event captured from http://wiki.openstack.org/SystemUsageData that 
occurs every 5 minutes, the uptime will be 5 minutes more than the previous one.

While thinking about this use case I also had a simple idea to optimize the 
storage of events and added it to the wiki
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=77&rev1=76

Cheers

-- 
Loïc Dachary Chief Re

Re: [Openstack] [Metering] Meeting agenda for today 16:00 UTC (May 10th, 2012)

[Loic Dachary]

On 05/10/2012 02:14 PM, Loic Dachary wrote:
> Hi,
>
> The metering project team holds a meeting in #openstack-meeting, Thursdays at 
> 1600 UTC 
> <http://www.timeanddate.com/worldclock/fixedtime.html?hour=16&min=0&sec=0>. 
> Everyone is welcome.
> I propose an agenda based on the discussions we had on this list.
>
> http://wiki.openstack.org/Meetings/MeteringAgenda
> Topic: external API definition
>
>  * API defaults and API extensions
>  * API extension
>* extension= loads the  python module
>*  method query is called with the
>  * QUERY_STRING
>  * a handler to the storage
>  * a pointer to the configuration
>  * API calls common arguments
>* Datetime range : start and end
>  * Transparent cache for aggregation
>  * API defaults http://wiki.openstack.org/EfficientMetering#API
>* GET list components
>* GET list components meters (argument : name of the component)
>* GET list accounts
>* GET list of meter_type
>* GET list of events per account
>* GET sum of (meter_volume, meter_duration) for meter_type and account_id
>* other ?
>  * open discussion
>
For the record. There were too many issues raised during the meeting to agree 
on the API. Instead, another meeting was scheduled and the agenda calendar 
postponed for a week. 
http://wiki.openstack.org/Meetings/MeteringAgenda?action=diff&rev2=20&rev1=19

==
#openstack-meeting Meeting
==


Meeting started by dachary at 16:00:22 UTC.  The full logs are available
at
http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-10-16.00.log.html
.



Meeting summary
---

* LINK: https://lists.launchpad.net/openstack/msg11523.html  (dachary,
  16:00:22)
* actions from previous meetings  (dachary, 16:00:22)
  * LINK:

http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-03-16.00.html
(dachary, 16:00:22)
  * dachary removed obsolete comment about floating IP
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=70&rev1=69
(dachary, 16:00:22)
  * dachary o6 : note that the resource_id is the container id.
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=71&rev1=70
(dachary, 16:00:23)
  * The discussion about adding the source notion to the schema took
place on the mailing list
https://lists.launchpad.net/openstack/msg11217.html  (nijaba,
16:01:25)
  * The conclusion was to add a source field to the event record, but no
additional record type to list existing sources.  (nijaba, 16:01:25)
  * jd___ add Swift counters, add resource ID info in counter
definition, describe the table
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=57&rev1=54
(jd___, 16:03:08)

* meeting organisation  (dachary, 16:04:37)
  * This is 2/5 meetings to decide the details of the architecture of
the Metering project https://launchpad.net/ceilometer  (dachary,
16:04:37)
  * Today's focus is on the definition of external REST API  (dachary,
16:04:37)
  * There has not been enough discussions on the list to cover all
aspects and the focus of this meeting was modified to cope with it.
(dachary, 16:04:37)
  * The meeting is time boxed and there will not be enough time to
introduce inovative ideas and research for solutions.  (dachary,
16:04:37)
  * The debate will be about the pro and cons of the options already
discussed on the mailing list.  (dachary, 16:04:38)
  * LINK: https://lists.launchpad.net/openstack/msg11368.html  (dachary,
16:04:38)

* API defaults and API extensions  (dachary, 16:05:28)
  * AGREED: Ceilometer shouldn't invent its own API extensions
mechanism... it should use the system in Nova.  (dachary, 16:09:20)
  * LINK: https://github.com/cloudbuilders/openstack-munin  (dachary,
16:10:18)
  * LINK: https://github.com/sileht/openstack-munin  (dachary, 16:10:23)
  * ACTION: dachary add info to the wiki on the topic of poll versus
push  (dachary, 16:12:17)

* API defaults  (dachary, 16:13:08)
  * GET list components  (dachary, 16:13:14)
  * GET list components meters (argument : name of the component)
(dachary, 16:13:14)
  * GET list [user_id|project_id|source]  (dachary, 16:13:14)
  * GET list of meter_type  (dachary, 16:13:14)
  * GET list of events per [user_id|project_id|source] ( allow to
specify user_id or project_id  (dachary, 16:13:14)
  * GET sum of (meter_volume, meter_duration) for meter_type and
[user_id|project_id|source]  (dachary, 16:13:15)
  * other ?  (dachary, 16:13:16)
  * GET list of events per user_id && project_id  (dachary, 16:14:20)
  * LINK: http://wiki.openstack.org/OpenStackRESTAPI  (dachary,
16:15:48)
  * LINK: http://wiki.openstack.org/EfficientMetering#Meters 

Re: [Openstack] [Metering] API Extensibility (was: External API definition)

[Loic Dachary]

> Another item that we need to discuss is extensibility of this API.

Hi,

Here is a proposal, which we could discuss further during the meeting.

GET extension=¶m1=foo¶m2=bar

The API looks up /usr/share/ceilometer/extensions/.py and loads it. The 
 module defines a query function that takes the following arguments:

* QUERY_STRING (i.e. extension=¶m1=foo¶m2=bar )
* a handler to the storage
* a pointer to the configuration (assuming there is a /etc/ceilometer.ini file, 
for instance)

The query function would return the result. For instance { 'in': 20001, 'out': 
489324 } if asked for aggregated network usage.

Multiple extensions directories could be specified and searched, allowing a 
mixture of extensions provided in ceilometer and custom extensions to address 
specific needs or to mature an new extension.

The primary benefit of defining extensions in this way is to avoid complex 
conventions for aggregations or other advanced operations. If the API was to 
impose a syntax or conventions to say "sum this field and this one and display 
the result ordered in this way and grouped by this field and this one", it 
would be redundant with the query language of the underlying data. For 
instance, if using mongodb, it would be difficult to expose all the features 
provided by http://www.mongodb.org/display/DOCS/Aggregation or 
http://www.mongodb.org/display/DOCS/MapReduce

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] Meeting agenda for today 16:00 UTC (May 10th, 2012)

[Loic Dachary]

On 05/10/2012 02:14 PM, Loic Dachary wrote:
> Hi,
>
> The metering project team holds a meeting in #openstack-meeting, Thursdays at 
> 1600 UTC 
> <http://www.timeanddate.com/worldclock/fixedtime.html?hour=16&min=0&sec=0>. 
> Everyone is welcome.
> I propose an agenda based on the discussions we had on this list.
>
> http://wiki.openstack.org/Meetings/MeteringAgenda
> Topic: external API definition
>
>  * API defaults and API extensions
>  * API extension
>* extension= loads the  python module
>*  method query is called with the
>  * QUERY_STRING
>  * a handler to the storage
>  * a pointer to the configuration
>  * API calls common arguments
>* Datetime range : start and end
>  * Transparent cache for aggregation
>  * API defaults http://wiki.openstack.org/EfficientMetering#API
>* GET list components
>* GET list components meters (argument : name of the component)
>* GET list accounts
>* GET list of meter_type
>* GET list of events per account
>* GET sum of (meter_volume, meter_duration) for meter_type and account_id
Based on the discussions on the list I changed the list to:

#info GET list components
#info GET list components meters (argument : name of the component)
#info GET list [user_id|project_id|source]
#info GET list of meter_type
#info GET list of events per [user_id|project_id|source] ( allow to specify 
user_id or project_id
or both )
#info GET sum of (meter_volume, meter_duration) for meter_type and 
[user_id|project_id|source]
#info other ?

>* other ?
>  * open discussion
>
> Cheers
> -- 
> Loïc Dachary Chief Research Officer
> // eNovance labs   http://labs.enovance.com
> // ? l...@enovance.com  ? +33 1 49 70 99 82


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] [Metering] Meeting agenda for today 16:00 UTC (May 10th, 2012)

[Loic Dachary]

Hi,

The metering project team holds a meeting in #openstack-meeting, Thursdays at 
1600 UTC 
. 
Everyone is welcome.
I propose an agenda based on the discussions we had on this list.

http://wiki.openstack.org/Meetings/MeteringAgenda
Topic: external API definition

 * API defaults and API extensions
 * API extension
   * extension= loads the  python module
   *  method query is called with the
 * QUERY_STRING
 * a handler to the storage
 * a pointer to the configuration
 * API calls common arguments
   * Datetime range : start and end
 * Transparent cache for aggregation
 * API defaults http://wiki.openstack.org/EfficientMetering#API
   * GET list components
   * GET list components meters (argument : name of the component)
   * GET list accounts
   * GET list of meter_type
   * GET list of events per account
   * GET sum of (meter_volume, meter_duration) for meter_type and account_id
   * other ?
 * open discussion

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] External API definition

[Loic Dachary]

On 05/10/2012 05:46 AM, Daniel Dyer wrote:
> Is it your assumption that there will be one metering service per 
> "installation" or one per service (i.e swift, nova)? My assumption would be a 
> single metering service, so the API would need to handle some additional use 
> cases:
> -list services supported
> -list metrics for a service type
> -get metric details
>
Hi,

I added the "list services supported" assuming service == OpenStack component 
(nova, swift etc.)

http://wiki.openstack.org/EfficientMetering?action=diff&rev2=66&rev1=65

I added the "list meters for a component"

http://wiki.openstack.org/EfficientMetering?action=diff&rev2=67&rev1=66

I'm not sure what you mean by "metric details", could you expand ? It could be 
a description (human readable ?) of a given meter. Since we're using a fixed 
form storage, I'm not sure what else needs to be returned.

Cheers
> I would also consider separate use cases for accessing raw events vs. 
> aggregated metrics.
>
> Dan Dyer
> dan.d...@hp.com 
>
> On Wed, May 9, 2012 at 10:44 AM, Nick Barcet  > wrote:
>
>
>
> Doug Hellmann  > wrote:
>
> >On Wed, May 9, 2012 at 11:27 AM, Nick Barcet
> >mailto:nick.bar...@canonical.com>>wrote:
> >
> >> On 05/08/2012 08:27 AM, Nick Barcet wrote:
> >> [..]
> >>
> >> Thinking about this, I think we need to expend the API a bit to
> >reflect
> >> the evolutions of the schema that we decided last week.  Here are my
> >> proposals:
> >>
> >> > * Requests allow to
> >> >   GET account_id list
> >>
> >> change to: GET [user_id|project_id|source] list
> >>
> >
> >Does the [value|value] syntax mean "choose one" or "combine"? I assume
> >"choose one" and you are using square brackets because parens are used
> >in some of the other queries.
>
> You assumed correctly :)
>
> >>
> >> >   GET list of counter_type
> >> >   GET list of events per account
> >> > optional start and end for counter_datetime
> >> > optional counter_type
> >>
> >> change to: GET list of events per [user_id|project_id|source]
> >> optional start and end for counter_datetime
> >>optional counter_type
> >>
> >
> >Users may cross projects, so I'm not sure it makes sense to ask for the
> >events generated by a user without restricting it by the project. At
> >the very least we may need to allow them to specify user_id or project_id
> >or both.
>
> Good point. Thanks for catching this.
>
> >>
> >> >   GET sum of (counter_volume, counter_duration) for counter_type
> >and
> >> > account_id
> >> > optional start and end for counter_datetime
> >>
> >>   GET sum of (counter_volume, counter_duration) for counter_type and
> >> [user_id|project_id|source]
> >>  optional start and end for counter_datetime
> >>
> >> Hope this makes sense.
> >>
> >> Another item that we need to discuss is extensibility of this API.
> >>
> >> Nick
>
>
> ___
> Mailing list: https://launchpad.net/~openstack 
> 
> Post to : openstack@lists.launchpad.net 
> 
> Unsubscribe : https://launchpad.net/~openstack 
> 
> More help   : https://help.launchpad.net/ListHelp
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/09/2012 11:11 PM, Doug Hellmann wrote:
>
>
> On Wed, May 9, 2012 at 3:07 PM, Tomasz Paszkowski  > wrote:
>
> On Wed, May 9, 2012 at 8:02 PM, Doug Hellmann
> mailto:doug.hellm...@dreamhost.com>> wrote:
> >
> > Nice!
> >
> > For production code I think we are going to want to separate collection 
> from
> > storage, aren't we? We don't want each compute node to require access 
> to the
> > database server (that's an issue with nova that they are trying to fix
> > during the folsom release, IIRC).
>
> Yes. Part of the code responsible for amqp support is not functional yet 
> :(
>
>
> OK, that's what I thought.
>
> We all seem to be reinventing different parts of the services that we will 
> eventually need, which is good for education but may be wasting a bit of 
> energy. Is it premature to start talking a little more about architecture so 
> we can start splitting up the implementation work and focusing that energy 
> differently? There is a lot of work we can do independently of the remaining 
> decisions outlined in http://wiki.openstack.org/Meetings/MeteringAgenda.
Hi,

It looks like the architecture of metering is indeed always implemented in 
similar ways. I had discussions with a company yesterday about their own 
metering implementation (which will be used in production soon) and it also has 
an architecture matching what has been proposed so far in ceilometer. I added a 
few points to the architecture chapter in the wiki:

http://wiki.openstack.org/EfficientMetering#Architecture

including a note summarizing the conclusions of the discussion regarding need 
for an independent ceilometer agent in addition to the existing meters provided 
by the OpenStack components.

What do you think ?
>  
>
>
>
>
> --
> Tomasz Paszkowski
> SS7, Asterisk, SAN, Datacenter, Cloud Computing
> +48500166299 
>
> ___
> Mailing list: https://launchpad.net/~openstack 
> 
> Post to : openstack@lists.launchpad.net 
> 
> Unsubscribe : https://launchpad.net/~openstack 
> 
> More help   : https://help.launchpad.net/ListHelp
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] Bootstrapping, first counter implementation

[Loic Dachary]

On 05/09/2012 05:38 PM, Doug Hellmann wrote:
> What is the difference between review.stackforge.org 
>  and review.openstack.org 
>  and why aren't we using the latter?
There is no technical difference (to my knowledge ;-). Only 
review.stackforge.org  is for projects that are 
not incubated yet.

Cheers
>
> On Wed, May 9, 2012 at 11:13 AM, Julien Danjou  > wrote:
>
> Hi there,
>
> I've added a first script that's able to connect to the AMQP
> notification queue using Nova RPC module. Later it will be able to treat
> them when we'll know what to do with them.
>
>  https://review.stackforge.org/#/c/26/
>  https://review.stackforge.org/#/c/27/
>  https://review.stackforge.org/#/c/28/
>
> I wish I could use nova.service.Service, but the code is too RPC
> oriented so that it can't grab notification from the notifier, so for
> now the daemon is rather simple.
>
> --
> Julien Danjou
> // eNovance  http://enovance.com
> // ? julien.dan...@enovance.com   ? 
> +33 1 49 70 99 81 
>
> ___
> Mailing list: https://launchpad.net/~openstack 
> 
> Post to : openstack@lists.launchpad.net 
> 
> Unsubscribe : https://launchpad.net/~openstack 
> 
> More help   : https://help.launchpad.net/ListHelp
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/06/2012 10:36 PM, Tomasz Paszkowski wrote:
> Hi,
>
> I'd like to share my thoughts on metering openstack resources usage.
> Except those data available from SystemUsageData and those mentioned
> in blueprints some of the cloud providers charge for I/O on disk
> drives (to prevent users from dd if=/dev/zero nosense and to teach
> them properly implementing cache strategies). 
Hi Tomasz,

I could not agree more and this is the reason why I/O shows in the list of 
meters shown in http://wiki.openstack.org/EfficientMetering (c5) "disk IO in 
megabyte per second has a high impact on the service availability and could be 
billed separately ".


> Those data along with
> network card usage can be gathered from libvirt using domblkstat and
> domifstat. 
Thanks for the hint
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=60&rev1=59
> My idea is to gather data using agent (or modified
> nova-compute)  and send them to messaging queue using following jsoned
> data schema:
>
> {'instance': 'instance-003b',
> 'host':'tytan-1','zone':'r4cz1','counters': {'interface': {'vnet0':
> (80796L, 1212L, 0L, 0L, 53403L, 621L, 0L, 0L)}, 'disk': {'vda': (629L,
> 11699200L, 58L, 219136L, 0L)}}}
>
> interface is a result of: interfaceStats(), disk is a result of: blockStats()
>
>
> Those messages are consumed from queue and stored in mysql tables. I
> assume that instance is a parent resource for each disk (ephemeral or
> volume) and for each network interface.
>
> So for message mentioned earlier we have three resources:
>
> 1) instances resource: instance-003b
> 2) child resource: vnet0 (network)
> 3) child resource: vda 'ephemeral
>
>
> Mysql table for resources is like below:
>
> CREATE TABLE resources (
> id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
> parent  BIGINT UNSIGNED,
> type VARCHAR(255) NOT NULL,
> value VARCHAR(255) NOT NULL,
> zone VARCHAR(255) NOT NULL,
> added TIMESTAMP,
> ) ENGINE=INNODB;
>
> Counters are stored also in Mysql using following table:
>
>
> CREATE TABLE counters (
> resource BIGINT UNSIGNED NOT NULL,
> type VARCHAR(255) NOT NULL,
> value BIGINT UNSIGNED,
> delta BIGINT UNISGNED,
> added TIMESTAMP NOT NULL,
> prev TIMESTAMP NOT NULL,
> ) ENGINE=INNODB;
>
> Where prev is a reference to previous counter value. Process which is
> reading data from queue is puting raw counter value into the table and
> if possible (reference to previous entry present) evaluates delta
> value.
>
>
> By using this model of stroing usage counter's it very easy for
> billing system to evaluate charges. We just run SUM(delta) on each
> counter for given time range.
>
> This model could be very easy adopted to other counters (IP Traffic
> external/internal counters from iptables).
>

It looks like you already have a codebase that could be useful for the metering 
implementation. Would you be willing to share it ?

Cheers
>
>
>
> On Mon, Apr 30, 2012 at 12:15 PM, Loic Dachary  wrote:
>> Hi,
>>
>> To prepare for the next meeting ( thursday 3rd, may 2012 
>> http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and 
>> reorganized the Metering blueprint so that it ( hopefully ) incorporates all 
>> the information temporarily stored in the etherpad ( 
>> http://etherpad.openstack.org/EfficientMetering revision 67 in case it is 
>> vandalized ).
>>
>> We could start a discussion from the content of the following sections:
>>
>> http://wiki.openstack.org/EfficientMetering#Counters
>> http://wiki.openstack.org/EfficientMetering#Storage
>>
>> and come up with a list of the counters that should exist by default and how 
>> they should be stored.
>>
>> This morning we had a discussion with Zhongyue Luo on 
>> irc.freenode.net#openstack-metering about how Dough could use the metering 
>> service. Since it already knows about instance creations, counter c1 that 
>> records how long a given instance was up is of no interest. However, other 
>> counters such as the external bandwidth used would be useful. I advocated 
>> that one of the advantages for Dough to rely on metering to collect counters 
>> is that it does not need to know about each OpenStack component and can rely 
>> on metering to figure out how to extract such counters from nova-compute, 
>> nova-network soon to be quantum, nova-volume soon to be cinder, swift, 
>> glance and free it from the burden of tracking structural changes.
>>
>> Cheers
>>
>

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/04/2012 11:50 AM, Thierry Carrez wrote:
> Robert Collins wrote:
>> On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services)
>>  wrote:
>>> Hi - I think a flexible aggregation scheme is needed; the levels of
>>> aggregation available should be definable in the meter independent of the
>>> sources of usage data themselves. If invoices need to be very granular down
>>> to the lowest possible level, then this drives higher data requirements all
>>> through the processing chain, including the rating engine. Traditional
>>> systems tend to pass less granular (more highly aggregated) data into the
>>> rating engine so that bill runs and invoices can be generated efficiently.
>>> At cloud-scale, this can be problematic. Given some “big data” approaches,
>>> though, this could be handled in a more granular and real-time fashion.
>> Has anyone looked at what statsd does? It has very similar
>> requirements (simple to use, no hard a-priori definition of things to
>> count, a few base types to track), and needs to be horizontally
>> scalable.
> Also Swift has plans to use statsd for instrumentation/monitoring, so
> it's definitely worth a look to see if it could be used here as well.
>
> http://folsomdesignsummit2012.sched.org/event/d9135eabdd775432c74c3f1d32a325d3
> http://etherpad.openstack.org/FolsomSwiftStatsd
>
Thanks :-) Just saved the etherpad as
http://etherpad.openstack.org/ep/pad/view/FolsomSwiftStatsd/9cy8Uxtp2U
in case it is vandalized.

Cheers


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] Meeting agenda for today 16:00 UTC (May 3rd, 2012)

[Loic Dachary]

On 05/03/2012 02:22 PM, Loic Dachary wrote:
> Hi,
>
> The metering project team holds a meeting in #openstack-meeting, Thursdays at 
> 1600 UTC 
> <http://www.timeanddate.com/worldclock/fixedtime.html?hour=16&min=0&sec=0>. 
> Everyone is welcome.
> I propose an agenda based on the discussions we had on this list.
>
> http://wiki.openstack.org/Meetings/MeteringAgenda
> Topic : schema and counter definitions
>
>  * counter definitions
>* Proposed http://wiki.openstack.org/EfficientMetering#Counters
>  * schema definition
>* Proposed http://wiki.openstack.org/EfficientMetering#Storage
>  * discuss storage assumptions
>* the storage will store all events
>* no aggregated value is permanently stored
>  * discuss API assumptions
>* the API provide a sum() function to aggregate values
>* the API may transparently store results of the sum function in a cache
>  * discuss event collection
>* events are collected from a components when possible
>* ceilometer agent is installed on a node when the a component does not 
> provide the value
>* contribute to the component instead of developping a ceilometer agent 
> plugin
>  * engaging discussions with core components
>* nova
>* cinder
>* glance
>* swift
>* quantum
>  *  open discussion
>
For the record, the first two points used all the time but that was the goal of 
the meeting. The other points would have been nice to discuss but can each be 
turned into a mailing list thread ;-)

==
#openstack-meeting Meeting
==


Meeting started by dachary at 16:00:16 UTC.  The full logs are available
at
http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-03-16.00.log.html
.



Meeting summary
---

* actions from previous meetings  (dachary, 16:00:36)
  * creation of the ceilometer project  (dachary, 16:00:36)
  * The repository for the ceilometer project has been created
(dachary, 16:00:36)
  * LINK: https://github.com/stackforge/ceilometer  (dachary, 16:00:36)
  * and the first commit was successfully reviewed and merged today
https://review.stackforge.org/#/c/25/  (dachary, 16:00:37)

* meeting organisation  (dachary, 16:01:03)
  * This is 1/5 meetings to decide the architecture of the Metering
project https://launchpad.net/ceilometer  (dachary, 16:01:03)
  * Today's focus is on the definition of the counters / meters and the
associated schema for the storage  (dachary, 16:01:03)
  * It is the conclusion of the discussions held on the mailing list and
the goal is to make a final choice that will then be implemented.
(dachary, 16:01:03)
  * The meeting is time boxed and there will not be enough time to
introduce inovative ideas and research for solutions.  (dachary,
16:01:03)
  * The debate will be about the pro and cons of the options already
discussed on the mailing list.  (dachary, 16:01:03)
  * LINK: https://lists.launchpad.net/openstack/msg10810.html  (dachary,
16:01:03)

* counter definitions  (dachary, 16:02:10)
  * Proposed http://wiki.openstack.org/EfficientMetering#Counters
(dachary, 16:02:10)
  * ACTION: dachary fix the note for net_float still talks about "number
of floating IPs"  (dachary, 16:09:18)
  * ACTION: jd___ include Number of object in Swift, Number of
containers in Swift, Number of GET/HEAD/PUT/POST requests in Swift
in the table  (dachary, 16:10:11)
  * ACTION: dachary add note about the fact that the resource_id for the
object count is the container_id  (dachary, 16:21:44)
  * LINK: http://wiki.openstack.org/EfficientMetering#Counters is agreed
on, provided the actions listed above are carried out.  (dachary,
16:25:35)
  * ACTION: jd___ document the resource_id for each counter  (dachary,
16:30:33)
  * ACTION: jd___  describes the general table schema and then something
that says for each counter exactly what goes in the fields of that
table and show how secondary field counters are recorded in the in
the schema too  (dachary, 16:33:27)
  * AGREED: s/counter/meter/  (dachary, 16:37:11)
  * LINK: http://wiki.openstack.org/EfficientMetering#Counters is agreed
on, provided the actions listed above are carried out. ?  (dachary,
16:37:38)
  * AGREED: http://wiki.openstack.org/EfficientMetering#Counters is
agreed on, provided the actions listed above are carried out. ?
(dachary, 16:38:52)

* schema definition  (dachary, 16:39:05)
  * Proposed http://wiki.openstack.org/EfficientMetering#Storage
(dachary, 16:39:05)
  * ACTION: jd___ clarify / document the fact that the secondary field
or the resource_id field carries the IP/VIF for the meters related
to network so that they can be "per IP"  (dachary, 16:40:42)
  * ACTION: nijaba describe the source field 

Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)

[Loic Dachary]

On 05/03/2012 09:58 AM, Robert Collins wrote:
> On Wed, May 2, 2012 at 10:28 AM, Loic Dachary  wrote:
>
>> As I wrote in a previous mail, once we manage to provide an implementation 
>> that proves useful, we will be in a position to approach the core OpenStack 
>> components. Integrating the metering agents as part of the core component, 
>> much in the same way it's currently done in nova. That will reduce the 
>> overall complexity of deploying OpenStack with metering (which must not be 
>> mandatory). However, there is very little chance that all components 
>> developed around OpenStack are convinced and there will always be a need for 
>> a metering that is external to the component. Therefore, even if metering 
>> eventually manages to become a first class concern for the core OpenStack 
>> components, the proposed architecture of the metering project ( per node 
>> agents when necessary and a collector harvesting them into a storage ) will 
>> keep being used for other components.
>>
>> Do you think I'm wrong ? We're at a very early stage and now is the time to 
>> question everything :-)
> I would avoid node agents as a primary interface: they can always be
> written to workaround components that don't provide metering
> themselves -  like the nagios plugins example given by Andrew. node
> agents are more complex than implementing metering in each component
> because they require a handoff, parsing of local logs (or whatever
> relevant store the component uses), and they are another process to
> keep running; they probably devolve to polling, and polling is a good
> way to use a lot of resources up :(.
>
> The key thing I see is having a clear contract for how metering is
> done, so that anyone writing a new component can add metering easily.
> This is easier for the component author than having to write their
> component *and* write a metering node agent for that component.
>
> E.g.
>  - define the signals, extension points, and python API for
> implementing metering in a component.
>  - implement a no-op implementation of that metering API which
> discards all data.
>  - implement a AMQP based implementation which shoves all the data out
> to some queue somewhere.
>  - submit patches to existing components to add metering
>  - and if a component owner rejects metering altogether, you can still
> implement a node agent to gather the data and push it out via the
> metering API - that will always be an option.
>
> Then we will have a situation where:
>  - new components can meter easily
>  - folk with different delivery constraints can change the network
> layer easily (drop in a different implementation of the Python API)
> [e.g. to log directly to a NoSQL store]
I could not agree more. This way we have the best of both worlds : integration 
in components when possible and standalone agents when it can't be done for 
whatever reason.

Cheers


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] [Metering] Meeting agenda for today 16:00 UTC (May 3rd, 2012)

[Loic Dachary]

Hi,

The metering project team holds a meeting in #openstack-meeting, Thursdays at 
1600 UTC 
. 
Everyone is welcome.
I propose an agenda based on the discussions we had on this list.

http://wiki.openstack.org/Meetings/MeteringAgenda
Topic : schema and counter definitions

 * counter definitions
   * Proposed http://wiki.openstack.org/EfficientMetering#Counters
 * schema definition
   * Proposed http://wiki.openstack.org/EfficientMetering#Storage
 * discuss storage assumptions
   * the storage will store all events
   * no aggregated value is permanently stored
 * discuss API assumptions
   * the API provide a sum() function to aggregate values
   * the API may transparently store results of the sum function in a cache
 * discuss event collection
   * events are collected from a components when possible
   * ceilometer agent is installed on a node when the a component does not 
provide the value
   * contribute to the component instead of developping a ceilometer agent 
plugin
 * engaging discussions with core components
   * nova
   * cinder
   * glance
   * swift
   * quantum
 *  open discussion

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)

[Loic Dachary]

On 05/03/2012 05:25 AM, Andrew Clay Shafer wrote:
>
> Integrating the metering agents as part of the core component, much in 
> the same way it's currently done in nova.
>
>
> What specifically is done?
>
> If metering is not integrated in the beginning it will likely never be.
>  
Hi,

I'm refering to http://wiki.openstack.org/SystemUsageData . In another thread 
on this list ( Subject: Understanding SystemUsageData and its evolution ) I 
asked the author Dragon Monsyne about its evolution and how to get per-IP 
meters which does not seem to be possible at the moment.

Do you know who would be most interested in discussing metering among the swift 
or quantum developers ?

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/02/2012 07:39 AM, Mark McLoughlin wrote:
> On Tue, 2012-05-01 at 23:05 +0200, Loic Dachary wrote:
>> On 05/01/2012 06:13 PM, Mark McLoughlin wrote:
>>> Hi Loic,
>>>
>>> On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote:
>>>
>>>> To prepare for the next meeting ( thursday 3rd, may 2012
>>>> http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and
>>>> reorganized the Metering blueprint so that it ( hopefully )
>>>> incorporates all the information temporarily stored in the etherpad
>>>> ( http://etherpad.openstack.org/EfficientMetering revision 67 in case
>>>> it is vandalized ).
>>> I'm a bit late to the discussion, but some brief comments after reading
>>> up on what you guys have done so far:
>>>
>>>   - big +1 on separating billing from metering; there's no need to 
>>> conflate the two problems and doing it this way will allow for a 
>>> bunch of different ideas to be tried around billing
>>>
>>>   - I'd prefer to avoid adding a new node agents, so +1 on building on
>>> the notifications system
>> I would also prefer this option. I have a few concerns though:
>>
>> a) adding too many messages to the existing message queues
>> b) not all core components provide notifications
>> c) convincing all components to agree on a unified approach to metering
>>
>> Instead it might be more practical to implement node agents when
>> necessary to complete a first implementation. That is, taking advice
>> from core component developers and possibly run into problems as
>> opposed to convincing core component developers to adopt an approach
>> to metering that is not yet implemented anywhere.
> I'd start with metering using the notifications which are already there.
> I think that will get us a long way.
I've started a thread to check there is all we need and if not to figure out 
how it can be modified.
>
> My impression is that the notifications system is intended to cover all
> billable usage in at least Nova and Glance.
It's also my understanding. Regarding swift, how would you suggest we approach 
the problem ? I see two possible courses:

a) directly create something similar to nova 
http://wiki.openstack.org/SystemUsageData for swift (i.e. a swift blueprint and 
coding in swift ) so that there is no need to install a metering agent for swift
b) create a swift plugin for a metering agent and when it proves useful, port 
it to swift so that it is integrated and there is no longer a need for a 
metering agent plugin

What do you think ?

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/02/2012 07:19 AM, Mark McLoughlin wrote:
> Hey,
>
> On Tue, 2012-05-01 at 23:05 +0200, Loic Dachary wrote:
>> On 05/01/2012 06:13 PM, Mark McLoughlin wrote:
>>> Hi Loic,
>>>
>>> On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote:
>>>   - I agree that we don't want to go too far with aggregation and lose 
>>> useful data like which instances have been running as opposed to 
>>> just how many instance minutes a given tenant has consumed
>>>
>>> Another aspect of aggregation to think about is aggregation over 
>>> time - e.g. I might like to see my hourly network usage has varied 
>>> over the last week, or how my daily usage has varied over the last 
>>> month, but I probably don't care so much about my hourly usage on a 
>>> specific day 3 months ago
>>>
>>> oVirt's equivalent of a metering service does this kind of 
>>> aggregation as follows:
>>>
>>>   http://www.ovirt.org/wiki/Ovirt_DWH
>>>
>>> * Sample data is collected at the end of every minute and is
>>>   kept for up to 48 hours.
>>> * Hourly level is aggregated every hour for the hour before 
>>>   last and is kept for 2 months.
>>> * Daily level is aggregated every day for the day before last
>>>   and is kept for 5 years. 
>> Where can I read a description of the corresponding database ?
> Here FWIW: http://goo.gl/3Bqct
Thanks : http://wiki.openstack.org/EfficientMetering?action=diff&rev2=49&rev1=48
>
> Cheers,
> Mark.
>


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] [Metering] Understanding SystemUsageData and its evolution

[Loic Dachary]

Hi,

I would like to better understand the rationale for the choices made for 
http://wiki.openstack.org/SystemUsageData values. For instance, why 
*bandwidth*:keys are network labels instead of IPs ? How to account for the 
bandwidth for a given IP ?

If that has already been discussed on this list I would very much appreciate a 
link to the thread.

I also wonder if you plan to modify the messages when quantum is being used. I 
assume IPs, bandwidth (and more) will then be in messages issued by quantum. 
How would that work ?

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] how should it be done ? ( Was: schema and counter definitions)

[Loic Dachary]

On 05/01/2012 09:57 PM, Andrew Clay Shafer wrote:
> I'm glad to see people championing the effort to implement metering. Is there 
> someway to refocus the enthusiasm for solving the metering problem into 
> engineering a general solution in OpenStack?
>
> I'm just going to apologize in advance, but I don't think this project is 
> headed in the right direction.
>
> I believe metering should be a first class concern of OpenStack and the way 
> this project is starting is almost exactly backwards from what I think a 
> solution to metering should look like.
>
> The last thing I want to see right now is a blessed OpenStack metering 
> project adding more agents, coupled to a particular db and making policy 
> decisions about what is quantifiable.
>
> I think there are really three problems that need to be solved to do 
> metering, what data to get, getting the data and doing things with the data.
>
> From my perspective, a lot if not all of the data events should be coming out 
> of the services themselves. There is already a service that should know when 
> an instance gets started by what tenant. A cross cutting system for 
> publishing those events and a service definition for collecting them seems 
> like a reasonable place to start. To me that should look awful lot like a 
> message queue or centralized logging. Once the first two problems are solved 
> well, if you are so inclined to collect the data into a relational model, the 
> schema will be obvious.
>
> If the first two problems are solved well, then I could be persuaded that a 
> service that provides some of the aggregation functionality is a great idea 
> and a reference implementation on a relational database isn't the worst thing 
> in the world. 
>
> Without a general solution for the first two problems, I believe the primary 
> focus on a schema and db is premature and sub-optimal. I also believe the 
> current approach likely results in a project that is generally unusable.
>
> Does anyone else share my perspective?
It would be better if all OpenStack core components agreed on unified 
interfaces / messages for metering that would be easy to harvest without 
installing agents on nodes. This is also true for many services outside of the 
OpenStack eco-system. However, much in the same way munin and nagios plugins 
are developped outside of the project for which they provide graphing and 
monitoring (for instance we recently published swift munin plugins in the 
repository where ops usually look for them : 
https://github.com/munin-monitoring/contrib/tree/master/plugins/swift and there 
is no glance plugin or up-to-date nova plugins yet ), metering agents will be 
developed separately, most of the time.

As I wrote in a previous mail, once we manage to provide an implementation that 
proves useful, we will be in a position to approach the core OpenStack 
components. Integrating the metering agents as part of the core component, much 
in the same way it's currently done in nova. That will reduce the overall 
complexity of deploying OpenStack with metering (which must not be mandatory). 
However, there is very little chance that all components developed around 
OpenStack are convinced and there will always be a need for a metering that is 
external to the component. Therefore, even if metering eventually manages to 
become a first class concern for the core OpenStack components, the proposed 
architecture of the metering project ( per node agents when necessary and a 
collector harvesting them into a storage ) will keep being used for other 
components.

Do you think I'm wrong ? We're at a very early stage and now is the time to 
question everything :-)



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/01/2012 06:13 PM, Mark McLoughlin wrote:
> Hi Loic,
>
> On Mon, 2012-04-30 at 12:15 +0200, Loic Dachary wrote:
>
>> To prepare for the next meeting ( thursday 3rd, may 2012
>> http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and
>> reorganized the Metering blueprint so that it ( hopefully )
>> incorporates all the information temporarily stored in the etherpad
>> ( http://etherpad.openstack.org/EfficientMetering revision 67 in case
>> it is vandalized ).
> I'm a bit late to the discussion, but some brief comments after reading
> up on what you guys have done so far:
>
>   - big +1 on separating billing from metering; there's no need to 
> conflate the two problems and doing it this way will allow for a 
> bunch of different ideas to be tried around billing
>
>   - I'd prefer to avoid adding a new node agents, so +1 on building on
> the notifications system
I would also prefer this option. I have a few concerns though:

a) adding too many messages to the existing message queues
b) not all core components provide notifications
c) convincing all components to agree on a unified approach to metering

Instead it might be more practical to implement node agents when necessary to 
complete a first implementation. That is, taking advice from core component 
developers and possibly run into problems as opposed to convincing core 
component developers to adopt an approach to metering that is not yet 
implemented anywhere.
>
>   - I agree that we don't want to go too far with aggregation and lose 
> useful data like which instances have been running as opposed to 
> just how many instance minutes a given tenant has consumed
>
> Another aspect of aggregation to think about is aggregation over 
> time - e.g. I might like to see my hourly network usage has varied 
> over the last week, or how my daily usage has varied over the last 
> month, but I probably don't care so much about my hourly usage on a 
> specific day 3 months ago
>
> oVirt's equivalent of a metering service does this kind of 
> aggregation as follows:
>
>   http://www.ovirt.org/wiki/Ovirt_DWH
>
> * Sample data is collected at the end of every minute and is
>   kept for up to 48 hours.
> * Hourly level is aggregated every hour for the hour before 
>   last and is kept for 2 months.
> * Daily level is aggregated every day for the day before last
>   and is kept for 5 years. 
Where can I read a description of the corresponding database ?
>
>   - Lastly, bikeshed mode - since we're calling this "metering" and not 
> "counting", how about just using the term "meters" rather than 
> "counters"?
>
+1 ;-)

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/01/2012 05:49 PM, Doug Hellmann wrote:
>
>
> On Tue, May 1, 2012 at 10:38 AM, Nick Barcet  <mailto:nick.bar...@canonical.com>> wrote:
>
> On 05/01/2012 02:23 AM, Loic Dachary wrote:
> > On 04/30/2012 11:39 PM, Doug Hellmann wrote:
> >>
> >>
>     >> On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary  <mailto:l...@enovance.com>
> >> <mailto:l...@enovance.com <mailto:l...@enovance.com>>> wrote:
> >>
> >> On 04/30/2012 08:03 PM, Doug Hellmann wrote:
> >>>
> >>>
> >>> On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary  <mailto:l...@enovance.com>
> >>> <mailto:l...@enovance.com <mailto:l...@enovance.com>>> wrote:
> >>>
> >>> On 04/30/2012 03:49 PM, Doug Hellmann wrote:
> >>>>
> >>>>
> >>>> On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary
> >>>> mailto:l...@enovance.com> 
> <mailto:l...@enovance.com <mailto:l...@enovance.com>>> wrote:
> >>>>
> >>>> On 04/30/2012 12:15 PM, Loic Dachary wrote:
> >>>> > We could start a discussion from the content of the
> >>>> following sections:
> >>>> >
> >>>> > http://wiki.openstack.org/EfficientMetering#Counters
> >>>> I think the rationale of the counter aggregation needs
> >>>> to be explained. My understanding is that the metering
> >>>> system will be able to deliver the following
> >>>> information: 10 floating IPv4 addresses were allocated
> >>>> to the tenant during three months and were leased from
> >>>> provider NNN. From this, the billing system could add a
> >>>> line to the invoice : 10 IPv4, $N each = $10xN because
> >>>> it has been configured to invoice each IPv4 leased from
> >>>> provider NNN for $N.
> >>>>
> >>>> It is not the purpose of the metering system to display
> >>>> each IPv4 used, therefore it only exposes the aggregated
> >>>> information. The counters define how the information
> >>>> should be aggregated. If the idea was to expose each
> >>>> resource usage individually, defining counters would be
> >>>> meaningless as they would duplicate the activity log
> >>>> from each OpenStack component.
> >>>>
> >>>> What do you think ?
> >>>>
> >>>>
> >>>> At DreamHost we are going to want to show each individual
> >>>> resource (the IPv4 address, the instance, etc.) along with
> >>>> the charge information. Having the metering system aggregate
> >>>> that data will make it difficult/impossible to present the
> >>>> bill summary and detail views that we want. It would be much
> >>>> more useful for us if it tracked the usage details for each
> >>>> resource, and let us aggregate the data ourselves.
> >>>>
> >>>> If other vendors want to show the data differently, perhaps
> >>>> we should provide separate APIs for retrieving the detailed
> >>>> and aggregate data.
> >>>>
> >>>> Doug
> >>>>
> >>> Hi,
> >>>
> >>> For the record, here is the unfinished conversation we had on 
> IRC
> >>>
> >>> (04:29:06 PM) dhellmann: dachary, did you see my reply about
> >>> counter definitions on the list today?
> >>> (04:39:05 PM) dachary: It means some counters must not be
> >>> aggregated. Only the amount associated with it is but there
> >>> is one counter per IP.
> >>> (04:55:01 PM) dachary: dhellmann: what about this :the id of
> >>> the ressource controls the agregation of all counters : if it
> >>> 

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 05/01/2012 04:38 PM, Nick Barcet wrote:
> On 05/01/2012 02:23 AM, Loic Dachary wrote:
>> On 04/30/2012 11:39 PM, Doug Hellmann wrote:
>>>
>>> On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary >> <mailto:l...@enovance.com>> wrote:
>>>
>>> On 04/30/2012 08:03 PM, Doug Hellmann wrote:
>>>>
>>>> On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary >>> <mailto:l...@enovance.com>> wrote:
>>>>
>>>> On 04/30/2012 03:49 PM, Doug Hellmann wrote:
>>>>>
>>>>>     On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary
>>>>> mailto:l...@enovance.com>> wrote:
>>>>>
>>>>> On 04/30/2012 12:15 PM, Loic Dachary wrote:
>>>>> > We could start a discussion from the content of the
>>>>> following sections:
>>>>> >
>>>>> > http://wiki.openstack.org/EfficientMetering#Counters
>>>>> I think the rationale of the counter aggregation needs
>>>>> to be explained. My understanding is that the metering
>>>>> system will be able to deliver the following
>>>>> information: 10 floating IPv4 addresses were allocated
>>>>> to the tenant during three months and were leased from
>>>>> provider NNN. From this, the billing system could add a
>>>>> line to the invoice : 10 IPv4, $N each = $10xN because
>>>>> it has been configured to invoice each IPv4 leased from
>>>>> provider NNN for $N.
>>>>>
>>>>> It is not the purpose of the metering system to display
>>>>> each IPv4 used, therefore it only exposes the aggregated
>>>>> information. The counters define how the information
>>>>> should be aggregated. If the idea was to expose each
>>>>> resource usage individually, defining counters would be
>>>>> meaningless as they would duplicate the activity log
>>>>> from each OpenStack component.
>>>>>
>>>>> What do you think ?
>>>>>
>>>>>
>>>>> At DreamHost we are going to want to show each individual
>>>>> resource (the IPv4 address, the instance, etc.) along with
>>>>> the charge information. Having the metering system aggregate
>>>>> that data will make it difficult/impossible to present the
>>>>> bill summary and detail views that we want. It would be much
>>>>> more useful for us if it tracked the usage details for each
>>>>> resource, and let us aggregate the data ourselves.
>>>>>
>>>>> If other vendors want to show the data differently, perhaps
>>>>> we should provide separate APIs for retrieving the detailed
>>>>> and aggregate data.
>>>>>
>>>>> Doug
>>>>>
>>>> Hi,
>>>>
>>>> For the record, here is the unfinished conversation we had on IRC
>>>>
>>>> (04:29:06 PM) dhellmann: dachary, did you see my reply about
>>>> counter definitions on the list today?
>>>> (04:39:05 PM) dachary: It means some counters must not be
>>>> aggregated. Only the amount associated with it is but there
>>>> is one counter per IP.
>>>> (04:55:01 PM) dachary: dhellmann: what about this :the id of
>>>> the ressource controls the agregation of all counters : if it
>>>> is missing, all resources of the same kind and their measures
>>>> are aggregated. Otherwise only the measures are agreggated.
>>>> 
>>>> http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39
>>>> 
>>>> <http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39>
>>>> (04:55:58 PM) dachary: it makes me a little unconfortable to
>>>> define such an "ad-hoc" grouping
>>>> (04:56:53 PM) dachary: i.e. you actuall control the
>>>> aggregation by chosing which value to put in the id column
>>>> (04:58:43 PM

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 04/30/2012 11:39 PM, Doug Hellmann wrote:
>
>
> On Mon, Apr 30, 2012 at 3:43 PM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
> On 04/30/2012 08:03 PM, Doug Hellmann wrote:
>>
>>
>>     On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary > <mailto:l...@enovance.com>> wrote:
>>
>> On 04/30/2012 03:49 PM, Doug Hellmann wrote:
>>>
>>>
>>> On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary >> <mailto:l...@enovance.com>> wrote:
>>>
>>> On 04/30/2012 12:15 PM, Loic Dachary wrote:
>>> > We could start a discussion from the content of the following 
>>> sections:
>>> >
>>> > http://wiki.openstack.org/EfficientMetering#Counters
>>> I think the rationale of the counter aggregation needs to be 
>>> explained. My understanding is that the metering system will be able to 
>>> deliver the following information: 10 floating IPv4 addresses were 
>>> allocated to the tenant during three months and were leased from provider 
>>> NNN. From this, the billing system could add a line to the invoice : 10 
>>> IPv4, $N each = $10xN because it has been configured to invoice each IPv4 
>>> leased from provider NNN for $N.
>>>
>>> It is not the purpose of the metering system to display each 
>>> IPv4 used, therefore it only exposes the aggregated information. The 
>>> counters define how the information should be aggregated. If the idea was 
>>> to expose each resource usage individually, defining counters would be 
>>> meaningless as they would duplicate the activity log from each OpenStack 
>>> component.
>>>
>>> What do you think ?
>>>
>>>
>>> At DreamHost we are going to want to show each individual resource 
>>> (the IPv4 address, the instance, etc.) along with the charge information. 
>>> Having the metering system aggregate that data will make it 
>>> difficult/impossible to present the bill summary and detail views that we 
>>> want. It would be much more useful for us if it tracked the usage details 
>>> for each resource, and let us aggregate the data ourselves.
>>>
>>> If other vendors want to show the data differently, perhaps we 
>>> should provide separate APIs for retrieving the detailed and aggregate data.
>>>
>>> Doug
>>>
>> Hi,
>>
>> For the record, here is the unfinished conversation we had on IRC
>>
>> (04:29:06 PM) dhellmann: dachary, did you see my reply about counter 
>> definitions on the list today?
>> (04:39:05 PM) dachary: It means some counters must not be 
>> aggregated. Only the amount associated with it is but there is one counter 
>> per IP.
>> (04:55:01 PM) dachary: dhellmann: what about this :the id of the 
>> ressource controls the agregation of all counters : if it is missing, all 
>> resources of the same kind and their measures are aggregated. Otherwise only 
>> the measures are agreggated. 
>> http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39 
>> <http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39>
>> (04:55:58 PM) dachary: it makes me a little unconfortable to define 
>> such an "ad-hoc" grouping
>> (04:56:53 PM) dachary: i.e. you actuall control the aggregation by 
>> chosing which value to put in the id column
>> (04:58:43 PM) dachary: s/actuall/actually/
>> (05:05:38 PM) ***dachary reading 
>> http://www.ogf.org/documents/GFD.98.pdf
>> (05:05:54 PM) dachary: I feel like we're trying to resolve a non 
>> problem here
>> (05:08:42 PM) dachary: values need to be aggregated. The raw input 
>> is a full description of the resource and a value ( gauge ). The question is 
>> how to control the aggregation in a reasonably flexible way.
>> (05:11:34 PM) dachary: The definition of a counter could probably be 
>> described as : the id of a resource and code to fill each column associated 
>> with it.
>>
>> I tried to append the following, but the wiki kept failing.
>>
>> Propose that the counters are defined by a function instead of being 
>> fixed. That helps addressing the issue of aggregating the bandwidth 
>> associated to a given IP into a single counter.
>>
>> Alternate idea :
>>  * a counter is defin

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 04/30/2012 08:03 PM, Doug Hellmann wrote:
>
>
> On Mon, Apr 30, 2012 at 11:43 AM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
> On 04/30/2012 03:49 PM, Doug Hellmann wrote:
>>
>>
>> On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary > <mailto:l...@enovance.com>> wrote:
>>
>> On 04/30/2012 12:15 PM, Loic Dachary wrote:
>> > We could start a discussion from the content of the following 
>> sections:
>> >
>> > http://wiki.openstack.org/EfficientMetering#Counters
>> I think the rationale of the counter aggregation needs to be 
>> explained. My understanding is that the metering system will be able to 
>> deliver the following information: 10 floating IPv4 addresses were allocated 
>> to the tenant during three months and were leased from provider NNN. From 
>> this, the billing system could add a line to the invoice : 10 IPv4, $N each 
>> = $10xN because it has been configured to invoice each IPv4 leased from 
>> provider NNN for $N.
>>
>> It is not the purpose of the metering system to display each IPv4 
>> used, therefore it only exposes the aggregated information. The counters 
>> define how the information should be aggregated. If the idea was to expose 
>> each resource usage individually, defining counters would be meaningless as 
>> they would duplicate the activity log from each OpenStack component.
>>
>> What do you think ?
>>
>>
>> At DreamHost we are going to want to show each individual resource (the 
>> IPv4 address, the instance, etc.) along with the charge information. Having 
>> the metering system aggregate that data will make it difficult/impossible to 
>> present the bill summary and detail views that we want. It would be much 
>> more useful for us if it tracked the usage details for each resource, and 
>> let us aggregate the data ourselves.
>>
>> If other vendors want to show the data differently, perhaps we should 
>> provide separate APIs for retrieving the detailed and aggregate data.
>>
>> Doug
>>
> Hi,
>
> For the record, here is the unfinished conversation we had on IRC
>
> (04:29:06 PM) dhellmann: dachary, did you see my reply about counter 
> definitions on the list today?
> (04:39:05 PM) dachary: It means some counters must not be aggregated. 
> Only the amount associated with it is but there is one counter per IP.
> (04:55:01 PM) dachary: dhellmann: what about this :the id of the 
> ressource controls the agregation of all counters : if it is missing, all 
> resources of the same kind and their measures are aggregated. Otherwise only 
> the measures are agreggated. 
> http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39 
> <http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39>
> (04:55:58 PM) dachary: it makes me a little unconfortable to define such 
> an "ad-hoc" grouping
> (04:56:53 PM) dachary: i.e. you actuall control the aggregation by 
> chosing which value to put in the id column
> (04:58:43 PM) dachary: s/actuall/actually/
> (05:05:38 PM) ***dachary reading http://www.ogf.org/documents/GFD.98.pdf
> (05:05:54 PM) dachary: I feel like we're trying to resolve a non problem 
> here
> (05:08:42 PM) dachary: values need to be aggregated. The raw input is a 
> full description of the resource and a value ( gauge ). The question is how 
> to control the aggregation in a reasonably flexible way.
> (05:11:34 PM) dachary: The definition of a counter could probably be 
> described as : the id of a resource and code to fill each column associated 
> with it.
>
> I tried to append the following, but the wiki kept failing.
>
> Propose that the counters are defined by a function instead of being 
> fixed. That helps addressing the issue of aggregating the bandwidth 
> associated to a given IP into a single counter.
>
> Alternate idea :
>  * a counter is defined by
>   * a name ( o1, n2, etc. ) that uniquely identifies the nature of the 
> measure ( outbound internet transit, amount of RAM, etc. )
>   * the component in which it can be found ( nova, swift etc.)
>  * and by columns, each one is set with the result of 
> aggregate(find(record),record) where
>   * find() looks for the existing column as found by selecting with the 
> unique key ( maybe the name and the resource id )
>   * record is a detailed description of the metering event to be 
> aggregated ( 
> http://wiki.openstack.org/SystemUsageData#compute.instance.exists: )
>  

Re: [Openstack] [Metering] Metering repository in stackforge

[Loic Dachary]

On 04/30/2012 03:56 PM, Andrew Hutchings wrote:
> Hey Loic,
>
> On 30/04/12 13:46, Loic Dachary wrote:
>> On 04/28/2012 06:31 AM, Monty Taylor wrote:
>>> I'm including Andrew
>>> here, as he's been doing most of the StackForge work. I'll also file a
>>> bug on openstack-ci to make sure we don't lose this.
>> I assume the idea is to file a bug that looks like
>>
>> https://bugs.launchpad.net/openstack-ci/+bug/988073
>>
>> I was not able to find one for ceilometer, did you create it already or 
>> should I wait for it ?
> I don't see a bug either, feel free to open one and I'll get on it ASAP.
>
> Kind Regards
Here it is

https://bugs.launchpad.net/openstack-ci/+bug/992030

thank you !


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 04/30/2012 03:49 PM, Doug Hellmann wrote:
>
>
> On Mon, Apr 30, 2012 at 6:46 AM, Loic Dachary  <mailto:l...@enovance.com>> wrote:
>
> On 04/30/2012 12:15 PM, Loic Dachary wrote:
> > We could start a discussion from the content of the following sections:
> >
> > http://wiki.openstack.org/EfficientMetering#Counters
> I think the rationale of the counter aggregation needs to be explained. 
> My understanding is that the metering system will be able to deliver the 
> following information: 10 floating IPv4 addresses were allocated to the 
> tenant during three months and were leased from provider NNN. From this, the 
> billing system could add a line to the invoice : 10 IPv4, $N each = $10xN 
> because it has been configured to invoice each IPv4 leased from provider NNN 
> for $N.
>
> It is not the purpose of the metering system to display each IPv4 used, 
> therefore it only exposes the aggregated information. The counters define how 
> the information should be aggregated. If the idea was to expose each resource 
> usage individually, defining counters would be meaningless as they would 
> duplicate the activity log from each OpenStack component.
>
> What do you think ?
>
>
> At DreamHost we are going to want to show each individual resource (the IPv4 
> address, the instance, etc.) along with the charge information. Having the 
> metering system aggregate that data will make it difficult/impossible to 
> present the bill summary and detail views that we want. It would be much more 
> useful for us if it tracked the usage details for each resource, and let us 
> aggregate the data ourselves.
>
> If other vendors want to show the data differently, perhaps we should provide 
> separate APIs for retrieving the detailed and aggregate data.
>
> Doug
>
Hi,

For the record, here is the unfinished conversation we had on IRC

(04:29:06 PM) dhellmann: dachary, did you see my reply about counter 
definitions on the list today?
(04:39:05 PM) dachary: It means some counters must not be aggregated. Only the 
amount associated with it is but there is one counter per IP.
(04:55:01 PM) dachary: dhellmann: what about this :the id of the ressource 
controls the agregation of all counters : if it is missing, all resources of 
the same kind and their measures are aggregated. Otherwise only the measures 
are agreggated. 
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39 
<http://wiki.openstack.org/EfficientMetering?action=diff&rev2=40&rev1=39>
(04:55:58 PM) dachary: it makes me a little unconfortable to define such an 
"ad-hoc" grouping
(04:56:53 PM) dachary: i.e. you actuall control the aggregation by chosing 
which value to put in the id column
(04:58:43 PM) dachary: s/actuall/actually/
(05:05:38 PM) ***dachary reading http://www.ogf.org/documents/GFD.98.pdf
(05:05:54 PM) dachary: I feel like we're trying to resolve a non problem here
(05:08:42 PM) dachary: values need to be aggregated. The raw input is a full 
description of the resource and a value ( gauge ). The question is how to 
control the aggregation in a reasonably flexible way.
(05:11:34 PM) dachary: The definition of a counter could probably be described 
as : the id of a resource and code to fill each column associated with it.

I tried to append the following, but the wiki kept failing.

Propose that the counters are defined by a function instead of being fixed. 
That helps addressing the issue of aggregating the bandwidth associated to a 
given IP into a single counter.

Alternate idea :
 * a counter is defined by
  * a name ( o1, n2, etc. ) that uniquely identifies the nature of the measure 
( outbound internet transit, amount of RAM, etc. )
  * the component in which it can be found ( nova, swift etc.)
 * and by columns, each one is set with the result of 
aggregate(find(record),record) where
  * find() looks for the existing column as found by selecting with the unique 
key ( maybe the name and the resource id )
  * record is a detailed description of the metering event to be aggregated ( 
http://wiki.openstack.org/SystemUsageData#compute.instance.exists: )
  * the aggregate() function returns the updated row. By default it just += the 
counter value with the old row returned by find()

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] Metering repository in stackforge

[Loic Dachary]

On 04/28/2012 06:31 AM, Monty Taylor wrote:
> Hey!
>
> On 04/27/2012 06:20 PM, Loic Dachary wrote:
>> Hi,
>>
>> I would like to create a repository ceilometer in 
>> https://github.com/stackforge to host the code for the newborn Metering 
>> project ( https://launchpad.net/ceilometer , first meeting held this 
>> thursday http://wiki.openstack.org/Meetings/MeteringAgenda ).
> We would be more than thrilled to get you set up. 
Great :-)
> I'm including Andrew
> here, as he's been doing most of the StackForge work. I'll also file a
> bug on openstack-ci to make sure we don't lose this.
I assume the idea is to file a bug that looks like

https://bugs.launchpad.net/openstack-ci/+bug/988073

I was not able to find one for ceilometer, did you create it already or should 
I wait for it ?

Thanks for your help !

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Openstack Essex - Guide for Ubuntu 12.04

[Loic Dachary]

On 04/30/2012 10:52 AM, Martin Gerhard Loschwitz wrote:
> Am 30.04.12 09:41, schrieb Emilien Macchi:
>> Hi,
>>
>> I release my first documentation on OpenStack Essex for Ubuntu 12.04.
>>
>> I've been working for three weeks with StackOps for my internship, and my 
>> work is focused ont Quantum (Networking as a service in OpenStack).
>>
>>
>> It was quite difficult to have a working infrastructure because Quantum is 
>> only in "incubation" for Essex release. That's why I publish a documentation 
>> in which anyone can test this fabulous software.
>>
>> You can find this documentation in attachment and here 
>>  with all configuration files & 
>> scripts.
>>
>>
>> *Please let me know if I did some mistakes, and of course I will do by best 
>> tocorrect it.*
>>
>>
>> Best regards
>>
> Hello there,
>
> that's fabolous work, thanks a lot! I'd be interested in integrating parts of
> this into the hastexo OpenStack installation guide if you don't mind -- what
> license is your documentation published under, and would you agree to such a
> step? :)
>
> Best regards
> Martin G. Loschwitz
>
Hi,

stackops chose to release under Creative Commons – Attribution-ShareAlike 3.0 ( 
https://github.com/EmilienM/doc-openstack/blob/master/README ), which allows 
all redistribution as long as you keep the credits and the copyleft license.

Emilien, in order for people to improve your document you should add the 
sources of the document to https://github.com/EmilienM/doc-openstack . 
Otherwise it needs to be copied over and you'll have a hard time incorporating 
pull requests.

Good work :-)

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] schema and counter definitions

[Loic Dachary]

On 04/30/2012 12:15 PM, Loic Dachary wrote:
> We could start a discussion from the content of the following sections:
>
> http://wiki.openstack.org/EfficientMetering#Counters
I think the rationale of the counter aggregation needs to be explained. My 
understanding is that the metering system will be able to deliver the following 
information: 10 floating IPv4 addresses were allocated to the tenant during 
three months and were leased from provider NNN. From this, the billing system 
could add a line to the invoice : 10 IPv4, $N each = $10xN because it has been 
configured to invoice each IPv4 leased from provider NNN for $N.

It is not the purpose of the metering system to display each IPv4 used, 
therefore it only exposes the aggregated information. The counters define how 
the information should be aggregated. If the idea was to expose each resource 
usage individually, defining counters would be meaningless as they would 
duplicate the activity log from each OpenStack component.

What do you think ?

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] [Metering] schema and counter definitions

[Loic Dachary]

Hi,

To prepare for the next meeting ( thursday 3rd, may 2012 
http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and 
reorganized the Metering blueprint so that it ( hopefully ) incorporates all 
the information temporarily stored in the etherpad ( 
http://etherpad.openstack.org/EfficientMetering revision 67 in case it is 
vandalized ).

We could start a discussion from the content of the following sections:

http://wiki.openstack.org/EfficientMetering#Counters
http://wiki.openstack.org/EfficientMetering#Storage

and come up with a list of the counters that should exist by default and how 
they should be stored.

This morning we had a discussion with Zhongyue Luo on 
irc.freenode.net#openstack-metering about how Dough could use the metering 
service. Since it already knows about instance creations, counter c1 that 
records how long a given instance was up is of no interest. However, other 
counters such as the external bandwidth used would be useful. I advocated that 
one of the advantages for Dough to rely on metering to collect counters is that 
it does not need to know about each OpenStack component and can rely on 
metering to figure out how to extract such counters from nova-compute, 
nova-network soon to be quantum, nova-volume soon to be cinder, swift, glance 
and free it from the burden of tracking structural changes.

Cheers

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] [Metering] Metering repository in stackforge

[Loic Dachary]

Hi,

I would like to create a repository ceilometer in https://github.com/stackforge 
to host the code for the newborn Metering project ( 
https://launchpad.net/ceilometer , first meeting held this thursday 
http://wiki.openstack.org/Meetings/MeteringAgenda ).

I'm not sure how to proceed, could you please guide me ? My github account name 
is dachary

Thanks in advance

-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ l...@enovance.com  ☎ +33 1 49 70 99 82


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Metering] First meeting for the Metering project

[Loic Dachary]

On 04/27/2012 05:37 PM, Francis J. Lacoste wrote:
> On 12-04-27 10:50 AM, Nick Barcet wrote:
>> The meeting occurred as planned, and overran...  Nevertheless a lot of
>> good decisions were made, the obviously most important one  being
>> the project name: ceilometer.
>>
>> For a more complete meeting summary, go visit:
>> http://wiki.openstack.org/Meetings/MeteringAgenda/meeting-0-summary
>>
>> Next week we will have for objective to find an agreement on "schema and
>> counter definitions". If anyone does not do it before me  I'll fire an
>> introduction email on the subject soon (sometime this we), as we agreed
>> to start the discussion via mailing list and use the irc meeting to
>> finalize the choices.
> The Launchpad project was created:
>
> https://launchpad.net/ceilometer
Thank you :-) I'm not familiar with the launchpad ways. Should I wait for the 
git repository to be created ? Or can I help with that ?

Chers


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Monitoring / Billing Architecture proposed

[Loic Dachary]

On 04/24/2012 04:45 PM, Monsyne Dragon wrote:
>
> On Apr 24, 2012, at 9:03 AM, Loic Dachary wrote:
>
>> On 04/24/2012 03:06 PM, Monsyne Dragon wrote:
>>> Yes,  we emit bandwidth (bytes in/out) on a per VIF basis from each 
>>> instance The event has the somewhat generic name of 
>>> 'compute.instance.exists'  and is emitted on an periodic basis, currently 
>>> by a cronjob. 
>>> Currently, we only populate bandwidth data from XenServer, but if the hook 
>>> is implemented for  the kvm, etc drivers, it will be picked up 
>>> automatically for them as well. 
>>>
>>> Note that we could report other metrics similarly. 
>> Hi,
>>
>> Thanks for clarifying this. So you're suggesting that the metering agent 
>> should collect this data from the nova queue instead of extracting it from 
>> the system (interface, disk stats etc.) ? And for other openstack components 
>> ( as Nick Barcet suggests below ) the metering agent will have to find 
>> another way. Or do you have something else in mind ?
>
> If it's something we have access to, we should emit it in those usage events. 
>  As far as the other components, glance is already using the same 
> notification system.  (there was a thread awhile back about putting it into 
> openstack.common)  It would be nice to have all of the components using it.  
>
Hi,

I don't see a section in http://wiki.openstack.org/SystemUsageData about making 
sure all messages related to a billable event are accounted for. I mean, for 
instance, what if the event that says an instance is deleted is lost ? How is 
the billing software supposed to cope with that ? If it checks the status of 
all VM on a regular basis to deal with this, how can it figure out when the 
missed event occured ?

It would be worth adding a short section about this in 
http://wiki.openstack.org/SystemUsageData . Or I can do it if you give me a 
hint.

Cheers
>> Cheers
>>
>> On 04/24/2012 12:17 PM, Nick Barcet wrote:
>>> On 04/23/2012 10:45 PM, Doug Hellmann wrote:
>>>> > 
>>>> > 
>>>> > On Mon, Apr 23, 2012 at 4:14 PM, Brian Schott
>>>> > >>> > <mailto:brian.sch...@nimbisservices.com>> wrote:
>>>> > 
>>>> > Doug,
>>>> > 
>>>> > Do we mirror the table structure of nova, etc. and add
>>>> > created/modified columns? 
>>>> > 
>>>> > 
>>>> > Or do we flatten into an instance event record with everything?  
>>>> > 
>>>> > 
>>>> > I lean towards flattening the data as it is recorded and making a second
>>>> > pass during the bill calculation. You need to record instance
>>>> > modifications separately from the creation, especially if the
>>>> > modification changes the billing rate. So you might have records for:
>>>> > 
>>>> > created instance, with UUID, name, size, timestamp, ownership
>>>> > information, etc.
>>>> > resized instance, with UUID, name, new size, timestamp, ownership
>>>> > information, etc.
>>>> > deleted instance, with UUID, name, size, timestamp, ownership
>>>> > information, etc.
>>>> > 
>>>> > Maybe some of those values don't need to be reported in some cases, but
>>>> > if you record a complete picture of the state of the instance then the
>>>> > code that aggregates the event records to produce billing information
>>>> > can use it to make decisions about how to record the charges.
>>>> > 
>>>> > There is also the case where an instance is still no longer running but
>>>> > nova thinks it is (or the reverse), so some sort of auditing sweep needs
>>>> > to be included (I think that's what Dough called the "farmer" but I
>>>> > don't have my notes in front of me).
>>> When I wrote [1], one of the things that I never assumed was how agents
>>> would collect their information. I imagined that the system should allow
>>> for multiple implementation of agents that would collect the same
>>> counters, assuming that 2 implementations for the same counter should
>>> never be running at once.
>>>
>>> That said, I am not sure an event based collection of what nova is
>>> notifying would satisfy the requirements I have heard from many cloud
>>> providers:
>>> - how do we ensure that event are not forged or lost in the current nova
&g

Re: [Openstack] Monitoring / Billing Architecture proposed

[Loic Dachary]

On 04/24/2012 03:06 PM, Monsyne Dragon wrote:
> Yes,  we emit bandwidth (bytes in/out) on a per VIF basis from each instance 
> The event has the somewhat generic name of 'compute.instance.exists'  and is 
> emitted on an periodic basis, currently by a cronjob. 
> Currently, we only populate bandwidth data from XenServer, but if the hook is 
> implemented for  the kvm, etc drivers, it will be picked up automatically for 
> them as well. 
>
> Note that we could report other metrics similarly. 
Hi,

Thanks for clarifying this. So you're suggesting that the metering agent should 
collect this data from the nova queue instead of extracting it from the system 
(interface, disk stats etc.) ? And for other openstack components ( as Nick 
Barcet suggests below ) the metering agent will have to find another way. Or do 
you have something else in mind ?

Cheers

On 04/24/2012 12:17 PM, Nick Barcet wrote:
> On 04/23/2012 10:45 PM, Doug Hellmann wrote:
>> > 
>> > 
>> > On Mon, Apr 23, 2012 at 4:14 PM, Brian Schott
>> > > > <mailto:brian.sch...@nimbisservices.com>> wrote:
>> > 
>> > Doug,
>> > 
>> > Do we mirror the table structure of nova, etc. and add
>> > created/modified columns? 
>> > 
>> > 
>> > Or do we flatten into an instance event record with everything?  
>> > 
>> > 
>> > I lean towards flattening the data as it is recorded and making a second
>> > pass during the bill calculation. You need to record instance
>> > modifications separately from the creation, especially if the
>> > modification changes the billing rate. So you might have records for:
>> > 
>> > created instance, with UUID, name, size, timestamp, ownership
>> > information, etc.
>> > resized instance, with UUID, name, new size, timestamp, ownership
>> > information, etc.
>> > deleted instance, with UUID, name, size, timestamp, ownership
>> > information, etc.
>> > 
>> > Maybe some of those values don't need to be reported in some cases, but
>> > if you record a complete picture of the state of the instance then the
>> > code that aggregates the event records to produce billing information
>> > can use it to make decisions about how to record the charges.
>> > 
>> > There is also the case where an instance is still no longer running but
>> > nova thinks it is (or the reverse), so some sort of auditing sweep needs
>> > to be included (I think that's what Dough called the "farmer" but I
>> > don't have my notes in front of me).
> When I wrote [1], one of the things that I never assumed was how agents
> would collect their information. I imagined that the system should allow
> for multiple implementation of agents that would collect the same
> counters, assuming that 2 implementations for the same counter should
> never be running at once.
>
> That said, I am not sure an event based collection of what nova is
> notifying would satisfy the requirements I have heard from many cloud
> providers:
> - how do we ensure that event are not forged or lost in the current nova
> system?
> - how can I be sure that an instance has not simply crashed and never
> started?
> - how can I collect information which is not captured by nova events?
>
> Hence the proposal to use a dedicated event queue for billing, allowing
> for agents to collect and eventually validate data from different
> sources, including, but not necessarily limiting, collection from the
> nova events.
>
> Moreover, as soon as you generalize the problem to other components than
> just Nova (swift, glance, quantum, daas, ...) just using the nova event
> queue is not an option anymore.
>
> [1] http://wiki.openstack.org/EfficientMetering
>
> Nick
>
>

>
> On Apr 24, 2012, at 6:20 AM, Sandy Walsh wrote:
>
>> I think we have support for this currently in some fashion, Dragon?
>>
>> -S
>>
>>
>>
>> On 04/24/2012 12:55 AM, Loic Dachary wrote:
>>> Metering needs to account for the "volume of data sent to external network 
>>> destinations " ( i.e. n4 in http://wiki.openstack.org/EfficientMetering ) 
>>> or the disk I/O etc. This kind of resource is billable.
>>>
>>> The information described at http://wiki.openstack.org/SystemUsageData will 
>>> be used by metering but other data sources need to be harvested as well.
> --
>   Monsyne M. Dragon
>   OpenStack/Nova 
>   cell 210-441-0965
>   work x 5014190
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


-- 
Loïc Dachary Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ? l...@enovance.com  ? +33 1 49 70 99 82

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Monitoring / Billing Architecture proposed

[Loic Dachary]

On 04/23/2012 10:09 PM, Sandy Walsh wrote:
> Flavor information is copied to the Instance table on creation so the
> Flavors can change and still be tracked in the Instance. It may just
> need to be sent in the notification payload.
>
> The current events in the system are documented here:
> http://wiki.openstack.org/SystemUsageData
>
Hi,

Metering needs to account for the "volume of data sent to external network 
destinations " ( i.e. n4 in http://wiki.openstack.org/EfficientMetering ) or 
the disk I/O etc. This kind of resource is billable.

The information described at http://wiki.openstack.org/SystemUsageData will be 
used by metering but other data sources need to be harvested as well.

Cheers
> -Sandy
>
>
> On 04/23/2012 02:50 PM, Brian Schott wrote:
>> So, we could build on this. No reason to reinvent, but we might want to
>> expand the number of events.  I'm concerned about things like what
>> happens when flavors change over time.  Maybe the answer is, always
>> append to the flavor/instance-type table.  The code I remember and the
>> admin interface that Ken wrote allowed you to modify flavors.  That
>> would break billing unless you also track flavor modifications.
>>
>> -
>> Brian Schott, CTO
>> Nimbis Services, Inc.
>> brian.sch...@nimbisservices.com 
>> ph: 443-274-6064  fx: 443-274-6060
>>
>>
>>
>> On Apr 23, 2012, at 1:40 PM, Luis Gervaso wrote:
>>
>>> I have been looking at : http://wiki.openstack.org/SystemUsageData
>>>
>>> On Mon, Apr 23, 2012 at 7:35 PM, Brian Schott
>>> >> > wrote:
>>>
>>> Is there a document somewhere on what events the services emit?  
>>>
>>> -
>>> Brian Schott, CTO
>>> Nimbis Services, Inc.
>>> brian.sch...@nimbisservices.com
>>> 
>>> ph: 443-274-6064   fx: 443-274-6060
>>> 
>>>
>>>
>>>
>>> On Apr 23, 2012, at 12:39 PM, Monsyne Dragon wrote:
>>>
 This already exists in trunk.  The Notification system was
 designed specifically to feed billing and monitoring systems. 

 Basically, we don't want Nova/Glance/etc to be in the business of
 trying to determine billing logic, since it is different for
 pretty much everyone,  so we just emit notifications to a queue
 and the interested pull what they want, and aggregate according
 to their own rules. 

 On Apr 22, 2012, at 1:50 PM, Luis Gervaso wrote:

> Hi,
>
> I want to share the architecture i am developing in order to
> perform the monitorig / billing OpenStack support:
>
> 1. AMQP Client which listen to RabbitMQ / QPid (this should be
> interchangeable) (Own Stuff or ServiceMix / Camel)
  
> 2. Events should be stored on a NoSQL document oriented database
> (I think mongodb is perfect, since we can query in a super easy
> fashion)
 We have an existing system called Yagi
 (https://github.com/Cerberus98/yagi/) that listens to the
 notification queues and persists events to a Redis database.  It
 then provides feeds as ATOM formatted documents that a billing
 system can pull to aggregate data, It also can support PubSub
 notification of clients thru the pubsubhubub protocol, and push
 events to a long-term archiving store thru the AtomPub protocol. 

 That said, the notification system outputs its events as JSON, so
 it should be easy to pipe into a json document-oriented db if
 that's what you need. (we only use ATOM because we have a
 atom-based archiving/search/aggregation engine (it's open
 source: http://atomhopper.org/ ) our in-house systems already
 plug into. )



> 3a. The monitoring system can pull/push MongoDB
>
> 3b. The billing system can pull to create invoices 
>
> 4. A mediation EIP should be necessary to integrate a
> billing/monitoring product. (ServiceMix / Camel)
>
> This is to receive your feedback. So please, critics are welcome!
>
> Cheers!
>
> -- 
> ---
> Luis Alberto Gervaso Martin
> Woorea Solutions, S.L
> CEO & CTO
> mobile: (+34) 627983344 
> luis@ woorea.es 
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> 
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
 --
 Monsyne M. Dragon
 OpenStack/Nova 
>>>

Re: [Openstack] Providing packages for stable releases of OpenStack

[Loic Dachary]

On 12/07/2011 10:32 PM, Stefano Maffulli wrote:
> On Tue, 2011-12-06 at 23:56 +0100, Loic Dachary wrote:
>> I think there is an opportunity to leverage the momentum that is
>> growing in each distribution by creating an openstack team for them to
>> meet. Maybe Stefano Maffulli has an idea about how to go in this
>> direction. The IRC channel was a great idea and it could become more.
>>
> It seems that there is general consensus that OpenStack should be the
> good upstream provider and ease the job of downstream distributions.
> There is an informal team that is coordinating the packaging effort.
> Would it help if this team became more formal? What would it mean for
> the openstack-packaging team to be 'formally' recognized?
Thierry Carrez and Mark McLoughlin essentially asked the same question on IRC a 
few hours ago. You are on the same line ;-) I honestly don't know how to 
express myself more clearly than I did in my previous mails.

The Debian GNU/Linux packaging team is doing fine. Since you are all 
comfortable with the fact that it operates independently, there probably is no 
need to create tighter relationships. If you ever feel that this needs to be 
discussed again, let me know.

Cheers

<>

signature.asc
Description: OpenPGP digital signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Providing packages for stable releases of OpenStack

[Loic Dachary]

On 12/06/2011 09:24 PM, Thierry Carrez wrote:
> Duncan McGreggor wrote:
>> On 06 Dec 2011 - 14:28, Thierry Carrez wrote:
>>> So the general consensus so far on this discussion seems to be:
>>>
>>> (0) The "2011.3 release" PPA bears false expectations and should be
>>> removed now. In the future, we should not provide such PPAs: 0-day
>>> packages for the release should be available from the "last milestone"
>>> PPA anyway.
>>>
>>> (1) OpenStack, as an upstream project, should focus on development
>>> rather than on providing a production-ready distribution.
>>>
>>> (2) We could provide "daily builds" from the stable/diablo branch for a
>>> variety of releases (much like what we do for the master branch), but
>>> those should be clearly marked "not for production use" and be
>>> best-effort only (like our master branch builds).
>>>
>>> (3) This should not prevent a group in the community from working on a
>>> project providing an "openstack on Lucid" production-ready distribution
>>> if they so wishes. This project would just be another distribution of
>>> OpenStack.
>> This doesn't seem like enough to me. OpenStack isn't just a library;
>> it's a fairly substantial collection of software and services, intended
>> to be used as a product. If it can't be used as a product, what's the
>> use?
>>
>> Someone made the suggestion that a new OpenStack group be started, one
>> whose focus is on producing a production-ready, distribution-ready,
>> release of the software. So can we add one more (need some help with
>> wording, here...):
>>
>> (4) OpenStack will accept and foster a new project, one that is not
>> focused on development, but rather the distribution and it's general
>> stability. This distro project will be responsible for advocating on
>> behalf of various operating systems/distros/sponsoring vendors for bugs
>> that affect performance and stability of OpenStack, or prevent an
>> operating system from running OpenStack.
> I don't think you need a project (openstack projects are about upstream
> software): you need a *team* to coordinate distribution efforts and make
> sure openstack projects are packageable etc.
>
> Like zul said, that team actually already informally exists and has an
> IRC channel :)
Packaging is a tremendous amount of work, I'm sure you agree on this otherwise 
this thread would not exist ;-) It is not upstream code development indeed. 
However the people working to package openstack provide valuable input to 
developers and patches that are not only essential to packaging but also to the 
useability of the components.

Creating a packaging team that acknowledge their contribution to the upstream 
project will show that the packagers contributions are an integral part of the 
openstack development, it would motivate new packagers to contribute their 
changes upstream instead of keeping them in a patch directory within the 
package.

I think there is an opportunity to leverage the momentum that is growing in 
each distribution by creating an openstack team for them to meet. Maybe Stefano 
Maffulli has an idea about how to go in this direction. The IRC channel was a 
great idea and it could become more.

Good packages make a huge difference when it comes to deploying a solution made 
of numerous components. A packaging team that spans all openstack components 
would reduce the workload as intended while keeping the subject on the agenda.

Cheers

<>

signature.asc
Description: OpenPGP digital signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Events in 2012 that OpenStack should not miss

[Loic Dachary]

On 12/01/2011 02:45 AM, Stefano Maffulli wrote:
> Hello all,
>
> I'd like to compile a list of events, conferences and such, around the
> world where OpenStack should be represented. 
>
> I have started with the few events I'm already aware of. You'll see that
> most of them are US-centric and I'm interested in other events around
> the world where you think OpenStack should be.
>
> What event do you think the OpenStack community should be going to,
> presenting or sponsoring? Add them to the etherpad:
>
> http://etherpad.openstack.org/OpenStackEvents2012
Hi,

I've added LSM in Geneva.

Cheers

<>

signature.asc
Description: OpenPGP digital signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Providing packages for stable releases of OpenStack

[Loic Dachary]

Hi,
> TL;DR summary:
> The resources needed to do that properly are bigger than you think (and
> doing that will alienate some distro packaging resources), so we'll
> either do a terrible job at it, or lose focus on the development
> release. If there is a need, it should be done as an alternate
> distribution, not inside the OpenStack project.
>

As Julien Danjou wrote, we ( OpenStack Debian GNU/Linux packagers [1]) 
discussed your mail.  We are comfortable with the idea that OpenStack focuses 
on  development and that packaging is left to the packagers involved in  each 
distribution. 

Packaging related patches from Julien Danjou were recently accepted  upstream 
(https://review.openstack.org/#dashboard,1669). This is the  kind of 
cooperation that makes it possible to maintain packages matching  the Debian 
GNU/Linux quality standard. We are confident in our ability  to provide stable 
packages in the future.

OpenStack packaging is not an easy task. It currently fits nicely in Debian 
GNU/Linux. However,  as it evolves towards a system widely used in production, 
it will face  new challenges and the communities working on packaging for each  
distribution will provide valuable input to developers. Creating a  "packaging 
team" with representatives for each distribution and electing  someone to 
represent them in the Policy Board could achieve this. 

Cheers

[1] PKG OpenStack page : https://alioth.debian.org/projects/openstack/and 
corresponding packages: 
http://qa.debian.org/developer.php?login=openstack-de...@lists.alioth.debian.org
http://qa.debian.org/developer.php?packages=nova 
http://qa.debian.org/developer.php?packages=glance



<>

signature.asc
Description: OpenPGP digital signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp