Re: [openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
Hi Julien, You'll find attached to this mail two dump files. gnocchi_resource.txt is an example of resource requests and responses from gnocchi. gnocchi_measure.txt an example of a timeframe request. Metric (vcpus) measure == Data stored by ceilometer with default configuration (devstack). GET http://10.8.8.168:8041/v1/metric/9c26bbea-6041-4067-9384-f6aa9b4ce120/measures ← 200 application/json 1.71kB 295ms Host: 10.8.8.168:8041 Connection: keep-alive X-Auth-Token: 90cf2d940e464ae0aef733d5f124aa43 Accept-Encoding: gzip, deflate Accept: application/json, */* User-Agent: keystoneauth1 No request content Date:Thu, 26 Nov 2015 14:58:59 GMT Server: Apache/2.4.7 (Ubuntu) content-length: 1752 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type:application/json; charset=UTF-8 JSON [ [ "2015-11-23T00:00:00+00:00", 86400.0, 1.0 ], [ "2015-11-24T00:00:00+00:00", 86400.0, 1.0 ], [ "2015-11-25T00:00:00+00:00", 86400.0, 1.0 ], [ "2015-11-26T00:00:00+00:00", 86400.0, 1.0 ], [ "2015-11-25T15:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T16:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T17:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T18:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T19:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T20:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T21:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T22:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-25T23:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T00:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T01:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T02:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T03:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T04:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T05:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T06:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T07:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T08:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T09:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T10:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T11:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T12:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T13:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T14:00:00+00:00", 3600.0, 1.0 ], [ "2015-11-26T03:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T04:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T05:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T06:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T07:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T08:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T09:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T10:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T11:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T12:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T13:00:00+00:00", 300.0, 1.0 ], [ "2015-11-26T14:00:00+00:00", 300.0, 1.0 ] ] Request for a 1h timeframe == GET http://10.8.8.168:8041/v1/metric/9c26bbea-6041-4067-9384-f6aa9b4ce120/measures?start=2015-11-23T10%3A00%3A0.0%2B00%3A00=2015-11-23T11%3A00%3A0.0%2B00%3A00=max ← 200 application/json 2B 64ms Host: 10.8.8.168:8041 Connection: keep-alive X-Auth-Token: 948f7cf696d94c41908e819237112876 Accept-Encoding: gzip, deflate Accept: application/json, */* User-Agent: python-keystoneclient No request content Date:Thu, 26 Nov 2015 14:17:39 GMT Server: Apache/2.4.7 (Ubuntu) content-length: 2 Keep-Alive: timeout=5, max=86 Connection: Keep-Alive Content-Type:application/json; charset=UTF-8 JSON [m:Auto] [] We don't get any data, but there is data with a bigger
Re: [openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
On Thu, Nov 26 2015, Stéphane Albert wrote:
Here a first reply about resource handling.
> Resource data
> =
> Duplicated resource informations from ceilometer, only revision timestamps
> changing.
So I guess that's a bug in Gnocchi, we should not create new revision if
there is 0 change. Would you mind opening a bug?
> Search for active instances during a timeframe
> ==
[…]
> Here the revision is outside of the requested timeframe.
I don't get that comment. You didn't ask for any specific revision, you
asked for a start/end timestamps. So that looks correct to me. You get
the instances that was active between 10:33 and 17:33.
> Same request with a filter on the revision
> ==
[…]
> Empty response because the filter is not matching with the latest resource
> revision.
Yes, that's normal too.
Revision are about resources modification.
You don't need to search based on revision if you want to retrieve
active resources during a timeframe. Just started_at/ended_at.
> Workaround
> ==
> Search for every resource of type 'instances' active during the timeframe. The
> generic request is just to reduce the amount of data transfered as its
> useless.
I don't see what you are working around. You get _exactly_ the same
result that you got with "Search for active instances during a
timeframe" so what's the problem in the first place?
> Request the correct revision from the resource_id we found before.
I don't understand what you call a "correct revision".
If what you want is the list of active resource during a timeframe and
their revision within that timeframe, you can just do:
POST http://10.8.8.168:8041/v1/search/resource/instance?history=true
{
"and": [
{
"or": [
{
"=": {
"ended_at": null
}
},
{
">=": {
"ended_at": "2015-11-23T10:33:26.388112+00:00"
}
}
]
},
{
"or": [
{
"=": {
"ended_at": null
}
},
{
"<=": {
"ended_at": "2015-11-23T17:33:26.388112+00:00"
}
}
]
},
{
"<=": {
"started_at": "2015-11-23T17:33:26.388112+00:00"
}
},
{
"<=": {
"revision_start": "2015-11-23T17:33:26.388112+00:00"
}
}
]
}
--
Julien Danjou
-- Free Software hacker
-- https://julien.danjou.info
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
On Thu, Nov 26 2015, Stéphane Albert wrote: Now about measures. > We don't get any data, but there is data with a bigger granularity. We don't > have a way to know that but request the archive policy and parse it. Oh I see. I think we can consider that as being a bug in the API, that should be easy to fix, feel free to open one. ;) -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
On Thu, Nov 26, 2015 at 05:04:02PM +0100, Julien Danjou wrote:
> On Thu, Nov 26 2015, Stéphane Albert wrote:
>
> Here a first reply about resource handling.
>
> > Resource data
> > =
> > Duplicated resource informations from ceilometer, only revision timestamps
> > changing.
>
> So I guess that's a bug in Gnocchi, we should not create new revision if
> there is 0 change. Would you mind opening a bug?
I'll open a bug with a full history dump, sometime it's due to the host
changing (flapping) from name to hash, this might be due to ceilometer.
But most of the time there is no change.
>
> > Search for active instances during a timeframe
> > ==
>
> […]
>
> > Here the revision is outside of the requested timeframe.
>
> I don't get that comment. You didn't ask for any specific revision, you
> asked for a start/end timestamps. So that looks correct to me. You get
> the instances that was active between 10:33 and 17:33.
I didn't ask for a specific revision because I can't ;)
I get the list of instances active from 10:33 to 17:33, but with the
latest metadata which can be 2 or 3 days from the timeframe requested.
When you are doing rating based on metadata it's problematic.
>
> > Same request with a filter on the revision
> > ==
>
> […]
>
> > Empty response because the filter is not matching with the latest resource
> > revision.
>
> Yes, that's normal too.
>
> Revision are about resources modification.
> You don't need to search based on revision if you want to retrieve
> active resources during a timeframe. Just started_at/ended_at.
See above.
>
> > Workaround
> > ==
> > Search for every resource of type 'instances' active during the timeframe.
> > The
> > generic request is just to reduce the amount of data transfered as its
> > useless.
>
> I don't see what you are working around. You get _exactly_ the same
> result that you got with "Search for active instances during a
> timeframe" so what's the problem in the first place?
I do this to get the id of active instances. So I can then query gnocchi
for the correct revision filtering on the id.
>
> > Request the correct revision from the resource_id we found before.
>
> I don't understand what you call a "correct revision".
The correct revision is the revision matching the timeframe (10:33 to
17:33). So I can get the metadata that were applying to the instance in
this timeframe.
>
> If what you want is the list of active resource during a timeframe and
> their revision within that timeframe, you can just do:
>
> POST http://10.8.8.168:8041/v1/search/resource/instance?history=true
> {
> "and": [
> {
> "or": [
> {
> "=": {
> "ended_at": null
> }
> },
> {
> ">=": {
> "ended_at": "2015-11-23T10:33:26.388112+00:00"
> }
> }
> ]
> },
> {
> "or": [
> {
> "=": {
> "ended_at": null
> }
> },
> {
> "<=": {
> "ended_at": "2015-11-23T17:33:26.388112+00:00"
> }
> }
> ]
> },
> {
> "<=": {
> "started_at": "2015-11-23T17:33:26.388112+00:00"
> }
> },
> {
> "<=": {
> "revision_start": "2015-11-23T17:33:26.388112+00:00"
> }
> }
> ]
> }
If there is more than one revision, then I'll get multiple revisions for
a resource. This imply that you request all the revisions and then
filter them client side. It's not the most efficient way to proceed as
it could be directly done in the DB query.
Thanks for taking time to answer Julien.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
On Thu, Nov 26 2015, Stéphane Albert wrote: > I'll open a bug with a full history dump, sometime it's due to the host > changing (flapping) from name to hash, this might be due to > ceilometer. This would be a Ceilometer (or above) bug indeed. I think I saw something recently. It's possible the host differs between polling and notifications; feel free to also open a bug on the Ceilometer side. >> > Search for active instances during a timeframe >> > == >> >> […] >> >> > Here the revision is outside of the requested timeframe. >> >> I don't get that comment. You didn't ask for any specific revision, you >> asked for a start/end timestamps. So that looks correct to me. You get >> the instances that was active between 10:33 and 17:33. > I didn't ask for a specific revision because I can't ;) Why can't you? You can request revisions based on their timeframes. Or is it because the revision date is based no the upload timestamp rather than the actual date of the revision? This is something we could fix in the API and Ceilometer I imagine, giving the ability to provide a timestamp for the change. > If there is more than one revision, then I'll get multiple revisions for > a resource. This imply that you request all the revisions and then > filter them client side. It's not the most efficient way to proceed as > it could be directly done in the DB query. But but… there might be multiple revision between two timeframe. If you want to rate an instance that was up during 10:33 and 17:33, and that resource has been e.g. resized 3 times at 12:00, 13:00 and 15:00, you'll get 3 revisions from Gnocchi. Isn't that what you want? -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
On Thu, Nov 26, 2015 at 05:31:38PM +0100, Julien Danjou wrote: > On Thu, Nov 26 2015, Stéphane Albert wrote: > > > I'll open a bug with a full history dump, sometime it's due to the host > > changing (flapping) from name to hash, this might be due to > > ceilometer. > > This would be a Ceilometer (or above) bug indeed. I think I saw > something recently. It's possible the host differs between polling and > notifications; feel free to also open a bug on the Ceilometer side. I'll see if there's a bug declared in ceilometer's bug tracker and if not report a new one. > > >> > Search for active instances during a timeframe > >> > == > >> > >> […] > >> > >> > Here the revision is outside of the requested timeframe. > >> > >> I don't get that comment. You didn't ask for any specific revision, you > >> asked for a start/end timestamps. So that looks correct to me. You get > >> the instances that was active between 10:33 and 17:33. > > I didn't ask for a specific revision because I can't ;) > > Why can't you? You can request revisions based on their timeframes. By "I can't" I meant I can't without asking for history. I just want to have the same result as the standard query (the latest revision) but with a maximum revision timestamp. > Or is it because the revision date is based no the upload timestamp > rather than the actual date of the revision? > > This is something we could fix in the API and Ceilometer I imagine, > giving the ability to provide a timestamp for the change. > > > If there is more than one revision, then I'll get multiple revisions for > > a resource. This imply that you request all the revisions and then > > filter them client side. It's not the most efficient way to proceed as > > it could be directly done in the DB query. > > But but… there might be multiple revision between two timeframe. If you > want to rate an instance that was up during 10:33 and 17:33, and that > resource has been e.g. resized 3 times at 12:00, 13:00 and 15:00, you'll > get 3 revisions from Gnocchi. Isn't that what you want? Yes and no... In CloudKitty the collection period is our epsilon. So we won't process multiple values for it. That's the way CloudKitty is working at the moment, it might change in the future. We get the latest metas and most of the time get max value from the aggregated measures (vcpus, mem, disk, etc). It's just that at the moment ceilometer/gnocchi is returning way too much revisions for a timeframe (and they don't have relevant data most of the time). If we query all active instances for a tenant we can quickly get huge load of data. We'll have support for ceilometer events soon, so we'll start doing diff on gnocchi resources to detect changes and apply rate on these. But at the moment it's out of our scope. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
On Wed, Nov 25 2015, Stéphane Albert wrote: Hi Stéphane, > We can't directly query gnocchi resource indexer for a specific resource > revision. Let me explain it, if you want to search for every instances > active for a specific timeframe you'll get a result but the revision > will be the latest, even if it's after the timeframe you requested. If > you add resource revision to the request you'll get an empty request if > it doesn't match the latest version. We can't add history to the request > as we'll have more than one response when we only need latest revision > for the timeframe. One workaround is to first do a request to query > every active instance and then do a query for every instance with > history filtering on UUID and revision and limiting to one result. It's > suboptimal as it requires many requests to get the correct information. You really need to put actual data and request examples in your explications, as it's really hard to follow you. Best way is to describe what you have, what you do, what you get and what you would expect. But I think I see what you mean, and it should be just a few API call option to add. Best thing would be to open a bug with the details I just mentioned. > The other problem we are having is when requesting measures. Let me > first tell you how CloudKitty is collecting data to set the context. > We've got a collection period (default 3600 seconds), every period we > query the backend for a specific timeframe (start = last collection, > stop = last + period). If we're missing data we restart from where we > stopped, which can be pretty far in the past. > We're using gnocchi as ceilometer's storage and by default it's pretty > hard to exploit data stored. If we request data too far in the past that > is using a higher granularity than our collection period then we got an > empty dataset from gnocchi. It's pretty ambiguous as we don't know if > it's because there is no data, or because our timeframe is too short. > That means we need to query for the archive policy and find what should > be the minimum granularity for the period. In some way it's normal to no > get data because gnocchi is unable to provide accurate data for this > period, but at least get information that we are requesting too > precisely. Or send us the data, since we've got the timestamp and > the granularity, we can easily detect that the result is not accurate > and decide what we should do. If you specify only the timeframe and no granularity, it should send to you all it got for this timeframe. Isn't that good enough for you? -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [gnocchi][rating] Issues regarding gnocchi support in CloudKitty
Hi, We've started implementing gnocchi support in CloudKitty. But we're facing some issues. Some problems where client sided and are now fixed, but for some others it might doing wrong requests or not using gnocchi's full potential. Here are some problems we're actually facing: We can't directly query gnocchi resource indexer for a specific resource revision. Let me explain it, if you want to search for every instances active for a specific timeframe you'll get a result but the revision will be the latest, even if it's after the timeframe you requested. If you add resource revision to the request you'll get an empty request if it doesn't match the latest version. We can't add history to the request as we'll have more than one response when we only need latest revision for the timeframe. One workaround is to first do a request to query every active instance and then do a query for every instance with history filtering on UUID and revision and limiting to one result. It's suboptimal as it requires many requests to get the correct information. The other problem we are having is when requesting measures. Let me first tell you how CloudKitty is collecting data to set the context. We've got a collection period (default 3600 seconds), every period we query the backend for a specific timeframe (start = last collection, stop = last + period). If we're missing data we restart from where we stopped, which can be pretty far in the past. We're using gnocchi as ceilometer's storage and by default it's pretty hard to exploit data stored. If we request data too far in the past that is using a higher granularity than our collection period then we got an empty dataset from gnocchi. It's pretty ambiguous as we don't know if it's because there is no data, or because our timeframe is too short. That means we need to query for the archive policy and find what should be the minimum granularity for the period. In some way it's normal to no get data because gnocchi is unable to provide accurate data for this period, but at least get information that we are requesting too precisely. Or send us the data, since we've got the timestamp and the granularity, we can easily detect that the result is not accurate and decide what we should do. Sorry for the wall of text, and thanks to people that took time to read it. BTW, we're available in #cloudkitty channel. You can reach me on most openstack channels too, I'm in UTC+1 timezone. Hoping we can find a solution and ship gnocchi support in the next CloudKitty version, which is pretty much ready. Cheers Stéphane Albert - 05 82 95 65 36 Objectif Libre www.objectif-libre.com Infrastructure et Formations Linux __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
