Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
On Mon, Sep 19 2016, Sam Morrison wrote: > No performance numbers. Do you have a test in mind so I can compare. Is there > a > standard way to test this? Gnocchi client has a "benchmark" tool that you can use to have some basic numbers. It's not perfect because it's in Python and it is often too slow to generate enough traffic for the server (at least with the Carbonara drivers). -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
Hi Julien, > On 16 Sep 2016, at 7:46 PM, Julien Danjou wrote: > >> I could push it up to gerrit but I think something will need to change >> for it to run the influxdb tests? > > You can use pifpaf like we do for the indexer, InfluxDB is supported. > That should make it possible to run the unit tests in the gate right > away. OK I’ve made some changes to support this, will push this to my branch shortly. Just fixing up a coupe more things. > As for the functional tests, you can set up support via devstack and > we wouldhad a job in infra. Will look into this soon. >> It should act more like the carbonara drivers now as opposed to the >> old influx driver. It will do downsampling and retention based on the >> archive policies. > > That's great, and I imagine it'd be faster than doing it on the fly like > previously. > >> Currently it is failing one test [1] and that is to do with retention. >> This is because influxDB does retention based on the current time, e.g. a 1 >> day retention policy will be from the current time. >> The tests assume that the retention period is based on the data stored and >> so it will keep 1 day of data no matter how old that data is. > > lol, yeah the test assume it's a database that does not block you to > insert things as you want. I feel like that being a bad and funny design > decision (Whisper has the same defect). > >> I also had to disable retention policies in influx while running the tests as >> when I backfill data influx is too smart and won’t backfill data that >> wouldn’t >> meet the retention policy. > > I imagine that's because some of our tests are using date in year e.g. > 2014? :) Yeah exactly, I think it is ok with these disabled. We can just trust that influx retention policies work. > Great. Do you have performance numbers, scalability, or things that are > different/better/worse than using Carbonara based drivers? No performance numbers. Do you have a test in mind so I can compare. Is there a standard way to test this? Cheers, Sam > > Cheers, > -- > Julien Danjou > // Free Software hacker > // https://julien.danjou.info __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
On 16/09/16 05:46 AM, Julien Danjou wrote: > On Fri, Sep 16 2016, Sam Morrison wrote: > > >> Currently it is failing one test [1] and that is to do with retention. >> This is because influxDB does retention based on the current time, e.g. a 1 >> day retention policy will be from the current time. >> The tests assume that the retention period is based on the data stored and >> so it will keep 1 day of data no matter how old that data is. > > lol, yeah the test assume it's a database that does not block you to > insert things as you want. I feel like that being a bad and funny design > decision (Whisper has the same defect). > i think i know why they did this. i originally thought we should do this since it'd make scheduling aggregation/compression tasks easier but i like the fact we handled this without limiting us to possible issues of what 'current time' is in distributed environment. i've no idea how to fix this unless you could somehow mock time influx uses. you might want to just skip this test too? cheers, -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
On Fri, Sep 16 2016, Sam Morrison wrote: Hi Sam, > Been working a bit on this and have a patch based on master that is > working at: > > https://github.com/NeCTAR-RC/gnocchi/tree/influxdb-driver Great! > I could push it up to gerrit but I think something will need to change > for it to run the influxdb tests? You can use pifpaf like we do for the indexer, InfluxDB is supported. That should make it possible to run the unit tests in the gate right away. As for the functional tests, you can set up support via devstack and we wouldhad a job in infra. > It should act more like the carbonara drivers now as opposed to the > old influx driver. It will do downsampling and retention based on the > archive policies. That's great, and I imagine it'd be faster than doing it on the fly like previously. > Currently it is failing one test [1] and that is to do with retention. > This is because influxDB does retention based on the current time, e.g. a 1 > day retention policy will be from the current time. > The tests assume that the retention period is based on the data stored and so > it will keep 1 day of data no matter how old that data is. lol, yeah the test assume it's a database that does not block you to insert things as you want. I feel like that being a bad and funny design decision (Whisper has the same defect). > I also had to disable retention policies in influx while running the tests as > when I backfill data influx is too smart and won’t backfill data that wouldn’t > meet the retention policy. I imagine that's because some of our tests are using date in year e.g. 2014? :) > One way to fix all this would be to change all the test times to be > relative from now but then there could be other race conditions etc. I > think. Completely, imagine if the test takes 1 month to run, it would fail anyway. ;-) It's completely hypothetical and unrealaistic of course, but in principle I think it's better if we can keep things the way they are. > I’m still not 100% happy with the code, particularly around how the > continuous queries are created based on the archive policies. > > We are using this code in preprod and so far all is going well. Great. Do you have performance numbers, scalability, or things that are different/better/worse than using Carbonara based drivers? Cheers, -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
Hi Julien, Been working a bit on this and have a patch based on master that is working at: https://github.com/NeCTAR-RC/gnocchi/tree/influxdb-driver I could push it up to gerrit but I think something will need to change for it to run the influxdb tests? It should act more like the carbonara drivers now as opposed to the old influx driver. It will do downsampling and retention based on the archive policies. Currently it is failing one test [1] and that is to do with retention. This is because influxDB does retention based on the current time, e.g. a 1 day retention policy will be from the current time. The tests assume that the retention period is based on the data stored and so it will keep 1 day of data no matter how old that data is. I also had to disable retention policies in influx while running the tests as when I backfill data influx is too smart and won’t backfill data that wouldn’t meet the retention policy. One way to fix all this would be to change all the test times to be relative from now but then there could be other race conditions etc. I think. I’m still not 100% happy with the code, particularly around how the continuous queries are created based on the archive policies. We are using this code in preprod and so far all is going well. Should also note that you will need influxDB v1.0 to use this. To test all you should need is an influx server running locally. Would love some feed back particularly from people who know how influx works as there are a few ways it could be architected. Cheers, Sam [1] gnocchi.tests.test_rest.MetricTest.test_add_measures_back_window > On 6 Sep 2016, at 11:24 AM, Sam Morrison wrote: > > Hi Julien, > >> On 5 Sep 2016, at 5:36 PM, Julien Danjou wrote: >> >> On Mon, Sep 05 2016, Sam Morrison wrote: >> >> Hi Sam, >> >>> The issue I’m having are with the tests. Because the continuous queries are >>> asynchronous and there is no current way in influxdb to force them to run I >>> get >>> tests failing due to >>> them not having run yet. >>> >>> I’m not sure how to get around this issue, apart from the tests failing >>> everything is working quite well. I’m going to start some load testing soon >>> to >>> see what it’s like when pushing in a lot of metrics. >> >> Does it break the REST API, or only some storage tests? > > REST API is fine, in fact it fixes some tests that influx was failing on. > >> If it's just some storage test, you can change the tests so they are >> retrying until the operation are done. Either in the test, or via a >> special flag in the driver – we used to have that in the first version >> of the driver. > > OK good idea, I’ll work on that. > > >>> Wondering if there would be time to talk about this in Barcelona. >> >> Sure. > > Cheers, > Sam > > >> >> -- >> Julien Danjou >> -- Free Software hacker >> -- https://julien.danjou.info __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
> On 6 Sep 2016, at 11:14 PM, Maxime Belanger wrote: > > Hey Sam, > > Are the driver you are implementing stores the index in Gnocchi index or > directly in influx? > In other words, are you fully using influx under gnochi API? Hi Max, The index lives in gnocchi index much like the other drivers, all that is stored in influx db is the samples (time, metric_id, value) Sam > Max > From: Sam Morrison > Sent: September 5, 2016 9:24:21 PM > To: Julien Danjou > Cc: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb > > Hi Julien, > > > On 5 Sep 2016, at 5:36 PM, Julien Danjou wrote: > > > > On Mon, Sep 05 2016, Sam Morrison wrote: > > > > Hi Sam, > > > >> The issue I’m having are with the tests. Because the continuous queries are > >> asynchronous and there is no current way in influxdb to force them to run > >> I get > >> tests failing due to > >> them not having run yet. > >> > >> I’m not sure how to get around this issue, apart from the tests failing > >> everything is working quite well. I’m going to start some load testing > >> soon to > >> see what it’s like when pushing in a lot of metrics. > > > > Does it break the REST API, or only some storage tests? > > REST API is fine, in fact it fixes some tests that influx was failing on. > > > If it's just some storage test, you can change the tests so they are > > retrying until the operation are done. Either in the test, or via a > > special flag in the driver – we used to have that in the first version > > of the driver. > > OK good idea, I’ll work on that. > > > >> Wondering if there would be time to talk about this in Barcelona. > > > > Sure. > > Cheers, > Sam > > > > > > -- > > Julien Danjou > > -- Free Software hacker > > -- https://julien.danjou.info <https://julien.danjou.info/> > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org > <mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org > <mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
Hey Sam, Are the driver you are implementing stores the index in Gnocchi index or directly in influx? In other words, are you fully using influx under gnochi API? Max From: Sam Morrison Sent: September 5, 2016 9:24:21 PM To: Julien Danjou Cc: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb Hi Julien, > On 5 Sep 2016, at 5:36 PM, Julien Danjou wrote: > > On Mon, Sep 05 2016, Sam Morrison wrote: > > Hi Sam, > >> The issue I’m having are with the tests. Because the continuous queries are >> asynchronous and there is no current way in influxdb to force them to run I >> get >> tests failing due to >> them not having run yet. >> >> I’m not sure how to get around this issue, apart from the tests failing >> everything is working quite well. I’m going to start some load testing soon >> to >> see what it’s like when pushing in a lot of metrics. > > Does it break the REST API, or only some storage tests? REST API is fine, in fact it fixes some tests that influx was failing on. > If it's just some storage test, you can change the tests so they are > retrying until the operation are done. Either in the test, or via a > special flag in the driver – we used to have that in the first version > of the driver. OK good idea, I’ll work on that. >> Wondering if there would be time to talk about this in Barcelona. > > Sure. Cheers, Sam > > -- > Julien Danjou > -- Free Software hacker > -- https://julien.danjou.info __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
Hi Julien, > On 5 Sep 2016, at 5:36 PM, Julien Danjou wrote: > > On Mon, Sep 05 2016, Sam Morrison wrote: > > Hi Sam, > >> The issue I’m having are with the tests. Because the continuous queries are >> asynchronous and there is no current way in influxdb to force them to run I >> get >> tests failing due to >> them not having run yet. >> >> I’m not sure how to get around this issue, apart from the tests failing >> everything is working quite well. I’m going to start some load testing soon >> to >> see what it’s like when pushing in a lot of metrics. > > Does it break the REST API, or only some storage tests? REST API is fine, in fact it fixes some tests that influx was failing on. > If it's just some storage test, you can change the tests so they are > retrying until the operation are done. Either in the test, or via a > special flag in the driver – we used to have that in the first version > of the driver. OK good idea, I’ll work on that. >> Wondering if there would be time to talk about this in Barcelona. > > Sure. Cheers, Sam > > -- > Julien Danjou > -- Free Software hacker > -- https://julien.danjou.info __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
On Mon, Sep 05 2016, Sam Morrison wrote: Hi Sam, > The issue I’m having are with the tests. Because the continuous queries are > asynchronous and there is no current way in influxdb to force them to run I > get > tests failing due to > them not having run yet. > > I’m not sure how to get around this issue, apart from the tests failing > everything is working quite well. I’m going to start some load testing soon to > see what it’s like when pushing in a lot of metrics. Does it break the REST API, or only some storage tests? If it's just some storage test, you can change the tests so they are retrying until the operation are done. Either in the test, or via a special flag in the driver – we used to have that in the first version of the driver. > Wondering if there would be time to talk about this in Barcelona. Sure. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
Hi Julien, I’ve been working on the influx driver and have been re designing it to use continuous queries and retention policies so it acts more like the carbonara based drivers. Basically the continuous queries down sample the data. All metrics are stored in the same influx measurement with the metric_id being a tag. The issue I’m having are with the tests. Because the continuous queries are asynchronous and there is no current way in influxdb to force them to run I get tests failing due to them not having run yet. I’m not sure how to get around this issue, apart from the tests failing everything is working quite well. I’m going to start some load testing soon to see what it’s like when pushing in a lot of metrics. Wondering if there would be time to talk about this in Barcelona. Cheers, Sam > On 4 Aug 2016, at 2:59 PM, Sam Morrison wrote: > > OK thanks Julien, > > I’m about to go on holiday for a month so I’ll pick this up when I return. > One of our devs is playing with this and thinking of ways to support the > things currently not implemented/working. > > Cheers, > Sam > > >> On 2 Aug 2016, at 8:35 PM, Julien Danjou wrote: >> >> On Tue, Aug 02 2016, Sam Morrison wrote: >> >> Hi Sam! >> >>> We have been using gnocchi for a while now with the influxDB driver >>> and are keen to get the influxdb driver back into upstream. >>> >>> However looking into the code and how it’s arranged it looks like >>> there are a lot of assumptions that the backend storage driver is >>> carbonara based. >> >> More or less. There is a separation layer (index/storage) and a full >> abstraction layer so it's possible to write a driver for any TSDB. >> Proof, we had an InfluxDB driver. >> Now the separation layer is not optimal for some TSDBs like InfluxDB, >> unfortunately nobody never stepped up to enhance it. >> >>> Is gnocchi an API for time series DBs or is it a time series DB >>> itself? >> >> Both. It's an API over TSDBs, and it also has its own TSDB based on >> Carbonara+{Ceph,File,Swift}. >> >>> The tests that are failing are due to the way carbonara and influx handle >>> the >>> retention and multiple granularities differently. (which we can work around >>> outside of gnocchi for now) >>> >>> So I guess I’m wondering if there will be support for other drivers apart >>> from carbonara? >> >> Sure. We dropped the InfluxDB driver because nobody was maintaining it >> and it was not passing the tests anymore. But we'd be glad to have it >> in-tree I'd say. >> >>> We use influx because we already use it for other stuff within our >>> organisation >>> and don’t want to set up ceph or swift (which is quite an endeavour) to >>> support >>> another time series DB. >> >> That makes sense. If you don't need scaling, I can only encourage you >> taking a look at using Carbonara+file rather than InfluxDB in the >> future, which I think is still a better choice. >> >> But in the meantime, feel free to send a patch to include back InfluxDB >> in Gnocchi. As long as you're ready to help us maintain it, we'll all >> open on that. :) >> >> Cheers, >> -- >> Julien Danjou >> # Free Software hacker >> # https://julien.danjou.info > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
OK thanks Julien, I’m about to go on holiday for a month so I’ll pick this up when I return. One of our devs is playing with this and thinking of ways to support the things currently not implemented/working. Cheers, Sam > On 2 Aug 2016, at 8:35 PM, Julien Danjou wrote: > > On Tue, Aug 02 2016, Sam Morrison wrote: > > Hi Sam! > >> We have been using gnocchi for a while now with the influxDB driver >> and are keen to get the influxdb driver back into upstream. >> >> However looking into the code and how it’s arranged it looks like >> there are a lot of assumptions that the backend storage driver is >> carbonara based. > > More or less. There is a separation layer (index/storage) and a full > abstraction layer so it's possible to write a driver for any TSDB. > Proof, we had an InfluxDB driver. > Now the separation layer is not optimal for some TSDBs like InfluxDB, > unfortunately nobody never stepped up to enhance it. > >> Is gnocchi an API for time series DBs or is it a time series DB >> itself? > > Both. It's an API over TSDBs, and it also has its own TSDB based on > Carbonara+{Ceph,File,Swift}. > >> The tests that are failing are due to the way carbonara and influx handle the >> retention and multiple granularities differently. (which we can work around >> outside of gnocchi for now) >> >> So I guess I’m wondering if there will be support for other drivers apart >> from carbonara? > > Sure. We dropped the InfluxDB driver because nobody was maintaining it > and it was not passing the tests anymore. But we'd be glad to have it > in-tree I'd say. > >> We use influx because we already use it for other stuff within our >> organisation >> and don’t want to set up ceph or swift (which is quite an endeavour) to >> support >> another time series DB. > > That makes sense. If you don't need scaling, I can only encourage you > taking a look at using Carbonara+file rather than InfluxDB in the > future, which I think is still a better choice. > > But in the meantime, feel free to send a patch to include back InfluxDB > in Gnocchi. As long as you're ready to help us maintain it, we'll all > open on that. :) > > Cheers, > -- > Julien Danjou > # Free Software hacker > # https://julien.danjou.info __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Support for other drivers - influxdb
On Tue, Aug 02 2016, Sam Morrison wrote: Hi Sam! > We have been using gnocchi for a while now with the influxDB driver > and are keen to get the influxdb driver back into upstream. > > However looking into the code and how it’s arranged it looks like > there are a lot of assumptions that the backend storage driver is > carbonara based. More or less. There is a separation layer (index/storage) and a full abstraction layer so it's possible to write a driver for any TSDB. Proof, we had an InfluxDB driver. Now the separation layer is not optimal for some TSDBs like InfluxDB, unfortunately nobody never stepped up to enhance it. > Is gnocchi an API for time series DBs or is it a time series DB > itself? Both. It's an API over TSDBs, and it also has its own TSDB based on Carbonara+{Ceph,File,Swift}. > The tests that are failing are due to the way carbonara and influx handle the > retention and multiple granularities differently. (which we can work around > outside of gnocchi for now) > > So I guess I’m wondering if there will be support for other drivers apart > from carbonara? Sure. We dropped the InfluxDB driver because nobody was maintaining it and it was not passing the tests anymore. But we'd be glad to have it in-tree I'd say. > We use influx because we already use it for other stuff within our > organisation > and don’t want to set up ceph or swift (which is quite an endeavour) to > support > another time series DB. That makes sense. If you don't need scaling, I can only encourage you taking a look at using Carbonara+file rather than InfluxDB in the future, which I think is still a better choice. But in the meantime, feel free to send a patch to include back InfluxDB in Gnocchi. As long as you're ready to help us maintain it, we'll all open on that. :) Cheers, -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev