Re: [openstack-dev] Ceilometer, New measurements, Types
Meters are measurable value. So it cannot be a string. If you really need to have notifications about event that is exist but cannot be measured you can use Events. http://docs.openstack.org/admin-guide-cloud/telemetry-events.html Cheers, Igor Degtiarov Software Engineer Mirantis Inc. www.mirantis.com On Sat, Oct 10, 2015 at 6:19 AM, phot...@126.com <phot...@126.com> wrote: > There are three type of meters are defined in Ceilometer, including > Cumulative, Gauge, Delta. > In fact, these three types are numeric, but i need string. > How could i make ceilometer to support string. > > Doc: http://docs.openstack.org/developer/ceilometer/new_meters.html > image: > -- > phot...@126.com > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] Inconsistent timestamping of polled data
Hi! Looks good to me, especially for cases when after some incident we gather a great amount of notifications in queue and stated to work with it so some data will have incorrect timestamp if it set only when sample is created. Cheers, Igor Degtiarov Software Engineer Mirantis Inc. www.mirantis.com On Fri, Oct 9, 2015 at 1:00 PM, Wen Zhi WW Yu <yu...@cn.ibm.com> wrote: > Hi all, > > As Gordon descriped in https://bugs.launchpad.net/ceilometer/+bug/1491509 > , many of pollsters define the timestamp individually for each sample that > is generated rather than basing on when the data was polled. I agree with > Gordon on that the timestamping of samples should base on when the data was > polled. > > What's your opinion on this? > > Best Regards, > Yu WenZhi(余文治） > OpenStack on Power Development, IBM Shanghai > 2F, 399 Keyuan Rd, Zhangjiang Chuangxin No. 10 Building, Zhangjiang High > Tech Park Shanghai > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer][AODH] Timeout Event Alarms
Hi, As far as I understand we have a base agreement that both specs are appropriate and either of that two features are shifted to Mitaka cycle. Gordon probably that question to you: Are we going to create a new folder in spec's dir for next cycle, or we continue discussing new specs as part of liberty? And the second one: are we going to create special rep for AODH specs or ceilometer-specs is ok for all new projects? Thank you in advance, Igor Degtiarov Software Engineer Mirantis Inc. www.mirantis.com On Fri, Aug 7, 2015 at 10:59 AM, Ryota Mibu r-m...@cq.jp.nec.com wrote: Hi, Sorry for my late response and my absent in weekly meetings... I'm not sure whether I captured your idea correctly, but I prefer the second approach now. I agreed the point Igor and liusheng mentioned that the second approach enables end users to have configurable expire-time. In another point of view, the first approach may affect pipeline performance since it have to keep event sequence state or have to access DB for state querying when each event received. This is just my concern, but I think event pipeline should be simplest and limited to have only common features between event data storage, event alarming and other receiver like audit system. Thanks, Ryota -Original Message- From: liusheng [mailto:liusheng1...@126.com] Sent: Wednesday, August 05, 2015 1:12 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Ceilometer][AODH] Timeout Event Alarms Hi, Maybe the event transformer is needed in some use cases to generate new events or do transformations like the samples handling. but for this timeout event alarming requirement, the 'timeout' of alarms will be various, it not a good idea of changing event_pipeline.yaml to generate new events based on events timeout when we need an event-timeout alarm. and also, the access of event pipeline definitions to users is inadvisable. I personally think it'd better to implement the second option and based on Ryota's proposal. Best Regards Liusheng 在 2015/8/5 3:36, gord chung 写道: hi Igor, i would suggest you go with second option as i believe your implementation will overlap and reuse some of the functionality Ryota would code for his alarm spec [1]. also, since Aodh is working on an independent release cycle, it'll give you some more time as i don't think we'd be able to get this into Liberty if we went the pipeline route. [1] http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/event-alarm-evaluator.html On 04/08/2015 10:00 AM, Igor Degtiarov wrote: Hi folks, On our meatup we agreed to add timeout event alarms [1](Event-Base Alarming part). In ToDo task Сhoose the optimal way for timeout alerting implementation Now we have two proposition for implementation: - first is to add timeout param in event pipeline (transformer part) [2] -- weakness of this approach is that we cannot allow user change config files, so only administrator will be able to set rules for timeout events alarms, and that is not what we are expecting from alarms. - second is additional optional parameters in event alarms description like sequence of required events and timeout threshold. Event alarm evaluator looks thru getting events and evaluates alarm if even one event from required sequence isn't received in set timeout.[3] It seems that second approach is better it doesn't have restrictions for end user. Hope for your help in choosing optimal way for implementation. (In specs review there is silence now) [1] https://wiki.openstack.org/wiki/Meetings/Ceilometer/Liberty_Virtual_Mid-Cycle [2] https://review.openstack.org/#/c/162167 [3] https://review.openstack.org/#/c/199005 Igor Degtiarov Software Engineer Mirantis Inc. www.mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List
Re: [openstack-dev] [Ceilometer][AODH] Timeout Event Alarms
Gordon thanks for quick answer. I'll add a patch with new dir for mitaka specs and move my specs there. cheers, Igor Degtiarov Software Engineer Mirantis Inc. www.mirantis.com On Mon, Aug 17, 2015 at 7:02 PM, gord chung g...@live.ca wrote: good questions... On 17/08/2015 10:19 AM, Igor Degtiarov wrote: Gordon probably that question to you: Are we going to create a new folder in spec's dir for next cycle, or we continue discussing new specs as part of liberty? we can create a new dir for M* cycle specs. as mentioned in last week's meeting, even though we don't have a exact date for feature freeze, unless it's a small bug size spec, Liberty specs are closed. feel free to create a patch for new M* dir And the second one: are we going to create special rep for AODH specs or ceilometer-specs is ok for all new projects? we'll track aodh specs under ceilometer now -- i don't want to increase the number of repos we need to monitor unless we notice Aodh specs are too much to handle. cheers, -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer][AODH] Timeout Event Alarms
Hi folks, On our meatup we agreed to add timeout event alarms [1](Event-Base Alarming part). In ToDo task Сhoose the optimal way for timeout alerting implementation Now we have two proposition for implementation: - first is to add timeout param in event pipeline (transformer part) [2] -- weakness of this approach is that we cannot allow user change config files, so only administrator will be able to set rules for timeout events alarms, and that is not what we are expecting from alarms. - second is additional optional parameters in event alarms description like sequence of required events and timeout threshold. Event alarm evaluator looks thru getting events and evaluates alarm if even one event from required sequence isn't received in set timeout.[3] It seems that second approach is better it doesn't have restrictions for end user. Hope for your help in choosing optimal way for implementation. (In specs review there is silence now) [1] https://wiki.openstack.org/wiki/Meetings/Ceilometer/Liberty_Virtual_Mid-Cycle [2] https://review.openstack.org/#/c/162167 [3] https://review.openstack.org/#/c/199005 Igor Degtiarov Software Engineer Mirantis Inc. www.mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] Pipeline for notifications does not seem to work
I am just curious have you restarted ceilometer services after pipeline.yaml has been changed? Igor Degtiarov Software Engineer Mirantis Inc www.mirantis.com On Sat, Mar 21, 2015 at 1:21 PM, Tim Bell tim.b...@cern.ch wrote: No errors in the notification logs. Should this work with the default ceilometer.conf file or do I need to enable anything ? I’ve also tried using arithmetic. When I have a meter like “cpu” for the source, this fires the expression evaluation without problems. However, I can’t find a good way of doing the appropriate calculations using the number of cores. Sample calculation is below expr: $(cpu)*0.98+$(vcpus)*10.0 I have tried $(cpu.resource_metdata.vcpus) and $(cpu.resource_metdata.cpu_number) also. Any suggestions on an alternative approach that could work ? Any suggestions for the variable name to get at the number of cores when I’m evaluating an expression fired by the cpu time ? Tim From: gordon chung [mailto:g...@live.ca] Sent: 20 March 2015 20:55 To: OpenStack Development Mailing List not for usage questions Subject: Re: [openstack-dev] [ceilometer] Pipeline for notifications does not seem to work i can confirm it works for me as well... are there any noticeable errors in the ceilometer-agent-notifications log? the snippet below looks sane to me though. cheers, gord From: idegtia...@mirantis.com Date: Fri, 20 Mar 2015 18:35:56 +0200 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [ceilometer] Pipeline for notifications does not seem to work Hi Tim I've check your case on my devstack. And I've received new hs06 meter in my meter list. So something wrong with your local env. Cheers, Igor D. Igor Degtiarov Software Engineer Mirantis Inc www.mirantis.com On Fri, Mar 20, 2015 at 5:40 PM, Tim Bell tim.b...@cern.ch wrote: I’m running Juno with ceilometer and trying to produce a new meter which is based on vcpus * F (where F is a constant that is different for each hypervisor). When I create a VM, I get a new sample for vcpus. However, it does not appear to fire the transformer. The same approach using “cpu” works OK but this one is polling on a regular interval rather than a one off notification when the VM is created. Any suggestions or alternative approaches for how to get a sample based the number of cores scaled by a fixed constant? Tim In my pipeline.yaml sources, - name: vcpu_source interval: 180 meters: - vcpus sinks: - hs06_sink In my transformers, I have - name: hs06_sink transformers: - name: unit_conversion parameters: target: name: hs06 unit: HS06 type: gauge scale: 47.0 publishers: - notifier:// __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] Pipeline for notifications does not seem to work
Hi Tim I've check your case on my devstack. And I've received new hs06 meter in my meter list. So something wrong with your local env. Cheers, Igor D. Igor Degtiarov Software Engineer Mirantis Inc www.mirantis.com On Fri, Mar 20, 2015 at 5:40 PM, Tim Bell tim.b...@cern.ch wrote: I’m running Juno with ceilometer and trying to produce a new meter which is based on vcpus * F (where F is a constant that is different for each hypervisor). When I create a VM, I get a new sample for vcpus. However, it does not appear to fire the transformer. The same approach using “cpu” works OK but this one is polling on a regular interval rather than a one off notification when the VM is created. Any suggestions or alternative approaches for how to get a sample based the number of cores scaled by a fixed constant? Tim In my pipeline.yaml sources, - name: vcpu_source interval: 180 meters: - vcpus sinks: - hs06_sink In my transformers, I have - name: hs06_sink transformers: - name: unit_conversion parameters: target: name: hs06 unit: HS06 type: gauge scale: 47.0 publishers: - notifier:// __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer][MongoDB] Using native time to live feature in MongoDB
Hi stackers, I'm working on solving bug [1]. Time to live feature has native implementation in MongoDB thru index. Now we remove docs from resource table if they have no relations with existing samples in meter table while samples are removed when time to live is expired. So it seems that we can use ttl index also in resource table and remove reduce operation from the code. We update field last_sample_timestamp in resource table with every new sample that is received from certain resource. So adding ttl index to that field gives the same result as reduce operation in clear_expired_metering_data, but it will work in background with low priority and won't block database. Change request with implementation of ttl index in resource table [2]. [1] https://bugs.launchpad.net/ceilometer/+bug/1270779 [2] https://review.openstack.org/#/c/132988/ Cheers, Igor. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Adding pylint checking of new ceilometer patches
My points are next: 1. Pylint check will be very useful for project, and will help to avoid critical errors or mistakes in code. 2. At first step we won't implement pylint as gate job, but will add it at master to have a possibility to check code with pylint locally, if it is needed. 3. In future it could be added as a non-voting job. -- Igor On Sat, Oct 4, 2014 at 1:56 AM, Angus Lees g...@inodes.org wrote: You can turn off lots of the refactor recommendation checks. I've been running pylint across neutron and it's uncovered half a dozen legitimate bugs so far - and that's with many tests still disabled. I agree that the defaults are too noisy, but its about the only tool that does linting across files - pep8 for example only looks at the current file (and not even the parse tree). On 4 Oct 2014 03:22, Doug Hellmann d...@doughellmann.com wrote: On Oct 3, 2014, at 1:09 PM, Neal, Phil phil.n...@hp.com wrote: From: Dina Belova [mailto:dbel...@mirantis.com] On Friday, October 03, 2014 2:53 AM Igor, Personally this idea looks really nice to me, as this will help to avoid strange code being merged and not found via reviewing process. Cheers, Dina On Fri, Oct 3, 2014 at 12:40 PM, Igor Degtiarov idegtia...@mirantis.com wrote: Hi folks! I try too guess do we need in ceilometer checking new patches for critical errors with pylint? As far as I know Nova and Sahara and others have such check. Actually it is not checking of all project but comparing of the number of errors without new patch and with it, and if diff is more then 0 then patch are not taken. Looking a bit deeper it seems that Nova struggled with false positives and resorted to https://review.openstack.org/#/c/28754/ , which layers some historical checking of git on top of pylint's tendency to check only the latest commit. I can't say I'm too deeply versed in the code, but it's enough to make me wonder if we want to go that direction and avoid the issues altogether? I haven’t looked at it in a while, but I’ve never been particularly excited by pylint. It’s extremely picky, encourages enforcing some questionable rules (arbitrary limits on variable name length?), and repots a lot of false positives. That combination tends to result in making writing software annoying without helping with quality in any real way. Doug I have taken as pattern Sahara's solution and proposed a patch for ceilometer: https://review.openstack.org/#/c/125906/ Cheers, Igor Degtiarov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best regards, Dina Belova Software Engineer Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer] Adding pylint checking of new ceilometer patches
Hi folks! I try too guess do we need in ceilometer checking new patches for critical errors with pylint? As far as I know Nova and Sahara and others have such check. Actually it is not checking of all project but comparing of the number of errors without new patch and with it, and if diff is more then 0 then patch are not taken. I have taken as pattern Sahara's solution and proposed a patch for ceilometer: https://review.openstack.org/#/c/125906/ Cheers, Igor Degtiarov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] MySQL performance and Mongodb backend maturity question
Hi, Qiming Teng. Now all backends support events. So you may use MongoDB instead of MySQL, or if you like you may choose HBase. Cheers, Igor. -- Igor On Thu, Sep 25, 2014 at 7:43 AM, Preston L. Bannister pres...@bannister.us wrote: Sorry, I am jumping into this without enough context, but ... On Wed, Sep 24, 2014 at 8:37 PM, Qiming Teng teng...@linux.vnet.ibm.com wrote: mysql select count(*) from metadata_text; +--+ | count(*) | +--+ | 25249913 | +--+ 1 row in set (3.83 sec) There are problems where a simple sequential log file is superior to a database table. The above looks like a log ... a very large number of events, without an immediate customer. For sequential access, a simple file is *vastly* superior to a database table. If you are thinking about indexed access to the above as a table, think about the cost of adding items to the index, for that many items. The cost of building the index is not small. Running a map/reduce on sequential files might be faster. Again, I do not have enough context, but ... 25 million rows? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Complex resource_metadata could fail to store in MongoDB
Hi!
After some local discussions with Dmitiy Uklov we have found solution.
To solve the problem with dots inside keys in metadata dictionary we
propose to reconstruct it
before storing in MongoDB.
Ex.
If we get metadata: {'a.s.d': 'v'}
it could be stored in MongoDB after a reconstruction
{'a': {'s': {'d': 'v'}}}
After storing in MongoDB value 'v' easily could be found with the standard
query 'metadata.a.s.d'='v'.
Keys that start with '$' are quoted with the quote function from
urllib.parse
I have proposed change request with fix:
https://review.openstack.org/121003
Cheers,
-- Igor D.
On Mon, Sep 8, 2014 at 5:02 PM, Igor Degtiarov idegtia...@mirantis.com
wrote:
On Thu, Sep 4, 2014 at 1:16 PM, Nadya Privalova nprival...@mirantis.com
wrote:
IMHO it's ok and even very natural to expect escaped query from users.
e.g, we store the following structure
{metadata:
{ Zoo:
{Foo.Boo: ''value}}}
Yep but such structure couldn't be stored in MongoDB without exchanging
dot in Foo.Boo
and query should be metadata.Zoo.Foo\.Boo .
That could be a good solution, but it is needed only if MongoDB is chosen
as a backend. So the question is
should we specify query only for MongoDB or change queries for all
backends?
In this case it's not necessary to know deep of tree.
Thanks,
Nadya
Cheers,
Igor D.
On Fri, Aug 29, 2014 at 3:21 PM, Igor Degtiarov idegtia...@mirantis.com
wrote:
Hi, folks.
I was interested in the problem with storing of samples, that contain
complex resource_metadata, in MongoDB database [1].
If data is a dict that has a key(s) with dots (i.e. .), dollar signs
(i.e. $), or null characters,
it wouldn't be stored. It is happened because these characters are
restricted to use in fields name in MongoDB [2], but so far there is no any
verification of the metadata in ceilometers mongodb driver, as a result we
will lose data.
Solution of this problem seemed to be rather simple, before storing data
we check keys in resourse_metadata, if it is a dict, and quote keys with
restricted characters in a similar way, as it was done in a change request
of redesign separators in columns in HBase [2]. After that store metering
data.
But other unexpected difficulties appear on the step of getting data. To
get stored data we constructs a meta query, and structure of that query was
chosen identical to initial query in MongoDB. So dots is used as a
separator for three nods of stored data.
Ex. If it is needed to check value in field Foo
{metadata:
{ Zoo:
{Foo: ''value}}}
query would be: metadata.Zoo.Foo
We don't know how deep is dict in metadata, so it is impossible to
propose any correct parsing of query, to quote field names contain dots.
I see two way for improvements. First is rather complex and based of
redesign structure of the metadata query in ceilometer. Don't know if it is
ever possible.
And second is based on removing from the samples bad
resource_metadata. In this case we also lose metadata, but save other
metering data. Of course queries for not saved metadata will return
nothing, so it is not complete solution, but some kind of the hook.
What do you think about that?
Any thoughts and propositions are kindly welcome.
[1] https://bugs.launchpad.net/mos/+bug/1360240
[2] http://docs.mongodb.org/manual/reference/limits/
[3] https://review.openstack.org/#/c/106376/
-- Igor Degtiarov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Complex resource_metadata could fail to store in MongoDB
On Thu, Sep 4, 2014 at 1:16 PM, Nadya Privalova nprival...@mirantis.com
wrote:
IMHO it's ok and even very natural to expect escaped query from users.
e.g, we store the following structure
{metadata:
{ Zoo:
{Foo.Boo: ''value}}}
Yep but such structure couldn't be stored in MongoDB without exchanging
dot in Foo.Boo
and query should be metadata.Zoo.Foo\.Boo .
That could be a good solution, but it is needed only if MongoDB is chosen
as a backend. So the question is
should we specify query only for MongoDB or change queries for all
backends?
In this case it's not necessary to know deep of tree.
Thanks,
Nadya
Cheers,
Igor D.
On Fri, Aug 29, 2014 at 3:21 PM, Igor Degtiarov idegtia...@mirantis.com
wrote:
Hi, folks.
I was interested in the problem with storing of samples, that contain
complex resource_metadata, in MongoDB database [1].
If data is a dict that has a key(s) with dots (i.e. .), dollar signs
(i.e. $), or null characters,
it wouldn't be stored. It is happened because these characters are
restricted to use in fields name in MongoDB [2], but so far there is no any
verification of the metadata in ceilometers mongodb driver, as a result we
will lose data.
Solution of this problem seemed to be rather simple, before storing data
we check keys in resourse_metadata, if it is a dict, and quote keys with
restricted characters in a similar way, as it was done in a change request
of redesign separators in columns in HBase [2]. After that store metering
data.
But other unexpected difficulties appear on the step of getting data. To
get stored data we constructs a meta query, and structure of that query was
chosen identical to initial query in MongoDB. So dots is used as a
separator for three nods of stored data.
Ex. If it is needed to check value in field Foo
{metadata:
{ Zoo:
{Foo: ''value}}}
query would be: metadata.Zoo.Foo
We don't know how deep is dict in metadata, so it is impossible to
propose any correct parsing of query, to quote field names contain dots.
I see two way for improvements. First is rather complex and based of
redesign structure of the metadata query in ceilometer. Don't know if it is
ever possible.
And second is based on removing from the samples bad resource_metadata.
In this case we also lose metadata, but save other metering data. Of
course queries for not saved metadata will return nothing, so it is not
complete solution, but some kind of the hook.
What do you think about that?
Any thoughts and propositions are kindly welcome.
[1] https://bugs.launchpad.net/mos/+bug/1360240
[2] http://docs.mongodb.org/manual/reference/limits/
[3] https://review.openstack.org/#/c/106376/
-- Igor Degtiarov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer] Complex resource_metadata could fail to store in MongoDB
Hi, folks.
I was interested in the problem with storing of samples, that contain
complex resource_metadata, in MongoDB database [1].
If data is a dict that has a key(s) with dots (i.e. .), dollar signs (i.e.
$), or null characters,
it wouldn't be stored. It is happened because these characters are
restricted to use in fields name in MongoDB [2], but so far there is no any
verification of the metadata in ceilometers mongodb driver, as a result we
will lose data.
Solution of this problem seemed to be rather simple, before storing data we
check keys in resourse_metadata, if it is a dict, and quote keys with
restricted characters in a similar way, as it was done in a change request
of redesign separators in columns in HBase [2]. After that store metering
data.
But other unexpected difficulties appear on the step of getting data. To
get stored data we constructs a meta query, and structure of that query was
chosen identical to initial query in MongoDB. So dots is used as a
separator for three nods of stored data.
Ex. If it is needed to check value in field Foo
{metadata:
{ Zoo:
{Foo: ''value}}}
query would be: metadata.Zoo.Foo
We don't know how deep is dict in metadata, so it is impossible to propose
any correct parsing of query, to quote field names contain dots.
I see two way for improvements. First is rather complex and based of
redesign structure of the metadata query in ceilometer. Don't know if it is
ever possible.
And second is based on removing from the samples bad resource_metadata.
In this case we also lose metadata, but save other metering data. Of
course queries for not saved metadata will return nothing, so it is not
complete solution, but some kind of the hook.
What do you think about that?
Any thoughts and propositions are kindly welcome.
[1] https://bugs.launchpad.net/mos/+bug/1360240
[2] http://docs.mongodb.org/manual/reference/limits/
[3] https://review.openstack.org/#/c/106376/
-- Igor Degtiarov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Tox run failure during installation of dependencies in requirements
Hi,
Actually the same question, what is wrong with Tox now?
-- Igor
On Wed, Aug 6, 2014 at 1:19 PM, Narasimhan, Vivekanandan
vivekanandan.narasim...@hp.com wrote:
Hi,
Recently , the Tox runs started to fail in my workspace.
It fails consistently during installing dependencies with the following.
Downloading/unpacking PrettyTable=0.7,0.8 (from
python-keystoneclient=0.10.0--r
/home/narasimv/dev/bug1350485/neutron/requirements.txt (line 22))
Cleaning up...
Exception:
Traceback (most recent call last):
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/basecommand.py,
line 122, in main
status = self.run(options, args)
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/commands/install.py,
line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle,
bundle=self.bundle)
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/req.py,
line 1197, in prepare_files
do_download,
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/req.py,
line 1375, in unpack_url
self.session,
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/download.py,
line 546, in unpack_http_url
resp = session.get(target_url, stream=True)
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py,
line 468, in get
return self.request('GET', url, **kwargs)
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/download.py,
line 237, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py,
line 456, in request
resp = self.send(prep, **send_kwargs)
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py,
line 559, in send
r = adapter.send(request, **kwargs)
File
/home/narasimv/dev/bug1350485/neutron/.tox/pep8/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py,
line 384, in send
raise Timeout(e, request=request)
Timeout:
(pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection
object at 0x37e4790, 'Connection to pypi.python.org timed out. (connect
timeout=15)')
The TOX was earlier running in the same workspace successfully on the
machine
even though we were behind the proxy.
Could you please advise on how to resolve this problem?
--
Thanks,
Vivek
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Question of necessary queries for Event implemented on HBase
Hello, For today I have prepared a spec for events feature in HBase https://review.openstack.org/#/c/96417/, and it was successfully merged. In spec was described the details of event features in HBase. Yours, Igor D. On Wed, May 21, 2014 at 3:03 PM, Dmitriy Ukhlov dukh...@mirantis.com wrote: Hello Igor, Sounds reasonable. On 05/21/2014 02:38 PM, Igor Degtiarov wrote: Hi, I have found that filter model for Events has mandatory parameters start_time and end_time of the events period. So, it seems that structure for rowkey as ''timestamp + event_id will be more suitable. -- Best regards, Dmitriy Ukhlov Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Question of necessary queries for Event implemented on HBase
Hi, I have found that filter model for Events has mandatory parameters start_time and end_time of the events period. So, it seems that structure for rowkey as ''timestamp + event_id will be more suitable. I have started to work on bp https://blueprints.launchpad.net/ceilometer/+spec/hbase-events-feature Partial implementation of events into HBase is available by link https://review.openstack.org/#/c/91408/ By now it was added record method that writes events into HBase and get method with filtering by period of generation time of events. Sincerely, Igor D. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Question of necessary queries for Event implemented on HBase
Hello Dmitriy! Of course event_id could identify row. Actually Row structure now is a question for discuss. The main feature of HBase that data is stored in sorted rows. To filter events by time interval we will need to scan all stored data, when rowkey is started with event_id. Instead of adding timestamp in rowkey, we could set timestamps as a additional column value or qualifier inside row. Another decision is to start rowkey with timestamp + event_id. In this case we will have events stored by timestamps in HBase. What type of rowkey will give more efficiency is a question. On Wed, Apr 30, 2014 at 4:36 PM, Dmitriy Ukhlov dukh...@mirantis.comwrote: Hello Igor! Could you clarify, please, Why do we need event_id + reversed_timestamp row key? Isn't event_id identify row? On Tue, Apr 29, 2014 at 11:08 AM, Igor Degtiarov idegtia...@mirantis.comwrote: Hi, everybody. I’ve started to work on implementation of Event in ceilometer on HBase backend in the edges of blueprint https://blueprints.launchpad.net/ceilometer/+spec/hbase-events-feature By now Events has been implemented only in SQL. You know, using SQL we can build any query we need. With HBase it is another story. The data structure is built basing on queries we need, so to construct the structure of Event on HBase, it is very important to answer the question what queries should be implemented to retrieve events from storage. I registered bp https://blueprints.launchpad.net/ceilometer/+spec/hbase-events-structurefor discussion Events structure in HBase. For today it is prepared preliminary structure of Events in HBase: table: Events - rowkey: event_id + reversed_timestamp - column: event_type = string with description of event - [list of columns: trait_id + trait_desc + trait_type= trait_data] Structure that is proposed will support next queries: - event’s generation time - event id - event type - trait: id, description, type Any thoughts about additional queries that are necessary for Events. I’ll publish the patch with current implementation soon. Sincerely, Igor Degtiarov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best regards, Dmitriy Ukhlov Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceilometer] Question of necessary queries for Event implemented on HBase
Hi, everybody. I’ve started to work on implementation of Event in ceilometer on HBase backend in the edges of blueprint https://blueprints.launchpad.net/ceilometer/+spec/hbase-events-feature By now Events has been implemented only in SQL. You know, using SQL we can build any query we need. With HBase it is another story. The data structure is built basing on queries we need, so to construct the structure of Event on HBase, it is very important to answer the question what queries should be implemented to retrieve events from storage. I registered bp https://blueprints.launchpad.net/ceilometer/+spec/hbase-events-structurefor discussion Events structure in HBase. For today it is prepared preliminary structure of Events in HBase: table: Events - rowkey: event_id + reversed_timestamp - column: event_type = string with description of event - [list of columns: trait_id + trait_desc + trait_type= trait_data] Structure that is proposed will support next queries: - event’s generation time - event id - event type - trait: id, description, type Any thoughts about additional queries that are necessary for Events. I’ll publish the patch with current implementation soon. Sincerely, Igor Degtiarov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
